>> I WANT TO WELCOME YOU TO THIS
REALLY IMPORTANT WORKSHOP ON
NOVEL APPROACHES TO PREVENTING
PUBLICATION BIAS.
SO I'M THE DIRECTOR OF NINDS,
AND I AM PRIVILEGED TO
WORK WITH SOME REALLY GREAT
PEOPLE IN OUR OFFICE OF RESEARCH
QUALITY, AND THE PURPOSE IS TO
TRY TO MAKE SURE THAT THE
RESEARCH WE FUND ATTAINS THE
HIGHEST QUALITY STANDARDS.
SO I WAS FOR MANY YEARS A
PHYSICIAN, WE SPENT TONS OF TIME
DOING QUALITY ASSURANCE TO MAKE
SURE THAT THINGS DIDN'T GO WRONG
WITH OUR PATIENTS.
WHEN YOU GO ON AN AIRPLANE,
THERE'S LOTS OF QUALITY
ASSURANCE THAT THE DOORS ARE NOT
GOING TO FALL OFF WHILE YOU'RE
IN FLIGHT.
SOMETIMES IT WORKS, BUT WE SPEND
BILLIONS OF DOLLARS ON RESEARCH
AND WE HAVEN'T REALLY THOUGHT
ABOUT THAT EQUIVALENT.
HOW DO WE GET THE BEST QUALITY
OUT OF -- IN OUR RESEARCH, AND
THAT'S WHAT OUR OFFICE IS REALLY
ABOUT.
THERE'S A LOT OF DIFFERENT
THINGS, EDUCATIONAL PROGRAMS,
DEVELOPING GROUPS OF RIGOR
CHAMPIONS TO PROMOTE RIGOR IN
RESEARCH ACROSS THE COUNTRY,
ACTUALLY GAVE OUT PRIZES THIS
YEAR FOR PEOPLE WHO ARE REAL
CHAMPIONS AND DONE REALLY GOOD
WORK IN THEIR AREA.
BUT NOW WE'RE GOING TO BE
TALKING ABOUT A PARTICULAR AREA
TODAY, AND THAT'S PUBLICATION
BIAS.
YOU'LL HEAR MUCH MORE ABOUT IT.
I WAS JUST GOING TO OFFER JUST A
COUPLE OF TAKE-HOME POINTS FOR
YOU TO REMEMBER.
ONE IS THAT WHEN YOU READ A
PUBLICATION, THE REAL CRUX IS IN
THE DATA AND THE METHODS AND
EVERYTHING ELSE IS AN
ADVERTISEMENT.
SO I'M SAYING THAT SOMEWHAT
FACETIOUSLY, BUT IN TRUTH THE
DATA IS THE GOAL.
AND THE PUBLICATION IS BASED ON
THE GOAL BUT ALSO THE STORY.
YOU HAVE TO HAVE THAT BALANCE,
AND OFTENTIMES THAT BALANCE GETS
OUT OF WHACK, WHERE THE STORY
REALLY KIND OF TAKES OVER AND
THEN THERE'S TONS OF DATA THAT
WERE DONE IN PREPARATION AND
THAT WERE NEVER PUBLISHED, AND
WE DON'T KNOW WHAT THOSE THINGS
SHOWED.
I THINK THE FUTURE IS GOING TO
BE DIFFERENT.
THE REASON WE'RE IN THE SPOT
WE'RE IN NOW IS BECAUSE WE RELY
ON A TECHNOLOGY FOR
COMMUNICATING SCIENCE THAT'S
BASED ON THE TECHNOLOGY OF THE
PRINTING PRESS.
AND THAT'S CHANGED.
WE PUBLISH PAPERS, THAT'S BEEN
GOING ON SINCE THE PRINTING
PRESS.
BUT THE FUTURE, I THINK, IS
GOING TO BE DATA AND DATA IS
GOING TO BE THE GOAL, THEN THE
GOAL IS GOING TO BE THESE MINES
WHERE PEOPLE CAN GO AN LOOK AT
DATA AND WHEN YOU WRITE YOUR
STORY ABOUT YOUR DATA, YOU'RE
PROBABLY RIGHT BUT MAYBE NOT
RIGHT AND THERE MIGHT BE
ALTERNATIVE STORIES THAT SOME
PEOPLE COULD GENERATE BASED ON
THE DATA.
AND SO THAT'S WHY IT'S REALLY
IMPORTANT THAT WE FIGURE OUT A
WAY TO GET REALLY GOOD DATA IN
PLACES WHERE PEOPLE CAN LOOK AT
IT AND TRY TO UNDERSTAND WHERE
IT INTERSECTS IN WHAT THEY'RE
THINKING ABOUT AND MAYBE DEVELOP
ALTERNATE STORIES.
SO IT'S REALLY ENCOURAGING THAT
WE HAVE A DISIE VE DIVERSE GROUF
RESEARCHERS FROM A VARIETY OF
COUNTRIES AND SCIENTIFIC FIELDS
HERE, AND I WANT TO JUST REALLY
END OUT BY THANKING ALL FOR
THEIR LEADERSHIP IN THIS SPACE
AT NIH.
AND WITH THAT, I WANT TO
INTRODUCE MY GOOD FRIEND,
DR. LARRY TABAK.
LARRY IS THE DEPUTY DIRECTOR OF
NIH, AND HE'S JUST BEEN A MAJOR
CHAMPION FOR RIGOR AND RESEARCH,
ALONG WITH SAVING THE INSTITUTE
MULTIPLE TIMES, AND HE CAN'T
TALK ABOUT THAT BUT WHEN I
RETIRE, I CAN TELL YOU ABOUT IT.
HE PREVIOUSLY HE WAS THE
DIRECTOR OF THE NATIONAL
INSTITUTES OF DENTAL AND
CRANIOFACIAL RESEARCH, BEFORE
HERE, UNIVERSITY OF ROCHESTER,
PROFESSOR OF BIOCHEMISTRY AND
BIOPHYSICS.
STUDYING GLYCAN BIOLOGY, WHICH
IS REALLY IMPORTANT IN HEALTH OF
ALL THE MUCOSAL SURFACES OF THE
BODY.
IF YOU WANT TO HEAR ABOUT
GLYCANS, MAYBE DO THAT OFFLINE.
>> WELL GOOD MORNING, EVERYBODY.
THANK YOU, WALTER.
SO I DON'T TALK TO MY WIFE
WITHOUT POWERPOINTS, SO YOU HAVE
TO PUT UP MY FIRST SLIDE, OR DO
I JUST CLICK THIS?
THAT DIDN'T WORK.
SO SOMEBODY NEEDS TO PUT MY
SLIDES UP.
DO YOU HAVE MY SLIDES?
NO?
NO ONE HAS MY SLIDES?
BOY, NOW I'M REALLY IN TROUBLE.
NOW IF I GIVE YOU A FLASH DRIVE,
WILL YOU BE ABLE TO FIND MY
SLIDES?
THERE'S THE FLASH DRIVE.
SO, IF I HAD MY SLIDES, THE
FIRST THING THAT I WOULD DO IS I
WOULD GIVE YOU GREETINGS FROM
THE NIH DIRECTOR, DR. MONICA
BERTAGNOLLI.
I THINK YOU ALL KNOW THAT MONICA
WAS APPOINTED AS THE 17TH
DIRECTOR OF THE NATIONAL
INSTITUTES OF HEALTH.
HER SIX-MONTH ANNIVERSARY WAS
LAST WEEK.
OR MAYBE TWO WEEKS AGO NOW.
SHE IS FIRST SURGEON TO BE
APPOINTED AS DIRECTOR OF NIH.
THERE'S ALL SORTS OF
METAPHORICAL THINGS THAT YOU CAN
THINK ABOUT ABOUT THAT, BUT --
AND SHE IS ONLY THE SECOND WOMAN
TO BE APPOINTED NIH DIRECTOR.
SHE HIT THE GROUND RUNNING.
IT DOESN'T FIT?
WHY?
OKAY, THERE IT IS.
AND PICK WHICHEVER ONE YOU WANT.
ANY ONE OF THE TALKS THAT ARE ON
THAT DRIVE, AND I'LL ADAPT TO
WHAT WE'RE TALKING ABOUT HERE
TODAY.
IT'S LIKE GLYCANS, THEY'RE VERY
ADAPTABLE.
WAS THERE ONE THAT YOU FOUND
THAT MAYBE WOULD WORK?
THAT'S PROBABLY NOT THE ONE YOU
WANT.
ALL RIGHT.
AND IN ANY EVENT, MONICA --
THANK YOU.
MONICA WANTED TO BE HERE, BUT
UNFORTUNATELY, IS ON TRAVEL.
SO HERE SHE IS BEING SWORN IN.
THE PERSON ON THE FAR LEFT OF
COURSE IS SECRETARY BECERRA.
THE GENTLEMAN IN THE MIDDLE IS
MONICA'S HUSBAND, AL, AND I WAS
PRIVILEGED TO BE THERE DURING
HER SWEARING IN BECAUSE I WANTED
TO MAKE SURE THAT I DIDN'T HAVE
TO BE THE ACTING DIRECTOR
ANYMORE.
AFTER 23 MONTHS.
BUT WHO WAS COUNTING?
OKAY.
SO WALTER REALLY, YOU KNOW,
SUMMARIZED SOME OF WHAT I WAS
GOING TO SAY BUT HE DID IT IN A
MUCH NICER WAY THAN I WOULD
HAVE.
BACK IN 2014, FRANCIS COLLINS
AND I PUBLISHED THIS POSITION
PIECE IN NATURE, AND IT REALLY
OUTLINED WHAT THE ISSUES WERE
AND THE SORT OF BEGINNINGS OF
WHAT WE WERE TRYING TO DO.
NOW WHEN I SAY "WE," I HAVE TO
PAY HOMAGE TO SHAI.
BECAUSE BEFORE THIS BECAME A
THING, SHAI WAS TALKING ABOUT
IT, AND REALLY INFLUENCED ME,
INFLUENCED FRANCIS AND HAS
INFLUSSED THEINFLUENCED THE WHO.
SO SHAI, THANK YOU FOR YOUR
LEADERSHIP, NOT JUST AT NINDS
BUT REALLY ACROSS THE WHOLE
BIOMEDICAL RESEARCH SPACE.
IN THIS PIECE, FRANCIS AND I
WROTE THAT PRE-CLINICAL RESEARCH
IS REALLY SUSCEPTIBILITY TO
REPRODUCIBILITY ISSUES, BUT THEN
EMPHASIZING THAT SOME SOME EAR
REPRODUCIBLE REPORTS ARE
PROBABLY THE RESULT OF
COINCIDENTAL FINDINGS THAT
HAPPEN TO REACH STATISTICAL
SIGNIFICANCE COUPLED WITH BIAS
SO EVEN BACK THEN WE KNEW THIS
WAS A POTENTIAL ISSUE.
SO JUST TO LEVEL SET, AND YOU
ALL KNOW THIS BETTER THAN I DO,
BUT AGAIN JUST TO LEVEL SET,
PUBLICATION BIAS IS A TENDENCY
ON THE PARTS OF EVERYBODY IN THE
WHOLE ECOSYSTEM TO SUBMIT OR
ACCEPT MANUSCRIPTS FOR
PUBLICATION BASED ON WHAT
HAPPENED, SO TO SPEAK.
AND OFTEN, THIS RESULTS IN THE
ABSENCE OF NO RESULTS FROM THE
SCIENTIFIC RECORD.
YOU ONLY KNOW WHAT, QUOTE,
WORKED, YOU NEVER KNOW WHAT,
QUOTE, DIDN'T WORK.
AND THAT, OF COURSE, LEADS TO
THIS INCOMPLETE PICTURE, AND IT
DRAMATICALLY UNDERUTILIZES THE
NIH INVESTMENT IN BIOMEDICAL
RESEARCH, WHICH IS WHY AS AN
AGENCY, WE'RE SO STRONGLY
INTERESTED IN THIS.
SO WHY AM I SHOWING YOU THIS
PAPER ON PERIODONTAL DISEASE?
NOT BECAUSE I'M PAYING HOMAGE TO
NIDCR, WHERE I WAS THE DIRECTOR
FOR A DECADE.
BUT IT'S A PAPER THAT WAS A
LONG, LONG, LONG TIME IN COMING.
SO FOR MANY YEARS, MY COLLEAGUES
IN THE DENTAL INSTITUTE WOULD
FUND SMALL TRIEPS THAT WERE -- L
TRIES TTRIALS THAT WERE UNDERPOD
AND WHEN I ARRIVED AS A
GLYCOPATHOLOGIST I HAD TO ASK
WHY ARE YOU ONLY FUNDING THESE
TRIALS WHICH THE CONCLUSIONARY
SENTENCE OF EVERY PAPER THAT'S
PUBLISHED IS, "AND FUSHT
RESEARCH WILL BE NEEDED TO
DEFINITIVELY -- I WAS TOLD WE
CAN'T AFFORD TO DO DEFINITIVE
TRIALS, THEY'RE VERY EXPENSIVE.
AND I SAID, WELL, I'M ONLY AN
ENZYMOLOGIST, I STUDY
TRANSFERASES BUT I THINK PAYING
FOR THE TRIAL IS MY PROBLEM, NOT
YOURS, LET'S DO SOME DEFINITIVE
TRIALS.
AND THIS WAS ONE OF THE VERY
FIRST THAT THEY TOOK ON.
AT THE TIME, IT WAS RAGING BACK
IN THE MID 2000s, 2006 IS WHEN
THIS WAS PUBLISHED, THAT
PERIODONTAL DISEASE CAUSED
EVERYTHING.
FROM ALZHEIMER'S TO IN THIS CASE
PRETERM BIRTH TO CARDIOVASCULAR
ACCIDENT, ET CETERA, ET CETERA,
ET CETERA.
I MEAN, IT WAS THE CAUSE OF
EVERYTHING.
AND PERIODONTISTS WERE REALLY
EXCITED BECAUSE THEY WERE
CONTRIBUTING TO OVERALL
MAINTENANCE OF SYSTEMIC HEALTH.
NOW, IF YOU FAST FORWARD, THAT'S
ACTUALLY TRUE, BECAUSE IT DOES
REDUCE THE OVERALL BURDEN OF
CHRONIC INFLAMMATION.
BUT SPECIFICALLY, DID
PERIODONTAL DISEASE INDUCE THE
RISK OF PRETERM BIRTH?
SO BRIAN MICHALOW ICZ AND HIS
COLLEAGUES IN MINNESOTA ACTUALLY
DID THE DEFINITIVE TRIAL, AND IF
YOU CAN READ THE CONCLUSIONS,
TREATMENT OF PERIODONTITIS IN
PREGNANT WOMEN IMPROVED
PERIODONTAL DISEASE, THAT'S
GOOD, IT PROVES PERIODONTAL
THERAPY WORKS FOR PERIODONTAL
DISEASE, BUT DOES NOT
SIGNIFICANTLY ALTER PRETERM
BIRTH, LOW BIRTH WEIGHT OR FETAL
GROWTH RESTRICTION.
A NEGATIVE FINDING PUBLISHED IN
THE NEW ENGLAND JOURNAL OF
MEDICINE.
SO THE MESSAGE IS THAT IF YOU DO
A STUDY REALLY WELL, YOU CAN
PUBLISH NULL RESULTS.
BUT THE STUDY'S GOT TO BE DONE
REALLY, REALLY WELL.
NOW, I BECAME A PARIAH IN THE
PERIODONTAL RESEARCH COMMUNITY
TO THIS DAY, I AM NOT INVITED TO
ANY MEETINGS WITH PERIODONTISTS,
OKAY, BUT THAT'S OKAY.
BUT THE POINT OF THIS IS THAT
NULL RESULTS CAN BE PUBLISHED,
THEY CAN BE PUBLISHED IN
OUTSTANDING JOURNALS, BUT IT
ONLY IS IF IT'S DONE IN THE MOST
RIGOROUS WAY POSSIBLE.
AND THAT WAS AN EXPENSIVE BUT
IMPORTANT LESSON THAT AT LEAST
THE DENTAL COMMUNITY LEARNED,
AND I THINK OTHER COMMUNITIES
ARE LEARNING THIS ALONG THE WAY.
NOW THE QUESTION IS, WHEN THIS
GOES TO THE REVIEW BOARD FOR
TENURE AND PROMOTION AT YOUR
INSTITUTION, DOES THIS COUNT
BECAUSE IT'S A NULL RESULT?
NOW, IT'S A GOOD PUBLICATION,
IT'S NEW ENGLAND JOURNAL OF
MEDICINE.
BUT DOES IT COUNT?
BECAUSE WHAT DIFFERENCE HAVE YOU
MADE IN PRACTICE?
RIGHT?
AND THOSE OF YOU WHO SIT ON
THESE TENURE AND PROMOTION
BOARDS, IT WILL BE FASCINATING
WHAT YOU ALL THINK ABOUT THAT --
JUST AS AN EXAMPLE OBVIOUSLY BUT
THERE ARE MANY, MANY MORE
EXAMPLES LIKE THIS.
NOW, I JUST WANTED TO SHARE WITH
YOU SOME GUIDING PRINCIPLES THAT
DR. BERTAGNOLLI HAS BEEN
ARTICULATING BECAUSE IT REALLY
BEARS ON WHAT YOU ALL ARE GOING
TO BE DOING HERE DURING THIS
CONFERENCE.
MONICA HAS INDICATED THAT OUR
WORK IS NOT FINISHED WHEN WE
DIFFERENT A SCIENTIFIC
DISCOVERY.
THE WORK IS FINISHED WHEN PEOPLE
ARE LIVING LONG AND HEALTHY
LIVES.
SO IT'S NOT JUST THE
PUBLICATION, IT'S HOW THAT IS
TRANSLATED INTO PRACTICE.
THE RESEARCH COMMUNITY IS NOT
ONLY THE LABORATORY, IT'S NOT
ONLY THE CLINIC, BUT IT'S ALSO
THE COMMUNITY.
SHE REALLY WANTS TO STRETCH
THINGS OUT AT THE LEVEL OF WHERE
PEOPLE ARE.
AND FINALLY, PROGRESS IS
ACCELERATED WHEN ADVANCED
SCIENTIFIC METHODS SUCH AS THE
NEW DATA ANALYTICS, ARE APPLIED
TO DATA THAT INCLUDES EVERYONE,
AND WHEN NEW DISCOVERIES ARE
RAPIDLY AND EQUITABLY, EQUITABLY
ADOPTED IN CLINICAL CARE.
NOW EACH ONE OF THESE POINTS
BEARS ON THE CONVERSATION THAT
YOU ALL WILL BE HAVING OVER THE
NECK FEW DAYS, A -- THE NEXT FEW
DAYS, AND AS YOU THINK THROUGH
THE PERPLEXING, CHALLENGING
ISSUES THAT YOU'RE WRESTLING
WITH, SEE HOW THEY ALIGN OR
DON'T WITH SOME OF THE THINGS
THAT THE NEW NIH DIRECTOR HAS
BEEN DISCUSSING.
SO WITH THAT, MY EMAIL ADDRESS
IF YOU WANT TO SEND ME ANY HATE
MAIL.
I APOLOGIZE FOR PARACHUTING IN
AND PARACHUTING OUT, BUT I HAVE
TWO MORE THINGS LIKE THIS I HAVE
TO DO THIS MORNING, BECAUSE
DR. BERTAGNOLLI IS ON TRAVEL,
BUT IT WAS REALLY DELIGHTFUL TO
BE HERE WITH YOU THIS MORNING,
AND I REALLY HOPE THE CONFERENCE
GOALS WELL AND I LOOK FORWARD,
SHAI, TO HEARING WHAT YOU ALL
CAME UP WITH.
SO AGAIN, THANK YOU ALL VERY
MUCH.
>> GOOD MORNING TO YOU ALL.
WOW, THIS IS SO EXCITING.
IT'S BEEN SO LONG IN THE
WAITING.
EVEN THOUGH HE PARACHUTED OUT, I
WOULD LIKE TO THANK DR. TABAK
AGAIN FOR HIS VERY SUPPORTIVE
COMMENTS.
HE'S REALLY BEEN A TRUE CHAMPION
FOR RIGOR AT THE NIH FOR MANY
YEARS, AND HIS EFFORTS HAVE
REALLY INCREASED ATTENTION TO
SCIENTIFIC RIGOR BY NIH.
AS YOU MIGHT HAVE SURMISED, I'M
SHAI SILBERBERG.
I'VE BEEN FOR MANY YEARS
CHAMPIONING RIGOR AND
TRANSPARENCY AT NINDS.
AND THE ISSUE OF PUBLICATION
BIAS HAS BEEN A MAJOR CONCERN OF
THE OFFICE OF RESEARCH QUALITY
AT NINDS FOR QUITE SOME TIME, SO
WE'RE REALLY EXCITED ABOUT THE
DISCUSSION WE'RE GOING TO HAVE
IN THESE NEXT TWO DAYS.
SO I WANT TO START BY THANKING
ALL THE INVITEES WHO TOOK FROM
THEIR PRECIOUS TIME TO COME AND
DEVOTE TO THIS WORKSHOP.
AND YOUR BREADTH OF EXPERTISE,
ACADEMIC DISCIPLINES, AND CAREER
STAGES, I'M SURE WILL GREATLY
ENRICH OUR DISCUSSIONS ABOUT
BIAS AGAINST NULL RESULTS.
WHICH PERMEATE MOST IF NOT ALL
SCIENTIFIC FIELDS.
I ALSO WANT TO THANK NINDS
LEADERSHIP FOR THEIR CONTINUOUS
AND WHOLEHEARTED SUPPORT.
THE NINDS RIGOR WORKING GROUP,
WHICH PROVIDED GREAT ADVICE
DURING THE PLANNING STAGES OF
THIS WORKSHOP, THE LOGISTICS
TEAM, TOYA ROGERS, NINDS, HOLLY
RILEY AND THE REST OF THE TEAM,
AND LAST BUT CERTAINLY NOT
LEAST, DEVON AND MARIAH, WHO DID
MOST OF THE HEAVY LIFTING
LEADING UP TO THIS DAY.
SO I HAVE A DISCLAIMER TO MAKE
BEFORE WE MOVE ON, AND THAT'S
THE OPINIONS THAT THE THREE OF
US WILL LIKELY VOICE ARE NOT
OFFICIAL OPINIONS OF NIH.
THAT'S AN IMPORTANT POINT TO
MAKE.
SO IN A NUTSHELL, THE WORKSHOP
IS STRUCTURED AS FOLLOWS.
WE'RE GOING TO HAVE WHAT I HOPE
WILL BE A VERY EXCITING AND --
NOT HOPEFUL BUT I'M SURE A VERY
EXCITING KEYNOTE BY EDWARD
MIGUEL.
THIS WILL BE FOLLOWED BY FIVE
SESSIONS WHERE WE WILL DISCUSS
THE CURRENT STATE OF AFFAIRS.
THE FIRST FOUR WILL TAKE PLACE
TODAY, AND THE LAST ONE WILL BE
FIRST ONE TOMORROW MORNING.
THESE SESSIONS WILL BE FOLLOWED
BY WORKING GROUPS.
WE'LL BREAK OUT INTO WORKING
GROUPS, WHERE EACH WORKING GROUP
WILL BE CHARGED WITH ONE GROUP
OF STAKEHOLDERS TO DISCUSS WHAT
THAT STAKEHOLDER COULD POSSIBLY
DO TO HELP REDUCE THE BIAS
AGAINST NULL STUDIES.
THE WORKING GROUPS WILL THEN
REPORT BACK TO THE ASSEMBLY, AND
THAT'S WHEN, AFTER THAT, WHERE
THE RUBBER REALLY HITS THE ROAD
WHERE WE'LL HAVE THIS LAST
SESSION WHERE HOPEFULLY WE'LL BE
ABLE TO IDENTIFY NOVEL
STRATEGIES THAT SIGNIFICANTLY
REDUCE PUBLICATION BIAS.
AND THE HOPE IS THEN TO SHARE
THESE IDEAS WITH THE SCIENTIFIC
COMMUNITY TO ACCELERATE CHANGE.
I'M SHOWING YOU HERE THE
MASTERPIECE BY MATISSE TO MAKE
TWO IMPORTANT POINTS.
ONE IS THAT I'M SURE AS WE WILL
DISCOVER THROUGHOUT THIS
WORKSHOP, NO SINGLE STAKEHOLDER
CAN SOLVE PUBLICATION BIAS ON
THEIR OWN.
IT WILL TAKE A CONCERTED EFFORT.
AND THE SECOND POINT I WANT TO
BRING UP IS THAT THE STRUCTURE
OF THE WORKSHOP IS SUCH THAT
MOST OF THE TIME WILL BE DEVOTED
TO DISCUSSIONS.
AND HOPEFULLY HAND IN HAND, WITH
ALL THE GREAT MINDS WE HAVE
HERE, WE WILL BE ABLE TO
IDENTIFY NOVEL STRATEGIES THAT
COULD SIGNIFICANTLY REDUCE
PUBLICATION BIAS.
BUT FIRST WE HAVE TO DEFINE WHAT
WE MEAN BY PUBLICATIONS.
SO IN THIS REGARD, I WOULD FIRST
LIKE TO PERSONALLY DEDICATE THIS
WORKSHOP TO ROBERT ROSENTHAL.
ROBERT ROSENTHAL PASSED AWAY
EARLIER THIS YEAR, AND HE'S
FAMOUSLY KNOWN FOR COINING THE
TERM "THE FILE DRAWER PROBLEM."
HE WROTE, FOR ANY GIVEN RESEARCH
AREA, ONE CANNOT TELL HOW MANY
STUDIES HAVE BEEN CONDUCTED BUT
NEVER REPORTED.
THE EXTREME VIEW OF THE FILE
DRAWER PROBLEM IS THAT JOURNALS
ARE FILLED WITH THE 5% OF
STUDIES THAT SHOW TYPE 1 ERRORS
WHILE THE FILE DRAWERS ARE
FILLED WITH THE 95% OF STUDIES
THAT SHOW NON-SIGNIFICANT
RESULTS.
CLEARLY, HE HAD A CYNICAL SENSE
OF HUMOR.
BUT YOU CAN SEE HERE I
HIGHLIGHTED THE WORD "STUDIES,"
AND THAT'S REALLY IMPORTANT,
BECAUSE HE WAS TALKING ABOUT
STUDIES AND NOT RESULTS, AND
THAT IS IMPORTANT FOR HOW WE'RE
GOING TO DESCRIBE AND THINK
ABOUT PUBLICATION BIAS IN THE
CONTEXT OF THIS WORKSHOP.
TO UNDERSTAND THIS DIFFERENCE,
LET'S LOOK AT THE EARLIEST
ACCOUNT OF WHAT WAS CONSIDERED
TO BE PUBLICATION BIAS.
MARCUS CICERO, IN THE FIRST
CENTURY B.C., DESCRIBED AN EENT
THAT TOOK PLACE 300 YEARS
EARLIER.
AND HE WROTE, DIAGORAS, WHO IS
CALLED THE ATHEIST, BEING AT
SAMOTHRACE, ONE OF HIS FRIENDS
SHOWED HIM SEVERAL PICTURES OF
PEOPLE WHO HAD ENDURED VERY
DANGEROUS STORMS.
SEE, SAYS HE, YOU WHO DENY A
PROVIDENCE, HOW MANY HAVE BEEN
SAVED BY THEIR PRAYERS TO THE
GODS?
AY, SAYS DIAGORAS, I SEE THOSE
WHO WERE SAVED BUT WHERE ARE
THOSE PAINTED WHO WERE
SHIPWRECKED?
SO IS THIS PUBLICATION BIAS?
WELL, IT DEPENDS.
TO UNDERSTAND WHY IT DEPENDS,
LET'S DO AN EXPERIMENT.
LET'S PRETEND THAT WE'RE AN
ANCIENT GREEK SCIENTIST AND WE
WANT TO TEST THE HYPOTHESIS THAT
IF YOU PRAY TO THE GODS, YOU
WILL BE SAVED FROM DANGEROUS
STORMS.
SO THE SCIENTIST SENDS THREE
SHIPS TO SAIL THROUGH A VERY
DANGEROUS STORM AND INSTRUCTS
THE CREW TO PRAY TO THE GODS TO
SAVE THEIR LIVES.
TWO SHIPS MAKE IT ACROSS, ONE
GOES DOWN.
NOW SINCE TWO SHIPS MADE IT
ACROSS, THE SCIENTIST SAYS, OH,
WOW, MY HYPOTHESIS SEEMS TO
WORK, YOU KNOW, THE SHIP WENT
DOWN, MAYBE BECAUSE THEY DIDN'T
PRAY HARD ENOUGH OR MAYBE THEY
FORGOT TO PRAY.
SO LET'S JUST PUBLISH THE
RESULTS THAT SHOW THAT MY
HYPOTHESIS IS CORRECT.
AND HE PUBLISHES IT AS A MURAL
OR MOSAIC IN THE TEMPLE.
THIS IS NOT PUBLICATION BIAS IN
THE CONTEXT OF THIS WORKSHOP.
THIS IS SELECTIVE REPORTING.
OKAY?
BECAUSE NOT ALL THE RESULTS WERE
NULL.
IN CONTRAST, LET'S PRETEND THAT
ALL THREE SHIPS WENT DOWN.
IN THIS CASE, CLEARLY THE
HYPOTHESIS WERE NOT SUPPORTED BY
THE RESULTS OF THE EXPERIMENT.
SO EITHER THE INVESTIGATOR OR
THE TEMPLE CURATORS DECIDED THAT
THESE RESULTS ARE NOT
INTERESTING AND NOT WORTHY OF
PUBLICATION, AND THEY REMAIN
SILENT ABOUT IT.
THIS IS THE KIND OF PUBLICATION
BIAS WE'RE TALKING ABOUT HERE,
OKAY?
WHERE THE ENTIRE STUDY IS NULL.
NOW, AN ENTIRE STUDY COULD BE
EVEN A SINGLE EXPERIMENT, BUT
IT'S THE WHOLE STUDY.
WAS NULL.
OKAY.
SO THAT CONCEPT THAT IT'S
IMPORTANT TO REPORT ALL STUDIES,
NOT ONLY THOSE THAT SUPPORT THE
HYPOTHESIS, IS BY NO MEANS NEW.
SO CLAUDE BERNARD IN 1865 WROTE,
IF A HYPOTHESIS IS NOT VERIFIED
AND DISAPPEARS, THE FACTS WHICH
IT HAS ENABLED US TO FIND ARE
NONT THE LESS ACQUIRED AS
INDESTRUCTIBLE MATERIALS FOR
SCIENCE.
SO HOW OVE ARE INDESTRUCTIBLE
MATERIALS FOR SCIENCE NOT SHA
SHARED?
THAT'S THE TOPIC FOR DISCUSSION
IN THE FIRST SESSION.
SADLY, ERICK TURNER WHO WAS
SUPPOSED TO MODERATE THE SESSION
COULDN'T MAKE IT SO MARIAH AND
DEVON WILL CO-MODERATE THE
SESSION.
AND WE'LL HAVE VERY INTERESTING
DISCUSSIONS, I'M SURE.
WE'LL HEAR SOME DATA -- SOME --
WHY AM I BLANKING ON THE WORD?
WE'LL GET SOME DATA FROM REAL
EXPERIMENTS AS WELL AS MANY DATA
FROM MODELS, AND THE PROBLEM
WITH MODELS ARE THAT THEY ALL
ARE BASED ON ASSUMPTIONS, AND
WHEN YOU HAVE ASSUMPTIONS, WE'RE
PROBABLY GOING TO HEAR A BROAD
RANGE OF OPINIONS OF WHAT THE
MAGNITUDE OF PUBLICATION BIAS
IS.
I PERSONALLY PREFER GOOD OLD
DATA AS A SOURCE OF INFORMATION.
SO I'M GOING TO SHOW YOU TWO
STUDIES, EMPIRICAL STUDIES
RELATED TO PUBLICATION BIAS.
IN THE FIRST, THEY LOOKED AT
PROGNOSTIC MARKERS, AND THEY
CHECKED OUT TWO DATABASES.
IN THE FIRST DATABASE, THEY HAD
A TOTAL OF 304 ARTICLES AND THEY
FOUND THAT ONLY 4 WERE FULLY
NEGATIVE FOR ALL PRESENTED
RESULTS IN THE ABSTRACT.
A SECOND DATABASE THEY LOOKED AT
HAD A TOTAL OF 1,575 ARTICLES,
AND ONLY 21 OF THEM WERE
NEGATIVE.
THE SECOND STUDY FOCUSED ON
ANIMAL STUDIES OF ACUTE ISCHEMIC
STROKE, AND THEY FOUND THAT ONLY
6, 1.2%, DID NOT REPORT AT LEAST
ONE SIGNIFICANT FINDING.
CLEARLY, OUR WORKING HYPOTHESIS
CAN'T BE CORRECT 98% OF THE
TIME.
I MEAN, IF THAT WAS THE CASE,
WE'RE REALLY NOT DOING
CUTTING-EDGE SCIENCE, RIGHT?
SO THERE SEEMS TO BE, YOU KNOW,
AN ISSUE THAT THERE ARE STUDIES
THAT AREN'T PUBLISHED, MANY OF
THEM PROBABLY, AND THEY CAN HAVE
SEVERE CONSEQUENCES.
ONE SEVERE CONSEQUENCE IS,
ORVETION FINANCIAL.
OF
COURSE, FINANCIAL.
SO WHAT I'M SHOWING YOU HERE IS
THE DISTRIBUTION OF FUNDS BY
EACH OF THE INSTITUTES AT NIH IN
FISCAL YEAR 2023, DEVOTED TO THE
MOST COMMON FUNDING MECHANISM AT
THE NIH, THE R01 MECHANISM.
IN TOTAL, IN FISCAL YEAR '23,
NIH SPENT CLOSE TO $16 BILLION
TO SUPPORT THE R01 MECHANISM.
NOW LET'S BE VERY GENEROUS, AND
ASSUME THAT 80% OF THE TIME WHEN
WE COME UP WITH AN HYPOTHESIS,
WE'RE CORRECT.
AND OF THE 20% WHERE THE RESULTS
ARE NULL, HALF OF THEM, WE DO
NOT PUBLISH, OKAY?
IT MEANS TAKE 10% OF OUR TOTAL
STUDIES AREN'T PUBLISHED.
IN THAT CASE, IT'S NOT VERY
DIFFICULT MATH, CLOSE TO
$1.6 BILLION IN FISCAL YEAR '23
WERE WASTED DUE TO LOSS OF
KNOWLEDGE JUST FROM THE
R01 MECHANISM.
BUT IT GOES WELL BEYOND THE
DIRECT FINANCIAL LOSS.
IF RESULTS ARE NOT SHARED,
STUDIES ARE NOT SHARED, THEN
OTHER SCIENTISTS WILL PROBABLY
REPEAT THOSE EXPERIMENTS, AND
THEY ALSO WON'T SEE ANYTHING SO
THEY WON'T REPORT THAT.
AND THIS CAN GO ON SEVERAL
TIMES.
AND THAT'S TREMENDOUS LOSS OF
TIME, EFFORT AND RESOURCES.
FURTHERMORE, WHEN WE DO
META-ANALYSIS, PUBLICATION BIAS
AND P-HACKING WHICH WE'LL HEAR
ABOUT LATER ON, WILL LEAD TO
MISLEADING OR EXAGGERATED EFFECT
SIZES.
NOW IF EFFECT SIZE IS
EXAGGERATED, FUTURE STUDIES WILL
LIKELY BE UNDERPOWERED BECAUSE
THEY BASE WHAT THEY'RE GOING TO
USE BASED ON THIS FICTITIOUS
OVERESTIMATION OF THE EFFECT
SIZE.
AND THESE UNDERPOWERED STUDIES
NOW COULD LEAD TO LOTS OF FALSE
POSITIVES, AND, THEREFORE, MIGHT
LEAD -- THAT'S FUNNY.
WE'RE STUCK HERE.
THERE WE GO.
TO BASELESS CLINICAL TRIALS.
SO GIVEN THE GRAVITY OF
PUBLICATION BIAS, WHY ARE SO FEW
STUDIES ACTUALLY PUBLISHED?
SO THAT WILL BE THE TOPIC OF
DISCUSSIONS IN SESSIONS
2 THROUGH 5.
THE SECOND SESSION WILL FOCUS
SPECIFICALLY ON DISCUSSIONS
RELATED TO TRADITIONAL
MANUSCRIPTS, AND IN PREPARATION
FOR THIS, MARIAH DID A VERY
INTERESTING ANALYSIS.
WE ARE COMMONLY -- IT'S KIND OF
LIKE YOU KNOW, EVERYONE TALKS
ABOUT THAT THERE ARE JOURNALS
THAT ACCEPT NULL STUDIES.
AND YOU'RE LIKE OH, I DON'T KNOW
WHAT YOU'RE TALKING ABOUT, PLUS
ONE EXCEPTIONAL STUDIES.
SO THE MAJORITY OF THEM DO NOT
TRACK IT AND THE TWO THAT GOT
NUMBERS, AS YOU CAN SEE, THE
FRACTION IS VERY, VERY SMALL.
MARIAH ALSO DID A VERY
INTERESTING ANALYSIS, SHE WENT
THROUGH THE WEB PAGES OF 215
NEUROSCIENCE JOURNALS THAT
PUBLISHED PRE-CLINICAL RESEARCH,
AND HAD AN IMPACT FACTOR OF 2 OR
GREATER.
SHE FOUND THE MAJORITY OF THEM
DON'T SAY ANYTHING ON THE
WEBSITE ABOUT ACCEPTING NULL
STUDIES.
AND THERE ARE 35 THAT DO AND
THEY CAN BE DIVIDED INTO TWO
CATEGORIES, 14 ACCEPTED THEM
UNCONDITIONALLY, THEY JUST SAID
WE ACCEPT NULL STUDIES.
21 OF THEM HAD VARIOUS
CONDITIONS, LIKE YOU CAN ONLY
PUBLISH AS A BRIEF OR SHORT
REPORT, AND SO ON AND SO FORTH,
AND PERHAPS WHAT IS MOST
INTERESTING IS THESE TWO
JOURNALS THAT SAID SINCE THE NET
EFFECT OF NEGATIVE RESULTS IS TO
DISCOURAGE REPETITION, THE
STANDARD FOR ACCEPTANCE AS
NEGATIVE RESULTS WILL BE HIGHLY
DEMANDING.
I MEAN, THIS IS VERY STRANGE IN
MY OPINION.
BECAUSE IT SUGGESTS THAT
STANDARDS FOR ACCEPTING POSITIVE
RESULTS WILL NOT BE HIGHLY
DEMANDING.
SO THAT KIND OF LINKING INTO
WHAT DR. TABAK SAID, YOU KNOW,
IF YOU DO STUDIES REALLY
CAREFULLY, YOU CAN GET INTO --
YOU KNOW, BUT ALL STUDIES SHOULD
BE DONE VERY CAREFULLY AND ALL
STUDIES SHOULD BE REVIEWED VERY
CAREFULLY, ET CETERA.
SO MARIAH REACHED OUT TO THE
EDITORS OF THESE 35 JOURNALS
THAT MENTIONED NULL RESULTS.
ONLY FOUR PROVIDED NUMBERS ON
HOW MUCH THEY PUBLISH AND AS YOU
CAN SEE, THE NUMBERS ARE VERY
SMALL.
SO PEOPLE HAD -- THE QUESTION
IS, WHY IS THIS THE CASE?
WHY -- I MEAN, IF NULL RESULTS
ARE SO IMPORTANT, WHY ARE SO FEW
STUDIES PUBLISHED?
NULL RESULTS PUBLISHED?
AND ONE POSSIBLE REASON IS THAT
REVIEWERS MIGHT REVIEW THEM MORE
HARSHLY.
AND HERE I'M GOING TO DESCRIBE A
VERY INTERESTING STUDY BY
EMERSON AND COLLEAGUE.
THEY GENERATED TWO VERSIONS OF A
WELL-DESIGNED RANDOMIZED CONTROL
TRIAL, AND THEY ONLY DIFFERED IN
THE OUT COME, WHETHER IT WAS
POSITIVE OR NULL.
AND THEY SUBMITTED TO TWO
JOURNALS WHO WERE PART OF THIS
STUDY, AND THE JOURNALS IN PLACE
SENT THESE OUT TO REVIEWERS TO
REVIEW OBVIOUSLY EACH REVIEWER
GOT JUST ONE OF THE VERSIONS,
AND WHEN THEY TABULATED, THEY
GOT COMMENTS BACK FROM 210
REVIEWERS AND WHEN THEY
TABULATED THEM, YOU CAN SEE THAT
97% RECOMMENDED ACCEPTING THE
MANUSCRIPT FOR PUBLICATION IF
THE RESULTS WERE POSITIVE, WHILE
ONLY 80% DID THAT WHEN THE
RESULTS WERE NULL.
SO WE HAVE AN ISSUE HERE.
SO IN GIVING THAT FEW JOURNALS
ACTUALLY SPECIFICALLY SAY WE
REALLY WOULD LIKE YOU TO SUBMIT
NULL RESULTS AT LEAST IN THE
NEUROSCIENCES, AND THE NOTION,
WHICH IS VERY COMMON, THAT
REVIEWERS WILL TREAT IT MORE
HARSHLY, AS MY EDITORS AS WELL,
IT WOULD NOT BE SURPRISING IF
THE SCIENTIFIC COMMUNITY WOULD
BE RELUCTANT TO GO TO THE
TROUBLE TO WRITE UP PAPERS AND
SUBMIT THEM IF THE STUDY IS
NULL.
SO IN THIS REGARD, THIS IS A
VERY INTERESTING STUDY BY FRANCO
AND COLLEAGUES.
THEY LOOKED AT THE FATE OF
221 SURVEY-BASED STUDIES IN THE
SOCIAL SCIENCES.
I NEED MORE COFFEE.
THAT WERE ALL FUNDED BY THE
NATIONAL SCIENCE FOUNDATION.
AND WHAT THEY FOUND IS THAT IF
THE RESULTS OF THE SURVEY WERE
STRONG AND THEY DEFINE STRONG AS
ALL OR MOST OF THE HYPOTHESES
WERE SUPPORTED BY THE
STATISTICAL TESTS, ONLY 4% WERE
NOT WRITTEN UP BY THE
INVESTIGATORS.
HOWEVER, IF THE RESULTS WERE
NULL, 64% WERE NOT WRITTEN UP.
SO WE CAN SEE HERE, YOU KNOW,
THAT THERE IS AN ISSUE.
AND THIS KIND OF, I GUESS,
LED -- NOT THIS PAPER BUT BY THE
WAY, NEIL MALHOTRA IS THE
CORRESPONDING AUTHOR ON THIS
STUDY AND I'M SURE HE'LL GIVE --
ON THE STUDY WHEN HE SPEAKS.
BUT THE SCIENTIFIC COMMUNITY
FELT THAT MAYBE WHAT IS NEEDED
ARE DEDICATED JOURNALS TO NULL
RESULTS SO THAT THE REVIEWERS
WILL ALL BE ON THE SAME PAGE,
THEY KNOW THEY'RE GOING TO BE
LOOKING AT NULL STUDY, AND THAT
WILL SOLVE OUR PROBLEM.
AND OF COURSE THE FLIP SIDE OF
THAT IS SAYING OH, NULL RESULTS
SHOULD HAVE THEIR OWN SEPARATE
PLACE IN THE UNIVERSE.
TO OUR KNOWLEDGE, SINCE 2002,
THERE HAVE BEEN EIGHT CREDIBLE
JOURNALS FOR NULL STUDIES.
AND THEY DID NOT FARE VERY WELL.
IF YOU LOOK IN TOTAL, THEY
PUBLISHED JUST 604 ARTICLES
SINCE 2002.
TO PUT THINGS IN PERSPECTIVE,
PNAS ALONE PUBLISHED MORE THAN
77,000 RESEARCH ARTICLES IN THE
SAME TIME PERIOD.
OKAY?
SO CLEARLY, THIS IS NOT THE
SOLUTION.
SO PERHAPS WHAT IS NEEDED --
WHAT'S THIS?
AM I ADMITTING TOYA HERE?
WHO'S ADMITTING HER?
MIKE?
THANK YOU.
SO CLEARLY, YOU KNOW,
TRADITIONAL JOURNALS DON'T SEEM
TO BE DOING -- MAYBE WE COULD
FOCUS ON NON-TRADITIONAL
FORMATS.
SO THAT'S GOING TO BE THE TOPIC
OF DISCUSSION IN SESSION THREE.
AND WE'LL HEAR ABOUT INTERESTING
DIFFERENT MECHANISMS LIKE
PREPRINTS, LIKE REGISTERED
REPORTS, LIKE MICROPUBLICATIONS,
AND ALSO INTERESTING VENUES LIKE
OCTOPUS AND ARCADIA SCIENCE.
PAUL STERNBERG, WHO'S ON THIS
PANEL, HAD A VERY INTERESTING --
IS RUNNING A VERY INTERESTING
INITIATIVE WHICH I WANTED TO
BRIEFLY MENTION.
SO IT'S CALLED MICRO PUBLICATION
BIOLOGY.
AND IT PUBLISHES SINGLE
VALIDATED RESULTS THAT INCLUDE
NOVEL FINDINGS, NEGATIVE AND/OR
REPRODUCIBLE RESULTS, AND
RESULTS THAT ARE PERCEIVED TO
LACK HIGH IMPACT.
AND THEY DID EVERYTHING TO MAKE
IT EASY AND I WOULD SAY
WORTHWHILE FOR SCIENTISTS TO
SUBMIT TO MICRO PUBLICATION, SO
THERE'S JUST A NOMINAL FEE FOR
SUBMISSION.
IT'S PEER REVIEWED, IT'S
PUBLISHED ONLINE, IT CAN BE
FOUND ON PUBMED, AND THE DATA IS
INCLUDED, UNDERLYING DATA WHICH,
OF COURSE, ADDS CREDIBILITY TO
THE MICROPUBLICATION.
NEVERTHELESS, FROM OUR ANALYSIS,
FROM THE 808 MICROPUBLICATIONS
THAT HAVE BEEN PUBLISHED, ONLY
54% WERE DESIGNATE BY MICROPUBL
DAITION BIOLOGY AS NEGATIVE
RESULTS.
SO WE CAN HAVE A DISCUSSION
WHETHER THIS IS A LOT OR A
LITTLE.
I THINK GIVEN THE MAGNITUDE OF
THE LIKELIHOOD THAT HYPOTHESIS
WILL NOT BE CORRECT, I WOULD
GUESS THIS NUMBER SHOULD BE
HIGHER.
OKAY.
SO THE NEXT ISSUE I WANT TO
DISCUSS, WHICH IS SESSION FOUR,
WILL FOCUS ON INSTITUTIONS.
BECAUSE INSTITUTIONS CAN ALSO
CONTRIBUTE AS A NEGATIVE FORCE
TO SUBMISSION OF NULL RESULTS.
AND TO EXPLAIN THIS POINT, LET'S
JUST SAY THE FOLLOWING.
SO WE KNOW THAT RESEARCH IS --
WHAT DRIVES RESEARCH IS FUNDS.
AND THAT TYPICALLY COMES THROUGH
GRANT SUPPORT.
AND IN ORDER TO GET GRANT
SUPPORT, YOU HAVE TO SHOW THAT
YOU ARE A PRODUCTIVE SCIENTIST,
WHICH MEANS THAT YOU PUBLISHED A
LOT, AND IN THE ERA OF
BIBLIOMETRICS, THAT YOU HAVE TO
PUBLISH IN WHAT THEY CALL HIGH
PROFILE JOURNALS.
AND THAT MEANS THAT YOU HAVE
TO -- IT HAS TO HAVE A HIGH
IMPACT FACTOR, FOR THAT YOUR
RESULTS HAVE TO BE NOVEL,
INNOVATIVE AND SIGNIFICANT, AND
THIS CYCLE OCCURS AT LEAST IN
THE U.S. EVERY FOUR YEARS OR SO,
WHICH MEANS THAT IT'S UNLIKELY
THAT YOU WILL SPEND THE TIME TO
TRY TO WRITE UP A PAPER WHICH IS
NULL BECAUSE IT'S UNLIKELY THAT
IT'S GOING TO MAKE ITS WAY INTO
A HIGH PROFILE JOURNAL, AND EVEN
IF IT DOES, YOU DON'T KNOW HOW
THE INSTITUTIONS ARE GOING TO
TREAT IT BECAUSE THEY WANT YOU
TO PUBLISH IN HIGH PROFILE
JOURNALS, AND IF YOU PUBLISH IT
IN THE LOWER LEVEL JOURNAL, THEN
IT WON'T LOOK GOOD ON YOUR
TENURE AS DR. TABAK MENTIONED,
ON YOUR TENURE AND PROMOTION
COMMITTEES.
SO OF COURSE THIS IS AN
IMPORTANT ISSUE THAT WE'RE GOING
TO BE DISCUSSING.
SO THIS IS SESSION FOUR HERE,
WHICH WILL DISCUSS NOT ONLY
WE'LL HEAR ABOUT FROM ERIN
MCKIERNAN ABOUT ISSUES THAT WHAT
INSTITUTIONS DON'T DO TO SUPPORT
NULL STUDIES, BUT WE'LL ALSO
HAVE LOTS OF INTERESTING
DISCUSSIONS OF WHAT INSTITUTIONS
ARE DOING TO SUPPORT NULL
STUDIES.
IN SESSION FIVE, WE'LL TAKE A
MUCH MORE BROADER LOOK ON WHAT
CAN INCENTIVIZE REPORTING OF
NULL STUDIES, AND IN THIS
SESSION, WE'LL BE DISCUSSING
INITIATIVES SUCH AS THE
REPRODUCIBILITY NETWORKS THAT
ARE SPROUTING ACROSS THE WORLD,
AND VENUES SUCH AS -- SORRY, AND
OTHER INITIATIVES SUCH AS THE
PRIZE GIVEN OUT BY THE -- AT
VETERAN SCIENCES AND ALSO WHAT
SITE -- IS TRYING TO DO TO
ADDRESS PUBLICATION BIAS.
SO IN SUMMARY, WHAT I'VE SHOWED
YOU IN THIS BRIEF INTRODUCTION
IS LIMITED DATA SUGGESTING THE
PUBLICATION BIAS IS A BIG
PROBLEM THAT SHOULD BE SOLVED.
AND BACK TO OUR FAMOUS MATISSE
PICTURE, I THINK IT WILL BE
CLEAR THAT ALL THE STAKEHOLDERS
WHO SOMEHOW HAVE TO WORK
TOGETHER TO ADDRESS PUBLICATION
BIAS, AND THAT WILL BE THE TASK
AHEAD OF US IN THE NEXT COUPLE
OF DAYS AND IT WILL BE VERY
INTERESTING TO SEE WHAT WE COME
UP WITH.
I JUST WANT TO END WITH A LITTLE
BIT OF LOGISTICS, WHICH PROBABLY
SHOULD HAVE BEEN THE FIRST
SLIDE.
SO FOR COFFEE AND SNACKS AND THE
LUNCH BOXES FOR THOSE WHO
ORDERED THEM, WHEN YOU EXIT THE
ROOM, TAKE A RIGHT, GO DOWN THE
HALL, AND THAT'S WHERE IT IS.
RESTROOMS, YOU HAVE TO WALK BACK
TO WHERE YOU CAME IN AND THE
RESTROOMS ARE BEHIND THE
SECURITY DESK.
THEY'LL TAKE YOUR I.D. EVERY
TIME YOU GO.
FOR THOSE WHO DID NOT ORDER A
LUNCH BOX AND DID NOT SIGN UP
FOR DINNER, THERE ARE MANY
RESTAURANTS TO CHOOSE FROM
ACROSS THE STREET, PIKE AND
ROSE.
BUT SINCE YOU ONLY HAVE ONE HOUR
FOR LUNCH BREAK, WE SUGGEST THAT
YOU TAKE A TAKEOUT, BRING IT
BACK HERE AND EAT IT WITH THE
REST OF US HERE.
AND LASTLY, SINCE ALL THE
DISCUSSIONS IN THIS ROOM ARE
VIDEOCAST, WE KINDLY ASK THAT
YOU ALL USE MICROPHONES WHEN
ADDRESSING THE OTHERS.
WE HAVE TWO HERE IN THE ROOM AND
I'M SURE WE'LL RUN AROUND AND
HAND THEM FOR PEOPLE WHO CAN'T
GET UP BECAUSE THEY STUCK IN THE
MIDDLE.
AND PLEASE INTRODUCE YOURSELVES
WHEN YOU DO SPEAK AT THE
MICROPHONE.
SO WITH THAT, THANK YOU FOR YOUR
ATTENTION, AND I'M LOOKING
FORWARD TO A GREAT MEETING.
>> SO IT MY GREAT, GREAT PLEASUE
TO INVITE EDWARD MIGUEL AS OUR
KEYNOTE SPEAKER.
AS DISTINGUISHED PROFESSOR OF
ECONOMICS, PROFESSOR OF
ENVIRONMENTAL AND RESOURCE
ECONOMICS AT UC BERKELEY, HE
EARNED A BACHELOR OF SCIENCE IN
BOTH ECONOMICS AND MATHEMATICS
FROM MIT AND A PH.D. IN
ECONOMICS FROM HARVARD
UNIVERSITY.
ED'S MAIN RESEARCH FOCUS IS
AFRICAN AMERICAN ECONOMIC
DEVELOPMENT, INCLUDING WORK ON
THE ECONOMIC CAUSES AND
CONSEQUENCES OF VIOLENCE, THE
IMPACT OF ETHNIC DIVISIONS ON
LOCAL COLLECTIVE ACTION, AND
INTERACTIONS BETWEEN HEALTH,
EDUCATION, ENVIRONMENT, AND
PRODUCTIVITY FOR THE POOR.
PRETTY COOL.
NOW YOU KNOW WHY I HAD TO READ
IT AND NO, MA NOT MEMORIZE IT.
IN 2012, HE CO-FOUNDED THE
BERKELEY INITIATIVE FOR
TRANSPARENCY IN THE SOCIAL
SCIENCES TO IMPROVE THE
CREDIBILITY OF SCIENCE BY
ADVANCING TRANSPARENT
REPRODUCIBILITY, ETHICS AND
RESEARCH.
WE HAVE A LOT IN COMMON.
FURTHERMORE, IN 2021, HE AND
OTHERS PROPOSED A FRAMEWORK TO
REPORT ALL RESULTS EFFICIENTLY
IN THE SOCIAL SCIENCES, WHICH
OBVIOUSLY HAS DIRECT RELEVANCE
TO OUR WORKSHOP.
THIS COMPREHENSIVE FRAMEWORK
DEMONSTRATES THAT PUBLICATION
BIAS IS NOT UNIQUE TO BIOMEDICAL
RESEARCH, AND, HENCE, TO
MITIGATE PUBLICATION BIAS WILL
LIKELY REQUIRE CULTURAL REFORM.
THIS BRIEF INTRODUCTION DOES NOT
DO JUSTICE TO HIS CREDENTIALS,
ACHIEVEMENTS, BUT FOR THE SAKE
OF TIME, AND MY IGNORANCE WHEN
IT COMES TO SOCIAL SCIENCES, I
WILL STOP HERE AND PLEASE JOIN
ME IN WELCOMING DR. MIGUEL.
IS
>> THANKS SO MUCH FOR HAVING ME.
IT'S REALLY GREAT TO BE HERE.
THANKS FOR THE INTRODUCTION AND
I'M REALLY EXCITED ABOUT THIS
MEETING, GETTING TO SHARE WHAT
WE'RE DOING ACROSS FIELDS, WHERE
I THINK WE'LL FIND A LOT OF
COMMONALITY, SO AGAIN, I GO BY
TED, TED MIGUELING FROM
BERKELEY.
I'M EXCITED ABOUT WHAT MAY COME
OUT OF THIS MEETING.
ABOUT FIVE YEARS A WE HELD A
KIND OF RELATED MEETING FOR
SOCIAL SCIENTISTS THAT LED TO
THE FRAMEWORK THAT SHAI
MENTIONED AND JUST THE KIND
OF -- THE ENERGY AND THE
INTERACTIONS ACROSS FIELDS CAN
REALLY LEAD TO NEW IDEAS AND
HOPEFULLY BRING THINGS FORWARD
FOR THE FIELD.
ONE OF THE THINGS I'M GOING TO
TRY DO IS BRING OUT WHAT WE'VE
LEARNED IN ECONOMICS, TR TRY TO
CREATE POINT OF CONTACT.
I'M GOING TO HAVE THREE MAIN
THEMES.
SO I'M ACTUALLY GOING TO START
FROM FIRST PRINCIPLES FROM THE
BEGINNING OF WHAT WE'RE TRYING
TO DO AS SCIENTISTS, WHAT IS THE
SCIENTIFIC ETHOS, WHAT ARE OUR
GOALS.
THEN I'LL DISCUSS SOME OF WHAT
WE'VE BEEN DOING AND TRENDS IN
THE SOCIAL SCIENCES BEFORE
ENDING WITH SOME NEW DATA ON
PUBLICATION BIAS IN ECONOMICS,
WHICH I THINK IS JUST DIRECTLY
LINKED TO SOME OF THE IDEAS IN
THIS WORKSHOP.
AND I'LL TALK ABOUT THE
FRAMEWORK THAT SHAI MENTIONED AS
WELL FOR GOING FORWARD.
I THINK THERE'S A FEW MINUTES
FOR QUESTIONS AT THE END SO I'M
REALLY EXCITED TO ENGAGE ACROSS
FIELD AND THINK OF COMMONALITY
AND DIFFERENCES TOO.
SO LET ME START FROM THE
BEGINNING AND MAYBE IT'S
APPROPRIATE TO START THE
CONFERENCE, YOU KNOW, STEPPING
BACK A LITTLE BIT.
SO I'M GOING TO START BY TALKING
ABOUT WHAT OUR AIMS SHOULD BE AS
RESEARCHERS, WHAT SHOULD BE
GUIDING US, WHAT SHOULD OUR
PRINCIPLES AND VALUES BE.
ONE OF THE MOST INFLUENTIAL
ARTICULATIONS OF THE SCIENTIFIC
ETHOS WAS PUT TOGETHER BY ROBERT
MERTON, A FAMOUS SOCIOLOGIST OF
SCIENCE, WHO REALLY THOUGHT
ABOUT THE NORMS AT INSTITUTIONS
THAT LEAD SCIENTISTS IN A
CERTAIN DIRECTION AND THE KIND
OF VALUES THAT MOTIVATE THEM AND
HE TALKED ABOUT NORMS AS BINDING
THAT ARE BOTH EFFICIENT BUT ALSO
SEEMED TO BE SORT OF NORMATIVELY
RIGHT OR MORALLY RIGHT.
THERE WAS SOMETHING DEEPER THAN
JUST EFFICIENT SCIENTIFIC
PRODUCTION MOTIVATING
RESEARCHERS.
AND I THINK IT'S KIND OF
INSPIRING TO THINK SOME OF
THESE PRINCIPLES AND HOW THEY
LINK UP TO OUR WORK.
SO WHAT WERE THE FOUR CORE
VALUES THAT MERTON TALKED ABOUT?
I'LL VERY QUICKLY GO THROUGH
THEM IN TURN.
THEY'RE LISTED HERE:
UNIVERSALISM, COMMUNALITY,
DISINTERESTEDNESS AND ORGANIZED
SKEPTICISM.
I'LL HAVE A WORD OR TWO TO SAY
ABOUT EACH OF THESE AND HE AND
OTHERS HAVE WRITTEN ABOUT THE
ORIGINS OF THESE NORMS AND HOW
OFTEN THEY GO BACK CENTURIES
INTO THE KIND OF CULTURE OF
SCHOLARS EVEN CENTURIES AGO IN
EUROPEAN MONASTERIES.
SO WHAT IS THIS FIRST VALUE OF
UNIVERSALISM THAT MOTIVATES
SCIENCE?
THE FIRST IDEA IS THAT THE
RESEARCH FINDINGS AT THEIR BASE
ARE IMPERSONAL.
THE SCIENCE AND DATA STANDS ON
ITS OWN.
IT DOESN'T MATTER WHAT YOUR
IDENTITY IS, ETHNIC BACKGROUND,
GENDER IDENTITY, WHERE YOU'RE
FROM.
THE MATHEMATICAL PROOF IS THE
SAME WHOEVER WRITES IT, THE
SCIENTIFIC DATA SPEAKS
REGARDLESS OF WHO THE AUTHORS
ARE.
SO WE SHOULD BE JUDGING THE
SCIENCE FOR THE SCIENCE, NOT
JUST WHO THE PROTAGONIST IS OR
WHO THE AUTHOR IS.
AND HERE, THEN, YOU KNOW,
THERE'S AN IMMEDIATE IMPLICATION
WHICH IS RESEARCH COMMUNITIES OR
SOCIETIES THAT REALLY OPEN UP
THE SCIENTIFIC ENTERPRISE TO A
BROADER ARRAY OF PEOPLE ARE
GOING TO MAKE MORE PROGRESS.
THERE SHOULDN'T JUST BE ONE
SLICE OF PEOPLE WHO ARE KIND OF
WELCOMED INTO THE SCIENTIFIC AND
OTHERS NOD.
EVERYBODY CAN CONTRIBUTE,
EVERYBODY'S SCIENCE CAN MOVE THE
FIELD FORWARD.
SO ACTUALLY HE MAKES THE LINK
BETWEEN BEING IN AN OPEN,
DEMOCRATIC SOCIETY AND
SCIENTIFIC PROGRESS IN THE LONG
RUN.
AND I THINK WHEN WE THINK ABOUT
OUR COUNTRY HERE IN THE U.S.,
OUR DEMOCRATIC INSTITUTIONS, HOW
WE'VE TRIED TO MAKE THE
SCIENTIFIC ENTERPRISE MORE
INCLUSIVE, YOU KNOW, IMPERFECTLY
BUT EFFORTS TO DO THAT, THOSE
ARE THINGS THAT ARE GOING TO
ACCELERATE THIS SCIENCE GOING
FORWARD.
SO THAT'S ONE KEY VALUE,
UNIVERSALISM.
THE SECOND ONE IS COMMUNALITY.
OPEN SHARING OF SCIENCE IS
CRITICAL.
WE ARE A COMMUNITY, WE SHARE OUR
RESULTS WITH EACH OTHER, AND
THAT'S HOW WE ADVANCE AS A
COMMUNITY, HIDING OUR RESULTS
DOESN'T ADVANCE SCIENCE.
SO HERE WE'RE ALREADY STARTING
TO GET INTO THE ISSUES OF
PUBLICATION BIAS, IF SCIENTISTS
ARE HIDING RESULTS, SELECTIVELY
REPORTING RESULTS, NOT OPENLY
SHARING WHAT THEY'RE DOING, IT
REALLY GOES AGAINST THIS CENTRAL
NORM OF OPEN SHARING SO THE
WHOLE COMMUNITY CAN LEARN FROM
OUR FINDINGS.
AND IF WE'RE NOT DOING THAT,
WE'RE REALLY NOT CONSISTENT WITH
THIS PRINCIPLE.
ANOTHER KIND OF IMPLICATION HE
WRITES ABOUT AND OTHERS IS THE
TENSION, YOU COULD SAY, BETWEEN
THE GOALS OF SCIENCE AND
COMMERCIAL GOALS.
NOW, I KNOW IN BIOMEDICAL
RESEARCH AND ALSO IN SILICON VA
LESILICONVALLEY AND THE BAY AREI
COME FROM, THERE'S A LOT OF
COMMERCIALIZATION OF RESEARCH
FINDINGS, BUT ON SOME LEVEL
THERE'S A TENSION THERE BECAUSE
THEN IF YOU KIND OF TAKE
INTELLECTUAL PROPERTY OUT OF THE
SCIENTIFIC COMMUNITY AND TRY TO
COMMERCIALIZE IT, MAYBE YOU'RE
NOT FULLY SHARING ANYMORE.
YOU'RE KIND OF KEEPING SECRETS
FROM THE REST OF THE SCIENTIFIC
COMMUNITY.
SO THOSE ARE A TENSION.
THE THIRD VALUE, SCIENTIFIC
VALUE, IS DISINTERESTEDNESS.
WE AS RESEARCHERS SHOULD BE
MOTIVATED BY THE TRUTH.
WE SHOULDN'T BE THINKING ABOUT
THE IMPLICATIONS FOR OUR CAREERS
OR WHAT OTHER PEOPLE THINK ABOUT
US FROM REPORTING SCIENTIFIC
TRUTH.
OUR RESPONSIBILITY IS TO REPORT
THAT TRUTH TO THE REST OF THE
COMMUNITY AND DEAL WITH THE
CONSEQUENCES OF REALITY.
OF THE
TRUTH AND NOT OBSESS ABOUT
MONETARY GOALS OR OUR PERSONAL
MOTIVATIONS.
AND YOU KNOW, THIS IS A VERY
IMPORTANT ISSUE.
WE ALREADY HEARD IN A COUPLE OF
THE TALKS, SHAI'S TALK AND THE
EARLIER TALK, ABOUT HOW CAREER
MOTIVATIONS AND TENURE REVIEWS
REALLY FACTOR INTO WHAT
SCIENTISTS ARE DOING AND A LOT
OF THE CHOICES THEY MAKE.
THERE IS A TENSION THERE
POTENTIALLY WITH
DISINTERESTEDNESS.
WE SHOULD BE REPORTING RESULTS
REGARDLESS OF THE CONSEQUENCES.
EVEN IF THERE'S A NULL RESULT,
IT'S IMPORTANT FOR SCIENCE AND
THE COMMUNITY TO KNOW THIS, AND
THAT'S REALLY OUR RESPONSIBILITY
AS SCIENTISTS.
MERTON, AS A SOCIOLOGIST, WAS
THINKING ABOUT THE SOCIAL
STRUCTURES AND NORMS THAT REALLY
COULD PUSH PEOPLE TOWARDS THIS
ETHOS, AND I THINK THAT'S
SOMETHING VERY IMPORTANT TO
THINK ABOUT, AND WE'RE GOING TO
COME BACK TO, I THINK, IN BOTH
DAYS OVER AND OVER AGAIN, HOW
CAN STRUCTURES AND INCENTIVES
AND INSTITUTIONS BE SET UP TO
MAKE SURE SCIENTISTS REALLY LIVE
UP TO THIS VALUE.
THE FOURTH SCIENTIFIC PILLAR IS
ORGANIZED SKEPTICISM.
THE ABILITY TO VERIFY DATA,
VERIFY RESULTS, IS REALLY
CRITICAL FOR SCIENCE TO ADVANCE.
WE CAN'T JUST SAY OH, TRUST ME,
TRUST US, AGAIN, IF THERE'S A
MATHEMATICAL PROOF, OTHER PEOPLE
HAVE TO WORK THROUGH IT.
IF THERE'S DATA OUT THERE,
OTHERS NEED TO BE ABLE TO
REPRODUCE THE RESULTS, REPLICATE
THE RESULTS, CHECK THE RESULTS.
THAT'S REALLY CENTRAL TO WHO WE
ARE AS SCIENTISTS, AND ONE OF
THE THINGS THAT SETS US APART
FROM MANY OTHER FIELDS.
SO AGAIN, THIS OPENNESS, OPEN
SHARING, ABILITY FOR OTHERS TO
REALLY INTERROGATE RESULTS IS
CENTRAL TO PROGRESS OF OUR WHOLE
ENTERPRISE.
ALL RIGHT.
SO THESE FOUNDATIONAL SCIENTIFIC
NORMS, I FIND THEM INSPIRING.
I THINK IT'S EXCITING TO THINK
THAT WE'RE PART OF A PROCESS AS
RESEARCHERS ACROSS FIELDS THAT
GOES BACK CENT REE CENTURIES THS
VALUES WE CAN BELIEVE IN AND
VALUES THAT CAN ADVANCE THE
WHOLE COMMUNITY, BUT CENTRAL TO
THIS WHOLE ARTICULATION OF THE
SCIENTIFIC ETHOS IS OPEN
COMMUNICATION AND COMPLETE
SHARING OF FINDINGS.
IF WE'RE NOT DOING THAT, WE'RE
NOT LIVING UP TO THESE VALUES
AND WE'RE SLOWING DOWN SCIENCE.
WE HEARD THE VERY GOOD POINTS
ABOUT WE'RE WASTING RESEARCH
FUNDS POTENTIALLY IF WE'RE NOT
DOING IT.
MAYBE WE'RE LEADING OTHERS TO
TRY TO REPLICATE RESULTS OR
EXTEND RESULTS THAT AREN'T
REALLY SOLID TO BEGIN WITH.
WE COULD BE LEADING SCIENCE INTO
DEAD ENDS.
SO OPEN SHARING IS CENTRAL, NOT
JUST FOR US BUT FOR THE WHOLE
COMMUNITY.
SO THEN THE QUESTION IS, HOW
CLOSELY DO WE AS RESEARCHERS
LIVE UP TO THESE VALUES TODAY?
IN THE BIOMEDICAL SCIENCES, IN
THE SOCIAL SCIENCES WHERE I COME
FROM, THE DATA THAT SHAI JUST
SHOWED US OF HOW THE EXTENT OF
THE PUBLICATION BIAS AND LACK OF
NULL RESULTS IN THE LITERATURE
SUGGEST THERE IS A GAP BETWEEN
THESE VALUE AND OUR PRACTICES,
BUT SOME PEOPLE HAVE TRIED TO
QUANTIFY THESE SO I'LL BRIEFLY
MENTION ONE STUDY, ANDERSON ET
AL., AND I HAVE THE REFERENCES
FOR THOSE WHO ARE INTERESTED,
STUDIED THE DEGREE TO WHICH
PEOPLE SUBSCRIBE TO THESE NORMS,
THESE MERTONIAN NORMS.
DO WE AS SCIENTISTS SORT OF
BELIEVE IN THESE THINGS IN THESE
VALUES?
AND THEN ALSO, SURVEYED THEM ON
HOW THEY VIEW THEIR RESEARCH
COMMUNITIES.
HOW CLOSE IS BEHAVIOR TO THESE
NORMS, AND ACTUALLY IT'S BASED
ON A SAMPLE OF R01 RECIPIENTS
FROM NIH, SO THERE WERE A LOT OF
BIOMEDICAL SCIENCES, SOME
DEMOGRAPHERS AND SOCIAL
SCIENTISTS IN THE SAMPLE AS
WELL, AND THE KEY FINDING HERE,
AND I'LL SHOW YOU A FIGURE NEXT,
IS THERE'S ACTUALLY VERY HIGH
RATES OF ATTACHMENT TO THESE
VALUES AMONG SCIENTISTS.
WE WANT TO LIVE UP TO THESE
VALUES.
WE BELIEVE IN THEM.
WE THINK THEY'RE IMPORTANT.
WE THINK THEY'RE THE RIGHT WAY
FORWARD.
BUT PEOPLE ARE DISI DISILLUSIOND
BECAUSE THEY FEEL LIKE OUR
COMMUNITIES HAVEN'T ATTAINED
THESE VALUES, THAT OUR PRACTICES
AREN'T LIVING UP TO THEM, SO
WHAT THEY CALL NORMATIVE
DISSIDENCE, THIS GAP BETWEEN
NORMS AND PRACTICE.
LET ME JUST SHOW YOU SOME OF
THEIR DATA.
WHAT THEY HAVE HERE IS BOTH
AMONG MID CAREER AND EARLY
CAREER SCHOLARS, SUBSCRIPTION TO
THESE VALUES.
SO THEY ASKED A WHOLE SERIES OF
SURVEY QUESTIONS TO SEE IF
PEOPLE BELIEVED IN OPENNESS OR
SECRECY, SHARING OR NOT SHARING,
ET CETERA.
AND THE LIGHT GREY MEANS YOU
VERY MUCH ARE ATTACHED TO THESE
VALUES.
THE MIDDLE GREY MEANS YOU'RE
SOMEWHAT ATTACHED TO THEM, THE
DARK COLOR MEANS UROPOSED TO
THEM.
90-SOMETHING PERCENT OF SCHOLARS
THAT WERE VAIRED SAY THESE ARE
THE RIGHT VALUES, WE BELIEVE IN
THIS, WE THINK SHARING, OPEN
COMMUNICATION IS THE RIGHT WAY
FORWARD.
SO THAT'S EXCITING AND IT MEANS
THIS VALUE SYSTEM IS ALIVE AND
WELL, BUT WHEN YOU JUMP DOWN TO
THE BOTTOM PANEL AND PEOPLE ARE
ASKED, WHAT DO YOU THINK THE
BEHAVIOR OF COLLEAGUES IN YOUR
FIELD, HOW DOES THAT LOOK, ONLY
ABOUT 10% OF PEOPLE THINK THEIR
COLLEAGUES ARE LIVING TO THESE
VALUES.
SO THERE'S THIS HUGE GAP BETWEEN
WHAT PEOPLE WANT TO DO AND WHAT
WE THINK WE SHOULD BE DOING AS
SCIENTISTS, AND HOW THEY FEEL
THEIR COLLEAGUES ARE ACTUALLY
BEHAVING.
SO I THINK THIS IS THE KIND OF
GAP THAT COULD MAYBE EXPLAIN
SOME OF THE PATTERNS WE'VE SEEN,
AND THEN IT KIND OF BEGS THE
QUESTION, HOW CAN WE BRING OUR
PRACTICES BACK IN LINE WITH OUR
VALUES?
AND I THINK DEALING WITH
PUBLICATION BIAS AND PREVENTING
IT AND REALLY THE GOALS OF THIS
WORKSHOP ARE REALLY AN IMPORTANT
STEP IN THAT DIRECTION.
SO THAT'S THE KIND OF I THINK
HIGH LEVEL ABOUT WHY WE'RE HERE,
WHY THIS MATTERS, WHAT OUR
VALUES SHOULD BE.
I WANT TO NOW TAKE A TURN INTO
THE SOCIAL SCIENCES AND IN
PARTICULAR ECONOMICS.
MY OWN FIELD, JUST TO GIVE A
LITTLE BIT OF BACK STORY AS TO
OUR WORK -- I THINK THERE'S A
FEW OTHER SOCIAL SCIENTISTS IN
THE FIELD.
I SEE NEIL BACK THERE AND I'VE
BEEN AT A FEW OF THESE MEETINGS
THROUGH THE YEARS.
MY OWN WORK IN THIS FIELD
STARTED WHEN I WAS DOING WORK IN
WEST AFRICA AND SIERRA LEONE
RIGHT AFTER THE END OF THE CIVIL
WAR.
SO THE CIVIL WAR ENDED IN 2002,
AND IN 2004, 5, WE WERE WORKING
TOGETHER, MY TEAM AND BERKELEY
WITH THE WORLD BANK AND THE
GOVERNMENT OF SIERRA LEONE ON
SOME RECONSTRUCTION EFFORTS.
IT WAS A PRETTY EXCITING TIME TO
BE THERE.
THERE WAS A LOT OF HOPE AND
OPTIMISM THAT THERE COULD BE
ECONOMIC REFORMS, POLITICAL
REFORMS THAT COULD LEAD SIERRA
LEONE FORWARD AND REALLY KIND OF
PREVENT THE RETURN TO CIVIL WAR.
SO THE STAKES WERE VERY HIGH.
AND I THINK FOR EVERYBODY HERE
WHO'S DOING RESEARCH ON
NEUROSCIENCE, THE STAKES ARE
HIGH.
OTHER PARTS OF BIOMEDICAL
RESEARCH.
CERTAINLY IN THIS CASE WHEN
WE'RE TALKING ABOUT WAR AND
PEACE, SUPER HIGH STAKES TO
THESE REFORMS.
THE REFORM THAT THE GOVERNMENT
WITH FUNDING FROM THE WORLD BANK
WAS ADVOCATING WAS CALLED GOBIFO
WHICH IN CREOLE, THE LANGUAGE IN
SIERRA LEONE MEANS MOVE AHEAD,
MOVE FORWARD, WAS A LOCAL
GOVERNMENT REFORM THAT TRIED TO
IMPROVE REPRESENTATIVENESS OF
LOCAL GOVERNMENT COUNCILS.
SO TRADITIONALLY, HEREDITARY
CHIEFS HAD A LOT OF AUTHORITY IN
RURAL AREAS IN SIERRA LEONE AND
THERE WAS A FEAR THAT THAT MAY
HAVE CONTRIBUTED TO POLITICAL
INSTABILITY AND CONFLICT AND THE
HOPE WAS AFTER THE WAR, THERE
COULD BE ELECTED LOCAL COUNCILS
INCLUDING BRINGING WOMEN ON THE
COUNCILS, MEMBERS OF
MARGINALIZED GROUPS ON TO
COUNCILS, NOT JUST THESE
HEREDITARY LEADERS TO LEAD TO
MORE DEMOCRATIC LOCAL GOVERNANCE
AND HAVE BETTER OUTCOMES.
SO A PRETTY EXCITING REFORM.
BUT ALSO CONTROVERSIAL.
IT WAS VERY POLITICALLY CHARGED.
THE CHIEFS WEREN'T THRILLED BY
THIS.
AND ALSO, IN THE ACADEMIC
LITERATURE, THE NOTION OF HOW
FEASIBLE IT EVEN WAS TO REFORM
LOCAL GOVERNMENT INSTITUTIONS
WAS VERY CONTESTED.
VERY CONTROVERSIAL.
SO OUR TEAM WHEN WE WERE WORKING
ON IT DECIDED TO DO SOMETHING
THAT WAS INSPIRED ACTUALLY BY
WHAT HAPPENS IN CLINICAL TRIALS,
WE DECIDED TO PREREGISTER OUR
ANALYSIS BEFORE CARRYING OUT
THIS RCT OF LOCAL GOVERNMENT
REFORM WHICH TOOK PLACE ACROSS
HUNDREDS OF VILLAGES IN SIERRA
LEONE.
WE WROTE DOWN AN ANALYSIS PLAN,
WE WROTE DOWN THE HYPOTHESES
THAT MOTIVATED US, WHAT OUR OUT
COME MEASURES WOULD BE, WHAT OUR
STATISTICAL TESTS WOULD BE, AND
WE ACTUALLY SHARED THEM WITH OUR
GOVERNMENT AND WORLD BANK
PARTNERS AND SAID, HEY, ARE WE
ON THE SAME PAGE?
ARE THESE THE RIGHT OUTCOMES
THAT WE SHOULD BE LOOKING AT OR
NOT?
AND WE BUILT CONSENSUS, WE
MODIFIED OUR HYPOTHESES AND GOT
EVERYBODY TO AGREE.
OUR FEAR WAS IF WE FOUND
SOMETHING THAT WAS NOT
POLITICALLY PALATABLE, THERE
WOULD BE PRESSURE PLACED ON US
TO CHANGE RESULTS, NOT PUBLISH
RESULTS, AND THIS IS SOMETHING
THAT HAPPENED IN SOCIAL SCIENCE
RESEARCH, ESPECIALLY IF YOU'RE
STUDYING GOVERNMENT POLICY.
SO THAT WAS OUR INNOVATION.
THIS WAS ONE OF THE FIRST COUPLE
FREE ANALYSIS PLANS IN
ECONOMICS.
IT WAS SOMETHING A LITTLE
STRANGE AT THE TIME.
BUT THAT WE BELIEVED IN AND IT
TURNED INTO SOMETHING OF A
MOVEMENT IN THE SOCIAL SCIENCES.
SO OVER TIME IN ECONOMICS, IN
POLITICAL SCIENCE AND OTHER
FIELDS, IT'S ACTUALLY AS I'LL
SHOW YOU BECOME THE NORM WHEN
OUR FIELD RCTs ARE RUN,
RANDOMIZED CONTROLLED TRIALS, TO
WRITE PRE-ANALYSIS PLANS AND
PRE-REGISTER THEM.
SO IT WAS AN EXCITING PROCESS.
I WILL SAY JUST ANOTHER ONE
SECOND ON THIS STUDY, WE'VE
BASICALLY FOUND NULL RESULTS OF
THIS REFORM.
WE BASICALLY FOUND THIS ATTEMPT
TO IMPROVE INCLUSION, HAVE
GREATER LOCAL PUBLIC GOODS, ET
CETERA, THERE WAS NO IMPACT OF
THIS REFORM.
WE HAVE A NEGATIVE RESULT, A
NULL PAPER.
I THINK IT ONLY REALLY GOT
PUBLISHED BECAUSE WE WERE WELL
POWERED AND HAD THIS
PRE-REGISTRATION.
SO IT GETS BACK TO THE POINT OF
DIRECTOR TABAK THAT CERTAIN
TYPES OF NULL STUDIES ARE
GETTING PUBLISHED, OUR PARTNER
IN THE GOVERNMENT WERE NOT
EXCITED ABOUT THAT RESULT, BUT
WE'D ALL AGREED ON THE ANALYSIS
BEFOREHAND THAT KIND OF SAVED
US.
WHAT ENDED UP HAPPENING TO THE
COUNCILS IS ACTUALLY THE
TRADITIONAL CHIEFS TOOK OVER THE
COUNCILS.
THEY HAD THEIR OWN ALLIES
BASICALLY GET ELECTED AND THEY
CONTINUED TO CONTROL THE COUNCIL
SO NOTHING CHANGED WITH THE
REFORM.
BUT WE WERE ABLE TO PUBLISH
THAT, I THINK, IN PART BECAUSE
OF PRE-REGISTRATION.
OKAY.
SO IN THE LAST 20 YEARS, THINGS
HAVE CHANGED A LOT IN OUR FIELD.
I THINK IN OTHER FIELDS AS WELL,
THERE'S MUCH MORE AWARENESS
ABOUT THE IMPORTANCE OF OPEN
SCIENCE AND THE PROBLEMS OF
PUBLICATION BIAS.
SO THINGS HAVE CHANGED.
LET ME JUST SHOW YOU SOME OF THE
CHANGES THAT HAVE OCCURRED IN
ECONOMICS, AND MY OWN FIELD.
SHORTLY -- A FEW YEARS AFTER WE
PREREGISTERED OUR STUDY, THERE
WASN'T EVEN A REGISTRY WHEN WE
DID IT.
WE SENT OUR PRE-ANALYSIS PLANS
TO CERTAIN PEOPLE TO TIME STAMP
IN PUBLIC PLACES BASICALLY,
BECAUSE THERE WAS NO REGISTRY.
THE AMERICAN ECONOMIC
ASSOCIATION CREATED A REGISTRY,
AND AS I MENTIONED, IT'S BECOME
A NORM IN THEIR EXPERIMENTS TO
REGISTER, SO WE WERE INSPIRED BY
CLINICALTRIALS.GOV.
AND YOU CAN SEE THE KIND OF
CUMULATIVE NUMBER.
NOW THERE HAVE BEEN 8500
REGISTERED EXPERIMENTS IN
ECONOMICS OVER THE LAST DECADE
OR SO, AND THE PER-QUARTER
REGISTRATIONS JUST CONTINUE TO
CLIMB.
SO THIS IS SOMETHING THAT'S
REALLY TAKEN OFF, AND NORMS
SHIFTED RAPIDLY.
THERE WAS NO REQUIREMENT TO
REGISTER.
BUT PEOPLE UNDERSTOOD THE VALUE
OF IT AND NORMS SHIFTED WITHIN A
FEW YEARS IN OUR FIELD.
SOMETHING SIMILAR HAS HAPPENED
WITH EXPERIMENTS IN POLITICAL
SCIENCE AND PSYCHOLOGY, WHERE
PREREGISTRATION HAS BECOME A
NORM.
SO A LOT OF CHANGES IN OUR
FIELD.
AND OUR HOPE AT THE TIME WAS
THAT PREREGISTRATION COULD HELP
AVOID PUBLICATION BIAS.
NOW THAT YOUR STUDY IS
REGISTERED AND PEOPLE CAN SEE
WHAT YOUR HYPOTHESES ARE,
THERE'S GOING TO BE MORE
ACCOUNTABILITY AND PRESSURE TO
PUBLISH THOSE RESULTS.
THAT WAS OUR WORKING HYPOTHESIS.
I'LL HAVE A FEW WORDS TO SAY
ABOUT THAT IN A FEW SLIDES.
WHAT ELSE HAS CHANGED, THIS IS
FRI A 'NOFROM ANOTHER STUDY WITH
CO-AUTHORS WE PUBLISHED LAST
YEAR.
SORRY THE FONT IS KIND OF SMALL.
WE SURVEYED THOUSANDS OF
SCHOLARS IN ECONOMICS, POLITICAL
SCIENCE, PSYCHOLOGY AND
SOCIOLOGY.
IN THIS FIGURE, WE LOOKED AT
SCHOLARS WHO HAD BEEN ACTIVE IN
THE FIELD FOR AT LEAST ABOUT A
DECADE, AND WE ASKED THEM WHEN
IF EVER THEY HAD ADOPTED CERTAIN
OPEN SCIENCE PRACTICES LIKE
SHARING THEIR OWN DATA,
PREREGISTERING, AND YOU SEE
THESE UPWARD TRENDS OVER TIME.
THERE'S BEEN BIG SHIFTS IN
BEHAVIOR, PREREGISTRATION, DATA
SHARING, POSTING INSTRUMENTS,
AND AGAIN, SOME STUFF THAT WAS
RARE 15 YEARS AGO OR 20 YEARS
AGO HAS BECOME VERY COMMON.
NEIL WILL BE HAPPY TO HEAR THAT
POLITICAL SCIENTISTS ACTUALLY
HAVE THE HIGHEST RATES OF
ADOPTION EVEN A BIT HIGHER THAN
ECONOMISTS HERE.
AND FOR BIOMEDICAL AUDIENCE,
DATA SHARING HAS REALLY TAKEN
OFF.
THERE'S A LOT OF BARRIERS TO
SHARING INDIVIDUAL DATA IN THE
HEALTH FIELD AND OBVIOUSLY
THERE'S HIPAA CONCERNS, BUT IN
ECONOMICS AND POLITICAL SCIENCE
JOURNALS NOW, IT'S REALLY
STANDARD TO SHARE REPLICATION
DATA AND CODE, IT'S BECOME
STANDARD.
AND OTHER INNOVATIONS, I
MENTIONED STUDY PREREGISTRATION,
SHARING OF DATA CODE AND
MATERIALS.
THERE ARE NEW JOURNAL REVIEW
PRACTICES, REGISTERED REPORTS
HAVE TAKEN OFF.
THERE'S BEEN SOME GREAT
INNOVATION ON THE JOURNAL SIDE.
A LOT OF JOURNALS NOW HAVE WHAT
ARE CALLED DATA EDITORS, █WHERE
THE DATA EDITOR TEAM DOUBLE
CHECKS ALL STATISTICAL CODE AND
DATA BEFORE PUBLICATION TO
VERIFY THE ESTIMATES IN THE
PAPER ACTUALLY HOLD.
SO THERE'S THAT DEGREE OF LIKE
COLLABORATION BETWEEN THE
AUTHORS AND THE JOURNAL, THE
AMERICAN ECONOMIC ASSOCIATION
WAS ONE OF THE FIRST TO DO THAT.
AMERICAN JOURNAL OF POLITICAL
SCIENCE DID IT AS WELL.
THERE'S BEEN A LOT MORE META
SCIENTIFIC WORK IN OUR FIELDS
AND A BUNCH OF NEW ORGANIZATIONS
HAVE BEEN CREATED TO TRY TO
INSTITUTIONALIZE CHANGE AND
PROMOTE IT.
INCLUDING OUR GROUP AT BERKELEY,
BITS, THE CENTER FOR OPEN
SCIENCE AND THE INSTITUTE FOR
REPLICATION AMONG OTHERS.
SO IT'S BEEN A VERY EXCITING
TIME PERIOD AND IN MANY WAYS,
WE'VE BEEN OPTIMISTIC, WE'VE
SEEN A LOT OF CHANGES THAT ARE
POSITIVE.
WHAT I WANT TO DO NEXT IS TALK
ABOUT REMAINING CHALLENGES
BECAUSE THERE'S A LOT OF THEM.
AS WE'VE COLLECTED MORE DATA, WE
REALIZE THAT EVEN THOUGH WE'VE
TAKEN THREE OR FOUR STEPS
FORWARD IN THE SOCIAL SCIENCES,
WE STILL HAVE A LONG WAY TO GO.
SO MAYBE THERE'S SOME PARALLELS
THERE TO BIOMEDICAL RESEARCH.
SO ONE BIG CHALLENGE IS EVEN
THOUGH A LOT OF NEW TOOLS HAVE
BEEN ADOPTED IN EXPERIMENTAL
RESEARCH, THOSE CARRYING OUT
RANDOMIZED CONTROL TRIALS OR
EXPERIMENTAL LABORATORY TRIALS
LIKE PSYCHOLOGY TRIALS, THERE'S
BEEN A LOT LESS PROGRESS IN WHAT
WE CALL OBSERVATIONAL RESEARCH.
SO DATA NOT BASED ON
EXPERIMENTS.
DATA THAT'S BASED ON EXISTING
ADMINISTRATIVE RECORDS OR SURVEY
DATA.
THERE'S SOME PARALLELS HERE, I
THINK, BETWEEN THE KIND OF
PRE-CLINICAL LABORATORY PHASE OF
RESEARCH AND MAYBE THE CLINICAL
PHASE IN MEDICAL RESEARCH.
VERY OFTEN THE OBSERVATIONAL
RESEARCH THEN INSPIRES
RANDOMIZED CONTROL TRIALS.
TO TEST CERTAIN IDEAS.
BUT YOU KNOW, EVEN IN OUR OWN
DATA, THE RATE OF
PREREGISTRATION, DATA SHARING,
ET CETERA, IS LOWER IN
OBSERVATIONAL RESEARCH.
NON-EXPERIMENTAL RESEARCH.
SO THAT'S A BIG GAP, AND IT'S
HARDER TO DO TO ADOPT A LOT OF
THESE TOOLS IN OBSERVATIONAL
RESEARCH.
IT'S HARDER, FOR INSTANCE, TO
PRE-REGISTER STUDIES THAT ARE
NOT BASED ON DATA THAT HASN'T
YET BEEN COLLECTED.
WHEN THE DATA IS ALREADY OUT
THERE, PUBLICLY AVAILABLE
ADMINISTRATIVE DATA, WHEN
EXACTLY DO YOU PREREGISTER?
THERE'S BEEN A LOT OF DEBATE
ABOUT THAT IN THE SOCIAL
SCIENCES AND A LOT OF PUSH BACK
SO THAT'S REALLY AN OPEN ISSUE.
ANOTHER OPEN ISSUE OR JUST
QUESTION IS GOSH, WE'VE ADOPTED
ALL THESE PRACTICES IN THE LAST
15 OR 20 YEARS, I SHOWED YOU
THAT FIGURE WITH ALL THOSE
CURVES GOING UPWARDS, HAVE WE
DEALT WITH PUBLICATION BIAS?
HAVE WE CRACKED THE CODE OR NOT?
AND -- OR IS THERE STILL A LOT
OF PUBLICATION BIAS AND
SELECTIVE REPORTING?
WE DON'T HAVE DEFINITIVE ANSWERS
TO THIS BUT THERE'S SOME NEW
DATA AND IN FACT MY GROUP AT
BERKELEY IS JUST STARTING TO TRY
TO QUANTIFY THIS IN ECONOMICS
AND WE HAVE SOME KIND OF
DISCOURAGING RESULTS.
MAYBE THIS IS JUST A GOOD SEGUE
INTO THIS WORKSHOP.
THERE'S' THERE'S A LOT MORE TO .
WE HAVE A GREAT TEAM LED BY MY
COLLEAGUE, DR. DELA GUARDIA.
WE DECIDED TO GO BACK TO OUR
EARLY QUESTION, HOW MUCH DOES
REGISTRATION DEAL WITH
PUBLICATION BIAS?
THAT WAS OUR OPEN WITH THE STUDY
IN SIERRA LEONE.
NOW THE REGISTRY IN ECONOMICS
HAS BEEN AROUND FOR OVER A
DECADE SO WE CAN ACTUALLY START
LOOKING AT THIS.
SO WE WENT BACK TO THE STUDY
REGISTRY AND STARTED CODING UP
STUDIES STARTING WITH THOSE
REGISTERED IN 2015, 2016, 2017,
2018, WE'RE KIND OF WORKING OUR
WAY THROUGH THOSE EARLY STUDIES
ON THE REGISTRY.
A LOT OF TIME HAS PASSED SINCE
THOSE STUDIES WERE REGISTERED.
PLENTY OF TIME TO KIND OF GET
MANUSCRIPTS OUT AND GET THINGS
PUBLISHED.
AND WHAT WE'RE DOING IS GOING
THROUGH THOSE REGISTRIES, AND
CODING UP THE PRIMARY HYPOTHESES
THAT THEY LIST.
SO IN THEIR PLAN, THEY'RE LIKE,
HERE ARE OUR PRIMARY HYPOTHESES,
OUR MAIN OUTCOMES, OUR MAIN
MODELS, AND WE'VE ACTUALLY CODED
UP 136 REGISTRATIONS SO FAR.
THIS IS A LOT OF WORK BECAUSE
THERE'S NOT STANDARDIZED
PRE-ANALYSIS PLAN FORMATS
ACTUALLY IN ECONOMICS, AND
THERE'S A LOT OF JUDGMENT CALLS,
AUTHORS AREN'T ALWAYS CLEAR, SO
WE HAVE AN AMAZING TEAM THAT'S
DOING THIS CODING, AND THEN
WE'RE FOLLOWING UP THESE AUTHORS
IN THESE STUDIES TO SEE WHAT GOT
PUBLISHED NOW, EIGHT, NINE,
SEVEN YEARS LATER.
COULD BE A JOURNAL PUBLICATION,
IT COULD BE A PRE-PRINT WORKING
PAPER, IT COULD BE A REPORT FOR
A GOVERNMENT.
WE'RE JUST TRYING TO FIND IT.
IS THIS RESULT IN THE PUBLIC
DOMAIN?
SO REGISTRATIONS IN ECONOMICS ON
AVERAGE HAVE ABOUT 17 PRIMARY
HYPOTHESES.
YOU MIGHT SAY WOW, THAT'S A LOT,
BUT IF A TRIAL HAS LIKE THREE
ARMS, WHICH IS PRETTY TYPICAL
FOR ECONOMIC IS STUDIES, THERE
COULD BE DIFFERENT VERSIONS OF A
TREATMENT AND THERE COULD BE
THREE MAIN OUTCOMES OR END
POINTS AND YOU THROW IN A COUPLE
OF SUBGROUP ANALYSES AND YOU'RE
THERE, KIND OF 18 HYPOTHESES, SO
THAT'S THE AVERAGE.
MEDIAN IS 8.
SO, SO FAR WE'VE REGISTERED 2400
HYPOTHESES.
HOW MANY OF THESE ARE PUBLICLY
AVAILABLE?
KIND OF A SIMPLE QUESTION.
CAN WE FIND THESE?
THESE THINGS WERE REGISTERED
ALMOST 10 YEARS.
CAN WE ACTUALLY FIND THESE
ESTIMATES?
IT ACTUALLY GOES BACK TO NEIL'S
WORK AND IT GOES BACK TO THE
FIGURES THAT SHAI WAS SHOWING,
LIKE HOW MUCH STUFF IS
DISAPPEARING FROM THE
LITERATURE?
I ALSO WANT TO SAY I THINK THIS
IS A BEST CASE SCENARIO.
THESE ARE EXPERIMENTAL
ESTIMATES, THEY'RE
PREREGISTERED, THERE'S SOME
ACCOUNTABILITY.
PROBABLY THE SHARE OF HYPOTHESIS
TESTS THAT ARE SHOWING UP IN
OBSERVATIONAL RESEARCH IS
LIKE -- I'M ALMOST SURE IT'S
LOWER, I DON'T HAVE DATA, BUT
THIS IS PROBABLY A BEST CASE
SCENARIO.
SO HAVE WE SOLVED THE PROBLEM?
AGAIN THIS IS PRELIMINARY DATA.
WE'VE ONLY COLLECTED -- WE'RE
GOING TO DO ABOUT 400 STUDIES,
WE'VE DONE ABOUT A THIRD.
THIS IS TENTATIVE, DO CITE, BUT
I WANTED TO SHARE WITH THIS
GROUP.
SO FAR WHEN WE LOOK EIGHT YEARS
LATER, WE CAN ACTUALLY FIND SOME
DATA FOR 79% OF THE
REGISTRATIONS.
I DON'T KNOW IF YOU SEE THAT AS
GOOD OR BAD.
IT'S DEFINITELY NOT 100.
IT'S A LITTLE HIGHER THAN I WAS
EXPECTING.
BUT THAT'S STILL LIKE ALMOST A
QUARTER OF THESE REGISTRATIONS
JUST, LIKE -- THERE'S NO DATA
THAT WE COULD -- THERE'S NO
PUBLICLY AVAILABLE -- NO
PUBLICATIONS, NO RESULTS THAT WE
COULD FIND.
WE'RE INVESTIGATING THROUGH
SURVEYS AND OTHER RESEARCH HOW
MANY OF THESE ARE JUST LIKE
TOTALLY NON-REPORTED VERSUS
MAYBE THE TRIAL NEVER HAPPENED.
SO SOME OF THIS COULD BE THEY
REGISTERED IT AND THEN THEIR
FUNDING DEPARTMENT COME THROUGH
OR THEIR GOVERNMENT PARTNER SHUT
THEM DOWN.
SO WE'RE TRYING TO SORT THAT
OUT.
SO WE COULD BE GENEROUS AND SAY
WELL, MAYBE THIS IS JUST --
THESE WERE STUDIES THAT DIDN'T
OCCUR, BUT PROBABLY A LOT OF
THEM, NOTHING GOT REPORTED.
SO IN THE KIND OF TERMINOLOGY
SHAI WAS USING, THIS WOULD BE
LIKE PUBLICATION BIAS.
NOTHING CAME OUT OF THESE
STUDIES.
WE'RE TRYING TO FIGURE OUT NOW
WHY.
THEN WE DIG INTO STUDIES, SO
THIS IS RELATED TO SELECTIVE
REPORTING AND PUBLICATION BIAS,
SOME COMBINATION, AS TO HOW MANY
OF THESE ESTIMATES, THESE 2400
HYPOTHESES, CAN WE FIND RESULTS
FOR IN THE PUBLISHED LITERATURE
IN THIS DATA SO FAR?
WE CAN FINE ABOUT HALF.
FIND ABOUT HALF.
SO 79% OF STUDIES ARE OUT THERE,
BUT NOT ALL THE PRIMARY
HYPOTHESES AND THE REGISTRATION
GET REPORTED IN THE PAPER.
AND THIS IS SOMETHING THAT'S
VERY COMMON.
THERE'S LIKE THIS SLIPPAGE WHERE
ONLY A SUBSET OF OUTCOMES ARE
REPORTED, A SUBSET OF
TREATMENTS.
YOU MIGHT SAY, WELL, EVENTUALLY
MAYBE THEY'LL PUBLISH SOMETHING
THAT REPORTS THESE ESTIMATES,
BUT AGAIN, THIS IS LIKE EIGHT OR
NINE YEARS LATER.
SO WE'RE NOT SURE HOW MUCH FLOW
OF NEW PUBLISHED STUDIES THERE'S
GOING TO BE FROM THESE
REGISTRATIONS.
IF WE ONLY LOOK AT THAT 79% OF
STUDIES WHERE WE FOUND
SOMETHING, THEN ABOUT 72% OF
HYPOTHESES GET REPORTED.
SO SOME OF THIS IS JUST PURE
PUBLICATION BUY A WE DON'T SEE
THE PUBLICATION.
SOME OF IT'S SELECTIVE REPORTING
WITHIN THOSE STUDIES.
BUT ABOUT 57% OF ESTIMATES ARE
FOUND.
THEN OUR THIRD FINDING, AND I'LL
JUST LEAVE IT THERE, I'LL START
WRAPPING IT UP, SO WE WENT TO
THOSE PUBLICATIONS.
WE HAD REGISTERED THESE
HYPOTHESES AND MADE SOME
JUDGMENT CALLS BASED ON THE
RULES AS TO HOW CLOSE.
SO EVEN WHEN THERE'S A RESULT
REPORTED FOR A GIVEN HYPOTHESIS,
MANY TIMES IT DEVIATES PRETTY
SHARPLY FROM THE ORIGINAL TASK.
MAYBE A DIFFERENT STATISTICAL
TASK, A DIFFERENT MEASURE.
SO IF WE USE EVEN A KIND OF
GENEROUS DEFINITION OF BEING
CONSISTENT WITH THE ORIGINAL TAS
TASK, ONLY ABOUT 35% OF THE
ORIGINAL HYPOTHESES WERE FOUND
IN A WAY THAT WE JUDGE TO BE
CONSISTENT WITH THE ORIGINAL
REGISTRATION.
SO THIS NUMBER DOESN'T LOOK AS
GOOD.
THIS LOOKS LIKE MORE LIKE MAYBE
ONLY ABOUT A THIRD OF WHAT
PEOPLE SAID THEY WOULD DO ALMOST
A DECADE AGO IS ACTUALLY SHOWING
UP IN THE PUBLISHED LITERATURE.
THAT IS A LOT OF MISSING RESULTS
THAT WE MIGHT WANT TO SEE.
SO THIS IS, AGAIN, EARLY DATA,
WE'RE STILL PUTTING THIS
TOGETHER.
IT BUILDS ON SOME OF THE STUDIES
WE'VE BEEN HEARING WIT ABOUT, BT
WE'RE HOPING THIS WILL GIVE YOU
A MORE QUANTITATIVE TAKE ON
PUBLICATION BIAS AND SELECTIVE
REPORTING IN ECONOMICS DESPITE
ALL THE PROGRESS WE'VE MADE, SO
THERE'S A LOT WE STILL NEED TO
DO.
SO LET ME JUST WRAP UP WITH KIND
OF A WAY FORWARD.
AT THE CONFERENCE WE RAN ABOUT
FIVE YEARS AGO WHEN WE BROUGHT
TOGETHER ECONOMISTS,
PSYCHOLOGISTS, POLITICAL
SCIENTISTS AND OTHERS, WE TRIED
TO THINK THROUGH HOW TO IMPROVE
THIS KIND OF REPORTING.
WE WERE ALREADY CONCERNED THAT
THERE WOULDN'T BE GREAT
REPORTING FROM THE REGISTRY.
AND TRIED TO THINK THROUGH HOW
THE ECOSYSTEM FITS TOGETHER.
THAT'S WHY I'M PARTICULARLY
EXCITED ABOUT THIS MEETING, THE
DIFFERENT SESSIONS REALLY TALK
ABOUT DIFFERENT PIECES IN THE
PUZZLE.
PEOPLE START WITH A PROPOSAL,
THEY HAVE TO GET FUNDING, MAYBE
GO THROUGH IRB, CARRY OUT THE
DATA COLLECTION AND ANALYSIS,
AND IF YOU LACK LOOK AT THAT RE
ON TOP, THERE ARE A LOT OF
HYPOTHESES AND FINDINGS THAT
JUST DON'T GET REPORTED.
THAT'S THE PROBLEM, MAYBE TWO
THIRDS OF HYPOTHESES ARE
DISAPPEAR.
WHAT WE PROPOSED HERE WAS A
SYSTEM TO TRY TO STREAMLINE THE
REPORTING OF PREREGISTERED
HYPOTHESES BACK INTO THE PUBLIC
REGISTRY.
SO WE'VE ACTUALLY BEEN TALKING
TO FOLKS IN THE ASI A.A. REGISTY
ABOUT WAYS TO FACILITATE THAT.
OUR TEAM HAS BEEN DEVELOPING A
STREAMLINED TEMPLATE TO
CREATE -- IT'S NOT QUITE A PAPER
BUT JUST TO REPORT THOSE
HYPOTHESES IN A STANDARDIZED
STREAMLINED WAY SO THEY APPEAR
SOMEWHERE ON THE REGISTRY.
THAT'S THE ORANGE LINE THERE.
SO WE SEE THAT AS EFFICIENT
REPORTING TO REALLY REDUCE THE
KOTION TO RESEARCHERS OF DOING
IT, BUT JUST TO MAKE SURE
RESULTS GET SHARED WITH A
SCIENTIFIC COMMUNITY, BUT
JOURNALS AND FUNDING AGENCIES
AND OTHERS CAN PLAY A ROLE
BECAUSE THEY CAN ASK TO SEE YOUR
COMPLETE REPORTING BEFORE THEY
GIVE YOU FUNDING FOR THE NEXT
ROUND.
THEY COULD ASK TO SEE YOUR
COMPLETE REPORTING OF THAT TRIAL
BEFORE THEY PUBLISH YOUR PAPER.
AND AGAIN, IF IN IS DESIGNED IN
A WAY THAT ISN'T TOO ONEROUS, IT
CAN GIVE INCENTIVES TO PEOPLE TO
REPORT THEIR RESULTS, AND THEN
ALLOW OTHERS TO REPRODUCE,
REPLICATE AND KIND OF EXTEND THE
FINDINGS.
SO THIS WAS OUR PROPOSAL, WHICH
AGAIN THERE MAY BE SOME
PARALLELS TO IN BIOMEDICAL
RESEARCH.
SO LET ME JUST WRAP UP THERE
BECAUSE I THINK I'M ABOUT AT
TIME AND THEN I'M REALLY EXCITED
TO DISCUSS FURTHER AND HAVE
QUESTIONS.
AGAIN, I'M VERY HAPPY TO BE WITH
THIS GROUP.
I THINK OUR PERSPECTIVES WILL
DIFFER BETWEEN THE SOCIAL
SCIENCES AND BIOMEDICAL
RESEARCH, BUT AT THE SAME TIME,
THERE'S ACTUALLY, I THINK, A LOT
OF POINTS OF CONTACT AND I'VE
CERTAINLY ALWAYS LEARNED A LOT
ACROSS FIELDS.
I WANT TO COME BACK TO SOMETHING
THAT SHAI MENTIONED.
THIS IS SOMETHING I'M A BIG
BELIEVER IN, WHICH IS MORE THAN
JUST IMPLEMENTING A CERTAIN
REFORM OR A CERTAIN POLICY
CHANGE.
THOSE ARE IMPORTANT.
WE REALLY HAVE TO CONTINUALLY
STRIVE FOR CULTURAL CHANGE IN
OUR FIELDS.
SO IT KIND OF SHOCKS ME, EVEN
WHEN I'M ADVISING AN UNDERGRAD
THESIS PROJECT, UNDERGRADS HAVE
ALREADY INTERNALIZED A LOT OF
NORMS AND IDEAS THAT MAY NOT BE
THE BEST.
SO I'M SUPERVISING EVEN THIS
YEAR A SENIOR HONORS THESIS AT
BERKELEY, AND STUDENT -- SHE'S
RUNNING HER ANALYSIS.
IT'S A REALLY INTERESTING
PROJECT TRYING TO UNDERSTAND
CRIME IN BRAZIL AND HOW ECONOMIC
CONDITIONS MAY BE CONTRIBUTING
TO HIGH CRIME RATES IN CERTAIN
AREAS, AND SHE PRESENTS HER
RESULTS TO ME AND SAYS, YOU
KNOW, THIS FINDING ISN'T QUITE
SIGNIFICANT AT 95% CONFIDENCE
AND ALREADY SHE'S LIKE TORMENTED
BY THIS.
AND I'M LIKE, WHERE DID THIS
COME FROM?
HOW IS THIS SUCH A STRONG, YOU
KNOW, FEELING AMONG A
20-YEAR-OLD WHO'S JUST DOING HER
FIRST RESEARCH PROJECT?
BUT THOSE FEELINGS ARE OUT
THERE, AND I THINK IN ALL OF OUR
WORK, WE HAVE TO ALMOST LIKE
DE-PROGRAM OUR STUDENTS, OUR
ADVISEES, FOLKS IN OUR CLASSES
TO EMPHASIZE THE IMPORTANCE OF
THE QUESTION, STRENGTH OF THE
DESIGN, STATISTICAL POWER, THE
STATISTICAL RIGOR AS WELL, AND
REALLY THE QUALITY OF THE DATA
RATHER THAN THE SIGN OF THE
FINDING, RATHER THAN THE
STATISTICAL SIGNIFICANCE.
THOSE THINGS HAVE TO BE
REPORTED, THEY'RE GOING TO
MATTER FOR INTERPRETATION, BUT
THE QUALITY OF THE STUDY
SHOULDN'T LIVE OR DIE BASED ON
THOSE THINGS.
SO I THINK WE ALL HAVE A LOT OF
WORK TO DO.
AT BITSS, THE BERKELEY
INITIATIVE FOR TRANSPARENCY IN
THE SOCIAL SCIENCES, FROM THE
BEGINNING WHAT WE'VE DONE IS
CARRY OUT TRAINING COURSES FOR
GRADUATE STUDENTS AND POSTDOCS
IN THE SOCIAL SCIENCES TO TEACH
THEM THESE VALUES, THE
SCIENTIFIC ETHOS AND THE TOOLS
THEY CAN USE TO DO MORE
REPRODUCIBLE SOCIAL SCIENCE AND
OUR HOPE IS THERE WILL JUST BE
MORE AND MORE YOUNG CADRES OF
SCHOLARS WHO SEE SCIENCE, YOU
KNOW, IN THIS TRADITIONAL WAY,
YOU KNOW, THAT'S REALLY
CONSISTENT WITH THE SCIENTIFIC
ETHOS RATHER THAN ALWAYS
OBSESSING OVER SIGNIFICANCE AND
NOT THE QUALITY OF THE SCIENCE.
SO LET ME JUST LEAVE IT THERE
AND I LOOK FORWARD TO ANY
QUESTIONS.
[APPLAUSE]
>> HOLD IT, THE MIC, THE MIC.
>> THE MOST CONTROVERSIAL PART,
WHICH IS OBSERVATIONAL STUDIES,
AND WHEN DO YOU REGISTER, SO
COULD YOU KIND OF TELL US ABOUT
THE CONTROVERSY AND HOW YOU FALL
IN THAT RANGE OF POSSIBLE
SOLUTIONS?
>> SO IN ECONOMICS AND POLITICAL
SCIENCE, THERE'S BEEN A LOT OF
PUSHBACK TO REGISTERING
OBSERVATIONAL STUDIES.
WHAT WE'VE DONE IN SOME OF OUR
OWN OBSERVATIONAL RESEARCH AND
OTHERS HAVE DONE IS START
REGISTERING ANALYTICAL DECISIONS
THAT WE'RE MAKING ALONG THE WAY.
I THINK THE CONCERN THAT SOME
PEOPLE HAVE, AGAIN, IS THAT DID
THEY LOOK AT THE DATA FIRST
BEFORE THEY MADE THOSE CHOICES?
IT'S REALLY HARD TO RULE THOSE
THINGS OUT.
WHAT WE'VE DONE AS A TEAM, I'M
WORKING ON A TEAM NOW WITH FIVE
CO-AUTHORS, WE ALL BASICALLY
CREATE A PREANALYSIS PLAN AND
SIGN OFF TO SAY LOOK, THIS IS
THE DATA WE'VE LOOKED AT, THIS
IS THE DATA WE HAVEN'T LOOKED AT
YET, THESE ARE THE ASSUMPTIONS
WE'RE GOING TO MAKE IN OUR
MODELING, AND WE REGISTER THAT.
PEOPLE CAN USE THAT AS THEY
WILL, BUT WE'RE TRYING TO CHANGE
NORMS THERE AS WELL.
THERE'S WHOLE FIELDS OF
ECONOMICS, ESPECIALLY LIKE MACRO
ECONOMICS, WHICH USES A LOT OF
BIG PUBLICLY AVAILABLE AGGREGATE
DATA WHERE THERE HASN'T BEEN ANY
PRE ANALYSIS PLANS AT ALL.
AND THAT'S A FIELD THAT WE'RE
TRYING TO PUSH FOR CHANGE.
SO I'M WORKING ON A DEVELOPMENT
MACRO PROJECT WHERE WE DID
REGISTER A PREANALYSIS PLAN AND
ACTUALLY SOME MACRO ECONOMISTS
ARE STARTING TO THINK ABOUT HOW
THEY CAN DO IT AS WELL.
SO I DON'T KNOW HOW QUICK CHANGE
WILL BE THERE, BUT THE CONCERN
IN THE FIELD IS PEOPLE ARE GOING
TO KIND OF LOOK AT THE DATA,
MANIPULATE THINGS AND THEN GO
BACK AND WRITE A PLAN AS IF THEY
HAD -- PRETENDING THEY DIDN'T
LOOK AT THE DATA.
I THINK THAT'S THE KIND OF
ETHICAL -- YOU KNOW, ETHICAL
CONCERN.
THERE'S STILL MANY OTHER OPEN
SCIENCE PRACTICES THAT FOLKS
DOING OBSERVATIONAL
NON-EXPERIMENTAL RESEARCH CAN
DO.
THEY CAN SHARE THEIR DATA, YOU
KNOW, FOR INSTANCE, THEY CAN
HELP OTHERS REPLICATE THEIR
RESULTS, SO I DO THINK THERE ARE
WAYS FORWARD.
BUT THAT'S BEEN A BIG STICKING
POINT FOR US, AND AGAIN, THERE
MAY BE SOME PARALLELS WITH
LABORATORY OR PRE-CLINICAL
RESEARCH RELATIVE TO TRIALS AND
BIOMEDICAL --
>> HI.
-- BERNARD, THANK YOU FOR THIS
FASCINATING TALK.
WE WOULD LOVE TO HEAR HOW IN
YOUR COMMUNITY, YOU MANAGE THESE
CONSCIOUSNESS IN ADOPTING THIS
FORMAT, AND PRODUCING REGISTERED
REPORTS AND DATA SHARING AND
WHERE IT CAME FROM, WHETHER
THERE WAS AN INCENTIVE OR
ANYTHING.
BUT MY QUESTION IS A BIT TRICKY.
IT'S ABOUT NULL RESULT.
I DISAPPROVE THE USE OF THIS
TERM.
BECAUSE THIS HAS A VERY STRONG
MEANING.
SO NOT FINDING STATISTICAL
SIGNIFICANCE IN NEARLY 80% OF
THE PAPERS, I CAN TAKE ANY PAPER
PUBLISHED YESTERDAY IN NATURE
AND SCIENCE AND I WILL SHOW YOU
THAT THEY AFFIRM THE NULL,
AFFIRMING THE NULL IS A COMMON
MISTAKE IN STATISTICS.
NOW TO AFFIRM THE NULL, YOU HAVE
TO DO A DIFFERENT TYPE OF
ANALYSIS.
SO NOT FINDING STATISTICAL
SIGNIFICANCE, WHICH WE WANT TO
BE REPORTED, HAS NOTHING TO DO
WITH AFFIRMING THE NULL, BECAUSE
YOU NEED TO DO OTHER EXPERIMENTS
FROM THE NULL.
SO DID YOU SOLVE THIS PROBLEM IN
YOUR FIELD?
BECAUSE THAT'S PERHAPS THE MOST
COMMON MISTAKE OF MISUSE OF
STATISTICS, AT LEAST IN
NEUROSCIENCE.
>> SO THERE WERE TWO PARTS
THERE.
THE FIRST WAS YOU MENTIONED HOW
DID WE GAIN PROGRESS IN OUR
COMMUNITY.
LET ME SAY A WORD ABOUT THAT AND
THEN GET TO THE KINDS OF ISSUES
OF THE TYPES OF HYPOTHESIS
TESTING THAT WE'RE DOING.
I THINK ECONOMICS IS AN
INTERESTING CASE.
POLITICAL SCIENCE HAS SOME
PARALLELS, THEY'RE FIELDS THAT
OFTEN COMMUNICATE CLOSELY, THOSE
TWO FIELDS.
THE EARLY PROGRESS ACTUALLY IN
ECONOMICS WAS MADE BY THE
AMERICAN ECONOMIC ASSOCIATION.
SO THE BIG KIND OF SOCIETY HAD
LEADERSHIP THAT EVEN 20 YEARS
AGO PUSHED FOR SHARING OF DATA
AND CODE.
SO IN 2005, THE AEA IMPLEMENTED
THIS POLICY, ACTUALLY SOMEONE
VERY FAMOUS WAS THE PRESIDENT OF
THE AEA, BEN BERNANKE, WHO ENDED
UP BEING, YOU KNOW, THE HEAD OF
THE FEDERAL RESERVE WAS A
PROFESSOR AT PRINCE TOLL AT THAT
TIME.
THERE WERE A COUPLE OF
CONTROVERSIES OF
NON-REPLICABILITY AND YOU KNOW
DATA NOT BEING SHARED.
HE SAID ALL AEA JOURNALS WILL
MANDATE DATA ON CODE SHARING NOW
IT ACTUALLY ENDED UP TAKING
SEVERAL YEARS TO TRULY BE
IMPLEMENTED BUT IT SET A
STANDARD AND OTHER JOURNALS
FOLLOWED AND TRULY IT'S ALMOST
UNIVERSAL NOW.
AT THE SAME TIME, YOU MENTIONED
REGISTERED REPORTS REALLY CAME
FROM THE COMMUNITY SO OUR GROUP
AT BITSS LEARNED ABOUT
REGISTERED REPORTS IN OTHER
FIELDS, THERE WAS NONE IN
ECONOMICS OR POLITICAL SCIENCE,
AND WE BROUGHT THE IDEA TO MY
OWN FIELD'S FIELD JOURNAL, THE
JOURNAL OF DEVELOPMENTAL
ECONOMICS.
THE EDITORS WERE COLLEAGUES AND
FRIENDS OF MINE, AND WE
CONVINCED THEM, TRY A PILOT WITH
REGISTERED REPORTS.
WE'LL SUPPORT YOU WITH STAFFING
FROM OUR INITIATIVE.
AND THEY DID IT AND THERE WAS A
LOT OF INTEREST IN REGISTERED
REPORTS.
LOTS OF SUBMISSIONS.
YOUNG SCHOLARS IN PARTICULAR
WERE EXCITED ABOUT THIS NEW
FORMAT, SO THEN THEY
INSTITUTIONALIZED IT AND OTHER
ECON JOURNALS HAVE BEEN COPYING.
SO I THINK IT'S BEEN A
COMBINATION OF TOP-DOWN SOCIETY
WORK, BUT THEN ACTUALLY BOTTOM
UP SCHOLARLY WORK.
ON THE SECOND POINT, THERE ISN'T
MUCH TO SAY BECAUSE WE CONTINUE
TO DO TRADITIONAL HYPOTHESIS
TESTING AND SO THE KIND OF
DIFFERENT SORTS OF TESTS TO
AFFIRM THE NULL AND REALLY FOCUS
ON IS IT A ZERO AND WHAT TEST DO
WE WANT TO DO AROUND
UNDERSTANDING HOW PRECISE THOSE
ESTIMATES ARE, ET CETERA, THERE
ISN'T SO MUCH PROGRESS.
SO I THINK THAT'S AN AREA WHERE
MORE WORK CAN BE DONE AND MAYBE
THERE'S PARALLELS AGAIN BETWEEN
OUR FIELDS OR NON-PROGRESS
BETWEEN OUR FIELDS.
>> HI.
-- TANZA, UNIVERSITY OF
FLORIDA.
MALU TANSY, UNIVERSITY OF
FLORIDA, OF NATURE PARTNER
JOURNAL, PARKINSON'S DISEASE.
A COUPLE OF THINGS ON THE
FUNDERS.
SOME OF THE FUNDERS LIKE MICHAEL
J. FOX FOUNDATION, ALZHEIMER'S
ASSOCIATION IN OUR BIOMEDICAL
FIELD, ARE NOW REQUIRING OPEN
SCIENCE.
SO IT IS PRETTY ONEROUS, THOUGH.
YOU HAVE TO GIVE OUR IDs AND
YOU HAVE TO PUT EVERYTHING IN
CERTAIN WORK SPACES, SO NODO, ET
CETERA.
IT'S A LOT OF LOT OF LOT OF WORK
FOR THE FIRST AUTHOR,
CORRESPONDING AUTHOR.
BUT I DO THINK THAT THE ONLY WAY
TO GET CREDIT FOR THAT
PUBLICATION BY THE FUNDER IS TO
BE IN COMPLIANCE, SO THEY RUN
COMPLIANCE REPORTS ON THAT.
AND THIS IS PART OF THE ALIGNING
SCIENCE ACROSS PARKINSON'S ASAP,
MICHAEL J. FOX.
SO AS A P.I., IT HAS GIVEN US A
LOT OF HEADACHES BUT I THINK IN
THE END, THIS IS GOING TO
GUARANTEE THAT YOU DO HAVE OPEN
SCIENCE AND YOU HAVE NO OTHER
CHOICE BUT TO DO IT OR YOU WILL
NOT GET FUNDING, YOU KNOW,
RETROSPECTIVELY TO YOUR
INSTITUTION AND PROSPECTIVELY.
WHAT I WAS GOING TO SAY IS IN
TERMS OF THE RIGOR AND
REPRODUCIBILITY COURSES THAT OUR
STUDENTS HAVE TO TAKE, THIS
SHOULD BE A PART OF IT.
THERE SHOULD BE A MODULE ON
FIRST OF ALL JUST HOW TO PROPOSE
A HYPOTHESIS.
I HAVE A BIG PET PEEVE IN
BIOLOGY BECAUSE WHEN I WAS A
GRADUATE STUDENT, LEON AVERY AT
U.T. SOUTHWESTERN TAUGHT US THE
HYPOTHESIS THAT THE STATEMENT IS
ABOUT BIOLOGY, IT'S NOT THE DRUG
THAT IS GOING TO CURE OR REDUCE
HYPERTENSION.
THAT IS SOMETHING THAT IS
PREDICTED IF THE HYPOTHESIS IS
CORRECT AND YOU DO SOMETHING TO
TEST IT.
I THINK WE HAVE TO -- WE HAVE TO
GO BACK TO BASICS -- HOW TO
GENERATE HYPOTHESES BECAUSE IF
THEY DON'T KNOW HOW TO DO THAT,
I DON'T THINK THEY EVEN KNOW
WHAT -- THE NULL OR --
>> THANK YOU FOR THAT.
I THINK ON THE FIRST POINT ABOUT
FUNDERS, MANDATING THINGS, I
MEAN, IT SOUNDS LIKE IT'S BEEN
EFFECTIVE IN YOUR COMMUNITY.
AGAIN, IT'S THE LIFE BLOOD OF
OUR WORK, IS FUNDING, SO I THINK
FUNDERS HAVE A LOT OF POWER AND
THAT'S WHERE I THINK SPEAKING AT
NIH IS VERY INTERESTING BECAUSE
THEY'RE REALLY A BIG FUNDER.
EVEN IN DEVELOPMENT ECONOMICS, A
LOT OF MY RESEARCH IN KENYA IS
FUNDED BY NIA AND NICHD.
WE CARRY OUT LONG-TERM
LONGITUDAL DATA COLLECTION AND
THEY'RE INTERESTED IN THOSE
HEALTH MEASURES.
SO I DO THINK THEY'RE VERY
INFLUENTIAL.
THIS WORK TAKES TIME, AND I
THINK YOUR POINT ABOUT, YOU KN
KNOW, HOW CAN WE IMPLEMENT OPEN
SCIENCE PRACTICES TO REDUCE
PUBLICATION BIAS AND HAVE OTHER
BENEFITS IN WAYS THAT ARE LESS
ONEROUS IS A HUGE AGENDA POINT.
I'VE WORKED SO OFTEN FOR MONTHS
PREPARING DATA TO BE PUBLISHED
BECAUSE THAT'S THE STANDARD IN
OUR FIELD.
IT TAKES A LOT OF WORK AND
THOUGHT.
IT HAS A LOT OF BENEFITS BECAUSE
THEN A LOT OF OTHER PEOPLE USE
THAT DATA, AND THEN THE TOTAL
IRONY AND THE FUNNY THING ABOUT
IT IS ALL THAT WORK WE DO RIGHT
AFTER WE FINISH DATA COLLECTION
MAKES IT SO MUCH EASIER FOR ME
TO USE MY OWN DATA TWO YEARS
LATER.
BECAUSE IF I DON'T PUT
EVERYTHING IN AND DOCUMENT IT
AND POST IT, I END UP USING THE
DATA NOT THAT'S LIKE IN OUR
SECURE SERVERS.
I JUST USE THE DATA THAT'S LIKE
PUBLICLY POSTED, BECAUSE THAT, I
KNOW WE REALLY CLEANED AND
ORGANIZED.
SO ACTUALLY YOU'RE BENEFITING
YOUR FUTURE SELF IN SOME WAYS.
EVEN WITH PREANALYSIS PLANS AND
PREREGISTRATION IN ECONOMICS,
THE INITIAL PUSHBACK WAS, OH,
THIS TAKES A LOT OF TIME, IT
TAKES YOU A MONTH TO WRITE ONE
OF THESE PLANS, BUT IT SAVES US
SO MUCH TIME ON THE BACK END
WHEN WE HAVE THE DATA.
SO THERE REALLY IS THIS KIND OF
FUTURE SELF VERSUS CURRENT SELF,
YOU KNOW,IS, ISSUE AROUND A LOTF
IT.
YOU TALKED ABOUT STUDENTS AND
TRAINING STUDENTS.
WE FACE A LOT OF SIMILAR ISSUES,
THERE'S A LOT OF DEBATE IN
ECONOMICS ABOUT HOW CLOSE SHOULD
WE BE LINKING OUR HYPOTHESES TO
CORE THEORY VERSUS SOME FIELD
TESTS THAT ARE KIND OF VERY
SPECIFIC ABOUT A PARTICULAR
INTERVENTION.
THERE'S ALWAYS THIS KIND OF BACK
AND FORTH TO DO A MAPPING
BETWEEN OUR CORE THEORIES AND
THE TEST, SO AGAIN IT FEELS LIKE
THERE'S A LOT OF PARALLELS
BETWEEN OUR FIELDS.
>> HE WAS BEFORE BUT HE HAS NO
MICROPHONE SO SHOULD I SEIZE THE
DAY?
TED, AMERICAN SOCIETY FOR
MICROBIOLOGY, GREAT TALK, YOURS
AND SHAI, THE MARGINAL UTILITY
OF YOUR COST GREATLY ECK SIDES
THE MARGINAL COSTS OF THE
ADMISSION TICKET, IT'S FAN TASTE
TICK.
>> THANK YOU FOR PUTTING IT IN
LANGUAGE I UNDERSTAND.
>> QUICK QUESTION.
THIS INSTITUTE, NINDS AND NIH,
HAVE DONE A STUDY DOCTOR --
WHERE'S WALTER -- MANY YEARS AGO
ON THE SPLIT BETWEEN BASIC
SCIENCE AND CLINICAL SCIENCE,
AND GREAT THAT YOU REFER TO THE
BIOMEDICAL SCIENCES FOR
CLINICALTRIALS.GOV THAT MAKES
PERFECT SENSE.
THE QUESTION IS THAT THE NIH AND
NINDS FUNDS A LARGE AMOUNT OF
SCIENCE WHICH IS NOT CLINICAL.
AND FOR WHICH DEFINING THE
HYPOTHESIS, LET ALONE
REGISTERING IT, MAKES IT VERY
COMPLICATED AND ON TOP, I WOULD
ARGUE -- AND I'M VERY ON YOUR
PAGE, I'M NOT CRITICIZING YOU,
BUT THE VALUE OF ASKING
QUESTIONS IS ACTUALLY VERY HARD,
YOU KNOW, PROBABLY TO PUT INTO A
REGISTER, BUT IT IS EQUALLY
IMPORTANT TO HAVE THE OUTCOMES
THAT YOU'RE ALLUDING TO.
IN ANY SUGGESTIONS?
>> IT'S A FUNDAMENTAL TENSION.
I THINK THIS GETS BACK TO THE
VERY GOOD QUESTION BEFORE.
I THINK ONE OF THE VALUES OF
REGISTRATION AT LEAST IS WE KNOW
WHETHER SOMETHING IS EXPLORATORY
OR CONFIRMATORY, AND I THINK
THAT IN ITSELF IS REALLY
VALUABLE.
WHEN WE STARTED PROMOTING
PREREGISTRATION IN ECONOMICS, WE
WERE FACING THIS ISSUE, AGAIN,
ARE YOU SAYING THAT BECAUSE WE
CAN PREREGISTER SOME TYPE OF
STUDIES, OTHER TYPE OF WORK
WON'T HAVE VALUE?
ABSOLUTELY NOT.
IN FACT IN MOST, THERE ARE
ADDITIONAL ANALYSES WE DIDN'T
PLAN.
THEY JUST NEED TO BE SORT OF
FLAGGED FOR WHAT THEY ARE.
THEY'RE MORE EX EXPLORATORY RATR
THAN CONFIRMATORY.
SO EXPERIMENTING WITH
PREREGISTER OBSERVATIONAL WORK,
WHICH I THINK COULD BE THE WAY
FORWARD.
IT'S NOT PERFECT, I UNDERSTAND
THERE'S ISSUES WITH THE TIME BUG
WHY NOT TRY AND SEE WHAT NORMS
DO DEVELOP IN THE COMMUNITY.
SO AGAIN, IT MAY TAKE SEVERAL
YEARS, BUT MAYBE FOR CERTAIN
TYPES OF LABORATORY WORK,
PRE-CLINICAL WORK, THERE REALLY
ARE WAYS FORWARD.
SO I'LL GIVE YOU ANOTHER
EXAMPLE.
EVEN WITH NON-EXPERIMENTAL WORK
NOW, SOMETIMES IN THE SOCIAL
SCIENCES, PEOPLE ARE
PREREGISTERING ANALYSES BEFORE
DATA GETS RELEASED.
SO IT DOESN'T HAVE TO BE
EXPERIMENTAL WORK, BUT IF
THERE'S A NEW CENSUS ROUND THAT
ISN'T BEING RELEASED, IT'S
PROSPECTIVE, YOUR WORK, YOU
HAVEN'T SEEN THE DATA, YOU CAN
REGISTER ANALYSES.
MAYBE THERE'S OTHER TYPES OF
WORK, PRE-CLINICAL WORK THAT HAS
THAT SIMILAR TIMING WHERE IT'S
REALLY CREDIBLE THAT THE KIND OF
HYPOTHESIS REGISTRATION CAME
BEFORE DATA APPEARED, BUT OTHER
WORK COULD BE PURELY EXPLORATORY
AND KIND OF DATA MINING
EXERCISES IN THE GOOD WAY, AND
THAT'S FINE AS LONG AS WE
UNDERSTAND THAT THAT'S WHAT IT
IS.
>> HI, STEVEN CARRY, IMPERIAL
COLLEGE, RESEARCH -- ON RESEARCH
INSTITUTE AND FORMALLY -- I HAVE
A RHETORICAL QUESTION AND AN
ACTUAL QUESTION.
MY RHETORICAL QUESTION REFERS TO
YOUR LAST SLIDE.
WE ARE WE LOOKING TO TRAINING
THE NEXT GENERATION OF
SCIENTISTS TO DEAL WITH THIS?
WHY AREN'T WE FOCUSING ON THE
CURRENT GENERATION OF SCIENTISTS
AND GETTING THEM TO GET THEIR
ACT TOGETHER?
MY ACTUAL QUESTION IS AROUND --
YOU SHARED THIS IMPRESSIVE
EXPONENTIAL RISE OF REGISTERED
STUDIES IN ECONOMICS OVER THE
LAST DECADE.
WHAT PERCENTAGE OF PAPERS
PUBLISHED IN ECONOMICS ARE
PREREGISTERED?
>> GREAT QUESTION.
SO WHY THE NEXT GENERATION?
WE MATTER TOO!
THOSE OF US WHO ARE NOT THE NEXT
GENERATION MATTER, YES, AND WE
WANT TO ALL -- WE WANT TO CHANGE
PRACTICES AND CERTAINLY THERE'S
A LOT OF THAT WORK GOING ON.
I THINK THE FEELING WITH THE
NEXT GENERATION IS MAYBE THE
NORMS AREN'T AS SOLIDIFIED.
THEY'RE LEARNING HOW TO DO
SCIENCE.
AND THAT COULD BE A PATH
FORWARD.
AGAIN, IF THE NEXT GENERATION'S
NORMS CHANGES, THAT WILL JUST
SOLIDIFY THE CHANGE AS WELL.
WE'RE HEARING FROM A LOT OF THE
TRAINEES, THE POSTDOCS AND
GRADUATE STUDENTS, THEY GO TO
THE BITSS TRAINING COURSES AND
THEN THEY GO BACK TO THEIR LABS
AND THEIR RESEARCH TEAMS AND
THEY'RE THE ONES PUSHING THROUGH
CHANGES IN THEIR GROUP.
SO THEY COULD BE VERY EFFECTIVE
AGENTS OF CHANGE.
BUT I THINK IT'S NEVER TOO LATE.
IT'S NEVER TOO LATE TO CHANGE.
IN TERMS OF PREREGISTERED
STUDIES, IN ECONOMICS JOURNALS,
THERE WAS JUST RECENT
PUBLICATIONS ON THIS, SOMETHING
LIKE ONLY 25% OF STUDIES ARE
EXPERIMENTAL.
75% ARE OBSERVATIONAL WORK.
NON-EXPERIMENTAL WORK.
SO AMONG OBSERVATIONAL WORK,
THERE'S NOT MUCH
PREREGISTRATION.
AMONG EXPERIMENTAL WORK, THE
PROPORTION OF PREREGISTERED
STUDIES HAS INCREASED
DRAMATICALLY, AND ACTUALLY IN
RECENT YEARS, NOW THE AEA HAS
MANDATED THAT EXPERIMENTAL
STUDIES PUBLISHED IN AEA
JOURNALS NEED TO BE PREREJ
PREREGISTERED ON THE AEA
REGISTRY.
SO THEY DIDN'T START WITH THAT.
THE NORM ACTUALLY CHANGED FIRST
AND THAT TAKEOFF YOU SAW HAPPEN,
AND THEN ONCE THE NORM WAS
ACCEPTED, THEY KIND OF CODIFIED
IT.
SO I THINK HIGH SHARES NOW ARE
BEING PREREGISTERED, AMONG THE
EXPERIMENTAL STUDIES.
>> WELL, THANK YOU ALL FOR THIS
VERY INTERESTING DISCUSSION.
THANKS, TED, FOR A FASCINATING
TALK.
WE'LL TAKE A BREAK NOW.
WE'LL RECONVENE AT 10:30, AND
LET'S THANK TED AGAIN.
[APPLAUSE]
>> HI, EVERYONE.
I'M DEVON CRAWFORD, PROGRAM
DIRECTOR IN THE NINDS OFFICE OF
RESEARCH QUALITY.
AT THE HEARD AT THE LAST MINUTE,
ERICK TURNER WAS NO LONGER ABLE
TO ATTEND THE MEETING SO I AM
SORT OF FILLING IN AS THE
GENERAL MODERATOR.
AND THIS SESSION IS REALLY
ABOUT -- IT'S CALLED QUANTIFYING
PUBLICATION BIAS, BUT IT'S
ASKING HOW DO WE QUANTIFY
PUBLICATION BIAS, HOW DO WE KNOW
IF WE'RE FIXING PUBLICATION
BIAS, HOW BAD IS PUBLICATION
BIAS, AND HOW DO WE THINK ABOUT
IT IN THIS WAY?
AND IN ORDER TO START OFF THIS
SESSION, MARIAH HOY, PROJECT
MANAGER IN THE NINDS OFFICE OF
RESEARCH QUALITY, BECAUSE WE
FELT THAT ERICK'S CONTRIBUTION
WAS REALLY IMPORTANT SHE'S GOING
TO BE PRESENTING SOME OF HIS
DATA TO START US OFF, SO MARIAH.
>> GREAT.
THANK YOU.
SO LIKE DEVON SAID, MY NAME IS
MARIAH HOY, I'M ALSO AN OFFICE
OF RESEARCH QUALITY AND I'M
GOING TO BE SHARING SOME OF
ERICK'S EMPIRICAL DATA THAT I
THINK JUST REALLY NICELY
DEMONSTRATES HOW PUBLICATION
BIAS CAN REALLY DISTORT REALITY.
SO THIS DATA CAME FROM HIS TIME
AS AN FDA REVIEWER BETWEEN 1998
AND 2001.
HE'S NOW A FACULTY MEMBER AT
OHSU, AND HE'S BEEN THERE SINCE
2005, BUT REMAINS REALLY
INTERESTED IN THE TOPIC OF
PUBLICATION BIAS AND OUR EFFORTS
TO ESTIMATE IT.
SO DURING HIS TIME AS AN FDA
REVIEWER, ERICK BEGAN TO NOTICE
THAT WHAT WAS REPRESENTED IN THE
PUBLISHED LITERATURE WAS VERY
DIFFERENT IN TERMS OF EFFICACY
OF ANTI-DEPRESSANTS, WAS VERY
DIFFERENT THAN WHAT THE FDA
COULD SEE WHEN THEY HAD THE
OUTCOMES OF THESE CLINICAL
TRIALS BEING REPORTED TO THEM.
SO IN THIS SEMINAL PAPER, ERICK
AND COLLEAGUES SHOW HOW
PUBLICATION BIAS CAN REALLY
DISTORT OUR UNDERSTANDING OF A
SCIENTIFIC FIELD, AND
PARTICULARLY OUR INTERPRETATION
OF EFFICACY OF VARIOUS
ANTI-DEPRESSANTS.
SO IN TOTAL, THEY LOOKED AT
74 FDA REGISTERED STUDIES.
THESE CLINICAL TRIALS WERE
CONDUCTED BETWEEN 1987 AND 2006.
THE MEDIAN BEING 1996.
SO THEY LOOKED AT THE OUTCOMES
OF THESE 74 FDA REGISTERED
STUDIES, AND THEY DID A REALLY
INTERESTING ANALYSIS.
SO THEY FOLLOWED UP ON WHAT WAS
THE JOURNAL VIEW, SO HOW ARE
THESE OUTCOMES REPORTED IN
JOURNALS, AND THEN COMPARED THAT
TO WHAT THEY SAW AS FDA
REVIEWERS.
OKAY.
SO HERE I'M SHOWING YOU THE
JOURNAL VIEW FOR THE OUTCOMES OF
THESE 74 FDA REGISTERED STUDIES.
SO AS YOU CAN SEE, THE
OVERWHELMING MAJORITY OF THEM
REPORTED POSITIVE FINDINGS.
SO 51 OF THE STUDYS WERE
PUBLISHED IN TRADITIONAL
JOURNALS.
48 OF THOSE REPORTED POSITIVE
FINDINGS.
3 REPORTED NEGATIVE FINDINGS.
THE WAY THAT I'M PR REPRESENTING
THIS IS EACH STUDY IS A BOX, AND
THEN BELOW IT, THE VARIOUS DRUGS
THAT THEY WERE TESTING.
ARE
REPRESENTED BE HE LO.
>> [INAUDIBLE]
>> OH, THAT'S JUST THE NUMBER OF
THE STUDY.
THERE WERE 74 TOTAL.
SO IT'S NOT GOING TO BE
FOLLOWING IN NUMERICAL ORDER
BECAUSE NOT ALL OF THE STUDIES
WERE PUBLISHED IN TRADITIONAL
JOURNALS.
HOWEVER, AS YOU CAN SEE, THE
FDA'S VIEW OF THEIR ASSESSMENT
OF THE OUTCOMES OF THESE
CLINICAL TRIALS WAS VERY
DIFFERENT.
SO THEY ACTUALLY FOUND THAT 38
OF THOSE 74 STUDIES HAD NEGATIVE
OUTCOMES.
AND -- OR HAD POSITIVE OUTCOMES,
AND THEN 36 OF THEM HAD EITHER
NEGATIVE OR QUESTIONABLE OR
INCONCLUSIVE.
AND SO YOU CAN SEE THAT THEY
ACTUALLY FOUND THAT ABOUT 50% OF
THE TRIALS ARE ACTUALLY
NEGATIVE.
SO A VERY DIFFERENT VIEW FROM
WHAT'S PUT OUT IN THE PUBLISHED
LITERATURE.
AND THEN SO WHEN YOU LOOK AT
THOSE 36 STUDIES THAT WERE
EITHER NEGATIVE, WHICH WAS 24 OF
THEM, OR QUESTIONABLE OR
INCONCLUSIVE, WHICH WAS 12 OF
THEM, SO THREE OF THOSE HAD
ACTUALLY BEEN PUBLISHED IN
TRADITIONAL JOURNALS AS
NEGATIVE.
22 OF THEM WERE NEVER PUBLISHED,
BUT THEN 11 STUDIES WERE
ACTUALLY PUBLISHED IN JOURNALS
TO HAVE POSITIVE FINDINGS AND
THAT DIRECTLY CONFLICTED WITH
THE CONCLUSIONS MADE BY THE FDA.
AND SO THIS IDEA THAT PEOPLE TRY
TO PUT A POSITIVE SPIN ON
SOMETHING AND PUT THAT OUT
THERE.
SO I JUST THINK THIS IS LIKE ONE
OF THE BEST EXAMPLES EMPIRICALLY
THAT WE HAVE IN KIND OF THE
BIOMEDICAL SPACE SHOWING HOW
PUBLICATION BIAS CAN JUST REALLY
DISTORT OUR ABILITY TO INTERPRET
THE EVIDENCE IN A SCIENTIFIC
FIELD, AND THAT'S BECAUSE WE
DON'T HAVE ALL OF THE DATA, ALL
OF THOSE BLACK BOXES, WE
TYPICALLY DON'T HAVE.
3% OF THOSE WERE PUBLISHED OR
3 OF THOSE WERE PUBLISHED IN
TRADITIONAL JOURNALS.
AND I THINK THIS IS A REALLY BIG
PROBLEM THAT WE HAVE TO TACKLE,
THAT WE HAVE TO DO SOMETHING
ABOUT.
AND I THINK WE'LL SEE FROM OUR
NEXT SPEAKER, THIS IS IS NOT
JUST A PROBLEM WE SEE IN
BIOMEDICINE, AS WE SAW IN SHAI
AND TED'S TALKS, THIS IS A
PROBLEM THAT AFFECTS ALMOST ALL
RESEARCH FIELDS.
SO WITH THAT, I WILL LET NEIL --
>> NEIL MA HOE TRA IS FROM
STANFORD UNIVERSITY.
>> I APPRECIATE THE INVITATION
TO JOIN A GROUP OF SCIENTISTS
AND NEUROSCIENCES, BIOLOGICAL
SCIENCES AND HOPEFULLY WE CAN
LEARN FROM EACH OTHER.
BASICALLY WHAT I'VE BEEN DOING
FOR MAYBE A DECADE IS TRYING TO
USE STUDY REGISTRIES TO MEASURE
PUBLICATION BIAS.
IN MANY WAYS, I THINK THIS IS
THE BEST WAY TO QUANTIFY HOW BAD
THE PROBLEM IS.
SO TED AT THE END OF HIS TALK
USED A VERY SIMILAR METHOD WHERE
YOU TRY AND TAKE A COHORT OF
STUDIES AND FOLLOW THEM THROUGH
THE PUBLICATION PROCESS, AND
THIS MAKES A LOT LESS
ASSUMPTIONS THAN USING THINGS
LIKE FUNNEL PLOTS OR TRIM AND
FILL, AND SO I THINK IF YOU LOOK
AT THE MOST CUTTING EDGE
RESEARCH ON THIS IS KIND OF
MOVING IN THIS DIRECTION.
THE DEPUTY DIRECTOR HAD THE
DICKERSON DEFINITION OF
PUBLICATION BIAS, WHICH I HAVE
USED BUT I'M NOT ACTUALLY A HUGE
FAN OF, SO THIS IS KIND OF THE
DEFINITION THAT I LIKE USING,
WHICH IS THAT PUBLICATION BIAS
OCCURS WHEN THE STATISTICAL
SIGNIFICANCE OF FINDINGS
INFLUENCES PUBLICATION
PROBABILITY, IMPORTANTLY
CONDITIONAL ON THE QUALITY OF
STUDY DESIGN.
SO THIS CONDITION IS IMPORTANT
BECAUSE ON AVERAGE, NULL STUDIES
ARE GOING TO BE LOWER QUALITY
THAN POSITIVE STUDIES.
AND THAT'S FOR MANY REASONS.
SO ONE, THEY'RE MORE LIKELY TO
HAVE MEASUREMENT ERROR IN THEIR
VARIABLES, THEY'RE GOING TO BE
MORE LIKELY TO HAVE LOW
STATISTICAL POWER, SO WHAT WE
ACTUALLY MEAN BY PUBLICATION
BIAS IS ONCE YOU CONDITION ON
THINGS LIKE PROPER MEASUREMENT,
GOOD DESIGN, STATISTICAL POWER,
THEN DOES THE SIGNIFICANCE OF
THE RESULTS.
SO THAT'S WHY STUDY REGISTRIES
ARE AN IMPORTANT WAY TO DO THIS.
SO WHAT WE DID IS WE USED THE
TIMESHARE EXPERIMENTS PROGRAMMED
IN THE SOCIAL SCIENCES.
THIS IS AN NSF-FUNDED PROGRAM
WHERE RESEARCHERS HAVE TO APPLY
TO OBTAIN SPACE ON VERY
EXPENSIVE, HIGH QUALITY
PROBABILITY SURVEYS TO RUN
RANDOMIZED EXPERIMENTS.
SO WHY DOES THIS COHORT SOLVE
SOME OF THE PROBLEMS THAT ALLOWS
US TO GET AT THIS DEFINITION?
SO ONE, YOU ACTUALLY HAVE A
SAMPLE OF COMPLETED STUDIES THAT
ARE OBSERVED PRIOR TO
PUBLICATION BIAS.
SO UNLIKE TED'S REGISTRY WHERE
WE MIGHT NOT KNOW WHETHER SOME
OF THESE THINGS WERE ACTUALLY
COLLECTED THE DATA, WITH TESTS
WE KNOW ALL OF THE DATA HAS BEEN
COLLECTED AND ALL OF THE DATA IS
POSTED ONLINE, AND BECAUSE YOU
HAVE TO APPLY, YOU DON'T JUST
HAVE TO HAVE LIKE A FACE AND
POST YOUR PREREGISTRATION PLAN.
THERE HYPOTHETICALLY IS A
MEANINGFUL QUALITY THRESHOLD
THAT'S BEEN ESTABLISHED SUCH
THAT THIS BOLDED LANGUAGE IS
MEANINGFUL, THAT WE HAVE SOME
CONDITION OF QUALITY, THAT HAS
BEEN PREVETTED.
SO SHAI ALREADY SHOWED THIS, BUT
WE'LL KIND OF SHOW IT AGAIN.
WHICH IS, ESSENTIALLY WHAT WE
FOUND IS THAT IF YOU KIND OF
COMPARE STRONG TO NULL STUDIES,
THERE WAS A MASSIVE DIFFERENCE,
THIS RATIO OF 65 TO 5 IN TERMS
OF NULL STUDIES NEVER BEING
WRITTEN UP AND STRONG STUDIES
BEING WRITTEN, AND YOU ALSO SEE
IN TERMS OF PUBLISHED PAPERS
THIS KIND OF PROBLEMATIC RATIO.
NOW ALL OF THESE DATA REALLY
COME FROM WHAT YOU WOULD CALL
THE BEFORE OPEN SCIENCE ERA FROM
2002 TO 2012.
SO I'M GOING TO TO SHOW YOU SOME
STUFF AFTER THE OPEN SCIENCE E
ERA, YOU CAN SEE POST OPEN
SCIENCE ERA IS CHARACTERIZED BY
MANY THINGS, SO ONE IS LIKE
STUFF THAT TED HAS DONE LIKE THE
BITSS PROGRAM, BUT I THINK THE
REAL CAUSE OF IT WAS A PAPER
PUBLISHED IN THE LATE 2000s BY
PSYCHOLOGIST DARRELL BEMM
PURPORTING TO SHOW THE EFFECT OF
ESP OR EXTRA SENSORY PERCEPTION.
AND ESSENTIALLY I THINK WHAT
THIS PAPER DID IS IT LIKE THIS
DEFINING EVENT THAT SAID WHAT IS
GOING ON HERE, AND IT LED TO A
BUNCH OF RESEARCH IN THE SPACE
INCLUDING URI SIMONSOHON'S WORK.
THIS IS ACTUALLY A QUESTION FROM
THE S.A.T. WHERE THEY ACTUALLY
TALKED ABOUT THIS PAPER.
SO IT'S KIND OF FUNNY WHERE THEY
SAY BASED ON THIS PASSAGE, WHICH
OF THESE FOLLOWING HYPOTHETICAL
SITUATIONS WOULD MALHOTRA
STRONGLY OBJECT TO?
I WOULD SAY THERE'S SUPPOSED TO
BE ONE ANSWER TO THIS, BUT I
OBJECT TO MANY OF THESE THINGS.
SO I WOULD OBJECT THAT IF A
RESEARCH TEAM EXCLUDES THE
PORTION OF DATA THAT PRODUCED
NULL RESULTS, WHICH IS SELECT
REPORTING, I WOULD ALSO OBJECT
TO A RESEARCH TEAM UNKNOWINGLY
REPEATING A STUDY THAT PRODUCED
NULL RESULTS FOR ANOTHER
RESEARCH TEAM.
AND THEN FINALLY, RESEARCH TEAM
PERFORMS A FOLLOW-UP STUDY THAT
EXPANDS ON THE SCOPE OF AN
INITIAL STUDY THAT PRODUCES NULL
RESULTS.
I THINK ALL THERE'S THREE OF THE
THINGS ARE REALLY BAD.
I'M NOT SURE WHAT THAT SAYS
ABOUT THE S.A.T. WHEN I CAN'T
EVEN ANSWER THE QUESTION ABOUT
MY OWN VIEWS CORRECTLY.
SO THREE RESEARCHERS, MONIZ,
DRUKMAN AND FRIES RECENTLY
REPLICATED THE ORIGINAL SCIENCE
PAPER WE WROTE IN 2014.
SO THEY'RE THE ONES WHO
ADMINISTER THIS PROGRAM FOR THE
NSF.
THE IMPORTANT THING THEY HAD WAS
THEY NOT ONLY HAVE THE PEOPLE
WHO GOT THESE GRANTS, THEY ALSO
HAVE THE PEOPLE THEY REJECTED
FOR THESE GRANTS.
SO THAT'S AN INTERESTING
COMPARISON.
NOW THIS ISN'T LIKE SOME
REGRESSION DISCONTINUITY DESIGN
OR SOMETHING, SO -- BUT YOU
COULD BASICALLY KIND OF COMPARE
TO SEE HOW THESE PUBLICATIONS
ASSESS, AND YOU CAN ACTUALLY SEE
THAT THINGS HAVE GOTTEN A LOT
BETTER.
SO IF YOU LOOK AT THAT TOP GAP
BETWEEN SIGNIFICANT AND NOT
SIGNIFICANT, IT'S NOT THE 64 TO
4 RATIO FROM THE PRE-OPEN
SCIENCE ERA.
IT'S ONLY ABOUT 28 TO 8.
SO IT'S SORT OF IN THE EYE OF
THE BEHOLDER LIKE THAT'S STILL
NOT PERFECT, SO THE QUESTION IS,
IS THAT ARE WE GOING TO JUST
EXPECT THINGS TO GET BETTER OR
IS THIS LAST MILE GOING TO
REQUIRE A LOT OF THE STUFF WE'RE
GOING TO BE TALKING ABOUT AT
THIS CONFERENCE.
SO ESSENTIALLY WHAT YOU CAN DO
IS KIND OF MAKE THESE
COMPARISONS.
THIS ISN'T A PERFECT DESIGN BUT
YOU CAN ESSENTIALLY SAY LET'S
LOOK AT THE ACCEPTED STUDIES
BEFORE OPEN SCIENCE AND AFTER
OPEN SCIENCE.
YOU CAN SAY WHAT EXPLAINS THAT
DIFFERENCE IS CULTURAL CHANGE.
AND THEN YOU CAN LOOK AT THE
MODERN AFTER OPEN SCIENCE ERA
AND COMPARE THE ACCEPTED AND
REJECTED STUDIES.
SO THESE TEST STUDIES, THEY KNOW
THEY'RE BEING WATCHED, THEY'RE
REGISTERED, WHEREAS THE REJECTED
STUDY, IT'S STILL THE WILD WEST,
THEY CAN KIND OF DO WHATEVER
THEY WANT.
SO YOU CAN SAY IF YOU'RE SUBJECT
TO THESE INSTITUTIONS, WHAT
ACTUALLY AFFECTS THE RESULTS?
SO YOU CAN SEE THAT FOR EXAMPLE,
OF THE NOT WRITTEN STATUS,
BEFORE OPEN SCIENCE AND AFTER
OPEN SCIENCE, I'VE SHOWN THIS,
YOU SEE THIS MASSIVE CHANGE,
THIS 60.2 PERCENTAGE POINT GAP
HAS NARROWED TO ABOUT 20.1
PERCENTAGE POINTS AND THE
DIFFERENCE BETWEEN SIGNIFICANCE
AND NOT SIGNIFICANCE, NOT BEING
WRITTEN.
BUT ON THE OTHER HAND, IF YOU
LOOK HE ILOOK AT THE INSTITUTIOS
CHANGE, IT'S A LOT SMALLER.
SO THERE'S A 38.7% BEGAN IN THE
REJECTED TEST STUDIES, AND --
YOU ALSO SEE THE CHANGE THERE
BUT IF YOU LOOK AT THE
INSTITUTIONAL CHANGE, IT'S A LOT
MORE MODEST.
SO WHAT DOES REGISTRATION DO, IT
ONLY DECREASES THE GAP BY ABOUT
8 PERCENTAGE POINTS.
THERE'S ACTUALLY AN INCREASE IN
THE PUBLICATION STATUS IF YOU
ACTUALLY HAVE REGISTRATION.
SO WHAT DOES THIS SHOW?
IT SHOWS THAT LIKE MOST OF WHAT
WE'VE GOTTEN IS THROUGH CULTURAL
CHANGE POTENTIALLY, AND MAYBE
NOT AS MUCH ON INSTITUTIONAL
CHANGE.
SO THAT'S NOT TO SAY THAT THE
LAST MILE IS NOT GOING TO BE
DONE THROUGH INSTITUTIONS, BUT
MAYBE WE HAVE TO FOCUS A LITTLE
BIT MORE ON KIND OF HOW THE
OVERALL CULTURE IS DONE.
NOW EVEN THOUGH I'M IN THE
BUSINESS SCHOOL, I DON'T
ACTUALLY STUDY THIS STUFF BY
ORGANIZATIONAL RESEARCHERS DO
STUDY THINGS LIKE HOW DO YOU
ACHIEVE CULTURAL CHANGE IN
ORGANIZATIONS WITHOUT
INSTITUTIONS, AND SO THERE IS
THEORY MIND THIS, AND MAYBE
THOSE PEOPLE SHOULD BE INCLUDED
IN FUTURE MEETINGS, NOT JUST
META SCIENTISTS AND
STATISTICIANS.
AND THEN THE FINAL THING I'LL
SAY IS THAT SOMETIMES LIKE
RANDOM EVENTS CAN BE VERY
POWERFUL.
SO IT'S KIND OF WEIRD THAT A
STUDY ABOUT ESP IS THE REASON
WHY SOCIAL SCIENCE STARTED
GETTING BETTER AT THIS STUFF,
BUT IT MIGHT BE THAT FUTURE
THINGS AND RANDOM EVENTS LIKE
THAT HAVE TO BE THE NUDGES THAT
CREATE SOME OF THESE CULTURAL
CHANGES.
SO THIS IS INTERESTING AND
LOOKING FORWARD TO THE
DISCUSSION.
>> NEXT WE WILL BE HEARING SOME
REPORTS FROM ARIELLE
MARKS-ANGLIN FROM MERCK.
>> GOOD MORNING.
I'M ARIELLE MARKS-ANGLIN, SENIOR
SCIENTIST IN REAL WORLD EVIDENCE
AT MERCK.
MY BACKGROUND TRAINING IS IN
BIOSTATISTICS, AND I'M HERE TO
GIVE A HISTORICAL AND
METHODOLOGIC PERSPECTIVE FOR
QUANTIFYING AND DETECTING
PUBLICATION BIAS, AND JUST A
QUICK DISCLAIMER THAT THE VIEWS
AND OPINIONS I EXPRESS HERE ARE
MY OWN AND DON'T NECESSARILY
REFLECT THE VIEWS OF MY EMPLO
EMPLOYER.
SO PUBLICATION BIAS, I WOULD
ARGUE AT LEAST AS WE STUDY IT
TODAY, IS REALLY A TALE AS OLD
AS NULL HYPOTHESIS SIGNIFICANCE
TESTING.
IN THE EARLY 100s, RONALD
FISHER PIERSON AND NAYMAN TRIED
TO MAKE MATHEMATICAL STATISTICS
MORE ACCESSIBLE TO RESEARCHERS,
SO THEY DEVELOPED, NOT TOGETHER,
VERY SEPARATELY, THEY DEVELOPED
A HYPOTHESIS TETHER TESTING
FRAME WORKS AND INTERTRE
PAITIONS OF THE P VALUE THAT CAN
BE USED MORE WIDELY BY
RESEARCHERS IN DIFFERENT FIELDS.
AND FISHER WAS PARTICULARLY
INSTRUMENTAL IN CATEGORIZING THE
SIGNIFICANCE OF STUDIES BASED ON
THAT .05 SIGNIFICANCE THRESHOLD.
THEN FROM THERE, ESPECIALLY IN
THE FIELD OF PSYCHOLOGY BUT ALSO
FOOD SCIENCE AND OTHER RESEARCH
AREAS, THE RESEARCHERS ADOPTED
THE MODERN FORM OF NULL
HYPOTHESIS SIGNIFICANCE TESTING,
AND PROFESSIONAL JOURNALS AT THE
TIME EVEN REQUIRED SIGNIFICANT P
VALUES FOR DISSEMINATION.
SO FAST FORWARD TO AS EARLY AS
1959, WHEN WE HAVE META-ANALYSIS
AND EVIDENCE SYNTHESIS STARTING
TO COME INTO THE RESEARCH
SPHERE, WE SEE EARLY INDICATIONS
OF SELECTIVE PUBLICATION BASED
ON THE SIGNIFICANCE OF RESULTS.
WHAT THIS HAS LED TO IS A
DECADES-LONG MISSING DATA
CHALLENGE.
HERE WE SEE A META-ANALYSIS OF
PSYCHOTHERAPY STUDIES SPANNING
DECADES -- I BELIEVE THE EARLY
STUDY INCLUDED IN THIS WAS IN
THE 60s, BUT WHAT YOU SEE HERE
ARE FUNNEL PLOTS AND THE IDEA IS
THAT IN THE ABSENCE OF
PUBLICATION BIAS, THE X AXIS
SHOWS THE EFFECT SIZE AND YOU
WOULD EXPECT AN EVEN
DISTRIBUTION OF STUDIES OF --
AROUND WHAT IS LIKELY THE TRUE
EFFECT SIZE.
BUT HERE WE SEE A SKEWNESS
TOWARDS MORE EXTREME STUDIES,
EXTREME EFFECTS, PARTICULARLY
FOR STUDYS THAT HAVE SMALLER
SAMPLE SIZES.
AND SO WHAT DO WE DO WHEN WE
HAVE A MISSING DATA CHALLENGE?
HOW DO YOU QUANTIFY THE UNKNOWN?
AND IN STATISTICS, TYPICALLY
THAT MEANS MAKING ASSUMPTIONS.
WE CAN DRAW FROM PATTERNS WE SEE
IN THE OBSERVED DATA.
WE CAN FORMALIZE THE KNOWLEDGE
AND BELIEFS WE HAVE ABOUT THE
PUBLICATION PROCESS.
AND WE'D ALSO LIKE TO ACCOUNT
FOR OTHER PROCESSES THAT MAY
LEAD TO THE OBSERVED DATA.
SO I DID A HISTORICAL REVIEW OF
THESE METHODS AND I WOULD ARGUE
THAT YOU CAN CLASSIFY THE
QUANTITATIVE METHODS FOR
PUBLICATION BIAS INTO TWO BROAD
CLASSES.
ONE OF WHICH IS GRAPH BASED.
AND WHICH WE INCLUDE THIS --
THAT INCLUDES THIS FUNNEL PLOD I
SHOWED PREVIOUSLY.
AND THE CENTRAL ASSUMPTION FOR
THAT IS THAT -- IS AN ASSUMPTION
CALL SYMMETRY, WHERE YOU ASSUME
A RANDOM SYMMETRIC DISTRIBUTION
OF STUDY EFFECT SIZES AROUND THE
TRUE EFFECTS.
AND THAT'S WHERE YOU GET METHODS
SUCH AS EGGERS TEST, TRIM AND
FILL, METHODS THAT YOU MAY HAVE
HEARD OF AND EVEN USED WIDELY IN
META-ANALYSES OR SYSTEMATIC
REVIEWS.
AND THEY'RE VERY BENEFICIAL
BECAUSE THEY'RE INTUITIVE AND
USER-FRIENDLY.
THEY'VE BEEN AROUND FOR A WHILE
SO WE HAVE VALIDATED CODE
PACKAGES AVAILABLE.
HOWEVER, THEY ARE PRONE TO
MISINTERPRETATION BECAUSE
PUBLICATION BIAS IS NOT THE ONLY
SOURCE OF ASYMMETRY YOU MIGHT
FIND IN A FUNNEL PLOT.
AND I THINK KNE NEIL ALLUDED TO
THIS, BUT DEPENDING ON STUDY
QUALITY OR HETEROGENEITY IN YOUR
STUDY DESIGNS, YOU CAN ALSO GET
DIFFERENT EFFECTS WHEN YOU'RE
TRYING TO ANSWER THE SAME
RESEARCH QUESTION.
SO ONE CHALLENGE IN THESE
METHODS IS HOW TO DISTINGUISH
BETWEEN PUBLICATION BIAS AND
OTHER SOURCES OF WHAT WE CALL
SMALL STUDY EFFECTS OR
ASYMMETRY.
ON THE OTHER HAND, WE HAVE
SELECTION MODELS, IN WHICH YOU
USE MATHEMATICAL MODELS TO
OUTLINE THE ASSUMPTIONS ABOUT
THE PUBLICATION PROCESS AND THEN
TRY TO ESTIMATE AN EFFECT BASED
ON THOSE MODELS.
AND THEY'RE VERY FLEXIBLE, YOU
CAN DEFINE MORE COMPLEX
PROCESSES, SO AT THE CRUDEST
LEVEL, YOU CAN ASSUME THAT
STUDIES WITH A P VALUELESS THAN
.05 ARE PUBLISHED WITH A
PROBABILITY OF 1.
AND IF THEY'RE GREATER THAN .05,
THEY'RE PUBLISHED WITH A
PROBABILITY OF ZERO.
AND OVER TIME, STATISTICIANS
HAVE DEVELOPED MORE COMPLEX
MODELS, THEY'VE TRIED TO EXTEND
TO MULTIVARIABLE SETTINGS, AND
THEY HELP TO EASILY DISTINGUISH
BETWEEN PUBLICATION BIAS AND
OTHER SOURCES OF ASYMMETRY OR
SMALL STUDY EFFECTS.
HOWEVER, BECAUSE THESE ARE
COMPLEX MODELS, AND
META-ANALYSES OFTEN HAVE VERY
SMALL SAMPLE SIZES, YOU CAN RUN
INTO COMPUTATION ISSUES,
IDENTIFIABILITY ISSUES.
THEY DON'T OFTEN -- THEY'RE NOT
OFTEN ACCOMPANIED BY CODE
PACKAGES SO THEY'RE LESS
ACCESSIBLE, SO YOU TEND TO NEED
A DEEPER KNOWLEDGE OF STATISTICS
IN ORDER TO IMPLEMENT THEM.
AND SO THERE'S NOT AN IDEAL
SOLUTION.
I WILL SKIP OVER THIS NEXT SLIDE
DUE TO TIME, BUT IF YOU'RE
INTERESTED, I DO INCLUDE IN THE
PAPER A TIMELINE OF THE
METHODOLOGIC DEVELOPMENTS, BUT .
WHAT YOU CAN GATHER FROM THIS IS
WE ARE CONTINUOUSLY ITERATING
AND TRYING TO IMPROVE UPON
METHODS FOR QUANTIFYING,
DETECTING, AND CORRECTING FOR
PUBLICATION BIAS, AS WE CONTINUE
TO IDENTIFY NEW COMPLICATIONS OR
NUANCES IN THE MISSING DATA
PROCESS.
SO I WOULD SAY THAT KEY
CHALLENGES FOR ANY
QUANTIFICATION OR DETECTION
METHOD FROM AT -- AT LEAST FOR
STATISTICAL METHODS IS THE
VALIDITY OF THE ASSUMPTIONS
MADE, HOW TO ISOLATE PUBLICATION
BIAS FROM OTHER SOUR SOURCES OF
ASYMMETRY, THE INTERPRETABILITY
OF YOUR APPROACH, HOW WELL YOU
CAN ESTIMATE THE PARAMETERS OF
YOUR MODEL, AND ALSO THE
COMPUTATIONAL FEASIBILITY.
I WOULD PARTICULARLY LIKE TO
HIGHLIGHT THESE TWO BECAUSE IF
WE'RE TRYING TO MAKE THIS MORE
ACCESSIBLE TO OTHERS,
COMPUTATIONAL FEASIBILITY IS A
MUST.
I THINK OUR NEXT SPEE SPEAK UR,,
HAS DONE EXCELLENT WORK IN THIS
AREA WITH HIS METHODOLOGIC
DEVELOPMENTS.
BUT IT'S IMPORTANT TO KNOW THAT
THESE ARE VERY AD HOC
APPROACHES.
THEY CAN HELP US IDENTIFY THE
PROBLEM, DO SENSITIVITY ANALYSES
BUT THEY'RE BY NO MEANS A
SUFFICIENT SOLUTION, SO I'M GLAD
THIS WORKSHOP IS TALKING ABOUT
PREVENTION BECAUSE THAT'S
ULTIMATELY WHERE THE SOLUTION
NEEDS TO COME FROM.
AND IN MY REDUCTIONIST
PERSPECTIVE, I SEE THAT THERE IS
A DESIRE FOR ACTIONABLE RESULTS
FROM RESEARCH TO ADVANCE THE
FIELD.
WE INVEST OUR TIME, WE INVEST
FUNDING INTO STUDIES BECAUSE WE
HOPE TO ADVANCE A CERTAIN AREA
OF KNOWLEDGE.
AND I ULTIMATELY THINK THAT'S A
GOOD DESIRE.
WE ALSO DESIRE GREATER RIGOR AND
OBJECTIVITY IN EXPERIMENTAL
STUDIES.
THAT'S WHY THESE EARLIER
STATISTICIANS, FISHER, NEYMAN
AND PEARSON, TRIED TO DEVELOP AN
EASY TO USE RULE FOR CLASSIFYING
EVIDENCE.
HOWEVER, WE'VE BECOME TOO
RELIANT ON IT.
WE DETERMINE WHETHER A STUDY
STANDS OR FALLS BASED ON WHETHER
YOU REJECT THE NULL BASED ON
WHETHER YOU HAVE THE P VALUE
SURPASSES OR GOES BELOW THIS
THRESHOLD OF .05, WHICH WAS
REALLY ONLY CONVENIENTLY
SELECTED BY FISHER BECAUSE IT
CORRESPONDS TO TWO STANDARD
DEVIATIONS FROM THE MEAN ON A
BELL CURVE.
I WOULD ENCOURAGE US TO CONSIDER
PERHAPS CAN WE RE-EVALUATE HOW
WE DEFINE EVIDENCE, OR TO TED'S
TALK EARLIER, HOW WE DEFINE
VALUE OF A STUDY.
THERE ARE OTHER MEASURES AND
FRAME WORKS APART FROM NULL
HYPOTHESIS TESTING, APART FROM P
VALUES, OR EVEN IF STICKING WITH
P VALUES AND CONFIDENCE
INTERVALS CAN WE MOVE AWAY FROM
HONING IN ON THIS .05 THRESHOLD
AND CLASSIFYING STUDIES ABOVE
THAT AS NULL STUDIES, WHICH I'M
GLAD SOMEONE POINTED OUT EARLIER
IS A BIT OF A PROBLEMATIC TERM.
SO THAT'S MY PERSPECTIVE WHERE
I'M COMING FROM.
THANK YOU.
>> THANK YOU SO MUCH.
SO NEXT, URI SIMONSOHON FROM
RAMON LLULL UNIVERSITY FROM
SPAIN AND YOU MAY ALSO KNOW HIM
FROM THE BLOG.
HE WAS UNABLE TO COME IN PERSON
BECAUSE OF A CONFLICT, BUT HE
DID SUBMIT A VIDEO AND WE WILL
BE WATCHING THAT NOW.
>> HI.
BECAUSE I COULDN'T MAKE THE
CONFERENCE, THEY ASKED ME TO
MAKE A FIVE-MINUTE VIDEO, WHICH
I'LL DO AND I'LL TALK ABOUT
THREE PERSPECTIVES ON
PUBLICATION BIAS I THINK MAY BE
SOMEWHAT NON-MAJORITY VIEWS AND
SO MAY BE BIGGER VALUE SHOWING
THOSE.
THE FIRST ONE IF YOU HAD TO
CHOOSE BETWEEN --
>> TURN IT UP.
>> AND I SEE LITTLE BENEFIT IN
ADDRESSING PUBLICATION BIAS
WITHOUT ALSO ADDRESSING P
HACKING.
SO LET ME CONTRAST THE
DEFINITION OF THE TWO.
PUBLICATION BIAS IS WHEN WE
SELECTED THE REPORT IN ENTIRE
STUDIES THAT WORK IN A
PARTICULAR WAY, FOR EXAMPLE, IF
THERE WERE STATISTIC -- P
HACKING, SELECTIVE REPORTING OF
PREFERRED ANALYSIS.
SO CONSEQUENCES OF HAVING MORE
STUDIES BEING AVAILABLE BUT NOT
MAKING SURE THE ANALYSIS OF
THOSE STUDIES WERE THE ONES THAT
WERE INTENDED OR COMPLEMENTARY,
THE LEVEL OF BIAS IN THE
LITERATURE COULD EASILY INCREASE
OR STAY THE SAME BECAUSE -- TH
THE -- IT DOESN'T MEAN -- YOU
CAN EASILY P-HACK A STUDY
SUCCESSFULLY LET'S SAY TO
PUBLISH P VALUES, DOESN'T MATTER
THE -- SIZE, FROM .4 TO .3, SO
STILL NOT SIGNIFICANT.
IF YOU FORCE ME TO REVEAL THEM,
I SHOW YOU THE .3 BASED ON BIAS
ESTIMATE, DIVIDE THE CORRECT --
OF THE EFFECT SIZE -- .4 AND
SOME -- OBSERVATION THAT'S BIAS
AND THE EXISTING -- STUDIES THAT
ARE PUBLISHED REMAINS THERE.
IT'S ALSO VERY HARD AS I'LL
ARGUE LATER TO -- PUBLICATION
BIAS WHILE IT'S VERY EASY TO
ADDRESS P-HACKING.
ON THE OTHER HAND, IF YOU REDUCE
P-HACKING AND -- PUBLICATION BUY
A THE BIAS ESTIMATES THAT ARE IN
THE LITERATURE WILL -- MANY WILL
DISAPPEAR BECAUSE WITHOUT
P-HACKING -- REPORTED.
NOW YOU HAVE TO WORRY ABOUT IS
WHETHER THE ONES THAT ARE
BIASED-FREE ARE SIMPLY BIASED
BECAUSE 1 IN 20 STUDIES ARE
WORKING BY CHANCE BUT IT'S HARD
TO HAVE A SUCCESSFUL KEER WHERER
WHERE 1 IN 20 STUDIES WORK.
-- MORE OF THE STUDIES BEING
PUBLISHED BECAUSE OF THE KIND OF
STUDIES THAT ARE BEING RUN.
THE SECOND POINT, AND THIS
RELATES TO THE FIRST ONE, IS
THAT PREREGISTRATION IS BETTER
AND MORE BENEFICIAL THAN
REGISTRATION.
-- MADE AN IMPACT IN MEDICAL
RESEARCH INCLUDED IN CLINICAL
TRIALS, CLINICAL TRIALS ARE
GOVERNING, SO PREREGISTRATION IS
WHEN YOU THINK OF THE PERFECT --
YOU PERFECTLY STIPULATE HOW
EVERY ANALYSIS WE CONDUCTED,
EVERY ASPECT -- HOW LONG IT WILL
LAST AND SO ON.
BUT IN REALITY, NOTHING IS
PERFECT.
SO YOU HAVE IMPERFECT
PREREGISTRATION, YOU STILL HAVE
A MEANINGFUL REDUCTION OF
P-HACKING, EVERYTHING YOU
STIPULATED YOU DON'T P-HACK AND
EVERYTHING YOU DON'T STIPULATE,
IT IS EXPLICITLY NOT STIPULATED
SO THE READER CAN IDENTIFY
THINGS THAT WERE AND WERE NOT
P-HACKED SO IT'S HIGHLY USEFUL
EVEN IF IMPERFECT.
NOW REGISTRATION, REGISTRATION
IS WHEN YOU STIPULATE AHEAD OF
TIME THAT YOU WILL RUN A STUDY
SO PEOPLE KNOW THAT YOU EAX AM D
THE IMPACT OF THIS PARTICULAR
TREATMENT, WHY DO WE WANT THAT,
WHY DO WE WANT PEOPLE TO TELL
THAWS AHEAD OF TIME, BECAUSE OF
PUBLICATION BIAS, WE VEN TIRE
STUDIES MISSING, BUT FOR THAT TO
BE USEFUL, DOWN THE LINE
SOMEBODY IS GOING TO BE FINDING
ALL THOSE STUDIES THAT WERE RUN
AND WERE PUBLISHED BR FOW --
ANALYST TO DO ANYTHING WITH THE
REGISTRATION, THEY NEED TO KNOW
NOT JUST THAT YOU PLANNED TO RUN
THE STUDY BUT ALSO WHETHER YOU
ACTUALLY RUN IT, HOW YOU
EXECUTED IT AND WHAT THE RESULTS
ARE, AND THAT HAPPENS VERY
RARELY.
IT HAPPENS VERY RARELY IN A STU
DAY THAT FAILS AND IS NOT
PUBLISHED, THE AUTHOR NEVER GOES
THROUGH THE TROUBLE OF SHOWING
ALL THE DETAILS OF THE STUDY.
SO THAT MEANS HAVING PERFECT
REGISTRATION WHERE PEOPLE
RESEARCH STUDIES AND -- METHODS,
THIS HAS ALMOST NO VALUE IN
CONTRAST TO IMPERFECT
PREREGISTRATION WHICH HAS HIGH
VALUE.
SO IF YOU REALLY NEED A
REGISTRY, IF YOU REALLY NEED
PEOPLE TO TELL YOU AHEAD OF TIME
WHICH STUDIES THEY RUN BECAUSE
THEY NEED TO KNOW THE OUTCOME,
WHAT YOU REALLY NEED IS -- THE
RORLTS.
PUBLICATION PROCESS WHERE PEOPLE
WILL PRE-COMMIT TO MAKING THE
RESULTS PUBLIC, REGARDLESS OF
THE OUTCOME.
THAT COULD BE IN THE WAY THEY
CURRENTLY WORK, WHICH IS A
JOURNAL HAS A COMMITMENT TO THE
AUTHORS TO PUBLISH IT BUT IT
COULD ALSO BE SOMETHING LIKE
CLINICALTRIALS.GOV REQUIRES THAT
AUTHORS -- THE INFORMATION, FOR
EXAMPLE, PREVENTING THEM FROM
CONDUCTING NEW REGISTRIES -- NEW
REGISTRATIONS IF THEY DON'T
HAVE -- THIS IS VERY IMPORTANT
AS FAR AS I KNOW HAS NOT BEEN
REALLY ADDRESSED.
AND THE LAST THING I'LL SAY IS
THAT META-ANALYSIS REALLY CANNOT
FIX PUBLICATION GU BIAS.
SO THE HOPE IS TO HAVE A REJ
INDUSTRIAL TO FIX IT.
META-ANALYSIS CANNOT REALLY
DELIVER FOR A FEW REASONS.
FIRST, META-ANALYSIS EXACERBATES
BIAS FROM P-HACKING IF EVERY
STUDY HAS A RI LITTLE BIT OF
P-HACKING THE -- IT'S VERY
DIFFICULT GIVEN -- STUDIES AND
LACK OF DETAIL IN THE ORIGINAL,
QUALITY CONTROL, AND THEY
INEVITABLY ADD LOW QUALITY
STUDIES THAT WEREN'T PUBLISHED
PERHAPS BECAUSE OF NO RESULTS,
BECAUSE PERHAPS BECAUSE OF
QUALITY CONCERNS.
THIS IS -- REVIEW, ALMOST ANY
SERIOUS META AL LIST WILL KNOW
THAT SMALL STUDY BIAS ISN'T
PUBLICATION BIAS, SO THE
FINAL -- OTHER TECHNIQUES AREN'T
REALLY DETECTING PUBLICATION
BIAS -- VERY NOISY TOOLS THAT
HAS A LOT OF WISHFUL THINKING --
PUBLICATION BIAS DETECTION.
THEN MAIN TOOLS LIKE TRIM-AND
TRIM-AND-FILL, THEY HAVEN'T
REALLY BEEN VALIDATED, DOESN'T
WORK AT ALL FOR PUBLICATION
BIAS.
SO IN SUMMARY, P-HACKING IS A
BIGGER PROBLEM THAN PUBLICATION
BIAS AND IS EASIER TO ADDRESS.
PUBLICATION BIAS CANNOT BE FIXED
WITH REGISTRY, META-ANALYSIS, IF
ONE MUST MIX IT, YOU NEED A
REGISTERED REPORT.
>> ALL RIGHT.
NEXT WE WILL HAVE SOME BRIEF
REMARKS FROM LAUREN VON
KLINGGRAEFF FROM AW AUGUSTA
UNIVERSITY.
>> WE HAD SOME GUIDING QUESTIONS
ABOUT THINGS WE CAN SPEAK TO
DURING SESSION ONE, AND THE
FIRST QUESTION BEING WHY OR
WHETHER IT IS IMPORTANT TO
ADDRESS PUBLICATION BIAS OR
QUANTIFY PUBLICATION BIAS.
AND I WOULD ARGUE THAT IN MY
FIELD, IT IS VERY IMPORTANT TO
QUANTIFY PUBLICATION BIAS AND TO
KEEP AN EYE ON IT.
SO I WORK IN THE FIELD OF
POPULATION LEVEL HEALTH BEHAVIOR
INTERVENTIONS, AND WHEN WE GO TO
DESIGN AN INTERVENTION TO HELP
PEOPLE STOP SMOKING OR EAT MORE
FRUITS AND VEGETABLES WE USUALLY
FOLLOW MOD DELLS LIKE THIS, THE
NIH STAGE MODEL OR NHLBI ORBIT
MODEL WHERE YOU DO A PRELIMINARY
FEASIBILITY TEST TO SEE WHETHER
THAT INTERVENTION HAS ANYTHING
WORTH PURSUING BEFORE YOU GO ON
AND RE-TEST IT ITTIVELY IN
HOPEFULLY LARGER SAMPLE SIZES.
THE ISSUE BEING THAT WITH THESE
EARLY SMALL STUDIES WE KNOW
THEY'RE UNDERPOWERED AND WE KNOW
EARLY STUDIES ARE LIKELY TO HAVE
LARGER EFFECT SIZES THAN THEY
MIGHT BE OTHERWISE IF THEY WERE
AT A DIFFERENT POINT IN THE
DEVELOPMENTAL PROCESS.
SO IF YOU TAKE THAT AND COMBINE
IT WITH PUBLICATION BIAS, WHAT
YOU COULD HAVE IS A SITUATION
WHERE YOU ARE PROMOTING AND
POSSIBLY INVESTING MORE INTO
THESE INTERVENTIONS THAT MAYBE
ARE BASELESS AND WE SHOULDN'T BE
TESTING IN THE FIRST PLACE
BECAUSE THAT POSITIVE RESULT IS
JUST COMING FROM A TYPE I ERROR.
SO THEN THE QUESTION IS HOW DO
WE QUANTIFY WHETHER OR NOT THIS
IS HAPPENING IN PRELIMINARY
BEHAVIORAL INTERVENTIONS?
WE DID A STUDY AND IT'S RIGHT
THERE, WHERE WE WENT AND JUST
LIKE YOU CAN LOOK AT THE FDA
REGISTRIES OR STUDY REGISTRIES,
PREREGISTRIES, YOU CAN LOOK AT
ABSTRACTS THAT ARE PRESENTED AT
CONFERENCES.
SO WE WENT AND LOOKED AT TWO
CONFERENCES, THE INTERNATIONAL
SOCIETY OF BEHAVIORAL NUTRITION
AND PHYSICAL ACTIVITY AND THE
SOCIETY OF BEHAVIORAL MEDICINE,
WHICH ARE TWO LARGE CONFERENCES
IN OUR FIELD.
WE LOOKED THROUGH ABOUT 18,000
CONFERENCE ABSTRACTS AND OF
THOSE, 800 WERE PRELIMINARY
BEHAVIORAL INTERVENTIONS.
WE THEN WENT THROUGH AND CODED
THOSE ABSTRACTS FOR LOTS OF
VARIABLES, BUT THE MAIN ONES
BEING WERE THEY RANDOMIZED
CONTROL TRIALS, WHAT WAS THEIR
SAMPLE SIZE, AND DID THEY HAVE
SOME FORM OF SIGNIFICANCE, AND
THIS IS SORT OF A MORE GENERAL
DEFINITION OF SIGNIFICANCE THAT
YOU MIGHT THINK, BEING THAT
ABSTRACTS ARE SHORT.
IF THE AUTHORS OF THE ABSTRACT
SAID THEY HAVE SOMETHING
SIGNIFICANT AND THEY'RE RESULTS,
THAT'S WHAT WE TOOK.
WE DIDN'T REQUIRE THEM TO HAVE A
P VALUE OR CONFIDENCE INTERVAL.
WE THEN TRACE THOSE FORWARD IN
TIME TO SEE IF THEY HAVE EVER
BEEN PUBLISHED AS FULL LENGTH
MANUSCRIPTS AND WHAT WE
DISCOVERED WAS THAT 50% OF ALL
PRELIMINARY BEHAVIORAL
INTERVENTIONS IN OUR FIELD GO
UNPUBLISHED, WHICH IS PRETTY
WEIGHTY.
WHEN WE WENT TO GO TEST WHETHER
WE SAW ANY INDICATION OF
PUBLICATION BIAS FAVORING
STATISTICAL SIGNIFICANCE, WHAT
WE FOUND IS THAT INITIALLY IT
APPEARS BUT ONCE WE CONTROLLED
FOR WHETHER STUDIES WERE RCT AND
WHAT THEIR SAMPLE SIZE WAS, THAT
THAT EVIDENCE OF PUBLICATION
BIAS -- AND STATISTICAL
SIGNIFICANCE DISAPPEARED SO I
THINK THAT KIND OF GOES BACK TO
THIS IDEA THAT IF YOU CONTROL
FOR STUDY QUALITY, PERHAPS
STATISTICAL SIGNIFICANT ISN'T
PLAYING QUITE AS BIG OF A ROLE
IN PUBLICATION BIAS.
SO THAT'S JUST ONE STUDY.
IF YOU GO AND YOU LOOK AT THE
LARGEST COLLECTION OF STUDIES
THAT USE THIS METHOD OF
CONFERENCE ABSTRACTS FOLLOW
FORWARD TO STUDY THE PUBLICATION
PROCESS, YOU'LL FIND THIS
REVIEW -- THIS META-ANALYSIS IN
SYSTEMATIC REVIEW.
SO THEY HAVE 461 STUDIES IN HERE
THAT USE THAT ABSTRACT FOLLOW
FORWARD PROCESS.
OF THOSE, ONLY 64 ACCOUNT FOR
WHETHER STATISTICAL SIGNIFICANCE
PLAYS A ROLE.
THEN IF YOU LOOK AND YOU SAY
DOES STATISTICAL SIGNIFICANCE
PREDICT AN INCREASED RISK OF
PUBLICATION, THE ANSWER IS YES.
BUT THEN IF YOU GO AND ONLY LOOK
AT RCTs AND QUASI EXPERIMENTAL
STUDIES, YOU SEE A LITTLE BIT OF
A DROP.
UNFORTUNATELY THEY DON'T HAVE
ANY SUBANALYSES THAT LOOK AT
SAMPLE SIZE AND STUDY DESIGN
ALONG WITH SIGNIFICANCE.
SO THE QUESTION BECOMES, ARE
THESE GOOD ESTIMATES OF
PUBLICATION BIAS?
AND I SAY YES, I DO THINK THE
PUBLICATION BIAS EXISTS, BUT I
THINK WE HAVE TO TAKE IT IN PART
WITH STUDY DESIGN, AND MY
ARGUMENT FOR THAT BEING THAT OUR
STUDY AND OTHER STUDIES THAT USE
THIS METHOD, WE GO ON AND WE
INTERVIEW OR SURVEY THE AUTHORS
AND SAY, HEY, WHY DIDN'T YOU
PUBLISH YOUR STUDY?
AND CONSISTENTLY AND I CAN
PROVIDE CITATIONS FOR THESE, WE
HAVE THE SAME ANSWER, SO AS A
RESEARCHER, WHEN YOU'VE GOT YOUR
RESULTS, YOU'RE GOING TO SIT
THERE AND THINK, WELL, HOW
DIFFICULT IS THIS GOING TO BE TO
PUBLISH, AND CONSIDERING ALL THE
OTHER RESPONSIBILITIES I HAVE,
IS IT WORTH THE EFFORT?
AND I THINK THAT IF YOU JUST
THINK ABOUT STATISTICAL
SIGNIFICANCE, THAT'S NOT THE
FULL STORY.
IF I'VE JUST DONE A SMALL SINGLE
ARM STUDY THAT WAS NULL OR SIG
KARNGHTS CONVINCING SOMEONE THAT
THERE'S MEANING IN THAT DATA IS
MUCH MORE DIFFICULT, BUT IF I
DID A LARGE RANDOMIZED CONTROL
TRIAL AND I DIDN'T HAVE
SIGNIFICANCE, I STILL COULD
CONVINCE SOMEONE THERE'S MEANING
IN THAT AND HOPEFULLY GET IT
PUBLISHED.
SO I THINK THAT WHEN WE TAKE
INTO ACCOUNT STATISTICAL
SIGNIFICANCE AND PUBLICATION
BIAS, WE ALSO NEED TO THINK
ABOUT STUDY QUALITY.
THAT'S ALL I HAVE.
>> THANK YOU VERY MUCH.
AND LAST BUT ABC LEET NOT LEASTT
LEAST IS VIRGINIA
ARECHAVALA-GOMEZA FROM
BIOBIZKAIA.
>>SO IN MY REAL I'VE, I AM A
RESEARCHER THAT HAS MADE HER
CAREER INTO QUANTIFYING
PROTEINS.
SO I DON'T HAVE ANY DATA HERE
TODAY, SO I FEEL A BIT LIKE A
SAMPLE OF AN INVESTIGATOR,, BUT
I'M GOING TO TAKE THOSE FIVE
MINUTES TO TELL YOU ABOUT HOW
HAPPY I AM TO BE HERE, BECAUSE I
THINK BEING ABLE TO PUBLISH
NEGATIVE RESULTS IS SOMETHING WE
SHOULD ALL STRIVE TO.
SO I'M HERE AS A RESEARCHER AND
I'M ONE OF THOSE RESEARCHERS
THAT IS EXCELLENT, AND I THINK
ALL OF US HERE ARE EXCELLENT.
BECAUSE RIGHT NOW, YOU CANNOT
NOT BE EXCELLENT TO DO RESEARCH.
IT'S EXCELLENT SCIENCE HERE,
EXCELLENT SCIENCE THERE.
AND IT'S IMPOSSIBLE TO DO IT
OTHERWISE.
SO WE'RE EXCELLENT.
AND I THINK WE HAVE TO STRIVE TO
BE PROFICIENT.
THAT'S PRETTY MUCH WHAT WE CAN
TO.
DO.
I DON'T THINK WE CAN ALL BE
EXCELLENT, I DON -- THIS IS
IMPOSSIBLE TO LIVE UP TO THAT
IDEA.
AND I THINK WE ALL STARTED DOING
RESEARCH BECAUSE WE WANTED
ANSWERS AND WE WANT THE WORLD TO
BE A BETTER PLACE.
IF YOU THINK ABOUT THIS AS A
COMMUNITY BEING IN THE MIDDLE OF
THIS MAZE, WHICH THERE IS NO
LIGHT, WE KIND OF SET OUT
TOGETHER TO SEE WHICH IS THE
RIGHT WAY FORWARD, AND THEN WE
RECONVENE BECAUSE THIS IS A
LARGE PLACE AND WE NEED TO TELL,
OKAY, THIS WAY IS GOING THERE
AND THEN WE NEED TO MAP IT AND
LEAVE THE TORCHES ON SO
EVERYBODY KNOWS WHERE WE HAVE TO
GO AND HOW TO GET THERE.
AND THAT INFORMATION IS
VALUABLE, WHETHER IT'S THE RIGHT
WAY OR THE WRONG WAY.
AND I THINK THAT'S SOMETHING
THAT WE TEND TO FORGET.
SO I WAS A BIT DISTRACTED WHEN I
READ THIS, IT WAS A PAPER YEARS
BACK SAYING THAT -- CONFIRMING
THE HYPOTHESIS THAT INCREASING
NOW 85% OF THE PUBLISHED RESULTS
WERE POSITIVE RESULTS.
SO I DID A VERY UNSCIENTIFIC
STUDY ON BIOARCHIVE, AND I JUST
CHECKED, BECAUSE THEY HAPPEN TO
HAVE CONTRADICTORY, CONFIRMATORY
AND NEW RESULTS, SO IT WAS A
PRETTY EASY THING TO DO, IN 10
MINUTES.
SO YOU SEE TONS OF DATA, AND
THEN YOU SEE THAT ACTUALLY
CONFIRMATORY AND CONTRADICTORY
RESULTS DO INCREASE OVER THE
YEARS, BUT WHEN YOU SEE THE NEW
RESULTS, IT'S A VERY SMALL
AMOUNT.
AND IT'S ONLY AROUND .3 OR 1% OF
ALL THE DATA.
WHICH IS NOT ENOUGH.
OBVIOUSLY WE DON'T KNOW THE
RESULT WHEN WE START DOING SOME
EXPERIMENTS, IT'S NOT 85% OF THE
TIME WE KNOW ALREADY WHAT'S
HAPPENING.
SO THAT COMES BACK TO
REPRODUCIBILITY.
I THINK PROBABLY MANY OF YOU
HAVE READ THIS PAPER ON WHY
THINGS ARE NOT -- WE ARE NOT
ABLE TO REPRODUCE THINGS THAT
OTHER PEOPLE DID, NOT EVEN
THINGS THAT WE DID IN OUR OWN
LABS, WHICH IS A PAIN AND IT'S A
BIG PROBLEM.
WE THINK ABOUT HONESTY BUT IT'S
NOT JUST THAT.
WE DID A -- WE'RE NOT SHARING
ALL THE ANSWERS THAT WE ARE
GETTING WHEN WE DO RESEARCH.
SORRY, THIS IS VERY ANNOYING, I
DON'T KNOW IF YOU CAN STOP IT
BECAUSE IT'S JUST GOING FURTH
FURTHER -- I THINK IT'S PROBABLY
ON AUTOMATIC AT SOME POINT.
SO I'VE BEEN THE CHAIR OF A
NETWORK OF EUROPEAN RESEARCHERS
THAT IS -- THAT WORKS ON NUCLEIC
ACIDS THAT ARE -- AND IT'S A
QUITE VARIED GROUP OF PEOPLE.
AND WE DID MAKE A SMALL SURVEY
AMONGST OUR MEMBERS, AND WE
REPORTED THOSE RESULTS, AND
BASICALLY VERY SIMILAR TO WHAT
THE PREVIOUS SPEAKER SAID, WE
HAD THIS ISSUE OF NOT HAVING
TIME OR DIFFICULTIES IN
PUBLISHING ARE NOT BEING VALUED.
AND I THINK THE MAIN PROBLEM IS
THAT WE ARE NOT GIVING THE SAME
VALUE TO GOOD RESULTS AND BAD
RESULTS.
AND I THINK THAT IS AN ERROR.
BECAUSE IF YOU ARE FINDING A WAY
TO -- IF YOU'RE FINDING A WAY
OUT OF THAT MAZE, IT'S USEFUL TO
YOU TO KNOW IF THIS IS THE RIGHT
WAY OR THIS IS A DE DEAD END.
ACTUALLY IF YOU'RE USING GOOGLE
MAPS TO GET SOMEWHERE, YOU DON'T
WANT TO GET INTO A DEAD END SO
YOU DO WANT THAT DATA AVAILABLE
FOR ALL OF US.
SO WHY SHOULD WE BE DOING IT?
I PROMISE I'M NOT DOING IT.
SO WITHIN THAT NETWORK, WE DID
PUBLISH -- I'VE BEEN LUCKY
ENOUGH TO BE INVOLVED IN TWO
SPECIAL JOURNALS DEDICATED
SPECIFICALLY TO NEGATIVE RESU
RESULTS.
ONE OF THEM WAS ON NEUROMUSCULAR
DISORDERS, THE OTHER WAS ON
NUCLEIC ACIDS BECAUSE MY
RESEARCH MATCHES BOTH OF THOSE.
WE WERE YO LUCKY TO DO THAT.
I THINK I WANT TO USE AN EXAMPLE
JUST TO SHOW HOW IMPORTANT THIS
IS TO US.
WE'VE BEEN TALKING ABOUT NOT
PUBLISHING RESULTS OF CLINICAL
TRIALS AND THAT IS REALLY BAD,
BUT SOMETIMES THE PRE-CLINICAL
WORK, WHICH IS MUCH MORE WORK
THAN THAT, IS ALSO VERY
IMPORTANT.
THIS ABOUT REPEATING SOMETHING
AND REPLICATING SOMETHING THAT'S
ALREADY BEEN DONE IS SOMETHING
THAT YOU DON'T USUALLY GET MONEY
OR FUNDING TO DO BECAUSE IT'S
NOT -- IT'S NOT NOVEL BUT IT'S
REALLY IMPORTANT.
THIS IS A STUDY THAT WAS
PUBLISHED YEARS AGO ON A DRUG
THAT COULD IMPROVE RARE DISEASE.
THAT WAS AN INTERESTING RESULT.
AND THESE PEOPLE WENT TO GET
FUNDING FOR A CLINICAL TRIAL ON
THIS PARTICULAR DRUG.
THIS IS A NEUROMUSCULAR
DISORDER, DUCHENNE MUSCULAR
DYSTROPHY, AND THOSE PATIENTS
HAVE NO TIME.
SO WHEN YOU HAVE NO TIME, WHAT
YOU HAVE TO DO IS DO THINGS
RIGHT.
SO WHAT THEY DID WAS TO FUND
ANOTHER COUPLE OF RESEARCH
GROUPS TO REPLICATE THAT DATA.
AND THAT IS SOMETHING THAT YOU
DON'T GET USUALLY.
BUT THEY DID.
ACTUALLY IN THE ORIGINAL PAPER,
YOU HAD AN INTERESTING RESPONSE.
AND THEY DID SEE THAT AS WELL IN
ANOTHER OF THE LABS.
BUT ACTUALLY WHEN THEY COMPARE
THAT TO THE WILD TYPE, IT WAS
NOT AS RELEVANT.
THERE WAS A P VALUE THERE.
BUT THAT MAKES NOT SUCH A BIG
DIFFERENCE FOR THE MICE.
THERE WAS LOTS OF INTERESTING
DATA IN THE ORIGINAL PAPER,
DON'T THINK THAT I'M CRITICIZING
THEM, BUT THERE WAS MORE
RESEARCH NEEDED BEFORE RUNNING
INTO A CLINICAL TRIAL.
WHEN YOU SEE WHERE THOSE THINGS
WERE PUBLISHED, THE ORIGINAL
DATA WAS PUBLISHED IN A HIGH
IMPACT JOURNAL AND THE OTHER WAS
PUBLISHED IN SOMETHING THAT AT
THE TIME HAD NO IMPACT FACTOR.
THAT MEANS FOR THOSE WHO DID THE
RESEARCH, ONE OF THEM, YOU KNOW,
MADE A BIG IMPACT IN THEIR
CAREERS, THE OTHER ONE BASICALLY
WAS VALUED AT ZERO IN MANY OF
THOSE ASSESSMENTS.
SO I THINK SOMETIMES THE VALUE
THAT WE GIVE THE RESULTS
DEPENDING ON WHERE THEY
PUBLISHED OR HOW THEY'RE
PUBLISHED AND HOW IMPACTFUL THAT
IS IS SOMETHING THAT WE SHOULD
TRY TO GET RID.
BECAUSE WE CANNOT AFFORD IT.
SO I THINK THAT'S A BIT OF THE
ELEPHANT IN THE ROOM.
WE NEED TO CHANGE THE VALUE OF
THAT, BECAUSE I THINK IT'S
VALUABLE TO KNOW WHEN SOMETHING
IS RIGHT AND WHEN SOMETHING IS
NOT CORRECT.
SO I THINK WE NEED TO CHANGE
THAT IN OUR OWN CAREERS AND ALSO
WHEN WE ARE TRYING TO CONVINCE
THE NEXT GENERATIONS INTO WHAT
IS -- WHAT THEY'RE DOING, AND WE
NEED TO BE PROFICIENT AND NOT --
THANK YOU VERY MUCH.
>> ALL RIGHT.
SO THE FORMAT OF THESE SESSIONS
IS MEANT TO BE PRIMARILY
DISCUSSION, SO AT THIS POINT, I
DON'T KNOW IF WE NEED TO HAND
OUT MICROPHONES OR IF PEOPLE
WOULD LIKE TO STAND UP AND SPEAK
AT THE MICROPHONE.
ASK QUESTIONS OR JUST START A
DISCUSSION.
BUT WHILE PEOPLE ARE DOING THAT,
MAYBE I CAN ASK, IT SEEMS LIKE
WE DON'T NECESSARILY HAVE A
DEFINITIVE WAY TO MEASURE
PUBLICATION BIAS, AND SO WHAT DO
YOU ALL THINK IS THE BEST PATH
FORWARD TO SEE WHETHER OR NOT
WE'RE FIXING THE PROBLEM?
>> I THINK IT WOULD HAVE TO BE A
MULTIPRONGED APPROACH WHERE YOU
INCLUDE STUDIES WHERE YOU'RE
LOOKING AT PUBLISHED RESULTS AND
COMPARING THEM TO RESULTS FROM
REGISTRIES OR ABSTRACTS OR OTHER
SOURCES, AND OVER TIME, SEE
WHETHER THOSE VARIOUS SOURCES
CONTINUE TO ALIGN OR IMPROVE IN
ALIGNMENT, AND YOU DON'T SEE
INCREASED NULL RESULTS PUBLISHED
COMPARED TO OTHER SOURCES.
I THINK THAT CAN BE SUPPLEMENTED
BY META-ANALYSES THAT USE
QUANTIFICATION OR DETECTION
METHODS AND TO SEE IF THOSE
REDUCE OVER TIME, I DON'T THINK
THERE'S ANY UNFORTUNATELY ONE
WAY TO DO IT.
BUT I THINK SOMETHING THAT IS
PERHAPS AT REGULAR INTERVALS
CONSISTENTLY MEASURED REPEAT
REPEATEDLY, I THINK WOULD BE
STRONG EVIDENCE OVER TIME.
>> HI, MY NAME IS LAURA SMITH
FROM TEXAS A & M UNIVERSITY.
I WANTED TO GIVE A LITTLE BIT OF
PUSH BACK AND ASK A BIGGER
QUESTION HERE.
SO I HEARD SOME INDICATIONS
MAYBE FROM DR. MALHOTRA THAT
NULL STUDIES OR NEGATIVE RESULTS
MIGHT BE THE RESULT OF
MISMANAGED RESEARCH OR SOME
MISTAKES IN THE RESEARCH, AND
THAT'S MORE LIKELY TO PRODUCE
NEGATIVE RESULTS.
AND I JUST WANT TO BRING A WORD
OF CAUTION HERE THAT THAT BRINGS
SOME SHAME TO SCIENTISTS FOR
PRODUCING NEGATIVE RESULTS, AND
THERE'S OTHER REASONS THAT
SCIENTISTS DON'T WANT TO OH
CAORCAN'T PUBLISH NEGATIVE RESUS
SO I THINK WE HAVE TO BE CAREFUL
BECAUSE WHENEVER A SCIENTIST
ACTUALLY IS TRYING TO GET THEIR
WORK PUBLISHED AND THEY'RE
LOOKING AT THE INCENTIVE
STRUCTURE OF WHAT THEY'RE DOING,
THEY'RE PRESSED INTO DOING
THINGS LIKE URI WAS TALKING
ABOUT, P-HACKING, ET CETERA.
SO I THINK, YOU KNOW, WE'RE NOT
ONLY LOOKING AT THE ISSUE OF
PUBLISHING NON-SIGNIFICANT
VERSUS SIGNIFICANT, BUT WE'VE
GOT A BIGGER PROBLEM.
WE DON'T EVEN SEE MOST OF THE
NEGATIVE STUDIES, AND THAT'S
MAYBE BECAUSE OF SHAME AND
INABILITY TO PUBLISH.
SO JUST COMMENTS, PLEASE.
THANKS.
>> OKAY.
I'LL RESPOND TO THAT.
SO IF I MEANT IT NORMTIVELY, I
DIDN'T MEAN IT AT ALL.
IT WAS A POSITIVE STATEMENT.
WHICH IS THAT JUST
STATISTICALLY, YOU CANNOT ASSESS
PUBLICATION BIAS SIMPLY BY
COUNTING HOW MANY PAPERS ARE
SIGNIFICANT VERSUS
NON-SIGNIFICANT.
BECAUSE THERE'S A LOT OF
UNOBSERVED HETEROGENEITY BETWEEN
SIGNIFICANT AND NON-SIGNIFICANT
STUDIES.
SO JUST THE PROPER WAY TO
MEASURE IT, NOT FROM A NORMATIVE
STATEMENT BUT JUST A POSITIVE
STATEMENT, IS YOU HAVE TO
CONDITION ON ALL THAT STUFF.
SO JUST TWO EXAMPLES.
WHICH IS SMALLER STUDIES ARE
GOING TO HAVE LESS SIGNIFICANT
RESULTS.
SO ONE THING YOU'D WANT TO
CONDITION ON IS WHAT IS THE
STATISTICAL POWER OF THESE
STUDIES.
SO IT WOULD BE A BAD THING IF
EQUALLY POWERED STUDIES, THE
SIGNIFICANT ONES, ARE THE ONLY
ONES GETTING PUBLISHED.
IT WOULD NOT NECESSARILY BE A
PROBLEM IF SMALLER STUDIES ARE
NOT BEING PUBLISHED AND LARGER
STUDIES ARE.
ANOTHER EXAMPLE IS MEASUREMENT
ERROR AND THE VARIABLES.
SO WE KNOW THAT IF YOU HAVE
MEASUREMENT ERROR IN THE
VARIABLES, THE ESTIMATES CAN BE
BUYS BYE-BYE ASSED TOWARDS ZERO.
SO I DON'T THINK IT'S A PROBLEM
WITH IF HE -- BUT CONDITIONAL ON
EQUIVALENT MEASUREMENT ERROR, WE
SHOULD NOT BE PUBLISHING
SIGNIFICANT VERSUS NOT
SIGNIFICANT STUDIES.
SO HOPEFULLY THAT CLARIFIES IT
AND TO SAY YES, THERE'S PLENTY
OF GOOD STUDIES THAT HAVE NULL
RESULTS, THERE'S PLENTY OF BAD
STUDIES.
THERE'S ALSO BAD STUDIES THAT
HAVE POSITIVE RESULTS BUT WE
WANT TO TRY TO CONDITION ON
THESE THINGS IF YOU WANT TO
ASSESS THE PROBLEM.
>> I JUST WANT TO -- ADD A
COMMENT TO THAT.
SO AT LEAST IN YOUR SCIENCE IN
PRE-CLINICAL RESEARCH, THE POWER
IS SOMEWHERE BETWEEN 8 AND 30%.
OKAY?
IN THE LITERATURE.
SO THE NOTION THAT AT LEAST IN
THE PRE-CLINICAL WORLD, THERE'S
THIS BIG DIFFERENCE IN HOW THE
STUDIES ARE POWERED, IF THE
RESULTS ARE POSITIVE OR NULL
DOESN'T SEEM TO EXIST.
SO WE'RE GOING TO -- PROBABLY BY
PRIMARILY DEAL HERE NOT WITH
STUDIES THAT ARE LARGE AND HAD
TO GO THROUGH AN IRB AND GET
APPROVAL, WE'RE GOING TO BE
TALKING MAINLY AS WE HEARD
ALREADY ABOUT THE PRE-CLINICAL
UNIVERSE WHERE MOST OF THE
STUDIES ARE SMALL, AND YET THEY
PUBLISHED THE ONES THAT REACH
THIS MAGICAL P VALUE, AND THE
ONES THAT DON'T, I DON'T SEE THE
LIGHT -- IN MY OPINION THERE'S
REALLY NO DIFFERENCE BETWEEN THE
TWO STARS THE TRUST THAT YOU
COULD PUT STATISTICALLY ON THE
RESULTS.
>> SO I HAVE TWO QUESTIONS, THE
FIRST IS JUST CHEEKY AND
DOESN'T -- IS THERE PUBLICATION
BIAS IN PUBLICATION BIAS
PUBLICATIONS?
THE SECOND QUESTION I HAVE, IT
RELATES TO THIS TABLE, WHICH I
WAS THINKING MAYBE, NEIL, OR
ANYONE UP HERE MIGHT BE ABLE TO
ANSWER, WHICH IS THE PERSEVERING
RESEARCHERS, THE ONES WHO DIDN'T
GET FUNDED ORIGINALLY, ARE
THEY -- IS THERE SOME POSITIVE
RESULTS HERE IN THE SENSE OF IS
THERE INDICATION IN THIS TABLE
THAT THE PERSEVERING RESEARCHERS
ARE PUBLISHING MORE NULL RESU
RESULTS?
I REALLY -- WE DIDN'T GET ENOUGH
TIME WITH THIS, MAYBE YOU CAN
EXPLAIN TO US A LITTLE MORE.
>> SURE, WE'LL LOOK AT IT A
LITTLE MORE SLOWLY.
SO LET'S LOOK AT THE BOTTOM ROW
IS WHAT YOU'RE MOST INTERESTED
IN, I GUESS; IS THAT CORRECT?
>> THE TOTAL.
>> THE TOTAL, YEAHMENT SO I
THINK THE RELEVANT THING IS FOR
THE PURPOSE OF THIS CONFERENCE
IS LIKE THE GAP.
SO IF YOU LOOK AT THE
PERSEVERING RESEARCHERS AND
THEIR PUBLICATION STATUS, THEIR
GAP IS ABOUT LIKE 20 PERCENTAGE
POINTS, WHEREAS IT'S 30
PERCENTAGE POINTS FOR THE OTHER
GROUP.
SO I BRIEFLY MENTIONED THIS AT
THE END.
THIS IS THE OPPOSITE DIRECTION
OF WHAT YOU WOULD EXPECT IF YOU
THINK THE INSTITUTION MATTERS.
IF THE INSTITUTION IS MATTERING,
YOU WOULD EXPECT THE GAP TO BE
LOWER ON THE LEFT-HAND SIDE.
SO I THINK THIS IS SOMETHING WE
SHOULD THINK ABOUT, WHICH IS
MAYBE THE INSTITUTIONS ARE NOT
AS POWERFUL AS WE THINK AT WHAT
IS DRIVING THE RESULT.
I'M NOT SAYING THAT'S FOR SURE
BUT IT'S A QUESTION WE MIGHT
WANT TO ASK.
BECAUSE YOU'RE CONDITIONING ON
CULTURE IN THIS ANALYSIS,
THEY'RE ALL OPERATING IN THE
SAME OVERALL POST OPEN SCIENCE
CULTURE, BUT ONE IS SUBJECT TO
INSTITUTIONS AND ONE IS NOT.
>> THANK YOU SO MUCH.
>> I HAVE A QUESTION THAT KIND
OF SPEAKS TO -- SORRY, I'M THE
EDITOR-IN-CHIEF OF THE JOURNAL
PLUS ONE.
IT SPEAKS TO THE QUESTIONS THAT
WE'RE GOING TO BE HEARING OVER
THE NEXT DAY OR TWO.
SORRY.
I HAVE TO SHRINK DOWN.
SO YOU MENTIONED THE PROBLEMS
WITH -- SORRY -- PUBLISHING
SMALL STUDIES THAT AREN'T
SUITABLY POWERED, BUT I JUST
WONDERED IF I COULD GET YOUR
VIEW ON THE VALUE OF RELEASING
THAT DATA INTO THE RESEARCH
ECOSYSTEM FOR OTHERS TO RE-USE
TO BUILD UPON.
IS THERE VALUE IN DATA THAT MAY
NOT BE STATISTICALLY SIGNIFICANT
ON ITS OWN BUT COULD FORM THE
BASIS OF FUTURE RESEARCH?
>> I THINK IT WILL BE GREAT,
SOMETIMES YOU DON'T HAVE THE
RESOURCES TO MAKE A LARGER --
STUDY BECAUSE YOU DON'T HAVE
NUSS MICE, YOU DON'T HAVE NUSS
CELLS OR YOU DON'T HAVE ENOUGH
MONEY TO RUN IT.
I THINK IF I COULD SPECULATE AND
KIND OF DREAM OUT LOUD, I THINK
MORE THAN OPEN SCIENCE, I THINK
IF YOU COULD HAVE OPEN LAB
BOOKS, THAT WOULD BE FANTASTIC.
BUT OBVIOUSLY TO BE ABLE TO DO
THAT, YOU NEED SOME KIND OF A
STRUCTURE TO LAB BOOKS THAT WE
ALL SHARED.
SO I THINK THERE COULD BE A WAY
THAT TO MAKE SURE WHAT WE DO
DOESN'T STAY, YOU KNOW, IN THE
LAB UNPUBLISHED FOR MANY YEARS,
BECAUSE THERE ARE LOTS OF VIABLE
DATA THERE THAT WE'RE NOT TAKING
ADVANTAGE.
>> THE HYPOTHESIS WAS THAT IT
WAS SUFFICIENTLY POWERED BUT IT
WAS AN INCORRECT HYPOTHESIS.
THANKS.
I JUST HAVE A QUESTION ABOUT --
OR COMMENT ABOUT WHAT ABOUT
STARTING FROM FUNDED GRANTS?
OR OTHER FUNDED SOURCES?
AND USING THAT TO TRACK WHAT
STUDIES ARE BEING PROPOSED?
RIGHT?
SORT OF THE OUTCOMES RESEARCH,
AND I DON'T KNOW HOW NIH HAN HAS
THAT OR ANYBODY DOES, AS ANOTHER
SOURCE OF THAT INFORMATION.
>> SO WE'VE BEEN TRYING TO DO
THAT IN MY RESEARCH GROUP.
THE TROUBLE IS THAT WE WANT SORT
OF THE -- SO WE'VE BEEN TRYING
DO THAT IN OUR RESEARCH GROUP.
THE TROUBLE IS THAT WE REALLY
WERE WANTING TO GET THE
PRELIMINARY DATA THAT WAS
PRESENTED IN THE GRANTS AND AT
LEAST FOR THE NIH, YOU HAVE TO
DO A FREEDOM OF INFORMATION ACT
TO GET EVERY SINGLE ONE OF THOSE
APPLICATIONS IF YOU WANTED TO
SORT OF FOLLOW THAT PRELIMINARY
DATA FORWARD.
WE JUST RAN INTO CAPACITY
ISSUES.
>> LET ME JUST CHIME IN, AS THE
GOVERNMENT HERE.
[LAUGHTER]
YOU'RE ALL UNDER ARREST.
[LAUGHTER]
SO AS YOU KNOW, THESE ARE GRANTS
AND NOT CONTRACTS.
AND AS A RESULT, CERTAINLY YOU
HAVE SPECIFIC AIMS BUT THEY CAN
CHANGE AND YOU CAN FOLLOW YOUR
HEART AND YOU CAN MAKE ALL --
YOU KNOW, SO IT'S ESSENTIALLY
IMPOSSIBLE TO KIND OF -- TO
RELATE NECESSARILY WHAT ENDS UP
WITH WHAT YOU WROTE IN YOUR
APPLICATION.
>> YEAH, GRANTS ARE NOT EXACTLY
PREREGISTRATION.
>> HI, MY NAME IS NID.
BALA, UNIVERSITY OF CALIFORNIA
SANTA CRUZ.
I REALLY APPRECIATED VIRGINIA'S
POINT ABOUT EXCELLENCE AND THE
WAY THAT WE'VE WEAPONIZED THE
WORD EXCELLENCE.
AND I APPRECIATED YOUR CONTRAST
WITH PROFICIENT OR COMPETENCE.
AND I THINK I WOULD LIKE TO
INTRODUCE TWO ADDITIONAL WORDS
THAT MIGHT BE USEFUL FOR US TO
THINK ABOUT FOR THE NEXT TWO
DAYS, WHICH IS INCENTIVES, I
THINK SOMETIMES WE USE
INCENTIVES AS AN EXCUSE BUT
THEY'RE ALSO INCREDIBLY
POWERFUL.
AND THEN STATUS.
I THINK THAT WE FAIL TO
RECOGNIZE THE IMPORTANT ROLE
THAT IS IT TUS PLAY STATUS PLACE
WOULD LIKE THEIR WORK TO BE
RECOGNIZED AND THAT OFTEN IS
EQUATED WITH STATUS IN BAD AND
GOOD WAYS.
AND BRINGING TOGETHER THOSE
IDEAS, I THINK ONE THING I'D
ALSO LIKE TO INTRODUCE IS THIS
IDEA THAT WHEN WE TALK ABOUT
INCENTIVES FOR SCIENCE, AND WE
TALK ABOUT AN ALTERNATIVE TO
EXCELLENCE, WHICH IS AN
INCREDIBLY VAGUE TERM, THAT WE
SHOULD THINK ABOUT SCIENCE AS
BEING CAPACITY-BUILDING.
RIGHT?
THAT WE WANT PEOPLE TO BUILD ON
OUR WORK, AND THAT MEANS,
NATURALLY, THAT WE SHOULD BE
PUBLISHING BOTH POSITIVE AND
NEGATIVE RESULTS FROM OUR WORK
BECAUSE THAT NEGATIVE RESULT --
THOSE NEGATIVE RESULTS WILL
PROVIDE THE CAPACITY FOR THE
PEOPLE TO MAKE THE NEXT JUMPS IN
LOGIC AND TECHNIQUES AND
TECHNOLOGY THAT MIGHT BE
NECESSARY WITHOUT HAVING TO
REPEAT THAT.
SO I WOULD JUST LIKE TO
INTRODUCE THOSE CONCEPTS AS WE
CONTINUE TO HAVE THIS
CONVERSATION.
>> THANK YOU.
I THINK THAT REALLY IS PROBABLY
WHAT I WANTED TO SAY WITH OTHER
WORDS.
I THINK WHEN YOU HAVE SOMEONE
THAT SAVES THE DAY, THAT PERSON
TAKES THE MERIT, AND SOMETIMES
THERE'S BEEN MANY OTHER PEOPLE
BEHIND THAT PERSON THAT
SOMETIMES DICTATE THE WRONG TURN
SO THAT PERSON COULD TAKE THE
RIGHT ONE.
AND WE DON'T -- FOR PEOPLE THAT
DIDN'T GET FIRST.
BUT IN SCIENCE, I THINK WE
SHOULD FOCUS ON DOING THE WORK
PROPERLY, AND NOT DOING THINGS
THAT ARE GOING TO BE ON THE
COVER OF A MAGAZINE.
BUT I THINK IT'S THE TYPICAL
THING, IT'S MORE LIKE A SOCIETAL
THING, PROBABLY NOT THE SCIENCE,
WE DON'T WANT TO BE THE, YOU
KNOW, LABORIOUS -- AND DOING
THINGS.
WE'D RATHER BE THE FAMOUS SINGER
OR RESEARCHER, BUT RESEARCH
SHOULD BE A BIT BORING AND JUST
METICULOUS, I THINK.
BUT WE DON'T SELL THAT.
>> I'D ALSO JUST LIKE TO ADD TO
THAT AND I THINK ALSO IN
RESPONSE TO AN EARLIER QUESTION
ON THE VALUE OF THE NULL
STUDIES, I THINK AS SCIENTISTS
WE CAN SOMETIMES HAVE A
SHORT-TERM VIEW OF THE IMPACT OF
OUR RESULTS, AND I THINK WE
SHOULD REMEMBER THAT RARELY ARE
DECISIONS SUCH AS POLICY OR
CLINICAL GUIDELINES THAT ARE
PUBLISHED, RARELY ARE THEY BASED
OFF OF ONE SINGLE STUDY, IT'S
USUALLY AFTER A PROCESS OF
EVIDENCE SYNTHESIS, SYSTEMATIC
REVIEWS OF META-ANALYSIS THAT WE
LOOK AT THE ENTIRE EVIDENCE AND
THEN DETERMINE OKAY -- I THINK
AS RESEARCHERS, WE SHOULD KEEP
THAT BROAD VIEW THAT WE ARE VERY
MUCH CONTRIBUTING TO A BODY OF
RESEARCH.
IT'S NOT JUST OUR OWN WORK AND
OUR OWN SINGLE RESULT THAT'S
GOING TO DETERMINE EVERYTHING.
AND I THINK ALSO, I MEAN, I'M
NOT GETTING INTO STATISTICS
HEAVY HERE, BUT PATIENT
STATISTICS ARE BECOMING
INCREASINGLY COMMON AND
ENCOURAGED TO USE IN THE FIELD,
AND A HUGE PART OF PAY SHAN
STATISTICS IS HAVING PRIOR
EVIDENCE, PRIOR KNOWLEDGE TO
HELP INFORM -- TO INFORM YOUR
ANALYSIS.
IF YOU BASE YOUR PRIOR BELIEFS
ON A BIASED EVIDENCE BASE,
THAT'S NOT GOING TO HELP YOUR
FUTURE STUDIES EITHER.
SO IT IS VERY MUCH IMPORTANT
THAT WE HAVE AN ACCURATE PICTURE
OF WHAT IS OUT THERE BOTH FOR
DECISION-MAKING FOR FUTURE
ANALYSIS, FOR EVIDENCE
SYNTHESIS, IT'S VERY IMPORTANT.
>> THANKS, EVERYONE.
I'M THE CO-FOUNDER AND CSO OF
ARCADIAN SCIENCE.
I WANTED TO MAYBE TALK OUT OF
BOTH SIDES OF MY MOUTH AND
HOPEFULLY SYNTHESIZE SOMETHING
THAT WE HAVEN'T YET TOUCHED ON,
WHICH IS THAT I COMPLETELY TAKE
THE POINT OF JUST ACTUALLY PR
PRECONDITIONING RESULT ON SOME
SORT OF SHARED ASSUMPTIONS ABOUT
RIGOR OF METHODOLOGY AND THINGS
LIKE THAT.
OBVIOUSLY YOU CAN IMAGINE
OMITTING HALF OF THE RELEVANT
COMPONENTS AND REACTION AND NOT
GETTING SOMETHING TO WORK AND
THAT'S NOT NECESSARILY SOMETHING
THAT'S USEFUL TO PEOPLE.
NOW THAT SAID, THE IDEA OF HAV
HAVING -- MAYBE TO SPEAK TO THE
QUESTION ABOUT DOES IT HELP TO
HAVE EITHER SMALLER OR
UNDERPOWERED OR OTHER TYPES OF
STUDIES OR EVEN INFORMATION
ABOUT INCOMPLETE WORK OUT THERE,
SUGGESTING THAT THERE ARE OTHER
USES FOR SOME OF THIS NEGATIVE
DATA BEYOND JUST SORT OF
CONFIRMING THE KNOWLEDGE, MAKING
SURE THAT WE CAN -- WHAT WE ARE
LOOKING AT THAT INFORMATION AS,
IS IT TRUTH, ARE WE LOOKING TO
GET INSIGHTS OR SOMEBODY ELSE'S
ASSESSMENT OF THAT OUT, OR ARE
WE SAYING OKAY, WE KNOW THIS
DIDN'T WORK AND NOW OTHERS KNOW
WHAT ELSE TO TRY, AND YOU KNOW,
TRYING TO NOT BASE OUR
ASSUMPTIONS ON IF WE START TO
RELEASE MORE NULL DATA, MORE
NEGATIVE DATA, HAVING PEOPLE
UNDERSTAND HOW TO INTERPRET THAT
INFORMATION, EVEN IF YOU WERE TO
BE EXTREMELY RIGOROUS IN YOUR
METHODOLOGY, RIGHT, AND THEN PUT
OUT A NULL RESULT, TECHNOLOGY IS
ALWAYS INCREASING, WE ALWAYS ARE
ACCELERATED OUR RESOLUTION,
WE'RE REDUCING OUR MEASUREMENT
ERROR. WHAT WE MIGHT FIND 10
YEARS FROM NOW MIGHT BE
DIFFERENT FROM WHAT WE FIND OT
EVEN IF IT WAS VERY RIGOROUSLY
DONE BEFORE JUST BECAUSE THE
NATURE OF OUR REALITY IS
CHANGING.
SO I THINK THE POINT I'M
SUGGESTING IS MAYBE THE WAY WE
INTERPRET OR THE WAY WE TEACH
OTHERS TO INTERPRET THOSE NULL
RESULTS IS JUST AS IMPORTANT AS
WHAT IT IS THAT WE PUT OUT THE
THERE.
>> I THINK THIS IS A VERY
IMPORTANT POINT AND I ALSO THINK
THIS TIES IT BIT INTO
REPRODUCIBILITY ISSUES OR WHY
PEOPLE DON'T ATTEMPT TO
REPLICATE STUDIES THAT HAVE BEEN
DONE BEFORE, ESPECIALLY IF IT
HAD A NON-SIGNIFICANT RESULT.
WE THINK OF STUDIES WHERE THEY
DIDN'T SAY OKAY, THIS QUESTION
HAS ALREADY BEEN ANSWERED BASED
ON THIS ONE STUDY CONDUCTED IN
THIS WAY AND GAVE THIS RESULT,
BUT THAT'S NOT THE CASE.
STUDIES SHOULD BE REPLICATED
BECAUSE REALLY ALL A P VALUE IS
SAYING THAT THIS IS THE
PROBABILITY THAT YOUR STUDY
CONDUCTED IN THIS WAY WITH THIS
DATA WOULD PRODUCE THAT RESULT,
GIVEN THAT THE NULL HYPOTHESIS
IS TRUE.
IF YOU CONDUCT THE STUDY ON A
DIFFERENT SAMPLE OR WITH
SLIGHTLY DIFFERENT CONDITIONS,
OR EVEN SIMILAR CONDITIONS, DUE
TO CHANCE, YOU CAN END UP WITH A
DIFFERENT RESULT.
SO IT IS VERY IMPORTANT TO
ATTEMPT TO REPLICATE AND TRY
DIFFERENT STUDIES, NOT JUST
ASSUME THAT WHAT'S ALREADY BEEN
DONE HAS TOLD THE WHOLE TRUTH.
>> HI, GIDA SWAMY DUKE
UNIVERSITY.
I WANTED TO MAKE TWO COMMENTS.
ONE, I NE WE HAVE A DIFFERENT
SESSION ON SORT OF
NON-TRADITIONAL PUBLICATION
ROUTE BUT JUST TO PICK UP ON THE
ANALYSIS BETWEEN -- ON THE
ANTIDEPRESSANT DATA.
I'M ASSUMING THAT FDA LOOKED AT
THAT OR THAT WAS LOOKED AT FROM
CLINICALTRIALS.GOV REGISTRATION,
AND SO WITH THE IMPROVEMENTS IN
COMPLIANCE WITH
CLINICALTRIALS.GOV REQUIREMENTS
AT LEAST IN THE UNITED STATES,
WE HAVE A LOT OF DATA THERE ON
RESULTS THAT CLEARLY IS NOT
GETTING OUT, RIGHT, IN A SUMMARY
FORMAT, IN A PRESS RELEASE, IN
THE WAY THAT WE WANT TO CONVEY
THAT INFORMATION.
SO IT'S REALLY ONLY IF YOU'RE
GOING TO DESIGN A STUDY THAT YOU
MIGHT GO AND LOOK AT
CLINICALTRIALS.GOV TO FIGURE OUT
WHAT'S OUT THERE AND WHAT HAS
BEEN DONE.
SO I THINK THAT WITH LOTS OF
THINGS COMING ON BOARD, WE
SHOULD REALLY BE CONSIDERING HOW
TO GET THAT INTO A BETTER
MAINSTREAM FORMAT THAT IS
SOMETHING THAT THE LAY PUBLIC
CAN UNDERSTAND AND TO BE HONEST,
EVEN SCIENTISTS WHO MAYBE AREN'T
DESIGNING THE TRIALS FROM AN EPI
STANDPOINT CAN UNDERSTAND AS
WELL BECAUSE RIGHT NOW THE
TABULAR FORMAT AND SO FORTH IS
CUMBERSOME TO FIGURE OUT BUT
IT'S THERE AND IT'S INFORMATION
WE REALLY SHOULD BE CONVEYING
AND USING.
ONE OTHER THING I WANTED TO MAKE
COMMENT THAT SORT OF PICKS UP ON
THE LAST DISCUSSION WAS ABOUT
PUBLISHING THINGS THAT I THINK
PEOPLE FEEL ARE REDUNDANT.
SO REPLICATION IS THE RIGHT
WORD, BUT WHEN PEOPLE SEE THAT
THEY'VE ALREADY BEEN SCOOPED AND
SOMETHING'S BEEN PUBLISHED, THEY
SORT OF LOSE FEITH AND THAT
ANYONE WILL ACTUALLY PICK IT UP
OR TAKE IT OR AS A MATTER OF
FACT, IF IT SHOWS SOMETHING THAT
THERE'S IS NOT SIGNIFICANT AND
THE OTHER ONE THAT'S PUBLISHED
IS, THEN IT BECOMES WHAT DOES
THAT MEAN?
I'D ARGUE IT SHOULD ALL BE OUT
THERE, RIGHT, THAT'S THE WHOLE
BENEFIT OF PEER REVIEW AND FOR
LOOKING AT THIS ACROSS THE
BOARD.
SO I DO THINK THAT'S ANOTHER
AREA THAT COMES INTO PLAY, IS
THAT PEOPLE SIMPLY DON'T PUBLISH
BECAUSE SOMEBODY ALREADY
PUBLISHED IT IN THE TIME COURSE
THEY WERE DOING THEIR WORK AS
WELL.
>> I DIDN'T WHAT TO MAKE A
SLIGHT COMMENT ON THE
CLINICALTRIALS.GOV EXEUN, AND IF
YOU THINK ABOUT IT, WHAT SH AT
LEAST NINDS FUNDS, CLINICAL
TRIALS ARE NOT THE MAJORITY OF
THAT.
SO IMAGINE HOW MUCH WORK WE HAVE
TO DO IN TERMS OF THE
PRE-CLINICAL WORK.
A LOT OF THE ANIMAL STUDIES
WE'RE DOING, IN VITRO STUDIES,
COMPUTATIONAL STUDIES, SO EVEN
IF CLINICALTRIALS.GOV WERE
PERFECT, WE WOULD STILL HAVE A
LONG WAY TO GO.
>> I ALSO WANTED TO SAY, WE WERE
LUCKY WHEN WE WERE DOING A
SPECIAL ISSUE -- RESULTS OF
BEING ABLE TO PUBLISH A CLINICAL
TRIAL AS WELL, AND I THINK IT
WAS VERY COMMENDABLE THAT THEY
DID MAKE A HUGE EFFORT TO
PUBLISH THAT EVEN THOUGH THEY
KNEW IT WAS NEGATIVE DATA.
AND IT WAS A SMALL JOURNAL.
AND I THINK IT MAY HAVE TO DO
WITH THE FACT THAT IT WAS RARE
DISEASE AND THERE ARE A LOT OF
STAKEHOLDERS AND THEY ARE
PUSHING AND IN MANY CASES
SPONSORING SOME OF THOSE TRIALS.
SO I THINK SPHIEMS SOMETIMES WE
ARE STAKEHOLDERS, BUT THEY HAVE,
YOU KNOW, THEIR OWN AGENDAS,
IT'S IMPORTANT SOMETIMES TO KNOW
WHO IS BEHIND A CLINICAL TRIAL,
AND THAT USUALLY MAKES IT MUCH
MORE ACCESSIBLE.
AND ALSO ON THE POINT OF -- IS
THE JOURNAL THE LAST DESTINATION
FOR SOMETHING?
IF THOSE DATA ON
CLINICALTRIALS.GOV WAS
ACCESSIBLE IN THE WAY THAT IT
SHOULD BE, PROBABLY AS THEY SAID
BEFORE, DATA AND METHODS ARE
WHAT ARE YOU'RE INTERESTED IN,
THE REST IS MARKET BEING, SO MAO
PROBABLY WE COULD DO WITHOUT THE
FINAL JOURNAL --
>> MALUDE TAN SEE, UNIVERSITY OF
FLORIDA.
I'M GOING TO REFLECT THE VIEWS
OF SOME OF THE PEOPLE THAT
REPLIED TO MY POST ON SOCIAL
MEDIA ABOUT THIS BIAS
CONFERENCE.
WORKSHOP.
AND ONE OF THEM IS THAT IT'S
BEEN THE EXPERIENCE OF SEVERAL
COLLEAGUES OF MINE IN THE
BIOMEDICAL RESEARCH THAT
SOMETIMES TO DISPUTE OR REJECT
OR PUBLISH CONTRADICTORY
FINDINGS TO WHAT HAS BEEN
PUBLISHED BY SOMEONE WITH
STATUS, IT TAKES SEVERAL LABS TO
DO IT TOGETHER.
AND I THINK -- I MEAN, I'M ALL
FOR MULTIDISCIPLINARY TEAM
SCIENCE, AND SOMETIMES THAT'S
WHAT IT TAKES BECAUSE A SINGLE
LAB MAY NOT HAVE THE, I GUESS,
CREDIBILITY ON THEIR OWN TO
DISPUTE SOMETHING THAT WAS
PUBLISHED IN A MAJOR JOURNAL.
SO I THINK EVEN THEN, WHEN THAT
HAPPENS, THERE'S ONE PARTICULAR
INSTANCE IN WHICH THERE'S NOW
BEEN SEVERAL PUBLICATIONS SAYING
THIS IS NOT THE CASE, THAT DOGMA
PERSISTS IN THE FIELD.
DESPITE THE FACT THAT YOU TOOK
THE TROUBLE TO PUBLISH THE
CONTRADICTORY FINDINGS BY 10
DIFFERENT LABS IN ONE PAPER.
SO HOW DO WE GET BEYOND THE NEED
TO CONFIRM OR DISPUTE IF THE
FIELD IS STILL SO STUCK ON
GIVING CREDIBILITY TO IDEAS OF
PEOPLE WITH STATUS THAT'S NOT
DATA-DRIVEN?
ANY THOUGHTS?
>> I DON'T HAVE AN ANSWER FOR
THAT.
I THINK THAT'S LIFE IN GENERAL.
I DON'T THINK IT'S ONLY -- I
DON'T THINK IT'S ONLY SCIENCE.
I THINK IT HAPPENS PRETTY MUCH
EVERYWHERE, AND I WISH IT
DIDN'T, BUT IT DOES.
>> I'LL JUST ADD, I FIND THAT
COMMENT SURPRISING COMING FROM
OUTSIDE THE BIOLOGICAL SCIENCES,
BECAUSE I GUESS PEOPLE IN THE
SOCIAL SCIENCES ARE LIKE, WELL,
WE DO STUFF THAT'S LOW STAKES
AND IT'S VERY HARD TO ASSESS THE
TRUTH.
BUT IN THE SCIENCES, EVERYTHING
IS PERFECT BECAUSE IF A DRUG
DOESN'T WORK, YOU HAVE THE BIAS
OF REALITY.
WHICH IS LIKE, HUMANS WILL
REALIZE THAT IT'S ACTUALLY
CAUSING PROBLEMS OR NON-EFFICACY
AND THEN LIKE THE REALITY OF THE
WORLD WILL CORRECT FOR ALL THESE
BIASES AND STATUSES.
SO YOUR COMMENT WAS SURPRISING
BECAUSE YOU KNOW, IT'S NOT WHAT
I WOULD HAVE EXPECTED, AND THEN
I JUST POINT THIS OUT WHICH IS A
LOT OF THE COMMENTS PEOPLE HAVE
BEEN MAKING ARE SOCIOLOGICAL
THEORIES.
EVEN TED MIGUEL OPENED WITH
ROBERT MERTON'S BELIEFS AS A
SOCIOLOGIST.
SO I DON'T KNOW HOW MANY
SOCIOLOGISTS ARE IN THE ROOM
RIGHT NOW BUT ACTUALLY KIND OF
SURPRISING IF YOU WANT TO SOLVE
A SOCIOLOGICAL PROBLEM, YOU HAVE
NO EXPERTS IN SOCIOLOGY IN THE
ROOM.
>> I WAS GOING TO SAY THAT
ACTUALLY ONE OF THE REASONS WE
HAVE A LOT OF PEOPLE FROM NOT
JUST THE BIOMEDICAL SCIENCES
HERE TODAY IS BECAUSE THIS IS
ACTUALLY SUCH A BROAD ISSUE, AND
THAT THE BIOMEDICAL SCIENCES ARE
NOT PERFECT AND WE NEED
PERSPECTIVES FROM OUTSIDE OF
BIOMEDICAL RESEARCH TO SOLVE
THIS PROBLEM.
>> I'M RACHEL MILES, VIRGINIA
TECH UNIVERSITY LIBRARIES.
I SPECIALIZE IN RESEARCH IMPACT
AND ASSESSMENT.
AND MY QUESTION MIGHT BE A
LITTLE BIT ABSTRACT BUT IT KIND
OF GOES OFF WHAT THE LAST
QUESTION WAS ASKING ABOUT
CULTURE AND DOGMAS.
AND I WONDER IF ANY OF YOU HAVE
HAD AN OPPORTUNITY OR MAYBE A
WAY OF FIGURING OUT A WAY TO
QUANTIFY DOGMAS OR CULTURE IN
YOUR RESEARCH.
AND IF THAT'S POSSIBLE BECAUSE I
KNOW WE CAN ALL SAY WE SHOULD BE
DOING THIS OR SHOULD BE DOING
THAT BUT IT'S EASIER SAID THAN
DONE TO SAY THAT WE SHOULD BE
DOING THESE THINGS BECAUSE I
WORK IN THE REALM OF
INSTITUTIONAL CHANGE, AND I
BELIEVE THAT NEIL'S RESEARCH
SHOWED THAT IBS TEUTIONAL CHANGE
IS LESS LIKELY TO OCCUR AROUND
CULTURAL CHANGE HAS TO HAPPEN
FIRST.
AND I AM A TRUE BELIEVER IN THAT
AND THAT'S WHY I KIND OF WORK IN
THIS REALM OF LOOKING AT
RESEARCH ASSESSMENT REFORM AND
THIS ALL GOES BACK TO CULTURAL
CHANGE AND IF YOU CAN LOOK AT --
IF WE CAN ZOOM OUT AT A MACRO
LEVEL AND KIND OF LOOK AT ROBERT
MERTON'S -- THAT KIND OF MATTHEW
EFFECT, AND LOOK AT THIS FROM A
MACRO LEVEL IN THINKING ABOUT
FROM A SOCIOLOGICAL PERSPECTIVE
OF HOW WE CAN QUANTIFY THESE
CULTURAL ASPECTS AND THESE
DOGMAS, CAN WE ACTUALLY TARGET
SOME OF THE CULTURAL ASPECTS AND
MAYBE FIGURE OUT WHERE WE NEED
TO CHANGE SPECIFICALLY, RATHER
THAN JUST SAYING WE NEED TO
CHANGE, WHERE DO WE NEED TO
CHANGE.
>> JUST ADDING TO THAT, IS THERE
A WAY ALSO TO TAKE INTO ACCOUNT
CULTURAL DIFFERENCES ACROSS
CONTINENTS, BECAUSE THE WAY THE
WESTERN WORLD IS DOING SCIENCE
IS NOT THE SAME AS THE RUSSIAN
WORLD IS DOING SCIENCE, AND THE
ASIAN WORLD IS DOING SCIENCE.
>> THE SOCIAL CHANGES, I CAN
ONLY TELL YOU WHAT WE TRIED TO
DO IN THE NETWORK I WAS
CHAIRING.
SO WE DID INCLUDE IN NATURE --
MEETINGS WE HAD A SECTION ON
NEGATIVE RESULTS, WE TRIED TO
INVOLVE ALL THE YOUNGER
RESEARCHERS TO PRESENT THEIR
DATA, EVEN IF IT WAS, YOU KNOW,
WE TRIED TO MAKE IT NON- --
TRIED NOT TO BE TOO HARSH ON
THEM, BASICALLY BECAUSE I THINK
THE MAIN IDEA IS THAT IF YOU
HAVE THE RIGHT CONTROLS AND THEN
YOU HAVE THE RIGHT DATA, YOU
DON'T NEED THE RIGHT RESULTS.
YOU JUST NEED TO SHARE THEM WITH
EVERYBODY ELSE.
SO WE INCLUDED SESSIONS ON
NEGATIVE DATA IN ONE OF OUR
MEETINGS AND WE ALSO ORGANIZED
WEBINARS AND WE ESPECIALLY WORK
ON NEGATIVE OR CONTRADICTORY
RESULTS.
AND I THINK THERE WERE QUITE A
LOT OF THINGS THAT CAME OUT OF
IT AND WE ARE QUITE HAPPY
BECAUSE I THINK IN MANY OF THE
RESEARCHERS THAT HAD COME
THROUGH THAT, I THINK THEY HAVE
A DIFFERENT VIEW AND AT LEAST
IT'S SOMETHING EMBEDDED IN THEIR
MINDS.
SO HOPEFULLY WE'LL BE ABLE TO
MAKE IT -- I THINK IT'S
HIGHLIGHTED, GIVEN A BIT -- IT'S
A WAY TO GO FORWARD.
BUT IT'S JUST ONE WAY.
>> THANK YOU.
SO I TOTALLY AGREE, THIS IS A
SOCIAL CULTURAL BEHAVIORAL
PROBLEM WITH MAJOR INSTITUTIONS,
PUBLISHERS, FUNDERS, ACADEMIC
INSTITUTIONS, AS THE
THREE-LEGGED STOOL OF THE
PROBLEM, SO WE CAN ALL
CONTRIBUTE TO THE PROBLEM OR THE
SOLUTION.
TO THE POINT, I THINK
REPLICATION RESEARCH IS, TO
START WITH, NOT ATTEMPTED OFTEN
BECAUSE NOBODY IN ACADEMIA WAS
PROMOTED THAT TENURE TRACK FOR
DOING REPLICATION OR
REPRODUCIBLE RESEARCH.
THEN WHEN IT IS ATTEMPTED, THERE
IS THAT PRIMACY OF PUBLISHING IN
THE FANCY PAPER AND PUBLISHING
FIRST, AND SORT OF THE BURDEN OF
EVIDENCE OR -- TO ACCEPTING THAT
THERE IS CONTROVERSY AND THAT
THE DATA -- THE CONCLUSIONS ARE
NOT SUPPORTED BY NEW DATA.
BUT THE THIRD ONE TO THE POINT
OF THE COLLEAGUE FROM PLUS ONE
IS THAT, YES, STUDIES ARE
IMPERFECT WHEN THEY'RE SMALL,
WHETHER THEY HAVE POSITIVE
CONCLUSIONS OR IPT PRE TAITIONSS
OR NEGATIVE INTERPRETATIONS,
THEY COULD BE USEFUL EXCEPT
THEY'RE NOT, AND THE REASON IS
THAT THE PROVENANCE OF THE
FINDINGS FROM THE PRIMARY DATA
METHODS, ET CETERA, IS VERY
OPAQUE.
NOT ONLY ANIMAL MODEL, RESEARCH,
WHERE IT'S ALMOST -- NEVER YOU
GET THE PRIMARY DATA, SO THE
IDEA OF OPEN NOTEBOOKS AND THIS
IS WHERE BOTH PUBLISHERS AND
PRINTERS CAN REALLY CONTRIBUTE,
BUT ALSO INCREASINGLY THE BIG
DATA RESEARCH, WHERE THERE IS A
LOT OF NEW FINDINGS, FAST
FINDINGS, WHERE OFTEN THE DATA
AVAILABILITY SECTION, EVEN IN
FANCY JOURNALS, IS KIND OF EMPTY
OR IT TAKES YOU -- PROVIDES A
LINK THAT LEADS TO NOWHERE.
FOR WHERE THE DATA IS.
SO ANYWAY, IT'S GREAT TO BE PART
OF THE DISCUSSION.
I SHOULD ALSO SAY FROM A -- I
DON'T KNOW IF I SAID THAT, BUT
ONLY FOSSAE T TO SAY THAT WE HAN
ANALYSIS IN TERMS OF THE QUALITY
OF THE RIGOR OF ANIMAL MODEL
EFFICACY STUDIES COMPARING --
AND NOT SO HIGH JOURNALS, NOT
MUCH DIFFERENCE, SPEAKING TO
THAT AGAIN.
THERE ARE SOME REAL CULTURAL
BARRIERS IN INERTIA.
LASTLY ON THE ACADEMIC
INSTITUTION SIDE, AS WELL AS THE
FUNDER SIDES, A GREAT
OPPORTUNITY TO IMPROVE AND BUILD
NEW TRAINING PROGRAMS, AND JUST
TEACH THE SCIENTIFIC METHOD.
IT FEELS LIKE I DON'T THINK
THERE IS A SUBJECT CALLED THAT.
IT SHOULD START IN HIGH SCHOOL
OR BEFORE AND CONTINUE THROUGH
UNDERGRAD, GRAD AND FOREVER.
>> OKAY, CAN I -- OH, YOU WANT
TO ANSWER?
SORRY.
>> MY BRAIN IS JUST -- THE
QUESTION ABOUT HOW TO MEASURE OR
ASSESS CULTURAL CHANGE, I WAS
JUST BRAINSTORMING THINKING
PERHAPS THAT COULD BE AN
OPPORTUNITY FOR A.I. OR NATURAL
LANGUAGE PROCESSING TO SCOUR
ARTICLES, MOST RECENT ARTICLES
COMPARED TO PREVIOUSLY AND SEE
IF THERE ARE ANY DIFFERENCES IN
HOW RESEARCHERS DISCUSS THEIR
RESULTS OR PRESENT THEM OR ARGUE
FOR THE STRENGTH OF THEIR
CONCLUSIONS.
JUST -- I DON'T KNOW, JUST
BRAINSTORMING AND THOUGHT I'D
THROW THAT OUT THERE AS A
POSSIBILITY FOR USING NOVEL
METHODS.
>> OKAY.
IT'S NEARLY LUNCHTIME SO MAYBE I
CAN THROW NOT A ROCK BUT A
BOULDER INTO THE POND.
CAN I DO THAT?
OKAY.
I'M GOING TO -- LET'S DO A
LITTLE BIT OF PHILOSOPHY OF
SCIENCE.
YOU'LL SEE WHERE I'M GETTING AT.
SO I'M COMING FROM THEORETICAL
PHYSICS.
THEORETICAL PHYSICS IS MATH, AS
WE JUST HEARD.
MATH, THEY DO NOT LIE.
WHEN YOU WANT TO DEMONSTRATE
SOMETHING, IT'S EITHER TRUE OR
FALSE.
THERE'S NO OTHER ALTERNATIVE.
BECAUSE IT FOLLOWS A SET OF
RULES AND AXIOMA.
IN BIOLOGY, WE DO NONT DO NOT Y
THEORY TO GUIDE US SO WE DON'T
KNOW WHAT WE'RE TALKING ABOUT,
THAT'S THE FIRST THING.
BUT THAT'S THE REALITIMENT
RIGHT?
REALITY,
OTHERWISE WE WOULD HAVE SOLVED
THE PROBLEM OF LIFE A LONG TIME
AGO.
WE STILL DO NOT UNDERSTAND HOW A
SINGLE TROA CARE YOT IS WORKING.
WE HAVE NO CLUE.
NOW, WE KNOW THAT THERE IS A
PROBLEM WITH REPLICATION, NULL
STUDIES, AND THE PROBLEM WITH
THE P VALUE.
THAT'S FOR SURE.
THIS IS A PROBLEM.
BUT I'M GOING TO ARGUE THAT IT
IS ALSO AN ILL-POSED PROBLEM,
LIKE IN MATH, ILL-POSED.
ILL-POSED MEANS THERE IS NO
SOLUTION TO IT.
AND FOR A SCIENTIFIC REASON.
SO YOU MENTIONED, SHAI, -- WHAT
IS CALLED HOMEOSTASIS, MEANS
THAT EVERYTHING IS IN
RELATIONSHIP WITH EVERYTHING,
AND MAINTAINS A KIND OF
EQUILIBRIUM.
BUT YOU DIDN'T FIND THAT, BUT
THIS LED LATER TO THE CONCEPT OF
DEGENRE SEE, THE MULTIPLE
SOLUTION TO THE SAME PROBLEM.
PROOF, YOU CAN REMOVE 70% OF THE
JEANS OF E. COLI AND NOTHING
HAPPENS.
THERE ARE REDUB DAN SEES AND YOU
CAN FIND SOLUTIONS ONE BY ONE,
NOT ALL OF THEM AT THE SAME
TIME, OF COURSE, BUT ONE BY ONE.
YOU SLOW DOWN METABOLISM, IT'S A
FAMOUS PAPER, TWO PAPERS.
PAPER SHOWING
THAT.
SO WHAT DOES IT MEAN?
IT MEANS THAT WHATEVER PARAMETER
YOU MEASURE IN A BIOLOGICAL
SAMPLE, YOU'RE GOING TO FIND A
LARGE SPECTRUM OF VALUES IN MANY
PARAMETERS.
THIS HAS BEEN FANTASTICALLY
DEMONSTRATED.
SO WHATEVER YOU MEASURE, YOU
HAVE A HUGE DISTRIBUTION.
NOW IF YOU PICK UP FIVE OR 10
SAMPLES FROM THIS HUGE
DISTRIBUTION, YOU MAY HAVE
SOMETHING WITH AN AVERAGE VALUE
IN THE STANDARD -- OF DOMAIN.
YOU DO SOME CHANGE, YOU MAY FIND
SOMETHING DIFFERENT OR NOT.
NOW, WHY DO WE HAVE SO MUCH
VARIABILITY IN CLINICAL TRIALS?
BECAUSE EACH INDIVIDUAL IS
UNIQUE, AND DENYING THAT, IT'S
DENYING BIOLOGY.
SO A DRUG WILL WORK IN A CERTAIN
NUMBER OF INDIVIDUALS WHERE THE
SET OF PARAMETERS IS SUCH THAT
THE DRUG IS TARGETING THEM, AND
NOT OTHERS.
THE PROBLEM THAT WE HAVE IS
TRYING TO FIND SOMETHING THAT
WORKS FOR EVERYTHING.
BUT THAT IS DENYING BIOLOGY,
BECAUSE BIOLOGY IS VARIABILITY.
AND THAT ALLOWED US TO EVOLVE
AND TO SURVIVE.
SO THE FACT THAT WE FIND STUDIES
WITH P VALUES WHICH ARE
DIFFERENT, SOMETIMES
SIGNIFICANT, SOMETIMES NOT
SIGNIFICANT, MOST OF IT FOR ME
REFLECTS THE VARIABILITY OF
LIFE.
AND NOT A MISTAKE IN THE DESIGN
OF THE EXPERIMENT, OF THE ME
MEASUREMENTS WHICH ARE NOT IN
THE SAME WAY.
IT'S BOUND TO BE VARIABLE.
SO PART OF WHAT EXPLAINS THIS
VARIABILITY OR THIS CRISIS IS
JUST BIOLOGY.
I'M NOT SAYING THAT EVERYTHING
EXPLAINS IT.
I'M JUST GIVING YOU SOME KIND OF
BROADER PERSPECTIVE THAT
PROVIDES A DIFFERENT APPROACH OF
THIS PROBLEM.
YOU CAN READ MY EDITORIAL ON
DEGENERACY THAT THEY PUBLISHED
IN NOVEMBER.
THAT WAS A BRIEF SUMMARY.
IT'S MORE COMPLEX THAN SAYING
IT'S TRUE OR NOT TRUE OR THERE
ARE PROBLEMS AND NOT PROBLEMS.
WE ARE DEALING WITH BIOLOGY AND
NOT WITH MATH.
DO YOU KNOW WHEN P VALUES
WERE -- WHERE STATISTICS ARE
COMING FROM?
IT'S ALREADY FROM BABYLON TIMES
AND EGYPTIAN TIMES WHEN THEY ARE
TRYING TO FIGURE OUT, PREDICT
THE TRAJECTORIES IN THE SKY TO
PREDICT AUTUMN, FALL, WINTER
HARVEST, AND EVERYTHING.
THESE ARE BASED ON PHYSICS.
YOU DO STATISTICS BECAUSE YOU
HAVE VARIABILITY IN YOUR
MEASUREMENT, AND YOU DESCRIBE
THIS UNCERTAINTY WITH
STATISTICS.
BUT IN BIOLOGY, THERE WAS NO
CERTAINTY.
IT'S ONLY VARIABILITY.
SO DO STATI STATISTICS REALLY AO
BIOLOGY?
>> I WOULD SAY THE AMERICAN
STATISTICAL ASSOCIATION HAS
DEFINITELY PUT OUT COMMENTARIES
ABOUT QUESTIONING WHETHER OR NOT
NULL HYPOTHESIS SIGNIFICANCE
TESTING IS EVEN APPROPRIATE AND
WHETHER OR NOT WE SHOULD BE
MAKING DI DICHOTOMOUS DECISIONS,
FOR EXAMPLE, AT P.05 BECAUSE OF
BIOLOGICAL VARIABILITY AND CAN
YOU REALLY MAKE A YES/NO ANSWER
WHEN THERE'S, YOU KNOW, A HUGE
VARIABILITY OF OUTCOMES AND SO
THAT IS AN OPEN QUESTION.
WERE YOU NEXT?
>> YEAH.
SO FISHER WAS -- A LOT OF THE
MODERN STATISTICS CAME OUT OF
GENETICS LIKE FISHER, BUT I TO
AGREE WITH THIS POINT FROM A
PRACTICAL PERSPECTIVE, I'M
PAUL -- CAL TECH.
THERE'S PRACTICAL REASONS TO
REPORT EVEN STUDIES THAT ARE NOT
COMING TO APPROPRIATE SUPPORTIVE
HYPOTHESIS BECAUSE THE
TECHNOLOGY AND INTERPRETATION
CHANGES.
FOR EXAMPLE, IF YOU'RE USING
PRE-CLINICAL STUDY WITH
BIOMARKERS, THE MEANING OF THOSE
BIOMARKERS OFTEN CHANGES BECAUSE
WE THINK WE'RE MEASURING
SOMETHING BUT WE'RE NOT ACTUALLY
DOING THAT.
ANOTHER MORE POINTIAN EXAMPLE,
THERE'S THOUSANDS OR TENS OF
THOUSANDS OF STUDIES THAT USE
GENE ONTOLOGY TO INTERPRET THEM
AND AS A PI OF THE GENE ONTOLOGY
CONSORTIUM I CAN TELL YOU THE
ONTOLOGY CHANGES SO THE ACTUAL
WAY THAT WE'RE DESCRIBING THINGS
CHANGES IN TIME AND IF YOU
REANALYZE DATA FROM 10 YEARS AGO
THAT WASN'T SIGNIFICANT OR
MEANINGFUL, IT NOW OFTEN BECOMES
MEANINGFUL OR VICE VERSA SO YOU
WANT TO DO THAT AND REPORT THOSE
AND AGAIN TO THE POINT OF
STRATIFICATION AS ANOTHER
EXAMPLE, YOU DON'T KNOW WHAT THE
STRATIFICATION IS AT THIS POINT,
BUT YOU MIGHT LATER ON.
AND IN THE LONG TERM, THE
JOURNAL IMPACT FACTOR IS LESS
IMPORTANT THAN WORKING IT UP IN
A DATABASE EVEN IN PUBMED OR A
KNOWLEDGE BASE.
>> I HAVE A VERY SHORT POINT.
THERE WERE SOME REALLY
INTERESTING POINTS, I THINK
RACHEL MADE THIS PINTPOINT AND
SOME OTHERS ABOUT DOGMA AND
STATUS.
I DON'T KNOW IF FOLKS ARE
FAMILIAR, IF SAYING SIMILAR
HAPPENING IN BIOMEDICINE BUT IN
THE SOCIAL SCIENCES THERE'S THIS
EFFORT OF GATHERING PREDICTIONS
AND FORECASTS, I DON'T KNOW IF
THEY'RE SAYING SIMILAR IN
BIOMEDICINE, BUT IT'S BEEN
PROVED TO BE REALLY USEFUL TO
QUANTIFY PEOPLE'S EXPECTATIONS
IN THE FIELD ABOUT WHAT THE
EFFECT OF A CERTAIN TREATMENT
WILL BE AND THEN YOU COMPARE
THAT TO THE ESTIMATE.
SO WE'RE NOT ALWAYS COMPARING
AGAINST THE NULL ANYMORE, WE'RE
COMPARING TO THE PRIOR JUST TO
GET TO THE KIND OF BAYESIAN
POINT, AND IT GETS YOU TO THE
STRENGTH AND UNIFORMITY OF
PEOPLE'S DOGMA, LIKE WHAT WAS
THE EXPECTATION OF EFFECT OF A
CERTAIN TREAT M.
SO THERE'S SOMETHING CALLED
SOCIAL SCIENCE PREDICTION
PLATFORM, SSPP AND SOME OTHER
EFFORTS TO TRY TO CROWD SOURCE
OR GET EXPERT PREDICTIONS.
I DON'T KNOW AS MUCH ABOUT
BIOMEDICINE BUT AT LEAST OUR
FIELD, THAT'S NOW.
>> ONE ANSWER HERE, IT'S THE
STOCK MARKET.
>> THANKS, SHAI.
YEAH, SO THAT ACTUALLY BRINGS US
UP TO -- UNLESS YOU HAD ONE
QUICK COMMENT YOU'D LIKE TO
MAKE?
>> I HAVE A QUESTION.
>> GO FOR IT, GO FOR IT.
>> OH, YES.
I JUST KIND OF HAVE MORE OF LIKE
A METHODOLOGICAL QUESTION.
MY NAME IS CHRISTINE LU, POSTDOC
AT UCSF.
I WAS A BIT MORE CURIOUS ABOUT
HOW YOU ACTUALLY QUANTIFY THE
PUBLICATION BUY A ESPECIALLY
WITH THIS RELATIONSHIP TO RIGOR
AND DESIGNING A GOOD EXPERIMENT,
YOU WANT PERHAPS A POSITIVE
CONTROL IN WHICH YOU MIGHT
EXPECT SORT OF STATISTICAL
SIGNIFICANCE.
SO JUST TO UNDERSTAND A LITTLE
BIT MORE ABOUT WHETHER NULL
RESULTS OR NEGATIVE RESULTS OR
REJECTING THE NULL HYPOTHESIS,
LIKE HOW DOES THIS PLAY INTO
WHAT YOU GUYS HAVE PUBLISHED,
HOW YOU TAKE INTO ACCOUNT THIS
RELATIONSHIP BETWEEN MAYBE
PEOPLE DESIGNING EXPERIMENTS
THAT WILL HAVE THE LITTLE
ASTERISK BUT THAT THEN IS IN
FAVORITE OF NULL HYPOTHESIS,
WHERE DOES THIS ALL FIT INTO
THIS FRAMEWORK OF PUBLICATION
BIAS?
>> I MEAN, MAYBE PEOPLE HAVE
DIFFERENT VIEWS ON THIS, BUT MY
VIEW OF WHAT PUBLICATION BIAS IS
IS IT'S THE CAUSAL EFFECT OF THE
SIGNIFICANCE OF THE RESULT ON
PUBLICATION, OKAY?
AND MY VIEW IS THAT'S A VERY
HARD THING TO ACTUALLY QUANTIFY,
BECAUSE WE DON'T HAVE RANDOM
ASSIGNMENT OF THE SIGNIFICANCE
OF THE EFFECT.
SHAI PRESENTED A STUDY WHICH I
THINK WOULD BE THE CLOSEST STUDY
THAT YOU WOULD WANT ON THAT,
WHICH IS YOU HAVE THE SAME PAPER
AND YOU KIND OF LIKE FAKE OR ARE
A LITTLE DECEPTIVE ABOUT WHAT
THE RESULTS ARE, BUT YOU KEEP
EVERYTHING ELSE IN THE STUDY THE
SAME.
YOU SEND IT OUT TO THE COMMUNITY
AND SEE HOW THE REVIEWERS REACT,
AND THAT STUDY SHOWED ABOUT A 20
PERCENTAGE POINT EFFECT, 17
PERCENTAGE POINT.
SO THE QUESTION IS, WITHOUT A
POTENTIALLY DECEPTIVE STUDY OR
WHERE YOU HAVE THE CONFEDERATES
BEING THE JOURNAL EDITORS, HOW
DO YOU TAKE THE BROAD SAMPLE OF
STUDIES IN ORDER TO USE THAT
RESEARCH DESIGN, THE LOGIC OF
IT, TO GET WHAT THE CAUSAL
EFFECT OF THE SIZE OF THE EFFECT
WITH THE SIGNIFICANCE IS.
THAT'S KIND OF MY VIEW OF WHAT
IT IS.
>> I WOULD ALSO LIKE TO ADD TO
THAT ON THE DIFFICULTY OF
QUANTIFYING PUBLICATION BIAS.
IT CAN'T SIMPLY BE THROUGH THE
NUMBER OF NON-SIGNIFICANT
STUDIES THAT ARE PUBLISHED,
BECAUSE SIGNIFICANCE, YOU CAN
REACH STATISTICAL SIGNIFICANCE
BY HAVING A LARGE ENOUGH STUDY
OR BY THE TRUE EFFECTUALLY BEING
LARGENESS.
IF YOU HAVE A TRUE EFFECT SIZE
THAT IS LARGE, YOU KNOW, LESS
THAN 5% OF THE TIME YOU PROBABLY
WILL GET A NULL RESULT.
OR A NON-SIGNIFICANT RESULT BY
DEFINITION.
AND SO IT WOULD BE EXPECTED THAT
FOR THAT QUESTION IN THAT AREA
FOR A CERTAIN TYPE OF STUDY,
YOU'RE GOING TO HAVE VERY, VERY
FEW NON-SIGNIFICANT RESULTS OUT
THERE.
THAT'S NOT PUBLICATION BIAS,
THAT'S JUST THE TRUE EFFECTUALLY
BEING LARGE.
SO IT IS VERY DIFFICULT TO
QUANTIFY AS HE SAID, AND I THINK
A KEY POINT HE MADE ABOUT STUDY
QUALITY, PERHAPS I WOULDN'T SAY
STUDY QUALITY, MAYBE STUDY
DESIGN AND POPULATION.
I THINK YOU NEED TO REALLY TRY
TO COMPARE APPLES TO APPLES TO
GET AN IDEA OF PUBLICATION BIAS,
COMPARING STUDIES THAT ARE
FUNDAMENTALLY DIFFERENT IN THEIR
SAMPLE OR DESIGN, IT'S NOT A
TRUE ACCURATE COMPARISON TO SAY
THAT ONE IS PREFERENTIALLY
PUBLISHED OVER A 'NOTHER BASED
ON THE SIGNIFICANCE.
>> WELL, THANK YOU VERY MUCH TO
OUR PANELISTS.
THIS WAS A GREAT OPENING SESSION
FOR -- AND THIS BRINGS US TO
LUNCHTIME, AND SO THOSE OF YOU
WHO ORDERED BOX LUNCHES, THEY
WILL BE AVAILABLE DOWN THE HALL
BY THE KITCHENETTE WHERE THE
COFFEE WAS.
AND WE WILL MEET BACK HERE AND
START AGAIN AT 1:00.
THANK YOU.
>> SESSION 2 IS ABOUT REPORTING
NULL STUDIES AND BARRIERS WE MAY
FACE, NOT ONLY IN THE PUBLISHING
WORLD BUT ALSO AS SCIENTISTS.
I'M CRISTTOPHE BERNARD,
NEUROSCIENTIST WORKING IN THE
FRENCH NIH, ALSO THE FUNDER AND
EDITOR IN CHIEF OF NEURO.
WE'LL GIVE YOU A FEW WORDS ABOUT
IT, I INTRODUCED FROM THE
BEGINNING THE POSSIBILITY TO
PUBLISH NEGATIVE RESULTS,
BECAUSE THINKING IT WAS VERY
IMPORTANT FOR THE COMMUNITY TO
HAVE ACCESS TO THAT KIND OF
KNOWLEDGE IN ORDER NOT TO
REPRODUCE THE SAME EXPERIMENTS
LEADING TO NEGATIVE RESULTS.
IT'S BEEN TOUGH AT THE BEGINNING
TO HAVE PEOPLE SUBMIT TO THE
JOURNAL FOR THAT BECAUSE THEY
ARE STILL UNAWARE OF THIS
POSSIBILITY.
I THINK IT'S A KIND OF CRIME
AGAINST THE POSTDOCS AND Ph.D.
STUDENTS WHO DID THE WORK, AND
ALL OF YOU HAD WITH TWO OR THREE
YEARS OF WORK DON'T GET
PUBLISHED, THE WORK IS NOT
ACKNOWLEDGED BECAUSE NOBODY
WANTS TO PUBLIC THESE THINGS.
SO, HAVING THE POSSIBILITY TO
PUBLISH NEGATIVE RESULTS IS VERY
IMPORTANT, YOU'LL HEAR ALSO
EXAMPLES FROM EXPERIMENTAL
NEUROLOGY AND PLOS ONE, WE'LL
INTRODUCE THESE POSSIBILITIES,
THERE'S I THINK A BARRIER AND WE
CAN DISCUSS THAT AMONG THE
SCIENTISTS NOT TO PUBLISH NULL
RESULTS, THERE MAY BE REASONS
FOR THAT.
AND WE'D LIKE TO KNOW HOW TO
INCREASE THE NUMBER OF
SUBMISSIONS.
THE OTHER THING THAT IS ALSO
IMPORTANT FOR NEGATIVE NULL
STUDIES OR NEGATIVE STUDIES IS
THAT IT'S REALLY INSTRUMENTAL I
THINK FOR PEOPLE TO KNOW ABOUT
THAT.
AND WE HEARD THIS MORNING THAT
WHO IS GOING TO ACKNOWLEDGE
THAT, RIGHT?
SO HOW DO YOU VALUE THIS WORK
THAT HAS BEEN DONE AND
SUPPOSEDLY DOESN'T MAKE SCIENCE
PROGRESS BUT SCIENCE ALSO
PROGRESSES WITH ERRORS, DOESN'T
ONLY PROGRESS WITH POSITIVE
RESULTS.
IF YOU LOOK AT THE HISTORY OF
SCIENCE OUTSIDE BIOLOGY IT HAS
PROGRESSED WITH ERRORS, NOT ONLY
WITH MAJOR DISCOVERIES.
OKAY.
THERE IS ANOTHER THING THAT
PREVENTS THE PUBLICATION OF NULL
STUDIES, TAKES SOME GUTS FOR A
JOURNAL TO PUBLISH BECAUSE THESE
STUDIES ARE USUALLY NOT CITED.
IF YOU'RE NOT CITED, YOUR IMPACT
FACTOR IS GOING DOWN, MOST
PEOPLE PLAY THE GAME, YOU WON'T
GET PAPERS SUBMITTED TO YOUR
JOURNAL.
SO, YES, I INSTRUCT THE
REVIEWERS AT NEURO, THAT WE'RE
HERE TO SERVE THE COMMUNITY.
IF A NULL RESULT IS IMPORTANT
FOR THE COMMUNITY IT MUST BE
PUBLISHED, NO MATTER WHAT.
OUR IMPACT FACTOR WILL DECREASE,
I DON'T CARE.
I DON'T BELIEVE IN IMPACT
FACTOR.
I DON'T MENTION IT ON THE
WEBSITE.
YOU CAN FIND IT IF YOU WANT,
IT'S THERE, BUT IT'S NOT
ADVERTISED BECAUSE IT'S A WRONG
METRIC ANYWAY.
AND I WOULD LIKE TO HEAR ABOUT
EXPERIMENTAL NEUROLOGY IN PLOS
ONE BECAUSE THEY DO THE SAME
THING.
WE ACCEPT IT.
YOU HAVE TO REALIZE THAT FOR A
JOURNAL, THIS IS BAD THING.
BUT IF YOUR GOAL IS TO SERVE THE
COMMUNITY, THEN YOU'RE DOING IT.
OKAY.
SO, THIS IS WHAT WE'RE DOING, IN
EUROPE WE HAVE ALSO REGISTRY
REPORTS, CONFIRMATION STUDIES,
WE ALSO PUBLISH CONFIRMATION
STUDIES, ALL THESE THINGS, THEY
ARE NOT CITED, RIGHT?
ANYWAY, THE ORDER FIRST WE'LL
HEAR FROM CHRIS, WHO INTRODUCED
THE REGISTRY REPORT BUT IS GOING
TO SHOW US SOME INITIATIVES, AND
NOVEL TOOLS THAT WOULD BE
HELPFUL FOR THE COMMUNITY.
THEN EMILY IS EDITOR IN CHIEF OF
PLOS ONE, WILL COVER ALSO WHAT
THEY DO AT THE JOURNAL.
WE DISCUSSED ALREADY AND
BASICALLY SHARE THE SAME
PHILOSOPHY.
KIF WILL PRESENT A CASE STUDY
BASED ON HIS OWN EXPERIENCE WITH
NEGATIVE RESULTS.
AND I THINK THAT YOU WILL SEE TO
GET THE POSITION, PUBLISHING A
NEGATIVE STUDY.
SO IT'S LIKE HELPFUL
REALIZATION.
AND THEN JOSE WHO IS GOING TO
PRESENT EXPERIMENTAL NEUROLOGY
WHERE THEY DO THAT KIND OF THING
IS GOING TO TALK MORE ABOUT THE
SAME TOPIC.
AND FINALLY LAURA FROM TEXAS
AMU, WHICH IS MEXICO ANYWAY,
WILL TALK ALSO ABOUT REAL LIFE
EXPERIENCE AND PUBLICATION BIAS
AND THE DIFFICULTY OF BEING A
NEUROSCIENTIST IN THE PRESENT W.
FIRST, CHRIS.
>> THANK YOU.
THIS IS THE ONLY SLIDE I NEED,
IF YOU WANT TO FIX PUBLICATION
BIAS USE THESE.
WE CAN TALK AROUND THE ISSUE.
IT WAS MENTIONED IN THE PREVIOUS
TALK WE HAVE MEETINGS EVERY FIVE
YEARS AND NOT MUCH CHANGES.
IF YOU WANT TO FIX PUBLICATION
BIAS, IF YOU WANT TO SEE MORE
NULL RESULTS IN LITERATURE, STOP
DECIDING WHAT GETS INTO
LITERATURE BASED ON RESULTS.
THE ONLY WAY TO DO THAT AT THE
MOMENT IS TO USE REGISTERED
REPORTS.
I DON'T NEED TO INTRODUCE THIS
OBVIOUSLY TO THIS AUDIENCE, BUT
THE TWO KEY INGREDIENTS OF
REGISTERED REPORT ARE PRE-STUDY
EVALUATION BEFORE RESULTS ARE
KNOWN AND THEN IN PRINCIPLE
ACCEPTANCE REGARDLESS OF
OUTCOME.
IF YOU DO THIS, AS WE'VE DONE,
WE CREATED REGISTERED REPORTS
ABOUT TEN YEARS AGO, AND WE'VE
SEEN IT INCREASE TO 350
JOURNALS, WE'VE SEEN GOOD
IMPACT, REGISTERED REPORTS ARE
POSSIBLE WITH CAREER
RESEARCHERS, IMPORTANT FOR OPEN
SCIENCE INITIATIVE TO BE POPULAR
WITH THE RESEARCHERS WHO ARE
DOING THE RESEARCH MOST OF THE
TIME, WHICH IS GOOD NEWS, WE'RE
FINDING THAT REGISTERED REPORTS
ARE MUCH MORE LIKELY TO
DISCONFIRM PREDICTIONS, YOU'VE
GOT A 5 TO 10 INCREASE CHANCE OF
DISCONFIRMING YOUR HYPOTHESIS IF
YOU DO A REGISTERED REPORT
COMPARED TO A REGULAR ARTICLE.
WE'RE FINDING THEY ARE MORE
COMPUTATIONALLY REPRODUCIBLE
THAN REGULAR ARTICLES, THEY ARE
RATED HIGHER IN QUALITY BY
EXPERTS IN THE FIELD COMPARED TO
REGULAR ARTICLES.
AND ALSO TO TOUCH ON YOUR POINT
ABOUT CITATION, FOR REGULAR
RESEARCH ARTICLES STUDIES
REPORTING NEGATIVE RESULTS ARE
CITED LESS THAN THOSE REPORTING
POSITIVE RESULTS.
WE FIND THAT TRENDS DISAPPEARS
INTERESTINGLY FOR REGISTERED
REPORTS.
THEY ARE CITED ON AVERAGE THE
SAME AS REGULAR ARTICLES IN THE
LITERATURE.
SO THAT ALL LOOKS GREAT.
IT'S GREAT, GOOD NEWS,
REGISTERED REPORTS SEEM TO BE
WORKING AS INTENDED.
WHAT I WANT TO TALK ABOUT IS THE
FUTURE, JUST FOR A COUPLE
MINUTES I WANT TO FLOAT TWO VERY
IMPORTANT CURRENT AND EMERGING
FUTURE STEPS.
THE FIRST OF THESE IS COMMUNITY
GOVERNED PRE-PRINT REVIEW WHICH
I THINK WILL BE THE FUTURE
BECAUSE OF EFFICIENCY SAVINGS.
WE'RE REGISTERED PEER COMMUNITY
AND REGISTERED REPORTS A FEW
YEARS AGO IN WHICH IT'S A FREE
NON-COMMERCIAL PLATFORM THAT
MANAGES THE PEER REVIEW OF STAGE
1 AND STAGE 2 REGISTERED REPORT
PRE-PRINTS.
BASED UPON EVALUATIONS OF THE
PRE-PRINTS WHICH ARE MADE
PUBLIC, WE HAVE A FLEET OF PCI
FRIENDLY JOURNALS WHICH COMMIT
TO ACCEPTING THOSE
RECOMMENDATIONS WITHOUT FURTHER
PEER REVIEW, WHICH MAKES -- IT'S
A CLEARINGHOUSE FOR CENTRALIZING
THE PEER REVIEW OF REGISTERED
REPORTS, CONVERGES REGISTERED
REPORTS EXPERTISE IN ONE PLACE,
SUBMISSIONS ACROSS THE WHOLE OF
SCIENCES, NOT JUST NEUROSCIENCE
BUT IN THE HUMANITIES AND SOCIAL
SCIENCES AND PHYSICAL SCIENCES.
AND IT PLACES CONTROL OF PEER
REVIEW AND DECISION MAKING IN
THE HANDS OF THE SCIENTIFIC
COMMUNITY RATHER THAN
PUBLISHERS, WHICH I THINK IS A
VERY IMPORTANT STEP.
IT ALSO TO LEAD TO MY NEXT POINT
MAKES IT EASIER FOR REGISTERED
REPORTS TO WORK IN TANDEM WITH
PUBLIC FUNDERS.
PUBLIC FUNDERS OFTEN HAVE LEGAL
RESTRICTIONS ON WORKING TOO
CLOSELY WITH PARTICULARLY
PROFIT-MAKING PUBLISHERS.
ON REGISTERED REPORTS
PARTNERSHIPS.
THIS IS ANOTHER VERY CRUCIAL
FUTURE STEP.
SO ONE OF THE CRITICISMS OF THE
REGISTERED REPORTS INITIATIVE IT
DOUBLES UP ON PRE-STUDY
EVALUATION.
WE ALREADY HAVE A LOT OF PHASES
OF PRE-STUDY REVIEW THAT WE GO
TO WHEN WE'RE APPLYING FOR A
GRANT GETTING ETHICS OR
REGULATORY APPROVAL, WE
INTEGRATE STAGE 1 PEER REVIEW
WITH GRANT REVIEW AND PERFORM AT
THE SAME TIME, A PROVEN MODEL
THAT'S WORKED BETWEEN PLOS ONE
AND THE FOUNDATION THAT EMILY
WILL TALK ABOUT AN EXAMPLE WHEN
SHE HAS HER WORDS SHORTLY.
I WANT TO HIGHLIGHT VERY
IMPORTANT FUTURE STEP.
WE NEED FUNDERS LIKE THE NIH TO
BE OFFERING REGISTERED REPORTS
GRANTS AS A STANDARD TRACK THAT
RESEARCHERS CAN USE AND WE'LL
SEE MASSIVE UPTAKE.
THANK YOU.
>> I DON'T HAVE SLIDES AND I'VE
HAD ENOUGH CAFFEINE TO SPEAK AS
TWICE SPY.
I'M THE EDITOR IN CHIEF OF THE
JOURNAL PLOS ONE, HEAD OF
EDITORIAL BOARD SERVICES AT
PLOS, I HAVE ULTIMATE
RESPONSIBILITY FOR OUR BOARD OF
ABOUT 10,000 ACADEMIC EDITORS
AND MAKING SURE WE'RE MAKING
CONSISTENT DECISIONS ACROSS THE
PORTFOLIO AND EVERYONE IS
ADHERING TO PUBLICATION
CRITERIA.
TO PICK UP ON THE POINT CHRIS
MADE ABOUT REGISTERED REPORTS
ONE OF THE THINGS THAT WE'RE
TRYING AT PLOS IS WAYS TO MAKE
THE PUBLICATION PROCESS EASIER
FOR AUTHORS AND MORE
TRANSPARENT.
WE WORK IN CONCERT WITH FUNDING
AGENCIES, ONE WE'VE DONE FOR
YEARS, IN THE U.S., ONE WE'VE
MORE RECENTLY STARTED WITH,
CANCER RESEARCH U.K., BOTH FUND
CANCER RESEARCH.
THERE ARE HURDLES TO AUTHORS, IT
TAKES MORE TIME, THERE'S A DELAY
BETWEEN THE TIME THE GRANT IS
ACCEPTED AND THEN START
RECRUITING PARTICIPANTS BECAUSE
THEY HAVE TO GET THEIR PROTOCOL
PEER REVIEWED AND PUBLISHED.
WE'RE WORKING WITH THE COMMUNITY
TO LOWER THE BARRIERS, TO GET
ADDITIONAL FUNDERS TO SIGN UP,
TO NORMALIZE THIS BECAUSE I
THINK THE END RESULT IS ONE
THAT'S MORE STRAIGHTFORWARD, A
CLEAR PATH FROM SUBMITTING
FUNDING PROPOSAL TO HAVING
RESEARCH FUNDED TO HAVING REVIEW
OF THE PROJECT, TO COMPLETING
THE STUDY TO THEN PUBLISH IN THE
RESULTS, IN A VERY TRANSPARENT
AND OPEN WAY.
I THINK THIS IS A REALLY
EXCITING OPPORTUNITY FOR FURTHER
INVESTIGATION, FURTHER
EXPERIMENTATION.
BUT I ALSO WANTED TO TALK ABOUT
THE GOAL OF THE SESSION.
WHAT ARE BARRIERS AND HOW DO WE
ENCOURAGE DISSEMINATION OF NULL
AND NEGATIVE RESULTS?
ONE IS AROUND PERCEPTION.
THERE IS THIS OVERARCHING
CONSIDERATION IN THE COMMUNITY
THAT A NULL OR NEGATIVE RESULTS
SOMEHOW MEANS LOWER QUALITY BUT
IF YOU'VE DONE THE STUDY
PROPERLY YOU WOULD HAVE SEEN
POSITIVE ASSOCIATION, CONFIRMED
YOUR HYPOTHESIS.
IT'S ALSO PROBLEMATIC WHEN WE
HAVE AUTHORS WHO SET OUT TO
CONFIRM A HYPOTHESIS RATHER THAN
TEST A HIGH POTENTIAL APPROACHES
SITS, PRE-REGISTRATION IS A WAY
AROUND THIS.
I WANT TO BE CLEAR THAT ONE OF
THE PROBLEMS WITH NULL AND
NEGATIVE RESULTS IS PEOPLE LIKE
ME, JOURNAL EDITORS.
SO WHEN WE'RE SELECTING FOR
STUDIES WITH AN EXCITING OR NEW
OR IMPORTANT RESULT WE'RE
EXHIBITING TO THE PROBLEM.
I WAS AN EDITOR FOR NATURE CELL
BIOLOGY AND I CONTRIBUTED TO THE
PROBLEM BY BEING PART OF THE
SELECTED JOURNALS LOOKING FOR
NEW INTERESTING NOVEL RESULTS
HIGHLY CITED.
BUT NOW IN MY CURRENT POSITION,
WE DON'T CARE ABOUT THE IMPACT
FACTOR.
WE TAKE ALL RESEARCH THAT IS
PERFORMED TO HIGH TECHNICAL AND
ETHICAL STANDARDS AND WE'LL
PUBLISH IT REGARDLESS OF THE
OUTCOME.
THAT'S A REALLY WONDERFUL
POSITION TO BE IN.
EDITORS WITH DO A LOT TO HELP
THE COMMUNITY ADDRESSING THE
PROBLEM BY ENSURING WE'RE
EVALUATING RESEARCH ON INTEREST
OF THE QUESTION, RIGHT?
HOW INTERESTING IS THAT
QUESTION, NOW HOW INTERESTING
ARE THE DATA.
THAT'S A FUNDAMENTAL CHANGE IN
TERMS OF HOW WE'RE EVALUATING
RESEARCH.
I THINK THE JOURNALS NEED TO
ENSURE THEY ARE SELECTING FOR
RIGOR, SO TECHNICAL AND ETHICAL
RIGOR.
THESE IDEAS OF NEW WAYS OF
WORKING WITH OUR COMMUNITIES
WITH PARTNERSHIPS ARE ONE,
ADDITIONAL INITIATIVES IN THE
COMING YEARS TO MAKE SURE WE'RE
MAKING IT EASY FOR AUTHORS, THE
DEFAULT OPTION, THE EASY OPTION
IS TO BE OPEN, TO BE
TRANSPARENT, IS TO PUBLISH THE
FULL SET OF DATA REGARDLESS OF
THE DIRECTIONALITY OF THE
OUTCOME.
I LOOK FORWARD TO DISCUSSING
THIS IN MORE DETAIL.
THANKS.
>> WONDERFUL TO BE PART OF THIS
PANEL.
MY NAME IS KIF LIAKETH-ALI, JUST
COMPLETE MY POSTDOC AT STANFORD
UNIVERSITY.
I STARTED MY NEW JOB AS
LECTURER, ASSISTANT PROFESSOR,
UNIVERSITY OF SOUTHAMPTON THREE
WEEKS AGO.
SLIGHTLY DIFFERENT COMPONENT
WANTED TO SHARE MY EXPERIENCE
PUBLISHING NEGATIVE RESULTS AS
SOMEONE DURING EARLY IN THEIR
CAREER IN MOLECULAR
NEUROSCIENCE.
SO, WE AT THE LAB STUDY HOW
NEURONS COMMUNICATE, A
FUNDAMENTAL UNIT OF INFORMATION
PROCESSING IN THE BRAIN.
WE PARTICULARLY ARE INTERESTED
IN HOW LIGANDS COMMUNICATE WITH
EACH OTHER, THE TWO IMPORTANT
PROTEINS IN THE SYNAPSE THAT
ALLOWS US, YOU CAN NAME ANY
FUNCTION, DAY-TO-DAY FUNCTIONS,
THAT IS IMPORTANT AND THESE
PROTEINS ARE KNOWN TO HAVE
MUTATED, SO THIS IS INTERESTING
QUESTION TO ASK HOW THEY
FUNDAMENTALLY FUNCTION IN THE
SYNAPSES.
WHEN I STARTED MY POSTDOC SIX
YEARS AGO, BEFORE I GO TO THE
ACTUAL CASE STUDY I WANT TO
BRIEFLY INTRODUCE THE ACTUAL
ASPECT OF HOW, YOU KNOW,
ALTERNATIVE SPLICING WHICH HAS
BEEN THE FOCUS OF OUR QUESTION,
SO IT CAN -- IT'S A VERY
FUNDAMENTAL MECHANISM BY WHICH
GENES CAN CHANGE THE OUTCOME OF
FUNCTION, SO IN A SENTENCE HOW
TO CREATE A DIFFERENT MEANING BY
SCRAMBLING THE WORDS.
SIMILARLY, THE GENES CAN CHANGE
THEIR FUNCTION BASED ON WHAT
SEQUENCE THEY INSERT.
NOW, THIS VERY SPECIFIC
MECHANISM, VERY FUNDAMENTAL IN
EUKARYOTE.
THE ACTIVITY, THE NEURONAL
INDUCED ACTIVITY SPLICING HAS
WIDE IMPLICATIONS NOT ONLY TO
FUNDAMENTAL BIOLOGY, ALSO
VARIOUS DISEASES.
WHEN I STARTED POSTDOC POSITION
IN THE LAB, THIS PAPER PUBLISHED
IN MARCH 2017 SHOWING REALLY,
REALLY INTERESTING RESULTS WHERE
YOU CAN CHANGE THE GENE PROTEIN
COMPONENT BY PUTTING THE MOUSE
IN A BEHAVIORAL PARADIGM OR ANY
BEHAVIORAL.
JUST INDUCING THE NEURONAL
ACTIVITY.
NOW, I WAS SO FASCINATED BECAUSE
I REALLY WANTED TO CONTRIBUTE A
VERY KEY DISCOVERY TO THE
COMMUNITY, SO THAT'S THE REASON
I STARTED POSTDOC IN THIS LAB.
AND I STARTED THIS POSTDOC WITH
THE TWO-YEAR LIMITED FELLOWSHIP
FROM EUROPEAN MOLECULAR BIOLOGY
ORGANIZATION WHICH MEANS MY TIME
WAS LIMITED.
HAVING A VERY AMBITIOUS PLANS,
ONLY TO PUBLIC THIS NEGATIVE
RESULT, THE TITLE SOUNDS LIKE
REVIEW BUT IT'S EXPERIMENTAL
PAPER WE PROVE IT IS WRONG, NOT
ENTIRELY WRONG BUT MAJOR
COMPONENT THEY USE, EXPERIMENTAL
BASIC COMPONENT IS FLAWED.
NOW, WHAT WE FOUND IS BASICALLY
A REPEAT FUNDAMENTAL PARADIGM
USED IN MANY NEUROSCIENCE LABS
ESPECIALLY AT THE CELL BIOLOGY
COMPONENT, TO MANIPULATE NEURAL
ACTIVITY EXPOSING TO CONDITIONS,
BASICALLY HAMMERING THE CELLS,
WHEN YOU INDUCE ACTIVITY
CREATING EPILEPSY, CELLS DIE,
NOT ALL CELLS, IT INDUCES
SPECIFIC POPULATIONS THAT LEADS
TO CELL DEATH.
IN A PHYSIOLOGY CONDITION YOU
HAVE NEURONS, TWO MAJOR CELL
POPULATIONS IN THE BRAIN, WHEN
YOU INDUCE CONDITIONS THAT LEADS
TO NEURONAL CELL DEATH,
ASTROCYTES PRESERVED, LEADS TO
SKEWING OF RESULTS ESPECIALLY
MOLECULAR FOCUSED, DIFFERENT
ALTERNATE SPLICING PATTERNS.
YOU COULD BUILD A STORY OUT OF
IT.
LIMITATIONS CREATIVE ENOUGH, YOU
CAN BUILD A STORY THAT'S
COMPLETELY FLAWED.
TO SUMMARIZE WHAT WE'VE FOUND IN
THIS CASE STUDY, IT'S VERY
SPECIFIC TO COMMUNITY, MOSTLY
WORKING ON THIS EXPERIMENTAL
PARADIGMS.
SO, OUR STUDY OR PAPER
UNDERSCORED VULNERABILITY OF
COMMON PROCEDURES, FUNDAMENTAL
IN VITRO TECHNIQUE, AND ALSO
EMPHASIZED PREVALENCE OF
EXPERIMENTAL OPTI-PLEX.
THERE'S TONS OF DATA, OTHER WORK
WAS VALIDATED BY OTHER GROUPS.
SO, FINALLY, IT ALSO UNDERSCORES
THE VALUE OF PUBLISHING NEGATIVE
RESULTS TO REFINE EXPERIMENTAL
PROTOCOL TO ADVANCE SCIENCE.
WHEN WE PUBLISHED SEVERAL MONTHS
LATER, WE GOT A HEARTENING IMAGE
FROM SOMEBODY WE DIDN'T
COLLABORATE WITH OR KNOW, SAYING
THAT THEIR GROUP ALSO HAVE SEEN
EXACTLY THE SAME RESULTS, BUT
THEY FAILED TO PUBLISH IT
BECAUSE THEY WERE TRYING TO
CHASE OFF THE ORIGINAL JOURNAL
WHO PUBLISHED THIS PAPER AND
THEN ULTIMATELY REJECTED.
THAT WAS THE BARRIER, BECAUSE WE
DON'T GO THAT ROUTE BECAUSE SOME
WARNED ME THAT ARE YOU READY TO
SPEND ANOTHER TWO YEARS WITH
HUNDREDS OFE MAILS BACK AND
FORTH?
WE JUST WANT THIS PAPER TO BE
OUT.
SO, EVENTUALLY WORKING WITH
GILES, WE MANAGED TO PUBLISH OR
MANAGED TO HELP OR PUBLISH THEIR
PAPER LATER.
NONE OF OUR PUBLICATION PAPERS
NOT CITED THE WAY ORIGINAL
PAPERS STILL BEING CITED TODAY.
A PAPER I'M REVIEWING RIGHT NOW
CITING THE ORIGINAL PAPER
IGNORING COMPLETELY WHAT WE DID
SHOWING NEGATIVE RESULTS, IT'S
COMPLETELY MISSING IMPORTANT
SCIENCE.
EVEN DISHEARTENING, ANY GENE
FUNCTION INVOLVED IN A DISEASE
AND WHERE IT IS LEADING TO.
NOW, WHEN I STARTED POSTDOC AT
STANFORD, I CAME FROM THE U.K.,
WHERE I HAD MY Ph.D., AND MY
Ph.D. SUPERVISOR GAVE ME AN
ADVICE WHICH WAS AN EXPERIMENT,
SHE SAID, IF YOUR EXPERIMENT IS
NOT COMPLETE, NOBODY WILL
PUBLISH, NO MATTER NEGATIVE OR
POSITIVE RESULTS.
I SACRIFICED MY -- I WOULDN'T
SAY SACRIFICED, TOOK ADVENTUROUS
ENCOUNTER, LAST TWO YEARS OF MY
POSTDOC.
THAT ACTUALLY ALMOST KILLED MY
CAREER.
BUT MY RESILIENCE AND SUPPORT
FROM MY POSTDOC ADVISER MADE ME
GO TO THAT.
AND WHAT I WANT TO DISCUSS TODAY
IS WHAT ARE THE ACTUAL BARRIERS
IN PUBLISHING NEGATIVE RESULTS
AS SOMEONE FOR THE VERY EARLY IN
THEIR CAREER, WHAT ARE THE
SUPPORTS IN PLACE?
WE'VE SEEN VARIETY OF SUPPORT IN
SPECIFICALLY TO SPECIFIC TO MY
CASE STUDY FROM P.I., FUNDERS,
WHAT ARE INCENTIVES, I WAS
FORTUNATE TO HAVE SOME FORM OF
RECOGNITION FOR THIS CASE STUDY.
BUT THE QUESTION IS DO ALL
PEOPLE WHO STRIVE TO DO THIS
HAVE THAT SUPPORT?
I'M HAPPY TO DISCUSS THIS IN
THIS PANEL.
THANK YOU.
>> HELLO, EVERYBODY.
I DON'T HAVE SLIDES.
I'M JOSE MERINO, EDITOR IN CHIEF
OF NEUROLOGY, THE JOURNAL OF THE
AMERICAN ACADEMY OF NEUROLOGY,
AND THE FAMILY OF JOURNALS UNDER
INFLAMMATION, ONE ON GENETICS,
CLINICAL PRACTICE, EDUCATION.
TO GIVE A LITTLE BIT OF
BACKGROUND, JUST FOR NEUROLOGY,
WE FOCUS ON CLINICAL RESEARCH,
WE DON'T PUBLISH BASIC STUDIES,
ANIMAL STUDIES, IT'S CLINICAL
RESEARCH ONLY.
WE GOT ABOUT 4,000 SUBMISSIONS A
YEAR AND REJECT HALF OF THEM AND
PUBLISH ABOUT 10% OF THE
SUBMISSIONS THAT WE HAVE, SO
THAT GIVES YOU AN IDEA OF MY
COMMENTS WILL COME FROM THAT
PERSPECTIVE.
IN 2018 WE ACTUALLY STARTED AN
ACTIVE INITIATIVE TO PROMOTE THE
SUBMISSIONS OF WELL CONDUCTED
NEGATIVE, INCONCLUSIVE OR
REPRODUCIBLE REPORTS, INITIALLY
AS A SEPARATE MANUSCRIPT
CATEGORY.
PEOPLE WOULD SUBMIT TO THE NULL
CATEGORY.
WHEN I BECAME EDITOR IN CHIEF I
SAID THAT DOESN'T MAKE SENSE TO
ME BECAUSE WE DON'T WANT TO
SIGNAL A DIFFERENT CATEGORY.
IT'S RESEARCH IS RESEARCH,
WHATEVER IT IS, IT SHOULD BE.
WE DID AWAY WITH THAT MANUSCRIPT
CATEGORY, LABELED THEM AS NULL
RESULTS COLLECTION BUT SUBMITTED
AS REGULAR RESEARCH ARTICLES.
WE DO STATE IN OUR INSTRUCTIONS
FOR AUTHORS IF THE QUESTION IS
IMPORTANT, THE METHODS ARE
SOUND, AND THE REPORTING IS
TRANSPARENT, FULL CONSIDERATION
IS GIVEN TO ALL PAPERS
REGARDLESS OF RESULT.
WE'RE ACTUALLY VERY PROMINENTLY
SIGNALING THE FACT THAT WE'RE
INTERESTED IN NULL STUDIES AND
THAT REPORT SO-CALLED NEGATIVE
REPORTS, THAT'S NOT AN IDEAL
TERM BUT WE'LL LEAVE IT AT THAT.
WE DON'T ACTUALLY TRACK HOW MANY
HAVE BEEN SUBMITTED ABOUT WE DO
KNOW HOW MANY HAVE BEEN
PUBLISHED.
ABOUT 40 OVER THE PAST FIVE
YEARS.
THERE'S BEEN MORE THAT NULL
STUDIES THAT HAVE BEEN
PUBLISHED.
IT'S JUST THEY HAVEN'T BEEN
LABELED AS SUCH.
WE FIND THOSE WE'VE LABELED HAVE
THE SAME NUMBER OF AVERAGE SIGH
CITATIONS, SAME ALT-METRICS,
DELAY ON PICKING UP ON THESE
INSTANCES.
I WANTED TO TALK ABOUT SOME OF
THE BARRIERS WE FOUND, SOME
LESSONS.
WHAT IS A NULL STUDY?
I FIND THAT'S SOMETIMES WE HAVE
DIFFICULTY DEFINING EXACTLY.
WE OFTEN GET PAPERS, WITH A
CLINICAL TRIAL, YOU HAVE A
REGISTERED PRIMARY OUTCOME AND
THEN THIS IS THE PRIMARY
OUTCOME, THERE WAS AN EFFECT,
THERE WAS NO EFFECT.
IT'S DIFFICULT IN OBSERVATIONAL
STUDIES PEOPLE SAY WE'RE LOOKING
AT THE BIOMARKERS FOR A GIVEN
OUTCOME IN PATIENTS WITH
ALZHEIMER'S, WE FOLLOWED THEM,
NO ASSOCIATION WITH SOME OF
THEM, DO YOU CALL THAT NULL OR
NOT NULL STUDY, THOSE ARE THE
ONES WHERE WE GET INTO DIFFICULT
AREA.
THAT LEADS TO ANOTHER THING TO
COMMENT ABOUT.
WHAT I'M FINDING IS AUTHORS HAVE
DIFFICULTY WITH FRAMING STUDIES
AS NULL STUDIES.
SO IN THE EXAMPLE, FOR EXAMPLE,
THE VARIOUS BIOMARKERS
ASSOCIATED WITH OUTCOME, THE
TITLE WILL SAY FACTOR X IS
ASSOCIATED, AND THAT'S WHAT THEY
HIGHLIGHT IN THE CONCLUSION, IN
THE FINDINGS, EVEN THOUGH AT THE
BEGINNING LOOKING AT SIX
FACTORS, FOUND THIS ONE
POSITIVE, ASKING PEOPLE DON'T
ANNOUNCE RESULTS, TELL US WHAT
THE QUESTION WAS AND THE TITLE
AND THEN YOU HAVE TO HIGHLIGHT
THE FACT THAT IT'S NOT.
WE ALSO HAVE THAT ISSUE WITH
TRIALS.
AS I MENTIONED, FOR US IT'S EASY
TO GO TO clinicaltrials.gov OR
ANY OF THE OTHER W.H.O.
REGISTRIES WHICH WE ACCEPT AND
SEE IF WHAT'S BEEN REPORTED IS A
PRIMARY OUTCOME OR NOT, WE'RE
GETTING IN EARLIER PHASE
CLINICAL TRIALS REPORTS WILL
HIGHLIGHT SECONDARY OUTCOMES AND
NOT THE PRIMARY OUTCOME.
WHAT WE TELL AUTHORS, YOU HAVE
TO REFRAME THE PAPER, TELL US
ABOUT 9 PRIMARY OUTCOME.
WE DON'T CARE THE RESULT WAS
NULL BUT YOU'RE REPORTING A
FINDING WHICH WE THINK IS
IMPORTANT BECAUSE WE THINK THE
RESEARCH QUESTION IS IMPORTANT.
THE OTHER ONE HAS TO DO WITH HOW
I BEGAN, I THINK, AS I
MENTIONED, ANNOUNCING THE
RESULTS, HIGHLIGHTING THE OTHER
SECONDARY OUTCOMES.
AND THEN I ALSO FIND THAT
REVIEWERS HAVE A DIFFICULTY WITH
NULL STUDIES.
SO WE OFTEN GET IN THE COMMENTS,
THIS IS A GREAT QUESTION, THIS
IS A GREAT STUDY, IT'S VERY WELL
DONE, BUT WE DON'T THINK IT'S OF
THE CALIBER FOR NEUROLOGY
BECAUSE THE RESULTS ARE
NEGATIVE.
OBVIOUSLY WE'RE NOT ACTING ON
THOSE COMMENTS.
WE ALWAYS SAY OUR ROLE IS NOT TO
ASK HOW MANY POINTS REVIEWERS
GAVE, IF ABOVE A CERTAIN GOAL WE
PUBLISH, IF NOT WE DON'T.
WE TAKE COMMENTS OF REVIEWERS
AND MAKE OUR OWN JUDGMENT BASED
ON THOSE.
SO WE TEND TO IGNORE THAT.
BUT THE POINT IS WE'RE HAVING
DIFFICULTY IDENTIFYING SOME NULL
STUDIES.
TWO, AUTHORS ARE RESISTANT TO
IDEA OF SUBMITTING NULL RESULTS,
REVIEWERS ARE RESISTANT.
MAYBE THAT'S SOMETHING WE CAN
TALK ABOUT IN THIS PANEL
SESSION.
>> HI, I'M LAURA SMITH,
NEUROSCIENCE DEPARTMENT, TEXAS
A & M UNIVERSITY, ASSISTANT
PROFESSOR.
AND SO I BRING A FEW
PERSPECTIVES, I THINK, TO THIS
PANEL.
THE FIRST IS, WELL, LET ME TELL
YOU ABOUT MY BACKGROUND.
I COME FROM A PSYCHOLOGY
DEPARTMENT, GRADUATE TRAINING
WAS IN BIOPSYCHOLOGY, BUT I WAS
DEEPLY STEEPED IN THE ETHOS OF
PROPER SCIENCE.
PROPER RESEARCH METHODS,
STATISTICAL METHODS, ET CETERA.
AS I WENT OUT INTO THE WORLD AND
TO REAL SCIENCE, WHAT I FOUND
WAS THAT THE THINGS I HAD
LEARNED, THE IDEALS I HAD
LEARNED ABOUT SCIENCE, WEREN'T
FITTING WHAT I WAS SEEING, WHAT
I WAS BEING TRAINED TO BE IN THE
REAL WORLD OF SCIENCE.
SO, HOW YOU BE A SUCCESSFUL
SCIENTIST WAS NOT LINING UP WITH
HOW YOU SHOULD BE A SCIENTIST.
SO, THAT CREATED A LOT OF
DISSONANCE FOR ME.
PERSPECTIVES I BRING, FIRST IN
THE FIELD I'M IN NOW, FRAGILE X
SYNDROME, WHICH IS A SMALL
CONDITION, A CONDITION THAT
AFFECTS A SMALL NUMBER OF
PEOPLE, A SMALL COMMUNITY OF
RESEARCHERS, BUT TIGHTLY KNIT
AND WITH SOME VERY INFLUENTIAL
LABS INVOLVED.
AND THIS GAVE ME AN INTERESTING
PERSPECTIVE BECAUSE WE HAD A
RECENT FAILURE OF CLINICAL
TRIALS, THAT WERE BASED ON VERY
STRONG PRE-CLINICAL WORK.
AND WE'RE A SMALL COMMUNITY BUT
SHOOK OUR COMMUNITY.
WE HAD FAITH IN THESE STUDIES.
WHEN THEY DIDN'T WORK OUT THIS
WAS A HUGE DISAPPOINTMENT, CLOSE
KNIT WITH PARENTS OF KIDS WHO
HAVE FRAGILE X, ET CETERA.
SO THIS REALLY HIT HOME.
IT ALSO MADE ME REFLECT ON THE
DISSONANCE THAT I HAD BEEN
FEELING ABOUT HOW TO BE A
SCIENTIST.
I WAS ASKED TO DO -- BECAUSE I'M
IN THIS FIELD WAS ASKED TO DO A
COMMENTARY A PUBLICATION BIAS.
I DID THAT.
AND I THINK THAT'S ONE OF THE
REASONS I'M HERE TODAY AT THIS
PANEL.
THE OTHER PERSPECTIVE I BRING IS
BEING AN ASSISTANT PROFESSOR
TRYING TO SURVIVE IN THE WORLD
OF RESEARCH THAT WE HAVE
CREATED.
SO, I DO PRE-CLINICAL WORK.
IF ANYONE FEELS LIKE THEY WANT
TO ASK QUESTIONS REGARDING THAT
AND HOW IT IS TO TRANSLATE YOUR
PRE-CLINICAL WORK TO CLINICAL
WORK, HOW HARD IT IS TO -- WHAT
IS INCENTIVE STRUCTURE WE'RE
FUNCTIONING UNDER, WHY IS IT
HARD TO BE HONEST IN YOUR
SCIENCE, I'M WILLING TO TALK
ABOUT ALL OF THOSE THINGS.
AND YEAH, I THINK THE LAST THING
I WANT TO SAY, SCIENTISTS HAVE
REALLY HEARD THE RULES OF THE
GAME LOUD AND CLEAR.
WE UNDERSTAND WHAT IT TAKES TO
BE A SUCCESSFUL SCIENTIST.
THIS IS REALLY A SYSTEMIC
PROBLEM.
IF WE DON'T CHANGE THE SYSTEM,
THEN WE'RE NOT GOING TO CHANGE
WHAT'S HAPPENING.
IT'S NOT JUST AS P-HACKING, IN
THE FIRST PANEL, THERE ARE OTHER
PROBLEMS, COLLECTING DATA,
ANALYZING IT.
THOSE ARE THE PERSPECTIVES I
BRING.
I'M EXCITED TO BE HERE.
SO I'LL CONTINUE OVER HERE.
>> LET'S OPEN THE CONVERSATION
FOR DISCUSSION, IF YOU HAVE
QUESTIONS.
>> IS IT ON NOW?
OKAY.
HI, I'M SAMUEL, I'M A TEAM
LEADER, SENIOR EDITOR OF BNC,
PART OF SPRINGER NATURE, I'M
ALSO JET LAGGED, BEAR WITH ME AS
I TRY TO COMPOSE A QUESTION.
THE BNC PROMOTES NEGATIVE DATA
AND NULL STUDIES BUT WE STRUGGLE
TO GET MANY SENT TO US.
FILLING MISSING DATA POINT
RAISED IN THE PREVIOUS, 4% OF
ALL OUR PAPERS.
AND I FIND EVEN THOUGH I PUSH
QUITE HARD ABOUT ONCE A YEAR FOR
IT TO HAPPEN WE GET CAUGHT IN
THE SAME CYCLE OVER AND OVER
AGAIN, SOME AUTHORS DO SUBMIT TO
US, NEARLY ALL GET REJECTED.
I THINK FOR TWO MAIN REASONS.
ONE, THERE'S A BIAS AGAINST
THEM, REVIEWERS ARE HARSHER ON
NULL OR NEGATIVE STUDIES, NOT
CERTAIN WHAT TO DO WITH THEM.
EVEN THOUGH MY JOURNAL IS A
SOUND SCIENCE ONLY JOURNAL.
WE DON'T JUDGE ON NOVELTY.
IT'S REALLY HARD TO DETERMINE
THE QUALITY OF A NEGATIVE OR
NULL STUDY.
IT'S EASY TO SAY WHY SOMETHING
IS AND WHY IT'S GOOD, MUCH
HARDER TO SAY WHY IT ISN'T.
THIS MEANS YOU REJECT A LOT, SAY
NO TO A LOT, A LOT OF AUTHORS
GET REJECTED AND DECIDE NOT TO
SUBMIT AGAIN, NOT TO REALLY TRY,
YOU GET EVEN LESS VISIBILITY, NO
ONE ELSE SEEMS TO BE PUBLISHING
ABOUT THIS, SO WHY SHOULD WE?
AND YOU GET CAUGHT IN A CYCLE
WHERE NOTHING HAPPENS OR GETS
BETTER.
I WONDER IF ANYONE HAS THOUGHTS
HOW WE CAN IMPROVE THIS AS
EDITORS AND PUBLISHERS.
THANK YOU.
>> I FEEL LIKE THAT'S SOMETHING
I'M HAPPY TO SPEAK TO YOU, BUT
WITH A DIFFERENT PERSPECTIVE.
PART HAS TO DO WITH EXPECTATIONS
IN THE COMMUNITY, RIGHT?
SO WE'RE EVALUATING RESEARCH
BASED ON HOW EXPERIMENT WAS
CONDUCTED, WHAT THE QUESTIONS
NOT RESULTS, REREGISTRATION
ADDRESSES THIS, BEFORE YOU EVEN
HAVE DATA, BEFORE YOU KNOW THE
DIRECTION OF THE RESULTS.
BUT I THINK A LOT OF THIS HAS TO
DO WITH BETTER TRAINING,
MANAGEMENT AND OVERSIGHT OF
ACADEMIC EDITORS WHO WILL BE
HANDLING THOSE PAPERS TO HELP
THEM, TO SUPPORT THEM IN
EVALUATING RESEARCH OBJECTIVELY.
SO THERE'S A BIG TRAINING AND
EDUCATION PIECE THAT COMES IN
BUT IT'S ESSENTIALLY ACADEMIC
EDITORS ARE RESEARCHERS.
EVERYONE IS AUTHOR, RESEARCHER,
EDITOR.
SO I THINK, YOU KNOW, HAVING
THAT CYCLE OF PUBLISHING
NEGATIVE RESULTS, HAVING A FAIR
REVIEW PROCESS, SEEING IT IN
PRINT AND THEN APPLYING YOUR OWN
TREATMENT THROUGH THAT CYCLE
WHEN YOU'RE REVIEWING AND
HANDLING PAPERS IS ONE WAY TO
BEGIN TO ADDRESS THIS.
>> THIS MAY BE A CONTROVERSIAL
COMMENT.
I DON'T THINK YOU'RE EVER GOING
TO FIX PUBLICATION BIAS WITH
INITIATIVES THAT CALL FOR
NEGATIVE RESULTS.
YOU'RE NEVER GOING TO GET IT.
TOO MANY SELECTIVE PRESSURES ON
EVERYONE TO REJECT, TURN THEM
POSITIVE, IT'S NEVER GOING TO
WORK.
IT'S JUST A BAD IDEA.
IT'S MUCH BETTER TO BLIND YOUR
REVIEW SYSTEM RESULTS.
WHETHER REGISTERED REPORT OR
SIMPLY PEER REVIEWING REGULAR
ARTICLES WITH RESULTS REDACTED
UNTIL YOU'VE DONE A PROPER
EVALUATION OF POLICY.
I DON'T THINK NULL RESULTS
INITIATIVES ARE GOING TO FIX THE
PROBLEM.
THE PROBLEM IS RESULTS-BASED
EVALUATION.
YOU DON'T FIX THAT BY DOING MORE
RESULTS-BASED EVALUATION.
>> I'M GLAD YOU SAID THAT.
ANOTHER JOURNAL IN THE BMC, WE
TRIED NO RESULTS REVIEW, IT HAS
CHALLENGES IN THE MIDDLE OF
COVID, NOT A LOT OF THINGS
WORKED AS YOU WANT DURING COVID.
BUT WE HAD ALMOST EXACTLY THE
SAME PROBLEMS WHICH IS REVIEWERS
STRUGGLED TO KIND OF GET TO GRIP
HOW TO ASSESS A MANUSCRIPT OF
BAD RESULTS.
THERE WAS A LOT OF BIAS.
AGAIN, I THINK TO PICK UP ON
WHAT YOU SAID, EMILY, IT'S A LOT
OF LEARNING REQUIRED BEFORE
EVERYONE BEFORE AN INITIATIVE
WILL WORK.
ANYTHING NEW INSTANTLY GETS
PUSHED AGAINST, HOW BEST TO
ADDRESS THAT ACROSS PUBLISHERS
AND AUTHORS AND FUNDERS AND
REGULATORS IS THE REASON WE'RE
HERE BUT THERE IS NO SIMPLE
SOLUTION ALTHOUGH I LIKE THE
SOUND OF YOURS BETTER THAN MINE
AT THE MOMENT.
>> I HAVE VERY DIFFERENT
EXPERIENCE.
WE INTRODUCED PUBLISHING
NEGATIVE RESULTS FROM THE VERY
BEGINNING SO WE PUBLISHED QUITE
A FEW NOW.
AND I NEVER SAW A PROBLEM LIKE
THAT.
SO, THE WAY WE DO IT REALLY
DIFFERENT SO WE HAVE A
CONSULTATION PROCESS.
THE REVIEWERS KNOW THEIR NAMES
READ EACH OTHER'S REVIEWS TO
ENSURE THAT ONE MESSAGE IS
CONVEYED TO THE OTHERS SO THEY
ALL TALK TO EACH OTHER.
IF THERE'S A MISCONCEPTION FROM
ONE REVIEWER THAT SHOULDN'T
PUBLISH NEGATIVE RESULTS,
DOESN'T GO FURTHER THAN THAT,
IT'S THE JOB OF THE EDITOR TO
ASK THEM IS THIS A MESSAGE THAT
IS USEFUL TO THE COMMUNITY,
THAT'S THE ONLY QUESTION YOU'RE
BEING ASKED, IF THE EXPERIMENTS
ARE DONE PROPERLY DOESN'T MATTER
WHAT THE RESULT IS EXACTLY, WHAT
EMILY SAID, BUT HAVING THIS
CONSULTATION PROCESS BECAUSE WE
ALSO PUBLISH REVIEWS.
I DON'T KNOW WHETHER IT'S
WORKING IN EUROPE BECAUSE OF
THAT, I DON'T KNOW, BUT THAT'S
THE MAIN DIFFERENCE I SEE WITH
OTHER JOURNALS BECAUSE WE'RE
FULLY TRANSPARENT.
>> I HAVE A SPECIFIC QUESTION.
[ GARBLED AUDIO ] YOU SPOKE
ABOUT PRE-PRINT COMMUNITY
REVIEW.
COULD YOU PROVIDE MORE DETAIL ON
THAT BECAUSE THAT SOUNDED
PROVOCATIVE.
>> SURE, YEAH.
ABSOLUTELY.
THE COMMUNITY AND REGISTERED
REPORTS INITIATIVE IS ONE
COMMUNITY WITHIN THE BROADER PCI
FAMILY.
SO PEER COMMUNITY IS A
PRE-PRINT-BASED NON-COMMERCIAL
PLATFORM FOR REVIEWING
PRE-PRINTS SO IT'S EXISTED FOR A
FEW YEARS NOW IN DIFFERENT
FIELDS.
THE IDEA IS THAT RATHER THAN
AUTHORS GETTING PEER-REVIEWED
JOURNALS WHY CAN'T WE DO PEER
REVIEW OF PRE-PRINTS, THIS IS AN
IMPORTANT -- WE'VE ALWAYS DONE
PEER REVIEW OURSELVES AS AN
ACADEMIC COMMUNITY.
WE MAKE UP ALL THE REVIEWERS,
VIRTUALLY ALL THE EDITORS AT
MOST JOURNALS.
WE'RE PERFECTLY CAPABLE OF DOING
THIS OURSELVES.
AND THEN THAT GIVES THE
COMMUNITY A LOT MORE CONTROL
OVER THE ARTICLES, WHAT PROBLEMS
WITH OPEN ACCESS BECAUSE
PRE-PRINTS HAVE BEEN PUBLICLY
AVAILABLE FOR EVERYBODY TO READ,
DEMOCRATIZE SCIENCE, IT'S CHEAP
AND COST EFFECTIVE.
WE CREATED A SEPARATE PCI FOR
REGISTERED REPORTS SPECIFICALLY
BECAUSE WE WANTED TO CREATE THE
SAME COMMUNITY MECHANISM FOR
REGISTERED REPORTS, SO
REGISTERED REPORTS EVALUATION
DONE ABOVE AND BEYOND JOURNALS
BY THE COMMUNITY, BY EXPERTS IN
REGISTERED REPORTS AT THE
PRE-PRINT REVIEWS, MADE PUBLIC
IN CASE OF POSITIVE
RECOMMENDATION.
WE HAVE A SET OF I THINK 36
PCI-FRIENDLY JOURNALS THAT
COMMIT TO ACCEPTING THE REVIEWS
AND RECOMMENDATIONS WITHOUT
FURTHER PEER REVIEW.
THESE JOURNALS ESSENTIALLY ALLOW
THE PCI REVIEW PROCESS TO
REPLACE THE REGISTERED REPORTS
THAT WOULD HAVE HAPPENED HAD
AUTHORS GONE TO THAT JOURNAL.
NOW, THIS IS REALLY GOOD FOR
EVERYBODY.
THIS IS GOOD FOR THE COMMUNITY
BECAUSE IT GIVES AUTHORS THE
OPPORTUNITY TO CHOOSE THEIR
JOURNAL BASED ON THE AUDIENCE
THEY ARE LOOKING FOR, BASED ON
ANY OTHER PARTICULAR FACTOR THEY
MAY HAVE, IT GIVES THEM A LOT OF
CONTROL OR POWER OVER THEIR
ARTICLE.
IT MAKES THE -- CENTRALIZED THE
PROCESS IN A VERY EFFICIENT WAY,
ONE REVIEW PROCESS CAN OPEN THE
DOOR TO MULTIPLE JOURNALS.
BECAUSE WE CONTROL THE PLATFORM,
WE CAN INNOVATE IN WAYS THAT
PUBLISHERS ARE UNABLE OR
UNWILLING TO DO.
WE IMPLEMENTED A SCHEDULED
REVIEW PROCESS, STAGE 1 REVIEW
TIME, A BARRIER FOR MANY AUTHORS
WAITING TO GO PRE-STUDY
EVALUATION, WE CAN PLAN AHEAD, A
PERIOD OF SEVEN DAYS, ONE TO
TWO MONTHS FROM SUBMISSION,
ACCELERATED TO A MATTER OF A FEW
WEEKS BY PLANNING THINGS IN
ADVANCE.
YOU THINK IT'S A SIMPLE
INNOVATION BUT ONE NO PUBLISHER
HAS ACCOMPLISHED.
WE'VE IMPLEMENTED PROGRAMMATIC
REPORTS, BASED ON MULTIPLE STAGE
2 REGISTERED REPORTS WILL BE
ACCEPTED IN ADVANCE.
SO ONE PROGRAMMATIC LINE OF
RESEARCH CAN LEAD TO MULTIPLE
REGISTERED REPORTS THROUGH ONE
CENTRALIZED REVIEW PROCESS.
IT'S COMMUNITY DRIVEN, FREE FOR
EVERYBODY, FREE TO READ, IT'S
OPEN.
AND IT'S INNOVATIVE, SOMETHING
WE CAN ALL SUPPORT IN DIFFERENT
WAYS, WHETHER YOU'RE A JOURNAL
EDITOR, JOIN AS RECOMMENDER,
WHETHER YOU'RE AN AUTHOR OR
REVIEW.
>> HELLO.
I'M A PROGRAM MANAGER AT
GLADSTONE.
NOT LONG BEFORE UP TILL LAST
YEAR WAS A POSTDOC, KIF'S STORY
RESONATED.
BEFORE I LEFT MY POSTDOC I
PUBLISHED A STORY ON NEGATIVE
RESULTS, MY VISOR WAS SUPPORTIVE
BUT NOT SURE IF EVERYONE IS
SUPPORTIVE OF HAVING STUDENTS OR
POSTDOCS OR WHOEVER PUBLIC
NEGATIVE RESULTS AND I THINK WE
HAVE ALSO SEEN THERE WAS RECENT
DATA THAT THE LENGTH OF POSTDOC
OR GRADUATE TIME INCREASED
SUBSTANTIALLY IN THE PAST YEARS.
THEY ARE PRODUCING DATA ABOUT
MAYBE IT'S NOT COALESCING INTO
HIGH IMPACT STORY OR WHATEVER.
COMING FROM JOURNALS OR REVIEW
SYSTEM, I WANT TO UNDERSTAND NOT
BIAS, IN TERMS OF PUBLISHING
NEGATIVE DATA LIKE PUBLICATION
BIAS IN GENERAL WE ALWAYS SORT
OF SAY LIKE SO-AND-SO ET AL.
PUBLISHED THIS, IT'S PEOPLE
FIRST, MAYBE SCIENCE LATER.
BUT LIKE CHANGING DYNAMIC TO
TALK ABOUT SCIENCE BEING IN
PERSON, I DON'T KNOW HOW MUCH IT
IS POSSIBLE TO BE IN PERSON BUT
MAYBE IN REGARDS TO SCIENTIFIC
DATA AND HOW IN TERMS OF THE
JOURNALS OR REVIEWERS WHICH ARE
ASSOCIATED WITH JOURNAL HOW CAN
THAT BE LIKE IS IT POSSIBLE TO
DO SCIENCE FIRST, PEOPLE LATER,
SOMETHING ON THOSE LINES?
>> TRICKY QUESTION GETTING AT
CREDIT, RECOGNIZING HOW WE
REWARD THE WORK IT'S VERY MUCH A
PERSON-DRIVEN THING.
THERE ARE WAYS WE CAN EXPLORE
GETTING CREDIT FOR DIFFERENT
PARTS OF THE RESEARCH QUESTION
BUT TRYING TO TEASE OUT CREDIT
IS A DIFFICULT PROPOSITION.
>> YEAH, I AGREE.
THAT'S THE CHALLENGE.
REALLY YOU SHOULD HAVE JUST THE
RESULT.
IN A WAY, YOU DON'T WANT TO
DOWNPLAY THE CONTRIBUTION OF
INDIVIDUALS.
>> I MEANT MORE TOWARDS THE SORT
OF LIKE PROCESS OF REVIEW LIKE
JOURNALS, GRANTS OR WHATEVER, IF
YOU KNOW THE PERSON WHO HAS THE
PAPER LIKE IN TERMS OF KIF THAT
PAPER IS MORE CITED, NATURE
NEUROSCIENCE MAYBE COMING FROM
YOU HAVE THE STATUS, IN TERMS OF
REVIEW PROCESS IN GENERAL DOES
THAT MATTER WHEN SOMEBODY IS
TRYING TO NEGATIVE RESULTS
CONTRADICTING LIKE NOT AS
SUPPORTIVE OF ANOTHER REPUTED
SCIENTIST IN TERMS OF THE BIAS
IN REVIEWING?
>> YEAH, THERE'S A BIAS, RIGHT?
WE HAD THAT IN THE LAST SESSION,
WHERE YOU'RE RECOGNIZING IT
TAKES A LOT OF EVIDENCE TO GO
AGAINST THE STATUS QUO.
AND THERE'S A LITTLE -- THERE'S
CERTAINLY A BIAS IF YOU'RE A
POSTDOC AND HAVE A FAMOUS PERSON
THERE IS WRONG, IT'S VERY
DIFFICULT.
THERE'S SOME FEAR YOU MAY HAVE,
IT MAY BE THE FAMOUS PERSON
REVIEWING THE PAPER, THAT TOPIC
AREA, SO THERE'S SOME BIAS
THERE.
>> THAT DIDN'T HELP.
YEAH, EXACTLY.
THE THING IS IT'S MORE -- IT'S
DISHEARTENING.
SINCE THE PUBLICATION OF OUR
PAPER IN 2021, LOOKING FOR
CITATIONS OF THE ORIGINAL PAPER,
HAD MORE THAN 40 CITATIONS, ALL
IN THE CONTEXT OF WHAT THEY HAVE
SHOWN IS RIGHT.
THAT MEANS mRNA CAN'T UNDERGO
SPLICING.
OUR PAPER CITED SIX TIMES, AND
THREE IS COMPLETELY DIFFERENT
CONTEXT.
PAPER WAS CITED FOR SOMETHING
ELSE BASICALLY.
IN MY OPINION I THINK THE VENUE
IDEALLY YOU WANT TO CHALLENGE
SOME STUDIES WHERE IT IS
ORIGINALLY PUBLISHED, WHICH
MISERABLY FAILED, WE WERE LUCKY
NOT TO RESULT IN THAT GROUP,
DIDN'T WASTE TIME.
I THINK IF WE PUBLISHED IN THAT
EXACT JOURNAL PROBABLY WE WOULD
GET SOME TENSION BUT THERE'S NO
BARRIER IN PUBLISHING NEGATIVE
RESULTS IN A WAY THAT HUNDREDS
OF VENUES THAT YOU CAN PUT IT
IN, RIGHT?
YOU CAN PUBLISH, YOU CAN ALSO
PUT IT IN -- THE PROBLEM IS NOT
PUBLISHING, IT'S IN THE
VISIBILITY.
NO MATTER WHERE IT IS COMING
FROM, EVEN A BIG LAB OR VISIBLE
LAB.
AND TALKING ABOUT CRITICS, I
ALSO KNOW FROM OTHER THINGS LIKE
ASTRONOMY AND PHYSICS, THEY
DON'T CARE ABOUT THE AUTHOR
ORDER, IT'S ALL ABOUT WHAT THE
SCIENCE THEY CONTRIBUTE TO.
I DON'T KNOW HOW MANY HUNDRED
YEARS IT WILL TAKE FOR US.
THAT I THINK IN THAT UTOPIAN
WORLD PEOPLE WILL SAY THE
SCIENCE FIRST RATHER THAN WHO
DID IT.
>> THANK YOU.
>> YOU ASKED MANY QUESTIONS BUT
IN SOME FIELDS, THEY WANT TO
ACKNOWLEDGE THE PRODUCTION OF
DATA, AND THEY ARE TRYING TO
INVENT A WAY BUT THAT'S BEYOND
JOURNALS, RIGHT?
ACQUIRED BY POSTDOCS, EVEN
ENGINEERS OR WHATEVER, THEY ARE
LABELED OTHER NAME, AND WHEN
THEY ARE CITED, USED BY OTHERS,
THAT IS BECOMING SOMETHING LIKE
A TRUE SCIENTIFIC PREDICTION,
THEY ARE THINKING ABOUT THAT
KIND OF APPROACH.
NOW, TO ADDRESS THE QUESTION OF
PUBLISHING NEGATIVE RESULTS OR
GOING AGAINST DOGMA, BECAUSE
YOU'RE FIGHTING A BIG NAME, FOR
EXAMPLE, CONTRADICTING A BIG
NAME, IT'S NOT AN YOU SHALL
BECAUSE WE'RE USING DOUBLE
BLIND.
IF THOMAS SENT THE PAPER TO
NEURO, THE REVIEWERS DON'T KNOW
WHETHER IT'S COMING FROM THOMAS
LAB OR SOMEBODY THEY HAVE NEVER
HEARD BEFORE.
SO, THAT REMOVES THAT PART OF
BIAS.
WHICH IS ALSO ANOTHER PROCESS
THAT SIMPLIFIES THINGS FOR US.
THAT'S WHY MAYBE IT'S WORKING.
YOU DON'T KNOW.
SO IF YOU DON'T KNOW, MAYBE IT'S
A BIG NAME.
>> ALEX FREEMAN, PUBLISHING
PLATFORM.
A COUPLE OF YOU MENTIONED
PROBLEMS OF HAVING REVIEWERS
COMMENTING ON THE FACT THAT THEY
DIDN'T THINK THAT NEGATIVE OR
NULL RESULTS SHOULD BE
PUBLISHED.
I WONDERED IF YOU COULD REFLECT
ON HOW MUCH GUIDANCE YOU GIVE TO
REVIEWERS AND HOW MUCH
STRUCTURED REVIEW YOU ASK AND
WHETHER IT MAKES A DIFFERENCE IF
REVIEWS WILL BE PUBLISHED OPENLY
AS WELL.
>> SO, IN TERMS OF THE GUIDANCE,
WE PROVIDE THEM, I DON'T KNOW
HOW MUCH THEY READ LETTERS IN
TERMS OF INSTRUCTIONS, RIGHT?
WHERE WE SAY WE'RE INTERESTED
MORE IN THE QUESTIONS THAN THE
RESULTS, SO ON.
OFTEN, THIS COMES -- IT'S NOT
SUITABLE FOR JOURNAL,
CONFIDENTIAL, SO IT'S NOT --
NOW, WE JUST ARE STARTING A
PILOT PROJECT AT THE MOMENT FOR
POSTING OPEN REVIEWS ON THE
JOURNAL, SO WE'LL SEE EFFECT OF
THAT.
WE HAD A LOT OF PUSHBACK FROM
THE COMMUNITY, THAT'S ANOTHER
TOPIC, IN TERMS OF MAKING IT
FULLY OPEN.
I WAS EDITOR AT THE U.S. EDITOR
FOR BMJ FOR SEVERAL YEARS, A
FULLY OPEN PROCESS, EVERYBODY
KNOWS WHO EVERYBODY IS.
AND THAT WORKED.
THAT WORKED WELL.
AND THEN YOU WOULD HAVE THAT
COMMENT, NOT SUITABLE, WE WOULD
TAKE IT OUT IF THAT COMMENT WAS
THERE FROM WHAT'S PUT OUT THERE
BECAUSE THAT'S NOT FOR THE
REVIEWER TO SAY.
THE REVIEWERS IS JUST TO COMMENT
ON THE QUESTION, AND METHODS AND
SO ON, NOT SO MUCH IN TERMS OF
WHAT GOES IN.
THAT MAY CHANGE THINGS.
I DON'T KNOW IF HAD YOU THAT
EXPERIENCE WITH PLOS OR --
>> YEAH, I THINK REVIEWERS
ARE -- WE DON'T HAVE A DIRECT
RELATIONSHIP WITH THEM, RIGHT?
THEY ARE SELECTED BY ACADEMIC
EDITORS.
WE RELY ON ACADEMIC EDITORS BUT
WE HAVE A BOARD OF 10,000ED
CORES, GETTING THEM TO ALL
FOLLOW THE SAME CRITERIA IS
REALLY TRICKY, THERE'S A BIG
EDUCATION PIECE AND BIG SUPPORT
PIECE THAT HAS TO COME INTO PLAY
HERE.
AUTHORS, REVIEWERS, EDITORS,
READERS, WE UNDERSTAND THIS FROM
PERSPECTIVE OF RESEARCHERS, HOW
FRUSTRATING NOT TO PUBLISH
RESULTS BUT THERE'S A DISCONNECT
WHEN WE GET THESE PAPERS AS
REVIEWERS AND AUTHORS, THIS
CAN'T BE PUBLISHED.
TOUGH NUT TO CRACK.
I DON'T KNOW THE ANSWER TO THAT.
>> JUST TO ADD, I THINK
SOMETIMES REVIEWERS APPLY A
DIFFERENT LENS TO METHODOLOGY
WHEN THERE'S ANYTHING RESULTS IN
FRONT OF THEM.
THEY MAY SAY WE SHOULD REJECT
BECAUSE RESULTS ARE NULL, THEY
WILL GO ON THE HUNT FOR THINGS
THAT MIGHT BE WRONG WITH
METHODS, IN A WAY THEY WOULD NOT
DO, ALL OF US ARE PRONE TO
CONFIRMATION BIAS, IT'S THE JOB
OF THE EDITOR TO KEEP A CLOSE
EYE AS MUCH AS POSSIBLE, BUT
IT'S AN IMPOSSIBLE TASK.
IF YOU'RE AN EDITOR, DETAILED
INFORMATION, YOU MAY FEEL
OBLIGATED TO RESPOND BY
REJECTING THAT PAPER.
IF THOSE COMMENTS WERE NOT MADE
BECAUSE THE RESULTS WERE
POSITIVE, YOU WOULD NOT FEEL THE
SAME.
SO I THINK EDITORS ARE GOING TO
FALL INTO THE SAME BLACK HOLE,
AS IT WERE.
I FOUND HAVING EDITED REGISTERED
REPORTS AND REGULAR ARTICLES,
THAT REGISTERED REPORTS ARE THE
ONLY SOLUTION.
YOU HAVE TO TAKE AWAY THE
RESULTS.
EVEN THEN, OCCASIONALLY AT STAGE
2, A REVIEWER WILL SAY IT'S A
SHAME THE RESULTS WERE NEGATIVE,
AND THEY WILL ASK FOR A LOT OF
EXTRA ANALYSES.
WITH A REGISTERED REPORT YOU
CAN'T SHIFT THE GOAL POSTS BUT
YOU CAN WITH A REGULAR ARTICLE.
WE HAVE TO BE MINDFUL OF THE
COGNITIVE LIMITATIONS OF EDITORS
AND REVIEWERS AND BUILD SYSTEMS
LIKE REGISTERED REPORTS WHICH
PREVENT BIASES FROM HAVING AN
EFFECT.
>> NEIL, STANFORD.
I'LL DIRECT MY QUESTION TO
LAURA, BUT IF YOU DON'T WANT TO
ANSWER THAT'S FINE.
CASE STUDY WAS ABOUT A
REPLICATION AND CRITIQUE FROM
KIF.
THERE'S EMERGING LITERATURE ON
NULL HACKING THAT SUGGESTS YOU
HAVE THE OPPOSITE BIAS THAT
WE'RE DISCUSSING WHEN IT COMES
TO REPLICATION AND CRITIQUED
PAPERS.
IT'S A LOT EASIER TO PUBLISH A
REPLICATION THAT SAYS SOMEONE'S
WRONG THAN PUBLISHING A
REPLICATION CRITIQUE THAT SAYS
SOMEONE IS RIGHT BUT ALL OF THIS
IS REALLY IMPORTANT FOR SCIENCE
TO SEE EVERYTHING.
IF YOU'RE BAYESIAN YOU WANT TO
KNOW IF SOMEONE DID SOMETHING
BEFORE THAT WAS CORRECT, SO WHAT
KIND OF INSTITUTIONS DO WE NEED
TO ENCOURAGE REPLICATIONS THAT
DON'T OVERTURN PRIOR RESEARCH
BUT SUPPORT IT?
>> I'M NOT SURE ABOUT
INSTITUTIONS.
I THINK WE NEED TO CHANGE THE
CULTURE SURROUNDING IS IT OKAY
TO REPLICATE AND TO, YOU KNOW,
DISAGREE OR AGREE, AND I AGREE
WITH YOUR COMMENT THAT IT'S
EASIER TO OVERTURN.
CERTAINLY EASIER BECAUSE
METHODOLOGIES DIFFER BETWEEN
LABS IN MANY CASES, IT'S EASIER
TO FRAME A QUESTION WE'LL NOT
SUPPORT WHAT'S ALREADY BEEN
SHOWN.
I'M NOT SURE IF I HAVE BEYOND
THAT A GOOD -- I DON'T THINK
THIS IS A BIG PROBLEM IN MY
UNDERSTANDING THAT PEOPLE ARE
OUT THERE, YOU KNOW, REPLICATING
AND TRYING TO BRING DOWN OTHER
LABS, NOT REPLICATING, I SHOULD
SAY.
IS THIS A BIG PROBLEM?
>> IF YOU'RE REPLICATING, YOU'LL
GO IN THE OTHER DIRECTION.
THE HARDEST THING TO PUBLISH IS
REPLICATING THAT SHOWS SOMEONE
IS RIGHT.
HOW DO WE GET THOSE PAPERS
PUBLISHED?
>> YOU MEAN ACTUAL PUBLICATION
PROCESS ITSELF?
YEAH, YEAH.
I THINK OUR WHOLE INCENTIVE
STRUCTURE NEEDS TO BE DIFFERENT.
WE NEED TO VALUE DIFFERENT
THINGS, INCLUDING SUPPORT,
EMERGING EVIDENCE SOMETHING IS
TRUE OR NOT.
THAT'S WHAT I HAVE TO SAY.
>> I WANT TO COMMENT ON THAT.
AS A BASIC RESEARCHER, IF YOU DO
THINGS CAREFULLY BEFORE YOU
START SOMETHING YOU REPEAT THE
EXPERIMENT TO SEE IS ANY
FOUNDATION FOR WHAT YOU'RE
TRYING TO DO, OKAY?
NOW, IF IT SUPPORTS, YOU KEEP ON
WITH YOUR RESEARCH AND ADVANCE
SCIENCE.
IF IT'S NULL YOU PUT IT IN THE
FILE DRAWER ACCOUNT , YOU DON'T
WANT TO GET INTO CONFRONTATION
WITH WHOEVER PUBLISHED THE
PAPER.
REPRODUCIBILITY, I DON'T LIKE
THE TERM AT ALL, BUT BUT
REPRODUCIBILITY IN THE SENSE YOU
TAKE A BIG STUDY AND REPEAT IT,
THAT'S VERY DIFFICULT TO DO
BECAUSE THAT TAKES A LOT OF
FUNDS, GOOD LUCK GETTING FUNDS
TO DO THAT.
THAT'S A MAJOR ISSUE.
>> NEIL?
>> INTERESTING QUESTION.
I REALLY LIKE IT.
IT'S NOT EASY TO DISPROVE
SOMEONE IS WRONG ESPECIALLY WHEN
PUBLISHED IN A TOP TIER JOURNAL.
WHEN I STARTED THE EXPERIMENT,
WHEN I SAW WHAT I SAW, I DIDN'T
BELIEVE IT.
I'M NOT AN EXPERT, JUST STARTING
MY POSTDOC CAREER.
TWO, THAT I COULD BE WRONG.
SO IT INVOLVED MULTIPLE
DIFFERENT EXPERIMENTS TO SUPPORT
THIS OBSERVATION.
I WOULD SAY IT'S EQUALLY
DIFFICULT TO DISPROVE SOMEONE IS
WRONG.
TO ANSWER YOUR QUESTION
BASICALLY IF SOMEONE IS RIGHT,
ESPECIALLY IN BASIC SCIENCE,
RECITE THEM IN A WAY WE'RE ABLE
TO REPRODUCE THEIR RESULTS, IN A
WAY WE'RE ABLE TO USE THEIR
TECHNIQUE OR TOOL.
THAT IN A WAY IS KIND OF
CORRECTING WHAT THEY HAVE.
IT'S NOT SEPARATE STORY LINE
THAT WE WANTED TO PUBLISH, YOU
KNOW, SO-AND-SO TOOL IS AMAZING.
>> CAN I CHALLENGE NIH AT THAT
POINT?
WE HEARD ABOUT THE PROBLEM WITH
PUBLISHING NULL RESULTS, AND
ALSO REPLICATIONS.
JUST SOME PUBLIC REPLICATIONS,
SEE WHAT HAPPENS.
WE DO PUBLISH REPLICATION
STUDIES.
WE VALUE THEM.
IT'S IMPORTANT FOR THE FIELD.
NOW, P.I.s MAY NOT WANT TO
PUBLISH NEGATIVE RESULTS,
REPLICATION STUDIES.
IN EUROPE, BECAUSE IT COSTS
MONEY, BUT IN THE U.S. YOUR LABS
ARE RICH.
IN EUROPE, SOUTH AMERICA, WE GET
MUCH LESS FUNDING, OR AFRICA AND
ASIA.
NOW, PUBLISHING SOMETHING IS A
LOT OF OUR BUDGET.
NOW, THE NIH IS NOT ENCOURAGING
THE PUBLICATION OF NEGATIVE
RESULTS AND REPLICATION BY
SUBSIDIZING DIRECTLY FROM THE
GRANT, OUTSIDE THE GRANT, BUT
THE PUBLICATION OF THE RESULTS?
AND THAT WOULD BE A VERY GOOD
INCENTIVE FOR THE LABS TO
PUBLISH THEIR NEGATIVE STUDIES
AND REPLICATION STUDIES.
I'M SERIOUS.
>> I CAN'T SPEAK FOR NIH.
BUT WHAT I CAN SAY IS, YOU KNOW,
I DON'T KNOW WHY THERE'S NO MORE
ENCOURAGEMENT FOR NULL STUDIES.
REPLICATION, I CAN TELL YOU THE
ISSUE, IF YOU'RE TRYING TO
REPRODUCE RESULTS.
AND THAT IS THAT YES, NIH HAS A
A LOT OF FUNDS, ALSO MANY MORE
INVESTIGATORS THAN IN EUROPE.
AT THE END OF THE DAY IT BOILS
DOWN TO THE SAME ISSUES, AND
EVERY STUDY THAT YOU SAY, YOU
KNOW, WE -- LET'S PRETEND FOR
ARGUMENT THAT YOU HAVE TO
REPRODUCE EVERYTHING.
SO EVERYTHING HAS TO BE
REPRODUCED.
THAT MEANS HALF THE FUNDS WILL
BE GONE FOR NEW RESEARCH, THERE
IS WHAT THERE IS.
AND SO YOU HAVE TO MAKE A
DECISION WHAT MEETS THE BAR THAT
YOU SAY SHOULD BE REPRODUCED?
THERE ARE ONGOING DISCUSSIONS
THE NIH.
YOU KNOW, WHAT THAT MIGHT BE.
THERE HASN'T BEEN DECISIONS, NOT
PART OF THIS CONVERSATION, BUT
AS FAR AS NULL RESULTS I FULLY
AGREE, YOU KNOW, THERE NEEDS TO
BE MORE.
>> PAUL SUNDBERG.
IF YOU DECREASE PUBLICATION
COSTS, BIOLOGY HAS LOW COST, YOU
KNOW, WOULD ALLOW MORE OF THAT
BUT PEOPLE STILL HAVE TO SPEND
THEIR TIME, RIGHT?
A LOT OF THESE AS WE HEARD, THE
FIRST THING YOU DO IS REPLICATE,
KEY PARTS OF THE STUDY, YOU'RE
NOT SPENDING TWO YEARS DOING
THAT.
YOU'RE SPENDING, I DON'T KNOW,
LET'S SAY A MONTH OR TWO WEEKS,
DEPENDING ON YOUR PLATFORM,
SMALL WITH RESPECT TO YOUR
TIMELINES.
THEN YOU CAN PUBLISH THAT.
IF YOU LOOK AT HOW MANY PAPERS
PEOPLE PUBLISH MOST PEOPLE DON'T
EVEN PUBLISH ONE PAPER A YEAR,
THERE'S LOTS OF CAPACITY TO
PUBLISH MORE.
WE HAVE TO MAKE IT EASIER AND
THEY WILL INCENTIVIZE IT.
THE COST, IN THE LIMIT, THAT'S A
PROBLEM.
RIGHT NOW I DON'T THINK THAT'S
THE LIMIT.
>> FOLLOWING ON THAT, IT'S ABOUT
HOW TO MAKE THE PUBLICATION MORE
DYNAMIC, IT TAKES A LONG TIME TO
GET FROM EXPERIMENT TO
PUBLICATION BUT ALSO FOLLOWING
ON THE OTHER COLLEAGUE'S
QUESTIONS ABOUT PEER REVIEW, AS
AN EARLY CAREER RESEARCHERS MY
COLLEAGUES, I HEAR THEM TALKING
ALL THE TIME LIKE WHY DO WE HAVE
TO DO ALL THIS WORK?
AND THEY SEE IT AS A CHORE THAT
THEY JUST WANT TO GET OUT OF THE
WAY, BUT I CAN SEE THAT IT'S
CENTRAL FOR EVERYTHING.
SO, HOW DO WE CHANGE THAT?
HOW DO WE ACKNOWLEDGE PEER
REVIEW IS AS IMPORTANT AS BEING
AN AUTHOR ON A PUBLICATION?
THANK YOU.
>> CHANGING THE WAY THAT THESE
CONTRIBUTIONS ARE VALUED, SO
MAKING SURE REVIEWERS HAVE THE
OPPORTUNITY TO DISCLOSE THEIR
IDENTITIES, REVIEWS WILL BE
PUBLISHED, THAT THEY CAN RECEIVE
CREDIT FOR THAT WORK, AND THAT
INSTITUTIONS AND UNIVERSITIES
ACTUALLY TAKE THIS ACTIVITY
SERIOUSLY AND SET ASIDE TIME IN
PEOPLE'S DAYS TO REVIEW
MANUSCRIPTS, SET ASIDE RESOURCES
FOR TRAINING, THAT'S THE ONLY
WAY WE CAN START TO MAKE HEADWAY
ON THAT.
>> ONE THE MOST INTERESTING
EXPERIENCES I HAD WHEN WE FIRST
LAUNCHED THE INITIATIVE WAS
NUMBER OF REVIEWERS WHO WOULD
WRITE TO US SPONTANEOUSLY, I'VE
NEVER SEEN THIS BEFORE, TO SAY
HOW MUCH THEY ENJOYED BEING PART
OF THIS PROCESS.
WHEN DID A REVIEWERRER EVERY SAY
THEY ENJOYED PEER REVIEW?
AND YOU CAN SEE WHY THAT IS,
RIGHT?
THEY HAVE A CHANCE TO CONTRIBUTE
CONSTRUCTIVELY TO A PROJECT THEY
ARE NOT INVOLVED IN WHEN THE
COMMENT CAN BE USED TO MAKE THE
RESEARCH BETTER OR AVOID
PROBLEMS.
I THINK THE REGISTERED REPORT,
HAS ANYBODY IN THE ROOM REVIEWED
A REGISTERED REPORT?
RIGHT, SMALL NUMBER.
I HOPE YOU FIND WHAT I FIND WHEN
YOU DO IT.
YOU DO FEEL LIKE YOU'RE A LOT
MORE VALUABLE TO THE PROCESS,
AND I THINK THERE'S A GREATER
SENSE OF SATISFACTION,
PARTICULARLY WHEN YOU ALMOST --
YOU ARE WITH THE AUTHORS ALL THE
WAY SO I'VE HAD LONGSTANDING
PROJECTS WHERE I'VE EDITED,
THERE'S ALL SORTS OF STUFF THAT
HAPPENS OF THE YOU'RE THERE WITH
THEM ON THIS JOURNEY.
THAT CAN CHANGE THE REVIEW
PROCESS FROM BEING COMBATIVE
TRIAL OR INQUISITION OF FLAW
FINDING INTO SOMETHING MORE
CONSTRUCTIVE AN COLLEGIAL.
AND ULTIMATELY POSITIVE.
>> ALEX RAMIREZ, UNIVERSITY OF
PENNSYLVANIA, POSTBAC SCHOLAR,
HAVEN'T STARTED GRAD SCHOOL YET.
I JUST WANTED TO GET SOME
COMMENTS OR THOUGHTS FROM YOU
ALL OR EVEN TIPS FOR US REALLY
EARLY CAREER SCIENTISTS ON HOW
TO NAVIGATE WHEN YOUR PROJECT
GOES IN A DIRECTION OF NEGATIVE
OR NULL RESULTS, AND FACING
BARRIERS SUCH AS GRADUATION
REQUIREMENTS, P.I.'S
EXPECTATIONS, IF YOU'RE A
POSTDOC, JOB REQUIREMENTS
GETTING PAPERS OUT IN
HIGH-IMPACT JOURNALS, DO YOU
HAVE ANY COMMENTS ON LIKE HOW TO
NAVIGATE THAT IN THAT SPACE WHEN
YOU'RE EARLY ON IN YOUR CAREER?
>> I WILL TRY TO TACKLE THIS
QUESTION.
I'M AFRAID I DON'T HAVE ANY GOOD
ANSWERS FOR YOU AT THIS STAGE,
UNTIL WE CHANGE OUR CULTURAL
EXPECTATIONS, I WOULD ENCOURAGE
YOU TO PRESS FOR DOING THE RIGHT
THING, WHICH IS PUBLISHING YOUR
NULL REPORTS, WHATEVER RESULTS
IN THE STATE THEY ARE IN, MAKING
SURE YOU'RE RIGOROUS IN YOUR
DESIGN PROCESS TO THAT POINT AND
GETTING OTHER OPINIONS OF WHAT
THAT SHOULD ENTAIL BEFORE YOU
EMBARK, IF POSSIBLE.
AND IF YOU'VE ALREADY BEEN
THROUGH THOSE STEPS THEN GO GET
OPINIONS ABOUT YOUR RESULTS AND
WHAT TO DO FROM THERE.
IF YOU HAVE A PARTICULARLY
DIFFICULT P.I. WHO REALLY IS
PUSHING FOR, YOU KNOW, LET'S
JUST SAY MISJUDGMENTS OF HOW YOU
SHOULD DO YOUR DATA ANALYSIS,
YOU SHOULD GO GET MORE HELP FROM
SOMEONE ELSE WHO CAN HELP STAND
UP FOR YOURSELF BECAUSE YOU'RE
NOT IN A POSITION TO STAND UP
FOR YOURSELF VERY WELL IN YOUR
ROLE.
YOU CAN TRY.
>> BUT YOU SHOULD HAVE A MENTOR,
RIGHT?
OUR STUDENTS HAVE MENTORS,
EXTERNAL MENTORS.
THESE EXTERNAL PEOPLE CAN REALLY
ACT IF THEY NEED TO AND YOU CAN
TALK TO THEM FREELY.
I DON'T KNOW WHETHER YOU HAVE
THIS MENTORSHIP IN THE U.S.
>> SHE JUST GOT HER P.I.
>> I WAS JUST SAYING IF HER P.I.
IS AGAINST.
>> SHE CAN REQUEST --
>> THEY ARE ONE AND THE SAME
USUALLY.
THESE ARE ONE AND THE SAME IN
THE U.S.
USUALLY.
YOU MIGHT HAVE MENTORS OUTSIDE
THE P.I. OF THE LAB ABOUT THE
P.I. IS THE MENTOR, RIGHT?
YES.
>> SORRY, MY JET LAG AND CAN'T
FIND THE WORD IN ENGLISH, LIKE
THE GODFATHER, SOMEONE WHO IS
EXTERNAL, AND TO WHOM YOU CAN
REFER TO AND ASK ADVICE OUTSIDE
THE LAB.
SORRY.
>> YEAH, ADVISER.
>> CAN I ADD TO THAT QUESTION?
IN MY EXPERIENCE I THINK THE
IMMEDIATE HELP I HAD WHEN I WAS,
YOU KNOW, LACKING MOTIVATION
PAST TWO YEARS OF MY POSTDOC,
WHEN I WAS SEEING NEGATIVE
RESULTS, THAT RESULTS THAT IS
NOT GOING TO GET ME A BIG PAPER,
OBVIOUSLY WHEN THE SHOWED THE
PAPER, RESULTS, PEOPLE SAID THIS
IS NOT GOING TO MAKE YOU FAMOUS.
THIS IS NOT GOING TO GET YOU A
JOB.
BUT I ALSO HAD COLLEAGUES WHO
REALLY ENCOURAGED ME, WHO MADE
ME STAND FOR THE TRUTH BECAUSE
THIS IS -- IF I HAD TO FOLLOW
THE SIMILAR ROUTE AS THE
ORIGINAL PAPER HAS SHOWED, AS I
SAID, ALL YOU NEED IS A LITTLE
BIT OF CREATIVITY, SEA OF DATA
IN FRONT OF YOU.
JUST NEED TO KNOW HOW TO PACKAGE
IT.
BUT IMMEDIATE HELP IN TERMS OF
LIKE DOING RIGHT THING THAT YOU
DO IS BASICALLY FROM YOUR
COLLEAGUE.
SO IDENTIFY LIKE-MINDED
COLLEAGUES, NOT TELLING IF YOU
HAVE TEN LAB MEMBERS THEY WILL
ALL BE ALIGNING, UNFORTUNATELY
EVERYBODY HAS THEIR OWN AGENDA,
THEIR OWN PLANS FOR THEIR
CAREER.
EVERYBODY WANTS TO ALIGN THE
SAME VALUE BUT UNFORTUNATELY WE
DON'T TALK INSIDE THE LAB.
YOU NEED TO HAVE LIKE-MINDED
PEOPLE.
YOU NEED TO TALK TO.
EVEN BEFORE P.I.
OBVIOUSLY HAVING SUPPORTIVE P.I.
IS VERY, VERY IMPORTANT, SO IN
YOUR STAIN OF YOUR CAREER I
THINK REALLY RIGHT TIME TO
IDENTIFY LABS THAT DOES THINGS
RIGHT.
>> HI, UNIVERSITY OF CALIFORNIA
SANTA CRUZ.
I WANT TO EMPHASIZE A COUPLE
THINGS I'M HEARING WHICH ARE
SUPER IMPORTANT, WHICH ARE WHAT
ARE THE ENSIGN TEFFS EN--
INCENTIVES AND BARRIERS,
INCENTIVE TO LOWER THE BARRIERS,
PUBLICATION BARRIERS.
I WANT TO POINT OUT MICRO PUB
PROVIDES EXPLICIT GUIDANCE HOW
TO REVIEW PAPERS, WHETHER OR NOT
ADDITIONAL EXPERIMENTS ARE
REQUIRED, IF ADDITIONAL
EXPERIMENTS ARE NOT REQUIRED
WHAT LANGUAGE NEEDS TO CHANGE?
I THINK THE VARIABILITY IN
ACADEMIC EDITORS, REVIEWERS,
THERE NEEDS TO BE
STANDARDIZATION TO LOWER
PUBLICATION.
I THINK FUNDING CAN BE A
POWERFUL INCENTIVE.
THERE'S A MECHANISM CALLED THE
R03 WHICH PROVIDES ONE YEAR
FUNDING FOR LAB OR P.I. TO SORT
OF FINISH UP A PROJECT OR DO A
SMALL SCALE SET OF PROJECTS, SO
THAT COULD BE SOMETHING THAT
COULD ON A SHORT TERM BASIS
INCENTIVIZE WHAT COULD BE THE
PUBLICATION OF NULL RESULTS, AND
POTENTIALLY ADDRESS LOWERING OF
PUBLICATION BARRIERS AS WELL.
REGISTRATION AND
PRE-REGISTRATION KEEPS COMING UP
AS A SOLUTION.
I'M FRUSTRATED BECAUSE I DON'T
SEE HOW THAT'S APPLICABLE TO
OBSERVATIONAL STUDIES WHICH IS
OFTEN WHAT STARTS OUT A LOT OF
OUR BIOLOGICAL RESEARCH.
AND SO I THINK WHAT I WOULD
APPRECIATE OVER THE NEXT TWO
DAYS IS WHAT A VERSION OF
PRE-REGISTRATION COULD LOOK LIKE
FOR OBSERVATIONAL STUDIES.
ONE THING I MIGHT PRESENT AS
OPTION IDEA OF STANDARDIZING OR
EMPHASIZING EXPERIMENTAL DESIGN
BEFORE YOU START YOUR
EXPERIMENTS, AND THAT BEING A
CRITICAL PART OF TRAINING.
THAT'S A NEW CLASS WE STARTED AT
U.K. SANTA -- UC SANTA CRUZ,
GRAD TRAINING, THAT MIGHT BE
BETTER SUITED FOR THE
OBSERVATIONAL BIOLOGICAL SCIENCE
THAN PRE-REGISTRATION MIGHT BE.
>> YEAH, THANK YOU.
ALL GREAT POINTS.
FIRST OF ALL, CONCERNING
STANDARDIZATION OF REVIEW
CRITERIA WE TAKE SERIOUSLY WITH
REGISTERED REPORTS.
WE HAVE SIX STAGE 1 AND STAGE 2
CRITERIA WHICH MUST BE MET TO
ACHIEVE STAGE 2 ACCEPTANCE, WE
ASK ASK REVIEWERS.
IF THEY GO BEYOND THEM, AND THEY
DO, IT'S THE JOB OF RECOMMENDER
OR EDITOR TO MAKE SURE
SUBMISSION IS ONLY EVALUATED
WITHIN THE SCOPE OF THE
CRITERIA.
AND RECOMMENDED AT PCI
REGISTERED REPORTS DO AN EXAM
BEFORE THEY HANDLE THE FIRST
SUBMISSION, TAKES AN HOUR, 95%,
REACTIVATES ONLY EXAM ANXIETY IN
PEOPLE LONG PAST DOING THAT,
WHICH IS KIND OF FUN.
ON THE SECOND POINT WE GET A LOT
OF OBSERVATIONAL STUDIES
SUBMITTED TO ROYAL SOCIETY OPEN
SCIENCE OR TO PCIRR.
AND I THINK IN GENERAL THE
FORMAT, REGISTERED REPORTS
FORMAT CAN BE VERSATILE.
IT'S ALWAYS UP TO THE COMMUNITY
TO SHAPE IT ACCORDING TO WHAT IT
NEEDS TO LOOK LIKE.
YOU'VE GOT THAT PRE-STUDY
EVALUATION AND THEN YOU'VE GOT
IN PRINCIPLES ACCEPTANCE
REGARDLESS OF RESULTS.
PROVIDED YOU CAN DEFINE A PLAN,
THAT SOMEONE ELSE CAN EVALUATE,
PROVIDED CAN YOU ARTICULATE THAT
IT'S SUITABLE IN ANY AREA.
WE GET SUBMISSIONS ACROSS THE
BOARD, SUBMISSIONS AND MORE IN
DRAMA STUDIES, NANOSCIENCE,
NEUROSCIENCE, EVERYTHING.
I THINK I'D BE INTERESTED TO
TALK TO YOU LATER ABOUT THE
CHALLENGES YOU FACE WITH
OBSERVATIONAL BIOLOGICAL
RESEARCH BUT I'M SURE THEY ARE
SURMOUNTABLE TO MAKE REGISTERED
REPORTS FORMATS COMPATIBLE.
>> THERE'S A NICE PIECE WRITTEN,
I HAVEN'T RED THAT IN PLOS
BIOLOGY.
PRE-REGISTRATION FOR EXPLORATORY
SCIENCE, I NEED TO READ THAT.
>> YEAH, AND THAT'S GREAT.
WE CAN -- YOU CAN -- A
REGISTERED REPORT CAN BE
ANYTHING.
THE MORE DETAILED IT IS,
OBVIOUSLY THE BETTER.
ONE OF THE CHALLENGES YOU DON'T
QUITE NOW WHAT YOU'RE GOING TO
BE DOING UNTIL IT'S DONE.
AND THAT'S FINE.
THERE'S ALWAYS A PLACE OUTSIDE
THE REGISTERED REPORTS FORMAT.
FOR A LOT OF WORK STILL
OBSERVATIONAL AND HAS AN
EXPLORER TO ANGLE, THERE'S STILL
A VERY DETAILED PLAN THAT GOES
INTO IMPLEMENTING THE RESEARCH.
EVEN IF THERE AREN'T STRONG
HYPOTHESES.
I'VE HANDLED SUBMISSIONS AT
ROYAL SOCIETY OPEN SCIENCE WHERE
IT WAS PURE OBSERVATION WITH NO
HYPOTHESES, AND YET A CLEAR
QUESTION, A VERY CLEAR ANALYSIS
PLAN AND CLEAR STATEMENT HOW
THEY WOULD INTERPRET OUTCOMES IS
IMPORTANT TO AVOID INTERPRETIVE
BIAS.
YOU CAN MAKE THE FORMAT WORK FOR
ANYTHING IF YOU WANT TO.
>> JUST TO BRIEFLY ADD TO THAT,
YOU KNOW, GRANTS REVIEW
COMMITTEES ARE IN SOME CASES
REVIEWING OBSERVATIONAL STUDIES.
THERE'S PRECEDENT OF HOW YOU'RE
GOING TO REVIEW A STUDY PLAN
WITHOUT DATA, YOU KNOW, THAT
WORKS ACROSS STUDY TYPES AS
WELL.
>> WHEN YOU'RE PLANNING FOR THAT
KIND OF REGISTERED REPORT YOU
CAN HAVE A DECISION TREE TRYING
TO IMAGINE POSSIBLE OUTCOME AND
THEN YOU TAKE A DECISION AND
TAKE A BRANCH AND A BRANCH, SO
YOU DON'T HAVE TO EXPLORE ALL
THE BRANCHES BECAUSE AT SOME
POINT OF YOUR OBSERVATION YOU
WILL HAVE TO MAKE CHOICES BUT IF
YOU ALREADY INTRODUCE THE
POSSIBILITIES THAT YOU THINK MAY
BE APPLICABLE TO YOUR PROJECT,
THEN IT IS REVIEWED LIKE THIS,
OF COURSE YOU NEVER KNOW WHERE A
PROJECT IS GOING TO GO BUT THEN
IF YOU DIVERT TOO MUCH FROM THE
REGISTERED, THAT'S OUR RULE, THE
PEOPLE BECOMES LIKE A NORMAL
PAPER BECAUSE IT WAS NOT
EVALUATED AS SUCH.
IF YOU TRY TO IMAGINE
EVERYTHING, THEN YOU'RE FINE.
>> YOU CAN ALSO HAVE NO ANALYSIS
PLAN AT ALL.
THIS IS GOING TO BLOW YOUR MIND.
YOU CAN HAVE A BLIND ANALYST,
RIGHT?
I'VE HANDLED A FEW REGISTERED
REPORTS INSTEAD OF SPECIFYING
THEY SAID WE'LL HAVE BLINDED
ANALYST THAT DOESN'T KNOW THE
DATA.
THEY HAVE LISTED VARIABLES, AND
THEY WILL IMPLEMENT ANALYSIS ON
THE DATA IN A WAY THAT IS
COMPLETELY PROTECTED AGAINST
BIAS.
IT'S DONE SO WITHOUT DETAILED
PRE-ANALYSIS PLAN.
WHICH I THINK IS A FASCINATING
APPROACH, QUITE POPULAR IN
PHYSICS.
>> I WANT TO CONTINUE THE THEME
ON INCENTIVES.
THE MORE I LISTEN TO TALKS AND
Q&A THE MORE I THINK THIS IS
JUST AS MUCH ABOUT CREDIT-WISE
AS PUBLICATION BIAS.
MAYBE WE MANAGE TO PUBLISH ALL
THESE NULL STUDIES.
IT COULD STILL LEAD TO PUBLISH
AND PERISH PARADIGM.
IT'S DISCOUNTED IF YOU PUBLISH
NULL STUDIES IN THE JOURNAL NOT
HIGH ON THE FOOD CHAIN.
I WANTED TO RUN AND IDEA PAST
YOU, WHETHER WE CAN INCENT --
SCIENTISTS MAY NOT BE INTERESTED
IN PUBLISHING THESE STUDIES SO
WHAT IF WE COULD INCENTIVIZE
PUBLISHING NULL STUDIES, THROUGH
TRAINING AND MENTORING, WE
EXPECT -- I'M NOT TALKING ABOUT
WHAT WE HEARD EARLIER, OH, TRAIN
THE NEXT GENERATION, I'M TALKING
ABOUT EXPECTING FROM LAB HEADS
THAT PART OF TRAINING AND
MENTORSHIP IS THAT TRAINEES ARE
PREPARED FOR THE NEXT CAREER
PHASE, WHATEVER THEY WANT TO DO,
BUT THAT INCLUDES OUTPUT FROM
TRAINING PERIOD.
IF THAT IS AN EXPECTATION, AND
WE ACTUALLY CHECK IT, WE ASK FOR
IT, IF EVERY TRAINEE HAS TO COME
OUT WITH SOMETHING, THAT MAY
HELP TO INCENTIVIZE PUBLISHING
THEIR PROJECT AS WELL BECAUSE
THE PUBLISH OR PERISH PARADIGM
WORKS VERY OFTEN IN BIG LABS
WHERE YOU HAVE SOME SUCCESSFUL
TRAINEES AND THE REST -- IF WE
EXPECT ALL TRAINEES HAVE A SHOT
AT SUCCESS, THAT THEY PRODUCE
SOMETHING, AN EXPERIMENT IS ONLY
DONE WHEN IT'S PUBLISHED, EVERY
TRAINING SHOULD LEAD WITH
SOMETHING THAT IS PUBLISHED,
THAT COULD BE AN INCENTIVE FOR
SCIENTISTS.
>> I THINK IT'S AN INTERESTING
IDEA.
I JUST DON'T THINK IT WILL WORK
BECAUSE I DON'T THINK WE CAN --
WHO IS THIS "WE" DOING THIS?
>> OH, SORRY.
INSTITUTIONS AND FUNDERS.
>> RIGHT, RIGHT.
>> IT WILL BE INSTITUTIONS AND
FUNDERS, THAT EXPECTATION.
WE'RE LOOKING AT YOUR TRAINING
AND MENTORING EFFORTS.
(INAUDIBLE)
>> USE THE MICROPHONES.
>> ISN'T THAT JUST SUPPORTING
PUBLISH OR PERISH CULTURE?
EVERYBODY HAS TO PUBLISH?
THIS GOES WITH THE IDEAS, WHAT
WE NEED IS LESS RESEARCH, WE
JUST NEED BETTER RESEARCH.
AND THAT'S A CHALLENGE I FOUND,
ALSO RESIDENCY PROGRAM WHO IS
WANT EVERY NEUROLOGY RESIDENT TO
PUBLISH A STUDY, MOST WILL NEVER
PUBLISH ANYTHING AGAIN, MAYBE
THEY SHOULDN'T BE FOCUSED ON
THAT.
WE SHOULD FOCUS ON GOOD QUALITY
RESEARCH RATHER THAN JUST KEEP
PROMOTING THIS PUBLISH OR PERISH
IDEA.
>> I THINK IT'S NOT ABOUT THE
QUANTITY NUMBER OF PUBLICATIONS,
LIKE IT'S ABOUT RESPONSIBILITY.
WHEN I AM USING EXPLICITLY THE
PUBLIC MONEY, I'M RESPONSIBLE TO
DISSEMINATE THAT.
THE RECENT ARTICLE, THERE'S A
LOT OF JUNK OUT THERE.
BUT OF COURSE PEOPLE WOULD ALSO
ARGUE YOU'RE GIVING ME THIS
MONEY, AND THAT EXPERIMENT DOES
NOT GO IN MY FILE DRAWER.
THE PAPER WE PRESENTED, IF WE
DIDN'T WANT TO SHOW THE RESULTS,
THERE WOULD BE CONTINUATION OF
PUBLICATIONS THAT WOULD CONTINUE
THAT.
YOU'RE CONTRIBUTING TO THAT.
>> I AGREE.
THAT'S THE IDEA BEHIND THE NEW
MANDATES, RIGHT?
IF YOUR WORK IS FUNDED BY NIH IT
HAS TO BE PUBLISHED IN A WAY
THAT'S OPEN ACCESS, AVAILABLE
IMMEDIATELY FOR EVERYBODY
BECAUSE YOU'RE A STEWARD FOR THE
TAXPAYER'S MONEY AND YOU HAVE TO
MAKE YOUR RESULTS PUBLIC.
>> THIS IS I THINK WE ALL HAD
SIMILAR LINES OF THOUGHT HERE
BUT THE WHOLE IDEA OF
PRE-REGISTRATION AND
REGISTRATION IS CLEAR, BUT I DO
SEE, AND I HAVE A ROLE IN
RESEARCH QUALITY, INTEGRITY AT
MY INSTITUTION AS WELL, I SEE A
STARK DIFFERENCE IN CLINICAL
RESEARCH OR PRE-CLINICAL, WE
DON'T SEE STUDENTS, TRAINEES,
GRAD STUDENTS, WHAT NOT, AND
MENTORS INVOLVED IN
PRE-REGISTRATION, REGISTRATION
EARLY ON.
ONE OF THE THINGS THAT COULD
HELP WITH IS ALSO THIS
NEVER-ENDING PLAN TO PUBLISH.
YOU SEE THIS GO ON, ON ONE MORE
EXPERIMENT, WE NEED TO SOLVE
THIS OTHER QUESTION, OH,
REVIEWERS ARE GOING ASK FOR
THIS, IT KEEPS GOING.
IT IS A SERIOUS DETRIMENT TO
TRAINEES, NOT ABLE TO GET WORK
PUBLISH, HELD TO THEIR
MENTOR/P.I. WANTS THEIR PAPER IN
"CELL" AND "NATURE."
NOT EVERYBODY IS GOING TO GET IN
"CELL" OR "NATURE," OTHERWISE WE
WOULDN'T HAVE THIS MANY OTHER
PUBLISHERS HERE, SOMEBODY IS
PUBLISHING IN THEIR JOURNALS.
GETTING SOMETHING OUT ON WHAT
HAS BEEN SAID HERE PUTTING FORTH
WHAT THEIR BASIS IS,
PRE-REGISTRATION, SOMETHING
THESE COMES INTO A DIFFERENT
FORMAT EVEN IF AT THE
INSTITUTION TO HAVE A
REPOSITORY.
SOMETHING THAT COULD GIVE THEM A
PROTECTION AND BE USED TO LOOK
FORWARD TO EXPERIMENTAL DESIGN,
WHAT SHOULD BE PUT FORWARD.
I OTHER THING I WANT TO SAY,
MANY HERE I CAN SAY THIS AS A
TENURED ENDOWED PROFESSOR, THERE
ARE SO MANY MYTHS ABOUT
PROMOTION AND TENURE.
WHEN I WAS ASSISTANT PROFESSOR
YOUR GUIDE SAID HAVE 15
PUBLICATIONS.
YOU NEED DOUBLE.
IT SAYS 15, WHY DO I NEED 30?
IF YOU PUBLIC NEGATIVE RESULTS
YOU'RE NOT GOING TO GET
PROMOTED.
IT MIGHT BE TRY.
PROBABLY IT'S IN BETWEEN, YOU'RE
ALSO GOING TO HAVE, IF HAVE YOU
TOP NOTCH PUBLICATIONS IN
"CELL," YOU HAVE TO HAVE
NEGATIVE RESULTS ALONG THE LINE,
OR SOMETHING THAT DIDN'T WORK.
IF YOU'RE TELLING ME YOU'RE
GOING TO WAIT UNTIL FOUR YEARS
AND PUBLISH NOTHING IN BETWEEN,
ONLY COME OUT WITH THAT, THIS IS
SOMETHING IN HERE IS NOT RIGHT,
THIS IS NOT JUST PUBLISHERS,
INSTITUTION, NOT JUST FUNDERS.
IT'S ACROSS THE BOARD WE
CONTINUE A LOT OF THIS FOLKLORE.
WHEN YOU DIG IN, SOME IS JUST
BIAS AND MYTHS WE CARRY ON.
WE HAVE TO DO MYTH BUSTING ALONG
THE WAY TOO.
>> GREAT COMMENT.
WITH A REGISTERED REPORT YOU'LL
NEVER BE EXPECTED TO DO EXTRA
EXPERIMENT AT STAGE 2.
AUTHORS BUY UNCERTAINTY, MUCH
HAS BEEN WRITTEN, REGISTERED
REPORTS PROCESS MAKES THE
TIMELINE MORE PREDICTABLE.
BECAUSE YOU KNOW IF YOU FOLLOW
YOUR PLAN, YOU'LL GET ACCEPTED.
REVIEWERS SAY ANOTHER
EXPERIMENT, YOU WOULD BE GREAT.
SURE, IT'S GREAT.
THERE HAS TO BE AN INFLUENCE.
IT'S HUGELY EMPOWERING FOR
CAREER SCIENTISTS.
>> I WORRY WITH REGISTERED
REPORTS, I'M FOR THEM, A BIG
ANSWER TO WHAT WE'RE FACING
HERE.
BUT IT'S A BARRIER, ESPECIALLY
FOR EARLY CAREER INVESTIGATORS,
IT TAKES TIME.
IT'S A HUGE TIME COMMITMENT, AND
COMMITMENTS SEEM TO BE RACKING
UP WITH COMPLIANCE FOR ANIMAL
CARE AND OTHER, YOU KNOW, FOR
HUMAN RESEARCH, ET CETERA.
NOW YOU'RE SAYING I CAN'T BE
ASKED FOR EXTRA EXPERIMENTS,
THAT ISN'T GOING TO DRAG ON,
THAT'S HELPFUL.
>> ON TIME ISSUE YOU HAVE TO GO
THROUGH STAGE 1 REVIEW PROCESS
BUT YOU SAVE TIME ON THE OTHER
END.
YOU'RE NOT GOING TO BE CHASING
JOURNALS, ONE AFTER ANOTHER,
AFTER YOU'VE RUN OUT OF FUNDING
AND YOU'RE TRYING TO PUBLISH
YOUR WORK.
THERE IS TIME SPENT AT THE
BEGINNING, WE PUT A LOT OF
EFFORT INTO SHORTENING THAT EVEN
MORE, PARTICULARLY WITH THE
SCHEDULED REVIEW TRACK.
WITH THE SCHEDULED REVIEW TRACK
WE START LINING UP REVIEW
PROCESS WHILE YOU'RE STILL
WRITING STAGE 1 MANUSCRIPTS.
WE DO THAT BASED ON A ONE-PAGE
PRO FORMA THAT YOU SUBMIT AS
AUTHOR.
BASED UPON THAT SNAPSHOT WE
RECRUIT REVIEWERS IN ADVANCE SO
WHEN YOU SUBMIT ON THE DUE DATE,
REVIEWERS ARE BOOKED IN, READY
TO GO, THE FIRST STAGE OF STAGE
1 REVIEW TAKES ON AVERAGE 18
DAYS.
>> I'VE LISTENED WITH A GROWING
SENSE OF FRUSTRATIONS, I HOPE WE
CAN LOB A BOULDER.
THIS IS ABOUT PUBLISHING, WE
HEARD SYSTEM OF MANUSCRIPTS HAS
GIVEN RISE IN THE FIRST PLACE.
I THINK THE PREMISE IS WRONG.
JUST TO RESPOND TO A COUPLE
THINGS I'VE HEARD IN THE
DISCUSSION, THE THING WE'RE
MISSING, I HAVE MISSED IN THIS
CONVERSATION IS RECOGNITION OF
HUMAN NATURE.
SO WE HEARD PEER REVIEW IS
IMPORTANT TO GIVE CREDIT,
ABSOLUTELY IT'S IMPORTANT TO
GIVE CREDIT FOR PEER REVIEW.
NO ONE IS GOING TO HIRE YOU
BECAUSE YOU'RE GREAT AT PEER
REVIEW, OKAY?
WE HEARD THERE'S CAPACITY FOR
WRITING NEGATIVE RESULTS IN THE
SYSTEM.
YOU SAID THIS.
WHERE IS THAT CAPACITY?
EVERYBODY I KNOW WORKING IN
ACADEMIA IS OVERLOADED WITH
WORK.
WE HAVE TO THINK ABOUT THAT
ASPECT.
AND NEGATIVE RESULTS, ABSOLUTELY
NEGATIVE RESULTS ARE IMPORTANT
BUT THEY ARE JUST NOT AS
EXCITING AS POSITIVE RESULTS.
OR DISCOVERING SOMETHING NEW,
NOBODY EVER RAN HOME TO SAY,
HEY, I DISCOVERED MY HYPOTHESIS
IS WRONG TODAY.
OKAY?
SO, YOU KNOW, WE ABSOLUTELY NEED
TO MAKE IT EASIER, I'M A FAN OF
PRE-REGISTERED REPORTS, THE
INNOVATION THAT MAKES A
DIFFERENCE, YET TO INFILTRATE
MANY FIELDS AND I THINK THERE
ARE STRUCTURAL ISSUES AROUND
THAT BUT I DO THINK WE NEED TO
INTRODUCE AN ELEMENT OF
COMPULSION OR OBLIGATION BECAUSE
YOU ARE NOT EVER GOING TO CREATE
A POSITIVE INCENTIVE TO PUBLISH
NULL RESULTS IN MY VIEW.
>> WHAT WE HAVE MISSING IN OUR
PANEL IS THE VIEW FROM THE
FUNDERS, OR THE PEOPLE WHO ARE
ON THE AWARD AND RECOGNITION
COMMITTEES.
SO THOSE ARE -- I CAN DO SO MUCH
AS A JOURNAL EDITOR, HAVE
POLICIES, MAKE IT EASIER, REDUCE
BARRIERS AS FAR AS WE CAN.
THAT'S AT THE END STAGE.
IT WOULD BE EXCITING TO HEAR
WHEN RESEARCH IS FIRST CONCEIVED
AND WHEN PEOPLE ARE WRITING UP
THE RESULTS HOW WE'RE GOING TO
DO THAT.
>> THAT'S AS GOOD OF
ADVERTISEMENT I CAN THINK OF FOR
LATER SESSIONS LATER TODAY AND
TOMORROW AND WORKING GROUPS.
WE LOOK FORWARD TO MORE
SESSIONS.
RIGHT NOW WE'RE ON BREAK, WE'LL
START AGAIN AT 2:45.
THANK YOU.
>> WELCOME BACK FROM THE BREAK,
THIS SESSION IS ABOUT
NON-TRADITIONAL FORMATS.
I'M ALEX FREEMAN, DIRECTOR OF
OCTOPUS.
RATHER THAN EXPLAINING, I PUT
LEAFLETS AROUND THE ROOM.
OBVIOUSLY ONE WAY TO GET RID OF
PUBLICATION BIAS IS GET RID OF
GATE KEEPERS, JOURNAL EDITORS.
AS WE'VE HEARD, THAT'S NOT THE
SOLUTION TO EVERYTHING BECAUSE
THEY ARE NOT THE ONES WHO ARE
CAUSING THIS PUBLICATION BIAS.
BUT I THINK EVERYBODY IN THIS
PANEL IS TALKING ABOUT THINGS
THAT DON'T INVOLVE BOTH INITIAL
GATE KEEPERS BUT ALSO WE'RE
TALKING ABOUT HOW TO CHANGE THE
MIND SETS OF RESEARCHERS WHEN WE
THINK ABOUT WHAT PUBLICATION IS
IS, WHO IS IT FOR, WHAT IS IT
FOR?
PROGRAMS LIKE OCTOPUS FIT
ALONGSIDE, TO DO SOMETHING
DIFFERENT, THAT IS TO
INCENTIVIZE BEST RESEARCH
PRACTICES ABOVE EVERYTHING ELSE.
OCTOPUS IS FUNDED BY UKRI AND
SHARES MANY FEATURES WE'LL TALK
ABOUT ACROSS THE PLATFORMS LIKE
HAVING IMMEDIATE PUBLICATIONS
WITHOUT THE GATE KEEPERS, SO
IT'S QUICK AND EASY.
HAS OPEN POST PUBLICATION PEER
REVIEW WHICH WE'LL TALK ABOUT
PROS AND CONS OF THAT.
IT HAS SMALLER UNITS OF
PUBLICATION SO IT DOESN'T
PUBLISH PAPERS.
AGAIN WE'LL BE TALKING ABOUT
MICROPUBLICATIONS BUT OCTOPUS'S
PUBLICATIONS I SAY IT'S A BIT
LIKE TAKING REGISTERED REPORTS
TO ULTIMATE CONCLUSION, WHICH IS
BREAKING UP THE PROCESS NOT THUS
BEFORE AND AFTER RESULTS BUT
DIFFERENT STAGES, EACH CAN BE
PUBLISHED, PEER REVIEWED, TREATS
PEER REVIEWS AS PUBLICATION IN
THEIR OWN RIGHT.
WE'RE GOING TO TALK ABOUT
METRICS OF QUALITY INSTEAD OF
IMPACT AND POPULARITY.
ABOUT AVOIDING STORY TELLING,
THINGS THAT CAN DRIVE
QUESTIONABLE RESEARCH PRACTICES,
IN THE DRIVE FOR POSITIVE
RESULTS.
BECAUSE GOOD STORY TELLING ISN'T
GOOD SCIENCE, SO WE WANT TO LOOK
AT HOW PUBLISHING PLATFORM CAN
FOCUS ON GOOD SCIENCE AND WHAT'S
IMPORTANT TO THE COMMUNITY,
RESEARCHERS, TO SOCIETY.
AND I THINK FUNDAMENTAL TO IT
REALLY TO ALL OF THESE PLATFORMS
IS RETHINKING WHAT WE'RE DOING
WITH PUBLICATIONS, AND THINKING
INSTEAD OF BEING INDIVIDUALS
THINKING ABOUT CAREERS, CAREER
STAGES, GETTING THOSE BOXES TO
WHERE YOU WANT TO BE IS TO THINK
COLLABORATIVELY, WE KNOW WHEN WE
START OUT IN SCIENCE WE WANT TO
BE ABLE TO CONTRIBUTE POSITIVITY
TO SOCIETY, FINDING OUT ABOUT
THE WORLD AND HELPING PEOPLE.
TO DO THAT IN A COLLABORATIVE
WAY TO PUBLISH INDIVIDUAL --
PUBLISH AN IDEA, SOMEBODY CAN
PUBLISH A PROTOCOL HOW TO TEST
THAT IDEA, THREE OTHER PEOPLE
COULD COLLECT DATA ACCORDING TO
THOSE PROTOCOLS AND THEN LINK
THOSE TO PROTOCOLS.
SO THINKING IN A COLLABORATIVE
WAY WHERE PEOPLE AREN'T THINKING
ABOUT WHAT ARE THE FINDINGS AND
IMPACTS BUT INSTEAD THINKING
ABOUT WHAT CAN I DO TO
CONTRIBUTE TO OVERALL PICTURE OF
KNOWLEDGE GENERATION.
AND THE FINDINGS ARE LIKE AN
EMERGENT PROPERTY OUT OF THIS.
THESE ARE PRINCIPLES THAT I
THOUGHT ABOUT WHEN I DESIGNED
OCTOPUS, HAPPY TO TALK FURTHER
ABOUT, ALSO PRINCIPLES THAT I
THINK ARE IN COMMON ACROSS A LOT
OF AUTHENTIC PUBLISH PLATFORMS
WE'RE GOING TO TALK ABOUT,
NON-TRADITIONAL FORMATS.
FIRST SPEAKER HAS PRE-RECORDED
HER TALK, TALKING ABOUT ARCADIA,
TRADEOFFS WITH TOOLS,
MICROPUBLICATION BIOLOGY,
ARCADIA SCIENCE, EXPERIENCES
FROM THEIR NEW FORMATS, AND
SUZANNA FROM THE NATIONAL
INSTITUTE ON AGING TALKING ABOUT
THEIR EXPERIENCE, HOW THEY ARE
ENCOURAGING FORMATS AND PROS AND
CONS.
HOPEFULLY WE CAN START WITH
JOY'S RECORDING FIRST.
>> GOOD MORNING, EVERYBODY.
GOOD AFTERNOON, GOOD EVENING,
WHEREVER YOU ARE.
FOR THOSE IN THE U.S., GOOD
MORNING.
IT'S UNFORTUNATE I COULD NOT
MAKE IT TO THE NIH MEETING BUT
THANK YOU FOR ALLOWING ME TO
SHARE THIS RECORDING ON MY VIEWS
ON THE PUBLISHING OF NULL
RESULTS, ESPECIALLY WITH
PRE-PRINTS.
I'M EXECUTIVE DIRECTOR OF
TRAINING CENTER IN
COMMUNICATIONS, WE'RE BASED AT
THE UNIVERSITY OF NAIROBI IN
KENYA, WE SUPPORT RESEARCHERS ON
HOW THEY CAN IMPROVE THEIR
RESEARCH.
THE POINT OF TODAY'S MEETING,
HOW DO PRE-PRINT PLAY
SIGNIFICANT ROLE IN REPORTING
NULL RESULTS IN AFRICA.
WITH THE RIGHT OF PRE-PRINT
COVERS WE HAVE AFRICA ARCHIVE,
WHICH IS A CONTINENTAL PRE-PRINT
PLATFORM, WE ALSO SEE
UNIVERSITIES PRODUCING PRE-PRINT
COVERS, A PRE-PRINT SERVER,
PRETORIA, AND ONE FOR ADDIS
ABABA UNIVERSITY.
WITH THIS RISE OF PRE-PRINTS, A
WAY TO LOOK WITH PHENOMENON OF
GROWTH OF PRE-PRINTS AND HOW
THEY ARE CONTRIBUTING IN THE
REPORTING OF NULL RESULTS.
FOR OBVIOUS REASONS RAPID
DISSEMINATION OF INFORMATION,
REDUCING PUBLICATION BIAS, AND
PUBLISHING NULL RESULTS IN
PRE-PRINT MAKES IT EASY FOR
COLLABORATION AND INCREASED
TRANSPARENCY.
INCREASED TRANSPARENCY YOU'RE
LOOKING AT POSTING NULL RESULTS
FOR RESEARCH WHICH ALLOW
RESEARCHERS TO SEE THE FULL
SPECTRUM OF FINDINGS RELATED TO
A PARTICULAR TOPIC, INCLUDING
THOSE THAT MAY NOT HAVE BEEN
PUBLISHED IN PEER-REVIEWED
JOURNALS, DUE TO PUBLICATION
BIAS.
SO THE RIGHT OF PRE-PRINTS,
CONTINENTAL OR INSTITUTIONAL,
ARE AIDING IN MITIGATING THROUGH
INCREASED TRANSPARENCY BY
PUBLISHING OF NULL RESULTS.
THIS TRANSPARENCY FOCUS, A MORE
COMPLETE UNDERSTANDING OF THE
SCIENTIFIC EVIDENCE BASE AND CAN
HELP PREVENT RECOGNITION OF
MISLEADING OR INCOMPLETE
INFORMATION SO LET'S LOOK FROM
AFRICAN PERSPECTIVE, A COMPLETE
PICTURE OF THE RESEARCH.
RESEARCH.
WE REALLY ACKNOWLEDGE POWER OF
PRE-PRINTS IN CONTRIBUTING TO
THE TRANSPARENCY AND VISIBILITY
OF NULL RESULTS AND MITIGATING
SKEWED PUBLICATION BIAS BASED ON
OUTCOMES OUT OF AFRICA.
THEN SPECIFIC DATA, THE TRENDS
AFRICAN RESEARCHERS PUBLISHING
NULL RESULTS, SEVERAL
INITIATIVES IN PROMOTING
TRANSPARENCY AND RESEARCH
TRANSPARENCY.
WHICH COULD POTENTIALLY LEAD TO
AN INCREASE IN PUBLICATION OF
NULL RESULTS.
PLATFORMS ADVOCATE FOR OPEN
ACCESS TO PROMOTE COLLABORATION
AMONG AFRICAN SCIENTISTS AND
RESEARCHERS.
AND WE HAVE INITIATIVES LIKE THE
COALITION OF AFRICAN RESEARCH
AND INNOVATION WHERE THEY ARE
WORKING TO STRENGTHEN CAPACITY
TO PROMOTE ACROSS THE CONTINENT.
WITH TRAINING CENTER IN
COMMUNICATION WE'RE ADVOCATING
FOR THIS ADOPTION OF OPEN STANCE
PRINCIPLES, WHETHER IT'S
REGIONAL, NATIONAL,
INSTITUTIONAL, AND THIS WILL AID
IN HAVING INSTITUTIONAL SYSTEMS
SET IN PLACE WHEREBY WHEN IT
COMES TO PUBLISHING, YOU HAVE
ALSO NULL RESULTS, BEYOND JUST
POSITIVE RESULTS THAT COME OUT
OF RESEARCH.
AT THE END OF THE DAY WE'RE
LOOKING AT TRANSPARENCY, TO GET
PUBLICATION BIAS, NOT ONLY THE
RISE OF PRE-PRINT REPOSITORY,
JUST THEY ARE ALSO INITIATIVES
THAT ARE TRYING TO PROMOTE
ADOPTION OF OPEN SCIENCE WITHIN
THE CONTINENT, WILL HAVE IMPACT
ON PUBLICATION OF NULL RESULTS,
BECAUSE OF DEMAND FOR
TRANSPARENCY AND DEMAND FOR
REDUCTION OF PUBLICATION BIAS
COMING OUT OF THE AFRICA.
THANK YOU SO MUCH FOR LISTENING
TO ME.
I HOPE YOU ENJOY THE REST OF
YOUR WATCHING.
GOOD-BYE.
>> THANK YOU FOR INVITING ME.
REALLY DELIGHTED TO BE HERE.
AS OF LAST MONTH, ASSISTANT
PROFESSOR AT SCHOOL OF MEDICINE,
DURING MY POSTDOCTORAL WORK
GAME INTERESTED IN MENTORSHIP,
IMPROVING FACULTY HIRING AND
PUBLISHING, AND SO I HAVE BEEN
INVOLVED WITH A FEW COMMUNITIES,
TODAY I HELP TO SHARE WITH YOU
MORE EARLY CAREER PERSPECTIVES
AND COMMUNITIES OF EARLY CAREER
RESEARCHERS PERSPECTIVE ON TWO
ASPECTS, ONE IS THEIR ROLE IN
OPEN SCIENCE AND IN PUBLISHING
ALL THEIR FINDINGS AT THE SPEED
OF RESEARCH, SECOND ON THE VALUE
OF THIS FOR EARLY CAREER
RESEARCHERS.
MY FIRST INVOLVEMENT, WE PUT
FORWARD A PERSPECTIVE ON HOW
PRE-PRINTS ARE ACTUALLY MORE
THAN JUST ALLOWING US TO
DISSEMINATE RESEARCH FAST.
THEY ARE SUPPORTING A PLETHORA
OF REALLY POSITIVE ASPECTS IN
OUR CAREERS AND RESEARCH LIVES.
OVER THE PAST FEW YEARS, ALL OF
THE STAKEHOLDERS, FUNDERS HAVE
APPRECIATED THIS APPROACH AND
HELPED RESEARCHERS IMPROVE THEIR
CAREERS IN MANY WAYS BY HOSTING
PRE-PRINTS.
SO THE SECOND COMMUNITY THAT I
HAVE BEEN INVOLVED WITH HAS BEEN
THE LARGE COMMUNITY OF 7,000
POSTDOCS, FOR THE PAST SEVEN
YEARS WE'VE HAD A LOT OF GOOD
DISCUSSIONS, THE COMMUNITY IS
FOCUSED ON THE LATE STAGE OF
CAREER, WHICH IS FOR POSTDOCS
INTERESTED IN ACADEMIC CAREER,
APPLYING FOR FACULTY POSITION
AND THESE ARE SOME OF THE
HIGHLIGHTS OF THE CONVERSATIONS
ON PUBLISHING DIFFERENT TYPES OF
FINDINGS AND PUBLISHING THEM AS
PRE-PRINTS, SO QUITE A FEW OF
THEM THINK THAT EVERYTHING
SHOULD BE PRE-PRINTED TO SHOW
PROGRESS, AND THERE ARE EXAMPLES
OF PEOPLE WHO HAVE BEEN ABLE TO
FIND FACULTY POSITIONS AND GET
INTERVIEWS IN THE JOB MARKET
BECAUSE THEY HAVE PRE-PRINTS,
BECAUSE POST DOCTOR CAN BE VERY
SHORT, ALSO QUESTIONS ON LISTING
PRE-PRINTS ALONG OTHER
PUBLICATIONS AND MANY THINK THAT
PRE-PRINTS ARE THE BEST
REPRESENTATION OF THEIR WORK,
ACTUALLY.
THERE ARE EVEN QUESTIONS AND
DISCUSSIONS ON WHETHER RESEARCH
FINDINGS SUCH AS THOSE THAT ARE
PRESENT AT CONFERENCES WHICH
TEND TO BE SHORTER, CAN BE
PRE-PRINTED, AND SHOWN AS
EVIDENCE OF PRODUCTIVITY, SO
EARLY CAREER RESEARCHERS SUCH AS
POSTDOCTORAL RESEARCHERS ARE
VERY INTERESTED IN PUBLISHING
THEIR FINDINGS.
NOW, I WAS ALSO INVOLVED IN
STUDY WITH FELLOW POSTDOCS ON
THE FUTURE PI SLACK PLATFORM.
WE LOOKED AT A FEW HUNDRED
POSTDOC, NOT NECESSARILY ON THE
PLATFORM BUT YOU THERE A --
THROUGH A SURVEY AND FOUND OVER
55% OF THEM HAD ACTUALLY POSTED
PRE-PRINTS.
THERE ARE ALSO OTHER STUDIES,
FOR EXAMPLE, ON THE RIGHT WHERE
IN THE FIELD OF ECOLOGY LOOKING
AT 500 PRE-PRINTS FOUND IT IS
POSITIVELY ASSOCIATED WITH EARLY
CAREER STATUS IN ECOLOGY.
ON THE LEFT OF THAT FIGURE A, IT
SHOWS PUBLICATIONS THAT ARE
PRE-PRINTED WITH EARLY CAREER
RESEARCHER AS FIRST AUTHOR.
ON THE RIGHT EARLY CAREER
RESEARCHER AS CORRESPONDING
AUTHOR.
YOU CAN SEE WHEN EARLY CAREER
RESEARCHERS HAVE THE POWER AND
BETTER CONTROL OVER THE FATE OF
THE PUBLICATION, THEY ARE VERY
MUCH INTERESTED IN POSTING
PRE-PRINTS AS OPPOSED TO WHEN
THEY ARE PRE-PRINTING AS A FIRST
AUTHOR.
IN THE SURVEY WE RAN THESE
FIGURES ARE FROM THAT PAPER,
THROUGH QUALITATIVE RESPONSES WE
FOUND PRE-PRINTING AS YOU CAN
SEE IN THE WORD CLOUD ASSOCIATED
WITH THE APPLICANT'S
PERSPECTIVE, WITH POSITIVE
OUTCOME, FACULTY JOB MARKET, AND
ON THE RIGHT A SAMPLE FOR THE
SEARCH COMMITTEE MEMBERS FOR
VARIOUS MODES.
YOU CAN SEE ABOUT 60% OF THEM
APPRECIATED PRE-PRINTS LISTED ON
THE C.V. OF THE JOB APPLICANTS.
FINALLY, THERE ARE MULTIPLE
STUDIES THAT SHOW THAT
PRE-PRINTING IS ASSOCIATED WITH
BETTER CITATIONS AND ATTENTION
TO THE FINDINGS, AND ALSO THIS
STUDY BY FRASEE FOUND THEY GET A
LOT OF SOCIAL MEDIA ATTENTION
AND FOUND THAT YOUNGER AUTHORS
TEND TO SUBMIT MORE OF THE
bioRxiv PRE-PRINTS AND
bioRxiv DEPOSITED PAPERS, SO
IT MAY IN FACT THAT EARLY CAREER
RESEARCHERS REALLY EMBRACE OPEN
SCIENCE AND ACTUALLY DRIVE IT.
I HOPE WE CONTINUE DISCUSSIONS
ON THE ROLE OF EARLY CAREER
RESEARCHES AND STAKEHOLDERS IN
THIS.
THANKS.
>> I'M PROFESSOR OF BUSINESS AND
MARKETING AT COLUMBIA, THE AREA
I SPECIALIZE IN IS CONSUMER
RESOURCES, IN THE AREA OF SOCIAL
SCIENCES, BULK HAS TO DO, FOR
EXAMPLE, WITH WHAT MAKES
CONSUMERS MORE LIKELY TO ADOPT
NEW TECHNOLOGIES, WHAT MADE THEM
RESPOND MORE TO CERTAIN MESSAGES
OR OTHER MESSAGES, AND SO ON AND
SO FORTH.
MOST OF THE WORK WE DO IS
EXPERIMENTAL, WE DO USE OTHER
TYPE OF PARADIGMS AS WELL.
IF I WAS TO CHARACTERIZE THE
TYPE OF RESEARCH THAT WE DO,
WE'VE HEARD SEVERAL TIMES TODAY
SOME KIND OF A CONTINUUM BETWEEN
CLINICAL AND PRE-CLINICAL OR
EXPLORATORY VERSUS CONFIRMATORY,
WE SORT OF HAVE THE SENSE OF
WHAT THAT MEANS.
EVEN THOUGH WITHIN OUR FIELD
MANY PEOPLE IN THE FIELD WOULD
NOT AGREE WITH MY
CHARACTERIZATION BUT WE TENDING
TO EXPLORATORY IN OTHER
DISCIPLINES THAN WE THINK WE
ARE.
SO, A NUMBER OF YEARS AGO I WAS
ASKED TO REFLECT, I WAS
PRESIDENT OF SOCIETY FOR
CONSUMER PSYCHOLOGY, AND WE HAD
REPRODUCIBILITY CRISIS, AND AT
THAT TIME AS PRESIDENT OF THE
SOCIETY I WAS ABLE TO PUSH FOR
MORE OPEN SCIENCE PRACTICES
BEING IMPLEMENTED IN THE
PRIMARY -- ONE OF THE PRIMARY
JOURNALS IN OUR FIELD WE WERE
ABLE TO CONTROL, BUT THE LAST
FEW YEARS LATER, I WAS ASKED TO
THINK ABOUT, OKAY, WHAT DO YOU
THINK ABOUT PRE-REGISTRATION?
THIS IS A TOPIC THAT HAS BEEN
MENTIONED A NUMBER OF TIMES
WITHIN THE TWO-DAY SESSION, AND
AS I WAS THINKING ABOUT IT, THE
THINKING ABOUT THE BIG PICTURE
WHAT IS GOOD FOR THE FIELD, NOT
JUST NECESSARILY WHAT IS GOOD
FOR THE RESEARCHER AND SO ON AND
SO FORTH.
AS I THOUGHT ABOUT THIS, I ENDED
UP HAVING A FEW OF PERHAPS LESS
GUNG HO, PERHAPS MORE SKEPTICAL
OF THE PROMISES OF
PRE-REGISTRATION, EVERYTHING
ELSE EQUAL TO USE A PHRASE THAT
NEIL USED, CONDITIONAL ON THE
RESEARCH BEING EQUAL,
PRE-REGISTRATION APPLIES.
THE QUESTION IS ONCE YOU THINK
ABOUT IMPLEMENTING AS A POLICY
OR MANDATE POLICY IN NEW
SCIENTIFIC PRACTICE OR NORM,
WHAT ARE SYSTEMIC CONSEQUENCES
OF A CHANGE IN POLICY.
IN AN ARTICLE IN CONSUMER
PSYCHOLOGY I TRIED TO STRUCTURE
A LITTLE BIT WHAT IS THE BIG
PICTURE OF SCIENCE WHAT WE'RE
TRYING TO DO HERE, AND THE WAY
I'M THINKING ABOUT IT, OKAY,
ESSENTIALLY PRE-REGISTRATION IS
JUST A TOOL.
THERE'S A LOT OF GOOD SCIENCE
TOOLS, DATA ANALYSIS, PEER
REVIEW PROCESS, WHAT DO YOU CALL
IT, PRE-REPORT?
AND PRE-PRINTS, THESE ARE ALL
TOOLS.
THESE ARE JUST TOOLS.
AND THE TOOLS ONLY VALUABLE TO
THE EXTENT THEY HELP US FULL
FELL A SERIES OF GOALS WE'RE
TRYING TO ACHIEVE COLLECTIVELY.
THE GOALS, WE WANT TO CONTROL
FOR TYPE 1 ERROR, THAT'S PART OF
THE REASON PEOPLE WERE CONCERNED
ABOUT P-HACKING AND SO ON, BUT
ALSO TALKED ABOUT FACTUAL
ERRORS, I DON'T WANT TO CREATE A
SYSTEM TWO AREAS INFLATED
BECAUSE YOU TRIED TO CONTROL
TYPE 1 ERROR, AND OTHER THINGS
THAT MATTER AS WELL IN MY FIELD,
MORE APPLIED, MOST OF THE CON
THE CITY CONSTITUENTS ARE
BUSINESSING, PEOPLE WHO COULD
POTENTIALLY USE WHAT WE HAVE,
THERE ARE ISSUES OF WHAT WE CALL
RELEVANCE, ARE YOU STUDYING
RESEARCH QUESTION THAT'S
SOMETHING THAT BUSINESSES, FOR
EXAMPLE, POLICYMAKERS FIND TO BE
IMPORTANT, THIS IS REAL
QUESTION, NOT SOMETHING TRIVIAL.
THERE'S AN ISSUE OF
CROSS-TRANSPARENCY WHICH OPEN
SCIENCE HELPS IN THAT,
GENERALIZABILITY, A CRITERIA,
ARE WE DISCOVERING SOMETHING WE
DIDN'T KNOW ABOUT?
IT'S IMPORTANT WHEN YOU THINK
ABOUT THIS KIND OF SYSTEM
CHANGE, POLICY CHANGE YOU WANT
TO IMPLEMENT, WHAT ARE THE TOTAL
EFFECT OF HAVING THESE NEW
POLICIES IN PLACE.
AND THIS IS WHERE -- I DO THINK
THIS WILL INFORM THE BACKGROUND
OF THE DISCUSSION, I THOUGHT
ABOUT, OKAY, WE ALSO HAVE
POTENTIALLY SOME DOWNSTREAM
UNINTENDED ADVERSE CONSEQUENCES.
ONE OF THE MAIN ONES I WAS
CONCERNED ABOUT AS THE PERSON
WHO IS BEING IN MY OWN FOR A
LONG TIME, PRE-REGISTRATION IF
IT BECAME A MANDATE WOULD CHANGE
THE NATURE OF THE RESEARCH
QUESTIONS THAT PEOPLE STARTED
INVESTIGATING.
INSTEAD OF ADDRESSING QUESTIONS
MOST USEFUL, MOST IMPORTANT,
MOST CRITICAL, THEY WILL MOVE
THE RESEARCH AGENDA TO THINGS
WHERE THEY EXPECT IT TO BE ABLE
TO FIND SUPPORT FOR WHATEVER
THEY TRY TO TEST AND COULD DO
THAT ON THE (INAUDIBLE) COST.
QUANTITATIVE ANALYSIS, OVER THE
LAST FEW YEARS OBSERVED RESEARCH
PUBLISHED IN MY DISCIPLINE WHERE
A LOT OF RESEARCH BEING DONE
ONLINE USING MECHANICAL
SURROGATES, RELATIVELY CHEAP TO
RUN, TO RERUN, SO ON AND SO
FORTH.
IF WE CAN PRE-REGISTER WHERE WE
ONLY END UP WITH EASY TO RUN
STUDIES THAT DON'T NECESSARILY
TEACH US ANYTHING, THAT WAS THE
KIND OF THING NOW, SO THAT'S
WHERE CONDITIONAL ON RESEARCH
BEING THE SAME IN TERMS OF
QUALITY, PRE-REGISTRATION HELPS
BUT YOU HAVE TO ALSO SAY ARE WE
CREATING SYSTEMS WHERE THERE ARE
COSTS THAT WERE NOT ANTICIPATED.
ANOTHER COST I FORESAW WAS
HAPPENING, PEOPLE READING
PRE-REGISTRATION AS HUERISTIC.
IT'S FOR RESEARCH THAT MUST BE
GOOD, IF YOU STOP PRE-REGISTER
MUST BE BAD, IN FACT YOU WANT TO
MAKE SURE YOU EVALUATE THE
RESEARCH IN ITS TOTALITY, THAT'S
THE SAME KIND OF THINKING YOU
NEED TO APPLY WHEN YOU INTERPRET
SIGNIFICANCE, NOT BECAUSE IT'S
SIGNIFICANT BELOW THE .05 LEVEL,
OR ABOVE IT'S BAD, THE SAME
THING HAPPENS, DICHOTOMOUS
THINKING USING SIMPLE
HUERISTICS, GOOD OR BAD
RESEARCH, HELPFUL, AND WHATEVER
PRACTICES WE COLLECTIVELY TRY TO
DEVICE HAS TO THINK ABOUT THE
BIG PICTURE, BALANCING ALL THE
GOALS THAT WE'RE COLLECTIVELY
PURSUING AND INTERESTED IN.
THANK YOU.
>> GREAT.
HELLO.
THANKS FOR INVITING ME TO
PARTICIPATE IN THIS INTERESTING
WORKSHOP.
I LOOK FORWARD TO SEEING THINGS
CONVERGE TOMORROW TO SOME
POSITIVE RESULTS.
WELL-DEFINED NEGATIVE RESULTS.
IN THE KIND OF MECHANISTIC BASIC
WORLD I LIVE IN, MOLECULAR CELL
BIOLOGY, NEURAL CIRCUITS, ET
CETERA, WE HAVE A TYPICAL
STUDIES, A LOT OF EXPERIMENTS,
AND SOME ARE POSITIVE AND SOME
ARE NEGATIVE.
SO EVEN IN ONE FIGURE THERE
MIGHT BE A TEST OF 30 GENES, TWO
OF WHICH GIVE RESULTS THAT ARE
POSITIVE AND FOLLOWED UP ON, THE
OTHER ONES ARE LEFT, THOSE ARE
NEGATIVE RESULTS.
WE STILL COUNT THAT AS A PAPER
WITH POSITIVE RESULTS.
FROM THE BIG DATA PERSPECTIVE,
I'M A PROPONENT OF THE IDEA THAT
AGGREGATION OF MANY SMALL
EXPERIMENTS, AGGREGATED PROPERLY
IN THE KNOWLEDGE BASE WHICH I
SPEND A LOT OF TIME DOING,
BECOMES BIG DATA, YOU CAN USE
MACHINE LEARNING AND A.I. TO
COMPUTE THAT INFORMATION.
WE WANT TO AGGREGATE AS MUCH
POSITIVE AND NEGATIVE DATA.
SO, THIS IS ONE OF THE
MOTIVATIONS, WE REALIZED ABOUT
HALF THE EXPERIMENTAL RESULTS,
IN SOMEONE'S Ph.D. THESIS,
WERE NEVER PUBLISHED.
IT'S FILE DRAWER PROBLEM WE
HEARD ABOUT.
ACCUMULATING THOSE IS A GOOD
VALUE, THE EXPERIENCE HAS BEEN
GOOD.
WE EVEN HAD WHAT I CALL VECRs,
EARLY CAREER RESEARCHERS, HIGH
SCHOOL STUDENTS, UNDERGRADUATES,
SOMETIMES COMPLETE ON THEIR OWN
AND PUBLISH IN PEER-REVIEWED
FORMAT.
ONE CRITERIA FOR
MICRO-PUBLICATION BIOLOGY,
TRYING TO BALANCE OUT
TRADITIONAL THINKING ABOUT
PUBLICATION AND USING HUMAN
NATURE, WHAT'S CURRENTLY THERE,
PEOPLE WANT CREDIT AND WANT PEER
REVIEW, THEY WANT UNITS OF A
CERTAIN SIZE, I WANT IT TO BE
MICRO.
MICRO PUBLICATIONS ARE DECI OR
MILLI, BUT NOT MICRO.
IT'S A DIFFERENT ISSUE.
SO THAT WE CAN CAPTURE THAT.
SO THE IDEA IS JUST TO REALLY
CONTROL THE PLATFORM AND MAKE IT
EASIER FOR PEOPLE TO GET OUT,
WHAT WE'RE DOING IS SINGLE
FIGURE PUBLICATIONS.
NOW, WE HAD HOPED THAT THESE
WOULD BE REALLY SIMPLE AND ONE
EXPERIMENT.
TURNS OUT PEOPLE WANT TO PUT
MORE IN THERE.
THAT'S JUST -- IT'S OKAY.
IT MAKES IT HARDER TO REACH THE
OTHER GOAL TO TRY TO CAPTURE ALL
THAT INFORMATION WITH ONTOLOGY
DIRECT INSERTIONS, CONNECTED
INSERTIONS, THAT'S LESS
SUCCESSFUL.
IN TERMS OF GETTING RESEARCH OUT
IT'S BEEN SUCCESSFUL.
MY PERSONAL FIELD, C. ELEGANS, A
WORM THAT'S THE DARLING OF LAB
SCIENTISTS, I THINK IT'S
STATISTICALLY SIGNIFICANT, I'M
P-HACKING, 5% MORE ARE
PUBLISHED, WE'RE GETTING EXCESS
PUBLICATIONS LAST FOUR YEARS
BECAUSE OF THIS.
THAT'S THE RESULTS THAT WOULDN'T
BE THERE.
THAT'S A SUCCESS.
I THINK THAT'S THE MAIN POINT,
LITTLE DATA BECOMES BIG DATA.
NEGATIVE RESULTS ARE IMPORTANT
BUT THERE'S ACTUALLY A GRANULAR
LEVEL STUDIES HAVE POSITIVE AND
NEGATIVE ONES, AND THEY ARE
REALLY IMPORTANT TO PULL OUT AND
TO GET INTO PUBLIC DOMAINS.
>> HI, EVERYONE.
I'M CO-FOUNDER AND CHIEF
SCIENTIFIC OFFICER OF ARCADIA
SCIENCE, HEAD OF OPEN SCIENCE AT
A NON-PROFIT WHICH I WON'T TALK
ABOUT TODAY BUT WE'RE FUNDING
BRAND NEW ZIRTA EXPERIMENTS IN
PUBLISHING, IF ANYONE HAS CRAZY
IDEAS TALK TO ME LATER.
IT'S A SCIENCE COMPANY, TRYING
TO RETHINK HOW WE DO SCIENCE,
HOW WE SUPPORT AND FUND SCIENCE,
HOW WE COMMUNICATE SCIENCE.
IT'S THE LARGER BROADER GOAL FOR
US.
AND ONE OF THE THEMES I WANTED
TO BRING UP, A KEY ISSUE FOR US,
IS SOMETHING THAT I WANTED TO
CHALLENGE THE NARRATIVE AND
FUNCTIONS AROUND SOMETHING THAT
MAYBE TED SAID EARLIER, WE'RE A
FOR-PROFIT COMPANY AND WE'RE
ALSO AN OPEN SCIENCE
ORGANIZATION.
WE ARE TRYING TO PUSH THE NEEDLE
AND CHALLENGE THIS IDEA THAT IS
IT THE CASE THAT THESE ARE LIKE
ALWAYS INTENTIONED OR WHAT WE
BELIEVE IS STRONG SYNERGY WITH
ONE ANOTHER.
WE'RE ALSO TRYING TO PUSH
AGAINST -- ALL OUR WORK, THE
SCIENCE IS DONE IN HOUSE,
PUBLISHED ON OUR OWN PLATFORM
OUTSIDE ACADEMIC JOURNAL SYSTEM.
IT GETS BACK TO SOMETHING
ALEXANDER SAID, WHY WE GOT INTO
THIS FOR COLLECTIVE SCIENTIFIC
BENEFIT, REGARDLESS OF ANY
SECTOR YOU'RE TALKING ABOUT,
SORT OF A LOT OF TIMES WE'RE NOT
PUBLISHING IN ACADEMIA BECAUSE
WE'RE HOLDING ON TO THINGS FOR
CREDIT OR REWARDS AND IN THE
PRIVATE SECTOR MIGHT BE TO
PROTECT PROFITS, WE LIKE TO
CHALLENGE THIS ASSUMPTION, CAN
WE DO THIS OUTSIDE ASSUMPTIONS
AND PUSH THE NEEDLE AND GET
EVERYONE TO RELEASE MORE SCIENCE
BECAUSE WE ALL NEED IT FOR
ACCELERATING ADVANCEMENTS FOR
ALL OF SOCIETY.
WE'RE USING A PUBLIC OPEN
PLATFORM BUILD BY KNOWLEDGE
FUTURES GROUP OUT OF M.I.T.
I PICKED THIS BECAUSE IT WAS
SOMETHING THAT COULD BE
AVAILABLE TO OTHERS SO WHEN
PEOPLE SAY HOW DO WE DO WHAT YOU
DO, IF IT WERE TO CATCH ON, WE
CAN POINT THEM AND GIVE THEM A
SOLUTION.
WE TRY TO PUBLISH EARLY AND
OFTEN IN A MODULAR WAY.
POST-PUBLICATION REVIEW MEANS WE
WANT ALL OF THE FEEDBACK ON IT
TO BE PUBLIC.
IN FACT IF SOMEONE E-MAILS US
PRIVATELY WE SAY WOULD YOU MIND
PUTTING THAT PUBLICLY AVAILABLE
BECAUSE IT BENEFITS THE READER,
NOT JUST US AS AUTHORS.
WHILE WE WANT PUBLIC FEED BACK,
WE'RE NOT DOING PRE-PUBLICATION
REVIEW, EVEN THOUGH IT IS
DIFFICULT, REQUIRES PRO-ACTIVE,
YOU KNOW, SEEKING TO GET THAT
PUBLIC FEEDBACK, YOU KNOW, THAT
FEEDBACK, WHAT PEOPLE THINK WHEN
THEY READ SOMETHING, THE MOMENT
IT DROPS, IS NOT AS IMPORTANT TO
US AS WE USE, STANDING TEST OF
TIME, ORTHOGONAL, THE REALLY
MEASURES THAT TELL US WHETHER OR
NOT SCIENCE IS RIGOROUS AND HIGH
QUALITY, THE SUBSTANCE OF THAT
IS SO MUCH MORE IMPORTANT THAN
APPEARANCE OF THAT.
SO EVEN THOUGH WE WANT THESE
THINGS TO BE GOOD PRACTICES, OUT
IN THE OPEN, THAT'S NOT THE
BE-ALL, END-ALL, WE'LL HAVE TO
TAKE DATA, METADATA, PROTOCOLS,
CODE, ALL OF THOSE THINGS
SERIOUSLY, 100% OF WHAT WE DROP
ON OUR PLATFORM HAS ALL OF THOSE
THINGS AVAILABLE THE MOMENT IT'S
AVAILABLE PUBLICLY.
AND SO TO GIVE SOME IDEA HOW
THAT'S GONE, IN THE LAST TWO
YEARS WE'VE HAD OUR PUBLICATION
PLATFORM ONLINE WE'VE PUBLISHED
46 DIFFERENT RESEARCH ARTIFACTS,
TWO PER MONTH, RIGHT NOW ABOUT
40 SCIENTISTS, STARTING WITH
MUCH FEWER SO GIVES YOU AN IDEA
OF THE CLIP WE'RE ABLE TO
RELEASE A AN IDEA, DATASET,
WHATEVER.
IT COULD BE ANY TYPE OF SHAPE,
REALLY TRYING TO MOVE THE NEEDLE
EARLIER.
JUST TO GET TO THE SORT OF HUMAN
NATURE PROBLEM, AS YOU'LL SEE WE
TRY TO PULL OUT LITERALLY ALL
THE STOPS THAT PREVENT PEOPLE
FROM SHARING RESEARCH, PROVIDE
SUPPORT AND INFRASTRUCTURE AND
REQUIRE THEM TO SHARE
EVERYTHING, EVEN SO WE HAVE A
LOT OF PERFECTIONIST SCIENTISTS
THAT WANT TO DO ONE MORE THING,
THEY WANT IT TO BE REALLY GOOD.
IT'S NOT DONE YET.
IT IS A THING WE CAN TEND WITH,
WE SIT OUTSIDE THE SYSTEM, TAKE
AWAY THE BARRIERS, THERE'S STILL
THIS PROBLEM WORTH MENTIONING.
FOR US, YOU KNOW, WE'RE DOING
THIS, OF COURSE IT'S A MISSION
FOR US TO MAKE SURE WE'RE
ADVANCING THIS SCIENCE
COLLECTIVELY BUT IT'S ALSO
PRAGMATIC.
WE DO THINK THAT OPEN SCIENCE IS
BETTER SCIENCE, FASTEST WAY, IF
WE WANT TO GO FROM BASIC TO
COMMERCIALIZATION QUICKLY I
WOULD DO IT IN THE OPEN, GOOD
IDEAS WHEREVER THEY LIVE, NOT
JUST IN OUR BUILDING.
ONE OF THE THINGS I MENTIONED,
WE MANDATE, WE SAY, WE EXPECT
ALL OF OUR SCIENTISTS TO SHARE
THEIR SCIENCE, EVERY PROJECT WE
DO IS BUILT INTO THE WORKFLOWS,
KICK OFF DISCUSSING WITH
PUBLICATIONS TEAM THE PLAN FOR
GETTING THIS OUT UP FRONT.
AND THAT INCLUDES SHARING THINGS
THAT DIDN'T WORK OR SHARING
THINGS THAT WE MAY DECIDE NOT TO
PURSUE.
AND SO WE HAVE A SECTION ON OUR
PLATFORM THAT'S CALLED THE ICE
BOX, THINGS WE'RE NOT PURSUING,
SO THAT'S ONE CASE IN WHICH WE
ARE JUST SAYING WE'RE NOT GOING
TO DO THIS, IT DIDN'T WORK.
IT'S NOT BECAUSE SCIENCE WAS
BAD.
FOR WHATEVER STRATEGIC REASON WE
DECIDED NOT TO PURSUE IT OR
COULD BE IT DIDN'T WORK AND OUR
ASSUMPTIONS WERE WRONG, WE TRIED
TO BE TRANSPARENT SO IT'S CLEAR
TO EVERYONE WHAT COULD BE THE
SORT OF REASONS WE DIDN'T MOVE
FORWARD.
WE ALSO WANT TO SHARE CLEARLY,
YOU KNOW, WHEN WE DO PUBLISH
SOMETHING EVEN IF IT IS A
POSITIVE RESULT WE ASK QUESTIONS
OF THE AUDIENCE AND REALLY
SUGGEST FUTURE DIRECTIONS
BECAUSE OFTEN THE ASSUMPTION
WHEN YOU PUBLISH SOMETHING IS
THAT YOU'RE PLANTING YOUR FLAG.
WHAT I DON'T WANT TO HAPPEN IS
PEOPLE TO SAY, OH, ARCADIA IS
DOING THIS WORK IN THIS AREA AND
THEY HAVE RESOURCES, AND A LOT
OF PEOPLE, THEREFORE WE SHOULD
STEER CLEAR.
NO, WE WANT YOU TO WORK OUT.
WE WANT EVERYONE, IT'S GOING TO
TAKE ALL OF US TO GET THERE.
WHEN WE PUBLISH SOMETHING, ALL
OF OUR SCIENCE GETS BETTER
FASTER, IT'S VERY -- WE'RE
TRYING TO BE AS TRANSPARENT AS
POSSIBLE WITH WHAT WE'RE DOING,
WHY, WE WANT TO DO THIS
TOGETHER.
AND SO ONE THING I WANTED TO
MENTION ABOUT DOING THINGS THIS
WAY IS THAT, YOU KNOW, WHEN YOU
START CHANGING ASSUMPTIONS, WE
EXPECT OUR SCIENTISTS TO PUBLISH
YOUR RESEARCH NO MATTER OUTCOME
AND MAKE SURE RESEARCH ARTIFACTS
ARE AVAILABLE, SIMILAR TO A
SITUATION, OSTP MEMO, FOR
EXAMPLE, YOU HEAR WHEN PEOPLE
SAY, OKAY, NOW YOU'RE REQUIRING
DATA SHARING, METADATA SHARING,
YOU HEAR CRIES ABOUT UNFUNDED
MANDATES, WHAT THAT IS IS AN
ARTIFACT OF THIS MOMENT IN TIME
WHERE WE GO FROM NOT EXPECTING
THAT DATA RELEASE TO EXPECTING
IT.
IF YOU HAVE THAT ASSUMPTION UP
FRONT THIS IS HOW WE DO SCIENCE,
IN A MINUTE THAT BECOMES
BASELINE ASSUMPTION, YOU NOW
CHANGE YOUR ENTIRE SCIENTIFIC
WORK FLOW.
IF YOU DON'T WORK, WORK, WORK,
AND WRITE, WRITE, WRITE, AND
HOPE IT'S WORTH IT TO PUBLISH,
IT'S DIFFICULT DEPENDING ON
RESULT.
PEOPLE ARE NOT RELEASING IT.
IF THE PRIMARY ASSUMPTION IS
YOU'RE RELEASING DATA ARTIFACTS
YOU NOW DO SOMETHING THAT
EVERYBODY DOES, TRY TO MAKE YOUR
LIFE EASIER.
YOU SAY WHAT SUCKS AT THE END WE
HAVE TO COLLECT THE INFORMATION
AND FIGURE OUT WHAT THE HELL WE
DID AND INSTEAD YOU SAY WAIT A
SECOND, AS WE GO COLLECT --
BETTER DATA HYGIENE, COLLECT
ARTIFACTS, DO CODE REVIEW ALONG
THE WAY, ALL THOSE OTHER THINGS.
NOW IT'S BUILT IN BECAUSE IT'S
EASIER.
BECAUSE WHEN THAT'S OUR BASELINE
ASSUMPTION IT'S EASIER TO HAVE
BETTER DATA PRACTICES AND FLIP
IT OPEN.
NO MATTER THE RESULTS.
WHICH GETS BACK AT OUR ABILITY
TO SHARE THINGS IN AN EASIER
WAY.
THAT'S THE POINT I WANTED TO
MAKE.
WE CAN CHANGE THE WAIT WE --
CHANGE THE WAY WE OPERATE.
WHEN YOU TAKE AWAY BOTTLENECKS
YOU LEARN THE REAL BOTTLENECKS.
THAT'S SOMETHING THAT'S HELPFUL
THAT WE LEARNED.
>> THANK YOU FOR THE SUPER
INSPIRING PRESENTATION.
THANK YOU ALL.
IT'S GREAT TO BE IN THE ROOM
WITH SO MANY CHANGEMAKERS.
I WAS INVITED TO SHARE NATIONAL
INSTITUTE ON AGING APPROACH TO
THE PROBLEM DU JOUR.
SO NIA IS LEADING FUNDING AGENCY
FOR ALZHEIMER'S RESEARCH AND
RELATED DEMENTIA RESEARCH.
EVERY FEW YEARS WE GO THROUGH A
STRATEGIC PLANNING EXERCISE,
CONVENE A LARGE MEETING.
WE HAD FOUR SUMMITS TO DATE.
YOU CAN SEE HALF OF THE KEY
RECOMMENDATIONS FROM THESE
SUMMITS SPEAK ABOUT THE NEED TO
CHANGE INCENTIVES, BRING OPEN
PRACTICES, AND IMPROVE RESEARCH
RIGOR, REPRODUCIBILITY,
TRANSLATABILITY.
SO, HERE IS, AGAIN, THE MAJOR
STAKEHOLDERS CONTRIBUTING TO THE
CURRENT SYSTEM OR TRYING TO
CHANGE THE SYSTEM FROM WITHIN
BUT ALSO INCREASINGLY FROM
OUTSIDE.
SO THE PUBLISHERS,
NON-TRADITIONAL PUBLISHERS,
FUNDING AGENCIES, SMALL AND
LARGE, ACADEMIC INSTITUTIONS, AT
THE CENTER INDIVIDUALS WITH
THEIR OWN AGENCY AND
RESPONSIBILITY.
AND SO SOME YEARS AGO, I THINK
2016, WE AT THE NIA LAUNCHED
ALZHEIMER'S DATABASE, A
KNOWLEDGE BASE THAT OUR GOAL WAS
TO SHINE THE LIGHT ON THE SCOPE
OF THE PROBLEM OF RIGOR AND
REPRODUCIBILITY OR LACK THEREOF
ANIMAL MODELS, AND THE TOOL TO
CONTRIBUTE TO SOLVING THE
PROBLEM, INCLUDING PUBLICATION
BIAS.
TO DATE WE HAVE CURATED OVER
1400 PUBLISHED STUDIES, OVER THE
SPAN OF 1996 TO 2023.
HERE IT GIVES YOU A VERY QUICK
LINE OF SIGHT INTO MANY ASPECTS
OF THESE PUBLISHED WORKS
INCLUDING VERY GRAN ALREADY
ASSESSMENT OF EXPERIMENTAL
DESIGN, INFORMALLY WHAT THIS
DISCUSSION, IT'S ALSO A PLATFORM
FOR CREATING CITEABLE REPORTS
AND PRE-PRINTS OF UNPUBLISHED
STUDIES INCLUDING STUDIES WITH
NEGATIVE FINDINGS, WORKING ON
BRINGING FUNCTIONALITY TO OFFER
PRE-REGISTRATION TO RESEARCH
COMMUNITY.
HERE ONE OF THE KEY FEATURES,
THERE IS THIS RIGOR REPORT CARD
FOR EVERY CURATED STUDY, WHERE
USE IT TO MON STORE RIGOR AND
STUDY DESIGN, CURATORS ASK THE
QUESTION, IS THE FOLLOWING
INFORMATION REPORTED IN THE
STUDY.
THESE ARE 24 ELEMENTS OF
RIGOROUS STUDY DESIGN FOR
TRANSLATIONAL RESEARCH STUDIES,
TESTING COMPOUNDS, NINE OF THESE
ARE ARE ACTUALLY ELEMENTS OF
JUST BASIC RIGOROUS STUDY
DESIGN.
ON MY RIGHT SIDE HERE, OUT OF
1200 STUDIES THERE'S REALLY LOW
PERCENTAGE OF REPORTING ON MANY
OF THE ELEMENTS OF STUDY,
RIGOROUS STUDY DESIGN, INCLUDING
THESE IN RED, THE BASIC ELEMENTS
OF SCIENTIFIC RESEARCH RIGOR.
A COUPLE WORDS AND DOES POLICY
MATTER, SO IN 2011 AND 2015 NIH
ISSUED TWO NOTICES, ONE
REQUIRING TRANSPARENCY REPORTING
OF FINANCIAL CONFLICT OF
INTEREST, THE OTHER REQUIRING
CONSIDERATION OF SEX AS A
BIOLOGICAL VARIABLE.
WHAT IS THE IMPACT?
WHEN WE SCAN TRENDS BY FIVE-YEAR
INCREMENTS, IN NINE BASIC
ELEMENTS OF RIGOR THERE'S A
POSITIVE SIGNIFICANT UPTICK ON
THESE TWO ELEMENTS FOR WHICH NIH
AT THAT INFLECTION POINT FOR
WHICH NIH ISSUED POLICY.
POLICIES DO MATTER.
AND THEY MATTER -- HERE I WANT
TO POINT OUT WHEN WE TALK ABOUT
SEX AS A BIOLOGICAL VARIABLE,
THAT IS REQUIRED TO BE REPORTED.
BUT WHAT IS NOT REQUIRED IS FOR
STUDIES TO BE BALANCED FOR SEX.
AND THE STUDIES ARE SERIOUSLY
LACKING IN THIS DOMAIN.
WHICH IS PARTICULARLY DAMNING
KNOWING THAT ALZHEIMER'S LIKE
MANY COMPLEX DISORDERS, THE
MOLECULAR ETIOLOGY OF THE
DISEASE IS PROFOUNDLY SEXUALLY
DIMORPHIC
HOW IT WORKS ACROSS SEXES, MANY
TRANSLATIONAL OUTCOMES.
WHAT WE'VE STARTED TO DO,
OBVIOUSLY WE'RE HOPING THAT
GIVEN INCREASED CONVERSATION
EMPHASIS ON RIGOR AND
REPRODUCIBILITY AND PUBLICATION
BIAS THERE'S MORE CHANGING
POLICY BUT IN THE MEANWHILE WE
START TO EMBED REQUIREMENT FOR
RIGOROUS STUDY DESIGN AND
TRANSPARENT REPORTING IN OUR
FUNDING INITIATIVES,
PARTICULARLY THOSE THAT ARE
TRANSLATIONAL, TARGETED, WHERE
THERE'S A LOT AT STAKE, A LOT IS
PLACED ON OUTCOME OF THOSE
STUDIES.
SO, WE HOPE THAT IS, AGAIN, WE
COMBINE CARROTS AND STICKS
APPROACH, WE PROVIDE TONIGHTS
FOR INVESTIGATORS TO REALLY HAVE
A PLACE TO SHARE DATA, HAVE A
TEAM TO SUPPORT THEM, WE EXPECT
THEY SHARE THE PROVENANCE OF
THEIR FINDINGS REGARDLESS OF THE
OUTCOME OF THE STUDY.
SO, SUPPORTS NOT JUST ACTUAL
RESEARCHERS IN ACADEMIA AND
INDUSTRY, IT'S A QUICK WAY TO
SURVEY WHAT'S BEEN PUBLISHED AND
RIGOR AND DECIDE WHETHER YOU
WOULD LIKE TO REPEAT A STUDY OR
REPEAT AND DO IT BETTER.
DATA SCIENTISTS CAN CONDUCT MET
ANNUAL SITS, LEARN FROM DIG AS A
GATED KNOWLEDGE, AND AGENCIES
ARE PARTNER, A NUMBER OF
NON-PROFITS, WITH SHARED
MISSION.
A TOOL FOR REQUIREMENTS FOR
RIGOROUS STUDY DESIGN.
AGAIN, THIS IS AVAILABLE TO ALL
ENGAGED IN ALZHEIMER'S RESEARCH,
REGISTRATION IS FREE.
MY COLLEAGUES ON THE LEADING
EDGE ARE HERE AND I HOPE YOU
ENGAGE WITH THEM OVER THE COURSE
OF TODAY AND TOMORROW.
I DO WANT TO END THAT WE HAVE --
IT'S NOT ONE SOLUTION TO THIS
BIG PROBLEM, WE'RE APPLYING A
MULTI-PRONGED APPROACH,
PARTICULARLY BECAUSE ANIMAL
MODELS RESEARCH IS NOT THE ONLY
RESEARCH WE FUND.
WE LEAN ON OPEN SCIENCE FOR THE
LAST TWO DECADES TO REALLY USE
IT TO IMPROVE RIGOR AND
REPRODUCIBILITY.
HERE TO ILLUSTRATE IN THE
PARTICULAR TRANSLATIONAL DOMAIN
WHERE WE BUILT LARGE PROGRAMS
THAT ARE BIG TEAM SCIENCE
EFFORTS CONSORTIA, ET CETERA,
EVERYTHING IS FROM THE DISCOVERY
TO MODERN TOOL DEVELOPMENT TO
CLINICAL TRIALS.
IT IS THE ONLY WAY TO DO FORWARD
AND REVERSE TRANSLATION, TO
LEARN AS FAST AS WE NEED TO
LEARN TO DO SOMETHING INVOLVED
WITH THIS TERRIBLE DISORDER.
WE DO THAT, WE BUILD ENTIRE
ECOSYSTEM, NOT JUST ACROSS LARGE
CENTERS DEPICTED HERE BUT
INCREASINGLY SO ACROSS
INDIVIDUAL GRANTS.
THESE ARE JUST SOME OF THE OPEN
SCIENCE PROGRAMS THAT ARE
FOCUSING ON BEING PRECISION
MEDICINE PRINCIPLES AND
PRACTICES IN DISCOVERY AND DRUG
DEVELOPMENT THROUGH A
CENTRALIZED DATA INFRASTRUCTURE
WHERE INVESTIGATORS CAN REALLY
SHARE AN ENORMOUS DIVERSITY OF
DATA TYPES WHERE THERE'S A TEAM,
SUPPORTING CURATION.
WE CONTINUE TO EVOLVE AND MAKE
IT BETTER.
IT'S NEVER PERFECT.
AND ALSO IN ADDITION TO
REPOSITORY OFFERING FULL
PROVENANCE OF DATA WE HAVE A
NUMBER OF OPEN SOURCE PLATFORMS
WHERE THINGS LIKE NEW NOMINATED
TARGETS DISCOVERED THROUGH BIG
DATA APPROACHES CAN SHALL PUSHED
OUT PRE-PUBLICATION WITH THE
PROVENANCE OF WHAT'S BEHIND
THEM.
WE'RE NOT SAYING THIS IS THE
BEST TARGET.
ALL WE'RE DOING IS SUR VASTING
THE BURDEN OF EVIDENCE, LETTING
THE COMMUNITY DECIDE.
THANK YOU AND LOOK FORWARD TO
THE DISCUSSION.
>> FEEL FREE TO JUMP TO THE
MICROPHONES TO START THIS
DISCUSSION.
>> ONE OF THE QUESTIONS WHEN IT
COMES TO PRE-REGISTRATIONS, AT
LEAST IN THE WORLD OF BIOMEDICAL
SCIENCES, WHEN THESE EVOLVE, ARE
THERE ISSUES RELATED TO
INTELLECTUAL PROPERTY THAT THIS
IS NOW IN THE -- PRIOR ART, AND
HOW ARE YOU TRYING TO MESSAGE
AND MANAGE THAT?
>> IT IS -- PUBLIC EXPOSURE
WOULD PREVENT YOU FROM FILING
AFTER THE FACT.
SO THERE ARE LOTS OF SOLUTIONS
AROUND THIS.
ONE IS, YOU KNOW, OF COURSE YOU
COULD FILE PROVISIONAL AND
IMMEDIATELY PUBLISH THE IDEAS TO
GET THEM OUT THERE.
WE REELEDDED FOR US OUR GOAL FOR
COMMERCIALIZATION IS VERY MUCH
TO BUILD A MODEL OF
SUSTAINABILITY TO SO OTHER
PEOPLE CAN START ORGANIZATION
IT'S TO DO SCIENCE IN DIFFERENT
WAYS.
WE DON'T WANT TO JUST TAKE OUR
INVESTMENT AND RUN IT TO ZERO.
IT'S DIFFICULT TO COMMERCIALIZE
SOME THINGS.
WE HAVE A NOT MAXIMAL USE.
INSTEAD OF SAYING THIS IS A
THING, YOU KNOW, IT'S NOT LIKE A
DEFENSIVE POSITION.
THE VAST MAJORITY OF THINGS WE
DO ARE NOT GOING TO BE THAT
NUGGET THAT'S GOING TO RESULT IN
REVENUE.
THE VAST MAJORITY IS MUCH MORE
USEFUL TO THE UNIVERSE THAN US.
IF ACADEMIC INSTITUTIONS GET A
WHIFF IT MIGHT BE OF VALUE, MOST
ARE UPSTREAM, WE TIE THE WHOLE
THING UP IN KNOTS FOR NO REASON.
CAN WE UPSTREAM FROM A LONG
COMMERCIALIZATION STRATEGY THAT
HIGHLIGHTS WHAT OR IS NOT
SOMETHING WE PURSUE?
IF IT'S LAYING THE GROUND WORK,
GETTING INFORMATION, THAT'S MORE
USEFUL, IT'S A MORE SELECTIVE
STRATEGY AND THEN EVEN THEN IF
IT IS AMONGST THINGS, THERE'S
STILL WAYS, TRY AND DO THAT IN A
DIFFERENT ORDER, IT CAN HAPPEN
FAST.
>> GINA POE, UCLA.
THIS IS ABOUT MICROPUBLICATIONS,
WHAT YOU'RE DOING AT THE
ALZHEIMER'S DISEASE RESEARCH,
YOU WERE SAYING, DR. STERNBERG,
THE MICROPUBLICATION VENUE HAS
RAISE IT'S ABOUT 5% OUTPUT OF C.
ELEGANS, IS THAT WHAT YOU'RE
SAYING?
>> YES, SO FAR.
WE STARTED OUT FIVE OR SIX YEARS
AGO, TAKE OFF, WATCHING IT
PROPAGATE WITHOUT ADVERTISING.
>> IT'S GOING.
>> IT'S WORKING.
>> I'M SURPRISED IT'S ALREADY
THAT MUCH.
>> YEAH, SO I WAS WONDERING IF
MAYBE PERHAPS WE CAN ADOPT THE
ALT-P PORTAL, MAKE
MICROPUBLICATIONS MICRO, AGAIN,
INSTEAD OF MILLI OR DECI,
PUBLISH ONE PANEL WITH FULL
METHODS AND THAT'S STATISTICAL
ANALYSES BUT NO INTRODUCTION.
IF WE DID THAT WOULD IT SPEED
RESEARCH EVEN FASTER THAN IT'S
GOING NOW?
AND SO THAT'S MY QUESTION.
>> USEFUL THING, PEOPLE HAVE
METHODS OR REAGENTS IF THEY WERE
GOING TO WAIT FOR A FULL PAPER
IT'S NOT GOING TO HAPPEN.
THAT'S REALLY HELPING DRIVE
ETHICS FORWARD.
AND IT'S GOING TO TAKE MORE
ONE-ON-ONE CONVINCING OF PEOPLE
THAT IT'S WORTH IT, RIGHT?
AND THAT'S GRADUALLY GOING.
BUT GO BACK --
>> THAT'S GOING FORWARD.
WHAT YOU'RE DOING IS GOING
BACKWARD THROUGH PUBLICATIONS
THAT HAVE ALREADY BEEN OUT
THERE.
DO YOU RUN INTO PROBLEMS WITH
PUBLISHERS AND COPYRIGHT FOR THE
ARTICLES WHEN YOU BREAK IT UP
INTO INDIVIDUAL UNITS?
>> WELL, I MEAN, WE PROVIDE THE
CITATION TO PubMed.
WE'RE JUST -- WE COLLECT GRAPHIC
INFORMATION, CONFLICTS OF
INTEREST, ET CETERA.
WE EXTRACT CERTAIN ELEMENTS.
WE LINK TO, FOR EXAMPLE,
CLINICAL TRIALS, TO OTHER
TRANSLATIONAL COMPONENTS RELATED
TO THE EVOLUTION OF THE
COMPOUND, AGGREGATE MORE THAN
WHAT'S IN THE PAPER.
FOR THE PAPER, CURATORS ASK THE
QUESTION, HAS THIS BEEN
REPORTED?
WE'RE NOT SAYING, OH, THIS IS A
BAD PAPER.
OH, WE DON'T LIKE IT.
OR LOOK AT THIS.
WE SIMPLY -- FULL DISCLOSURE,
SOME OF OUR OWN PAPERS ARE
THERE.
THEY HAVE QUITE A LOT OF RED.
SO WE'RE NOT -- AND SO THIS IS
JUST AN ENTRY POINT INTO THAT
MANUSCRIPT, A QUICK WAY FOR THE
COMMUNITY TO SURVEY IS IT
SEARCHABLE BY COMPOUND, BY
ANIMAL MODEL, BY TARGET, ET
CETERA, ET CETERA.
AND SO IT'S REALLY KIND OF
PORTFOLIO ANALYSIS AT YOUR
FINGERTIPS.
IT SAVES TIME TO INVESTIGATORS
WHEN THEY WANT TO START IN
ALZHEIMER'S RESEARCH, WE LIKE TO
KNOW WHAT'S OUT THERE, WHO ARE
THE TARGETS.
WE HAVEN'T RUN INTO ANY ISSUES.
>> WHAT I'M ASKING, CAN THE NIH,
FOR EXAMPLE, BREAK UP PAST
PUBLICATIONS INTO
MICROPUBLICATIONS IN THE SENSE
OF JUST SAY WHAT THE METHODS
WERE, THESE ARE THE AUTHORS, SO
TAKE ALL THE PAST PUBLICATIONS
THAT HAVE BEEN FUNDED BY THE NIH
THAT ARE NOW ALREADY ON
PubMed AND BREAK IT UP SO
PEOPLE CAN SEARCH FOR THINGS
WITHOUT NEEDING THE PERHAPS
BIASED STORY THAT GOES WITH IT.
>> FOR CURATED KNOWLEDGE BASES,
WHICH ARE NOT AVAILABLE FOR
HUMANS BUT FOR THE MODEL
ORGANISMS, MOUSE, RAT, FLY,
WORMSES THERE'S GRANULAR
INFORMATION THAT CAN BE LOCATED
AND SEARCHED AND COMPARED, EVEN
THOUGH THE PAPER IS NOT ABOUT
REPLICATE, RIGHT?
THAT'S VALUABLE AT A GRANULAR
LEVEL AND IS MAYBE THE RIGHT WAY
TO DO IT WITH CONTROLLED
VOCABULARIES SO METADATA IS JUST
NAILED.
THAT'S USEFUL.
>> THE FACT WE HAVE THE REPORT
CARDS AND OUTCOMES MEASURED,
THAT GIVES YOU THE PART THAT
MATTERS TO INVESTIGATORS.
ONCE THEY SEE THAT, THEY TEND TO
BE MESMERIZED BY THE NARRATIVE
AND JUST -- OR TAKE A CLOSER
LOOK.
STILL A FASCINATING STORY, GREAT
COMPOUND, BUT GIVES THEM
SKEPTICAL VIEW OF THAT
PARTICULAR PAPER.
>> CAN I ADD A COMMENT FROM NIH?
SO, DOING THAT TAKES A LOT OF
RESOURCES AND TIME.
THIS IS MANUALLY GOING OVER
EVERY PAPER, IT'S VERY, VERY
DIFFICULT.
THERE ARE NOW TOOLS TO TRY TO
AUTOMATE THESE THINGS.
THEY ARE NOT PERFECT.
AND THEY CERTAINLY ARE NOT AT
THE LEVEL WHERE THEY BREAK IT
DOWN PER EXPERIMENT.
IT'S LIKE PER PAPER.
THEY MENTIONED BLINDING IN THE
PAPER, WE'LL GIVE THEM CREDIT
FOR THAT.
IF FOR ANY ONE EXPERIMENT THEY
MENTION BLINDING THEY GET CREDIT
FOR THAT.
IT'S NOT PER EXPERIMENT AND SO
ON.
I THINK WE'RE FAR AWAY, SO FAR
AWAY FROM A.I. BEING ABLE TO
HELP US WITH THAT.
>> WE'RE ALWAYS LOOKING FOR WAYS
TO IMPROVE AND HAVE MACHINE MEET
HUMAN, YEAH, SYNERGY.
>> I THINK MAYBE THIS IS
SOMETHING WHICH HAS BEEN ALLUDED
IN THE LAST PANEL DISCUSSION
ABOUT FUNDING, TYING INTO SOME
NON-TRADITIONAL VENUES FOR
DISSEMINATION, FOR INSTANCE WE
TALKED ABOUT PRE-PRINT.
PRE-PRINTS HAVE BEEN AROUND BUT
REALLY TOOK OFF SINCE THE
PANDEMIC AND THE NIH SAID THEY
WILL BE PART OF THE REVIEW
PROCESS, WHO KNOWS IF YOU CAN
ADD, THAT'S WHEN PEOPLE STARTED
TAKING SERIOUSLY PEOPLE WHO
DIDN'T EVEN KNOW ABOUT
PRE-PRINTS.
YOU'RE NOT TIED TO AN EXTERNAL
FUNDING SOURCE SO YOU HAVE THESE
THINGS TO DEVELOP SOMETHING IN
HOUSE, REALLY SHAPE NOT JUST
PUBLICATION PROCESS BUT ALSO
SCIENTIFIC PROCESS ACCORDING TO
HOW YOU WANT.
SIMILARLY THE NIA TALKED ABOUT
THIS RESOURCE, YOU MAKE IT
MANDATORY FOR RESEARCHERS TO DO
THIS, YOU REQUIRE THEM TO -- SEX
AS A BIOLOGICAL VARIABLE OR
WHATEVER, THERE ARE MANDATED
REQUIREMENTS, ALLUDED TO, LIKE
IF THESE KIND OF THINGS WE
REALLY WANT RESEARCHERS TO
SHARE, THEY HAVE TO HAVE
MANDATES WHICH COME FROM FUNDING
SOURCES BUT AT THE SAME TIME
IT'S ALSO RAISING THE PROBLEM OF
THE BURDEN, SO MAYBE TYING WITH
VENUES OR TECHNOLOGY TO MAKE IT
EASIER FOR RESEARCHERS, DONE IN
A MANDATED FASHION SO IT'S A
REQUIREMENT, RESEARCHERS ALSO
GET THE CREDIT BUT ALSO GET THE
SUPPORT TO DO ALL OF THIS.
>> IT'S NOT ONE THING.
IT HAS TO BE ALL THESE THINGS.
I HEAR A LOT OF TIMES IF
EVERYONE WOULD REQUIRE
EVERYTHING THIS IS WHAT WOULD
HAPPEN.
YOU KNOW WHAT HAPPENS IF YOU SIT
IN THE ROOMS, NO ONE IS IN
SUPPORT OF ANY MANDATE FOR WHICH
THERE'S NOT COMMUNITY APPETITE
OR THEY HAVE NEVER SEEN OR DONE
IT.
I TEND TO TAKE A BOTTOM-UP VIEW,
HAVING THINGS EXIST IN WAYS THAT
NO ONE IS CERTAIN OF YET OR
WHATEVER, THINGS THAT DON'T
SCALE YET, TRYING TO DO
SOMETHING, SOME NEW EXISTS, IT
CAN MOVE YOU OVER TO WINDOW,
CHANGES THE NORMS, MAKES PEOPLE
THINK DIFFERENTLY, STARTS TO
CATCH ON.
NOW THERE'S A DISCUSSION --
THERE'S NO SUCH THING AS A COLD
MANDATE.
NOW I REQUIRE OF EVERYONE.
IT'S NOT A THING.
IT'S NEVER HAPPENED BEFORE BUT
THESE ARE THE SORTS OF THINGS
WHEN YOU THINK ABOUT ALL OF US
SOMEWHERE ALONG THIS
DISTRIBUTION THERE'S THE LEADING
EDGE THINGS THAT CAN MOVE THE
NEEDLE, THEN MANDATES BRING UP
THE REAR, ALL AT DIFFERENT PARTS
OF DISTRIBUTION, THEY ALL NEED
TO BE THERE.
IT'S NOT LIKE ONE SOLUTION OR
THE OTHER.
>> I POINT OUT NIH HAS MANDATED
THESE DATA MANAGEMENT AND
SHARING PLANS.
NOW, THE CHALLENGE RIGHT NOW IS
THAT NOBODY KNOWS WHAT THAT --
WHAT SHOULD BE DONE.
RIGHT?
SO, IT'S A GOOD OPPORTUNITY TO
THINK ABOUT WHAT WE REALLY WANT
TO DO, AND TO PROVE THE
PRINCIPLE BEFORE IT CAN BE
REALLY MANDATED IN A STRICT WAY.
SO WE NEED THESE EXPERIMENTS.
THE GOAL, THE SPIRIT OF THAT
MANDATE, IS TO MAKE THE DATA
FAIR, INCLUDING NEGATIVE
RESULTS.
JUST WE HAVE TO FIGURE OUT HOW
TO DO IT.
>> JUST TO ANSWER THAT FROM MY
PERSPECTIVE, I FIND WHEN I TALK
TO FUNDERS ABOUT ALL OF THESE
SORTS OF THINGS THEY ARE VERY
INTERESTED IN SEEING PRINCIPLES,
SOME HAVE MORE AT STAKE THAN
ANYBODY IN THE WHOLE ECOSYSTEM
GETTING GOOD SCIENCE.
THEY ARE THE ONES PAYING FOR IT.
I FIND FUNDERS ARE VERY OPEN TO
CHANGING PROCESS, ALBEIT SLOWLY,
DIFFERENT WAYS OF DOING THINGS,
BUT IT'S A STEP-BY-STEP THING,
MAKING IT POSSIBLE AND EASY AND
THEN ENCOURAGING IT.
WE'VE GOT TO THE STAGE WHERE
FUNDERS ARE ENCOURAGING IT AND
THERE WILL BE MANDATES COMING IN
AFTER THAT.
I THINK IT'S VERY EXCITING TIME,
WHERE THINGS ARE CHANGING AFTER
DECADES OF PEOPLE JUST MOANING.
>> ANOTHER COMMENT TO THAT
POINT, TIMING IS EVERYTHING.
THE ADVENT OF PRE-PRINTS, BUT
THEN NIH ALLOWING PRE-PRINTS TO
COME INTO THE BIOSKETCH, MADE IT
POSSIBLE FOR US TO SAY, OH,
LET'S ADD A PRE-PRINT FUNCTION.
WE'LL MAKE IT EASY, WE'RE NOT
SAYING YOU MUST HAVE HAVE A
PRE-PRINT WITH US, HAVE IT
ANYWHERE YOU WANT, BUT HERE IT
IS IF YOU CAN'T FIND ANOTHER
PLACE.
FOR THE DATA SHARING, BECAUSE
WHEN WE DECIDED TO REALLY BUILD
ROBUST OPEN SCIENCE
EXPECTATIONS, PRACTICES FOR
DISCOVERY RESEARCH, BIG DATA
SCIENCE, WE HAD TO CREATE -- WE
CAN'T SAY IT'S GOOD TO WALK BUT
NO SIDEWALKS.
YOU HAVE TO HELP -- ONE THING
NOT ONLY TO PROVIDE, MAKE THE
RIGHT CHOICE, THE EASIER CHOICE,
ALSO TO SHOW SUCCESS BECAUSE IF
YOU DON'T SHOW TWO YEARS, THIS
IS WORKING, THE COMMUNITY IS
BENEFITING, PARTICULARLY YOUNG
INVESTIGATE WORSE, I WAS
HEARTENED BY THE PRESENTATION
THAT YOUNG INVESTIGATORS
CONTRARY TO MYTHS OR POPULAR
BELIEF, THEY THRIVE -- OUR
EXPERIENCE, THEY THRIVE WITHIN
AN OPEN SCIENCE ECOSYSTEM.
THEY ARE THE NEW LEADERS WITH SO
MUCH MORE OPPORTUNITY TO LEAD,
OF A DIFFERENT GENERATION, TO
THEM SUPPORTING THINGS, SITTING
ON SOMETHING QUIETLY IS
SOMETHING NOT APPEALING.
>> ALSO MAKING IT EASIER, 2017,
ALLOWING PRE-PRINTS, BUT ALSO
COVID PRE-PRINTS, INDEX IN
PubMed, NIH PRE-PRINTS,
DISCOVERABILITY IS KEY TO MAKE
THINGS WE NEED TO HAPPEN TO
UTILIZE ALL OF THIS INFORMATION
IS LIKE MAKING IT EASY AND
HAVING THAT INFRASTRUCTURE
EXIST, USING THOSE SOLUTIONS TO
DEMONSTRATE THAT THEY ARE WORK
AND ARE EASY.
>> WE'D LIKE TO COME BACK TO THE
NEW FORMATS, THE TOPIC OF
INITIATIVES FOR OPEN SCIENCE.
JUST AS A DISCLAIMER, I'M A FULL
PROPONENT OF OPEN SCIENCE.
I LIKE PRE-PRINTS.
PUBLISHING EVERYTHING ON THE
ARCHIVE, bioRxiv.
BUT THERE IS A PROBLEM.
THAT WORKS PERFECTLY IN PHYSICS.
WHERE IT'S EASY TO CHECK THE
MATH OF SOMEBODY FOR
SIMULATIONS, NOW WE'RE DEALING
WITH BIOLOGY.
YOU CAN'T VERIFY THINGS EXCEPT
IF YOU DO YOURSELF THE
EXPERIMENTS.
IF YOU START TO DO OPEN SCIENCE
AND PUBLISH YOUR RESULTS, WHO IS
GOING TO SAY WHETHER WHAT
YOU'RE SAYING IS CORRECT OR NOT?
IT COULD BE A POSSIBILITY, HAVE
YOU NO WAY TO CHECK WHETHER IT'S
TRUE OR NOT TRUE.
SINCE WE'RE DEALING WITH BIOLOGY
AND THERE ARE PATIENTS, ALSO IN
THE POPULATION, WITH
EXPECTATIONS FOR THEIR KIDS TO
BE CURED OF A SPECIFIC DISEASE,
AND SOMEBODY'S INVENTING
SOMETHING PLAUSIBLE, THAT COULD
GENERATE MASSIVE DAMAGE.
I HAVEN'T SEEN ANYTHING IN THE
FIELD ARGUING OR DISCUSSING
ABOUT THIS POSSIBILITY.
I MEAN, THAT COULD BE FOLLOWING
YOUR BRIGHTEST MIND IN THE
UNITED STATES, SOMEBODY DOING
THE EXPERIMENT OF INJECTING
BELIEVE TO CLEAN THE BRAIN FROM
AMYLOID DEPOSITS, INSTEAD OF
TRYING TO CURE COVID, IT SOUNDS
STUPID BUT THAT COULD HAPPEN.
WELL, IT DID HAPPEN.
YOU REMEMBER "THE LANCET" STUDY?
TWELVE YEARS UNTIL RETRACTION,
THAT WAS PUBLISHED, PEER
REVIEWED.
HOW MANY CHILDREN WERE KILLED
BECAUSE THEY WERE NOT VACCINATED
AND GOT MEASLES?
THE U.K. DID A SURVEY.
THAT WAS PLAUSIBLE.
NOW, THAT GENERATED A LOT OF
DAMAGE.
WE HAD ALSO THIS EXPERIENCE WITH
GENDER DYSPHORIA, SOMEBODY
PUBLISHED A PAPER IN OUR
JOURNAL, A SMALL SENTENCE AT THE
END SAYING THERE ARE BIOMARKERS
OF GENDER DYSPHORIA, THEREFORE
THAT COULD BE AN AVENUE OF
RESEARCH TO FIND TREATMENT
AGAINST GENDER DYSPHORIA.
IT'S WHEN THE GENDER HAVE YOU AT
BIRTH IS NOT WHAT YOU FEEL LIKE
YOU ARE.
YOU HAVE MALE SEX BUT FEEL LIKE
A WOMAN.
SO EXACTLY A COUPLE MINUTES
LATER THE PAPER WAS PUBLISHED,
RELIGIOUS OR WHATEVER PEOPLE IN
THE UNITED STATES, THEY POSTED
BLOGS EVERYWHERE, AND A ON
TWITTER AT THE TIME.
A SCIENTIST PROVES GENDER
DYSPHORIA IS A DISEASE THAT CAN
BE CURED SO WE SHOULD PROHIBIT
AT THE STATE LEVEL TRANSGENDER
SURGERY.
BILLIONS OF SECONDS LATER THE
LGBTQ COMMUNITY REACTED
STRONGLY, THAT CREATED MASSIVE
UPHEAVAL.
THAT WAS REVIEWED BY THE
JOURNAL.
WE JUST MISSED IT.
NOW YOU DO OPEN SCIENCE, YOU
HAVE PEOPLE WHO ARE GOING
THROUGH THINGS LIKE THAT, HOW DO
YOU PREVENT THAT?
WE'RE DEALING WITH BIOLOGY,
WE'RE DEALING WITH DISEASES, SO
SOMEBODY CLAIMS ALL OF A SUDDEN
THEY HAVE FOUND A CURE, A
DISEASE CURE FOR COVID?
ALZHEIMER'S OR PARKINSON'S OR
EPILEPSY?
WE'RE NOT LIVING IN A FANTASY
WORLD.
WE'RE DEALING WITH HUMAN NATURE.
>> YOU'VE JUST MADE THE BEST
ARGUMENT FOR WORKING IN OPEN
PRACTICE BECAUSE THE WHOLE POINT
THINGS LIKE PEER REVIEWS ARE
OPEN, EVERYBODY CAN READ THEM,
ANYBODY CAN POST PEER REVIEW,
YOU CAN READ OPINIONS OF OTHER
PEOPLE.
ON OCTOPUS WE HAVE A RED
FLAGGING SYSTEM SO A READER CAN
RAISE A RED FLAG IF THEY THINK
IT'S IMPORTANT, SERIOUS CONCERN.
ALL THESE THINGS ARE OPEN,
TRANSPART, THE DATA IS OPEN AND
TRANSPARENT.
OTHER PEOPLE CAN LOOK.
IT MAY BE DIFFICULT FOR OTHER
PEOPLE TO COLLECT NEW DATA BUT
THE PROTOCOL IS THERE SO PEOPLE
CAN.
THEY CAN INVESTIGATE THE
VARIATION AROUND THE BIOLOGICAL
VARIATION AROUND THE VARIABLES,
COLLECTING DATA ACCORDING TO THE
PROTOCOLS.
THE WAY WE'RE THINKING OF
WORKING IN THIS TRANSPARENT AND
OPEN WAY, DOING IT QUICKLY,
DOING IT IN A WAY THAT EVERYBODY
CAN SEE WHAT'S BEEN DONE,
REPLICATED, REPRODUCE IT,
REANALYZE IT, THE BEST WAY TO
GUARD AGAINST THE PROBLEMS.
IT'S THE PEER REVIEWED BINARY
VIEW, PEER REVIEW THEREFORE IT'S
TRUE, NOT PEER REVIEW THEREFORE
IT'S FALSE, THAT'S CAUSING
PROBLEMS.
>> I'M SORRY, THIS IS A
FANTASTIC WORLD.
WHAT HAPPENED TO "THE LANCET"
STORY?
>> HE WASN'T PUBLISHING IN AN
OPEN WORLD.
>> SORRY?
YES, BUT IT WAS THEN
DEMONSTRATED AS WRONG.
IT WAS RETRACTED.
AND THEN HE'S STILL ACTIVE, ALL
THESE FAMILIES ARE AUTISTIC
CHILDREN, THEY STILL LISTEN TO
HIM, THEY BELIEVE HE'S RIGHT AND
THAT THE VACCINE IS THE REASON
FOR THAT.
[CROSSTALK]
>> ONCE THE DAMAGE IS DONE, IT'S
DONE.
>> THAT'S THE WORLD WE'RE TRYING
TO GET AWAY FROM WITH NEW
SYSTEMS WHERE PER REVIEW IS
OPEN, TRANSPARENT, NOT DONE IN A
BLACK BOX BEHIND THE SCENES
PUBLISH AND PEOPLE SAY THEREFORE
IT'S PUBLISHED IN A
PEER-REVIEWED JOURNAL EVEN
THOUGH WE CAN'T SEE THE DATA WE
CAN'T REANALYZE IT, WE BELIEVE
THAT'S THE TRUTH.
THAT'S A DANGEROUS WORLD.
>> I WOULD LIKE TO PLAY DEVIL'S
ADVOCATE.
JUST OPEN THE NEWSPAPER.
FORGET A STUDY.
LOOK AT ANY NEWSPAPER, DEPENDS
WHAT OUTLET YOU LISTEN TO OR
LOOK AT, YOU'LL HEAR WHATEVER
THE STORY IS, COMPLETELY
DIFFERENT INTERPRETATIONS OF THE
STORY.
THAT CAN GET DANGEROUS.
I AGREE THINGS CAN GET VERY
DANGEROUS, EVEN IF WE'RE
OPTIMISTIC AND SAY, OH, WE'RE
THE SCIENTIFIC COMMUNITY.
I'M WORRIED NOT ABOUT THE
SCIENTIFIC COMMUNITY BUT THE
OUTSIDE WORLD, WHATTER -- WHAT
THEY ARE GOING TO MAKE OF THIS.
>> THE CHALLENGE IS IF YOU SEND
SOMETHING TO A JOURNAL AND NOT
ONLY DO WE NOT HAVE BENEFITS OF
ALL THE OPEN INFORMATION TO
VERIFY THINGS, OPEN DISCUSSION
TO ACTUALLY SEE EXPERTS WEIGH IN
ON SOMETHING, WHICH IS ALSO KEY
AND WHY IT'S IMPORTANT TO HAVE
READERS HAVE ACCESS TO THAT
INFORMATION, NOT JUST A SECRET
CONVERSATION BETWEEN AUTHOR AND
REVIEWERS, BETTER TO HAVE THAT
BUT OFTEN YOUR PAPER THAT MIGHT
BE NONSENSE, WHATEVER OPINION
GETS REJECTED, GOES SOMEWHERE
ELSE, PUBLISHED AS IS, MOVES
DOWN THE CHAIN UNTIL SOMEONE
WILL HELP IT.
NOW THAT APPEARANCE OF TRUTH IS
MORE DANGEROUS.
THE BEST THING PRE-PRINTS DID
WAS FORCE US TO QUESTION
EVERYTHING.
>> NOT TO FORGET SCIENCE NOT NOT
HAPPEN AT NIH OR INSTITUTIONS,
IT HAPPENS IN SOCIETY.
IT'S ALWAYS A CONVERSATION IN
SOCIETY.
SOMETIMES WE SUFFER A SWOLLEN
HEAD AND THINK WE'RE ABOVE THE
CITIZENRY, THESE ARE STUPID
PEOPLE WHO ARE GOING TO
INTERPRET, MISINTERPRET.
THAT IS PART OF THE CONVERSATION
AROUND RESEARCH, AND MORE
TRANSPARENCY AND MORE EDUCATION,
MORE CONVERSATION, I THINK, WE
CANNOT WALL OFF SOME
INCONVENIENT DISCOVERY.
WE JUST HAVE TO MAKE SURE THAT
THE VERACITY OF THAT FINDING CAN
BE TESTED BY OTHERS IN THE
SCIENTIFIC COMMUNITY AND THAT
WE'RE HONEST AND CAN EXPLAIN
DIFFICULT SUBJECTS AND HAVE A
CONVERSATION, SOCIETAL
CONVERSATION, WHETHER SOCIAL
MEDIA, NEWSPAPERS, ET CETERA.
>> A LOT OF TIMES THEY DON'T
INTERVIEW THE AUTHORS OR OTHER
EXPERTS TO SEE WHAT THE STORY
ACTUALLY IS OR WHAT THE LIMITS
OF THE DATA ARE.
HOW MUCH IS EXAGGERATED IN
DISCUSSIONS, SO ON.
THERE ARE MULTIPLE CASES, COVID
OR NOT, WHERE PRE-PRINTS OR PEER
REVIEWED ARTICLES WERE REPORTED
IN THE WAY THAT WAS NOT
ACCURATE.
AND SO BECAUSE SCIENCE IS NOT
REALLY DONE OR REPORTED IN A
VACUUM I THINK WOULD BE
IMPORTANT FOR JOURNALISTS TO DO
THEIR JOB, A BIT BETTER TOO.
>> IF I MAY ADD ON, THERE'S
RESPONSE FROM RESEARCHERS
THEMSELVES HAS TO DO WITH
INCENTIVES.
TO THE EXTENT RESEARCHERS ARE
ENCOURAGED TO HAVE IMPACT,
DEMONSTRATE IMPACT IN
INSTITUTIONS, THEY TEND TO TALK
MORE TO MEDIA, SOMETHING PEOPLE
PUT ON THEIR C.V., WHEN THEY
SPEAK TO THE MEDIA NOT AS
CAREFUL, YOU SEE EXAGGERATED
CLAIMS BEING MADE THAT GET
FURTHER DISTORTED BY THE MEDIA.
>> HOW DOES THE NON-TRADITIONAL
VENUES DEFER FROM PRE-PRINT
SERVICE?
>> MICRO PUBLICATION IS PEER
REVIEWED.
PRE-PRINT CENTERS ARE NOT
REVIEWED.
>> OCTOPUS IS NOT.
>> A LOT OF PEOPLE ASK ME WHY WE
DIDN'T DO THIS.
TWO PRIMARY REASONS, WITH YOU
WE'RE VERY INTERESTED IN REUSE
OF THE DATA AND UTILITY OF THE
DATA, WE DIDN'T WANT TO BE STUCK
IN A SINGLE FORMAT LIKE PDF,
CONVERT TO XML, WE WANTED
ABILITY FOR FEMUR DIRECTLY
INTERACT AND REDISPLAY, PLAY
WITH DATA FORMATS BECAUSE WE
DON'T DO PRE-PUBLICATION OR PEER
REVIEW WE WANTED TO MAKE SURE
PEOPLE COULD USE THE DATA, NOT
LIKE AN EXCEL SPREADSHEET.
WE ON PURPOSE DON'T HAVE A
HIERARCHICAL BYLINE, SOMETHING
WE HAD TOE DO IF WE GO ON
PRE-PRINT OFFICE, WE USE CREDIT
TAXONOMY, WHO DID WHAT?
IT'S IN ALPHABETICAL ORDER ON
THE BOTTOM OF OUR PUBLICATION.
IT'S IMPORTANT BECAUSE
AUTHORSHIP AND CONTRIBUTORSHIP
IS NOT JUST ABOUT CREDIT.
IT'S ALSO ABOUT RESPONSIBILITY,
VERY IMPORTANT IT'S CLEAR WHO IS
RESPONSIBLE FOR WHAT, IN A
MODULAR WAY, WE HAVE
COLLABORATIVE RESEARCH, WE
WANTED TO GO OUTSIDE, FIRST
AUTHOR, LAST AUTHOR, MY PROJECT,
YOUR PROJECT.
I DON'T GET MAXIMUM CREDIT.
IT ACCELERATED OUR SCIENCE, TWO
REASONS WE DECIDED TO NOT OPT
FOR PRE-PRINT, I WANT THE AUTHOR
AGENCY OF PEOPLE POST WHEN
SCIENCE IS READY, THAT WAS KEY,
IN THE SPIRIT OF THAT WE'RE
STILL DISCOVERABLE, ALL THOSE
OTHER THINGS BUT THESE TWO
FEATURES ARE WHY WE DID NOT WANT
TO DO A TRADITIONAL PRE-PRINT.
>> THE OTHER QUESTION, HOW DO
THE FUNDERS RECOGNIZE THESE
PLATFORMS?
FOR EXAMPLE, FROM THE EARLY
CAREER RESEARCH POINT OF VIEW,
WHEN Ph.D. HAVE PAPER, WHEN I
STARTED MY POSTDOC OBLIGATIONS,
FUNDERS IN EUROPE WERE
EXPLICITLY ASKED FOR A PAPER
RESULTS FROM YOUR Ph.D.
STUDIES.
NOW THEY CHANGED EVEN IF YOU
HAVE A PRE-PRINT THAT'S ENOUGH
TO BE QUALIFIED FOR APPLICATION
OF POSTDOCTORAL FELLOWSHIPS, FOR
EXAMPLE.
IT'S VERY IMPORTANT.
SO bioRxiv HAS BEEN WELL
RECOGNIZED IN THAT ASPECT.
DOES YOUR PLATFORM ALSO
RECOGNIZED BY FUNDERS WHEN IT
COMES TO CAREER PROGRESSION?
>> I CAN'T SPEAK FOR INDIVIDUAL
FUNDERS BUT IN EVERY WAY THAT'S
MEANINGFUL WE HAVE THE SAME
TYPES OF, YOU KNOW, LIKE
PROPERTIES AS A PRE-PRINT
SERVER.
WE HAVE EXTRA BELLS AND WHISTLES
ON TOP OF THAT BUT WE MEET THE
SAME CRITERIA, SO THEORETICALLY
A LOT OF PLATFORMS TO BE
COMPLIANT, YEAH.
>> NOTHING TO ADD EXCEPT EXACTLY
THE SAME, YEP.
>> OKAY.
MY QUESTION IS FOR YOU.
YOU TALKED ABOUT IN YOUR STUDIES
YOU FOUND YOUNGER EARLY STAGE
SCIENTISTS MORE ENTHUSIASTIC AND
WILLING TO EMBRACE OPEN SCIENCE,
PARTICULARLY YOU TALKED ABOUT
POSTDOCS.
WAS THAT A MORE BROAD STUDY AND
DID YOU IN YOUR STUDY FIND THAT
MORE MID-LEVEL AND SENIOR LEVEL
PEOPLE, ARE THEY INDIFFERENT OR
RESISTANT OR SLOW ADAPTERS?
EVEN IN YOUNGER COHORT IS THERE
A DIFFERENCE IN BIOMEDICAL
VERSUS HOW YOU FOUND ECOLOGY AND
EVOLUTION?
>> DEFINITELY, THANK YOU.
STUDY ON THE FACULTY JOB MARKET
WAS FOR RESEARCHERS IN EUROPE
AND U.S.
BUT MAJORITY OF RESPONDENTS WERE
IN THE U.S.
AND SO IT WASN'T -- I WASN'T AT
U-CONN, I WAS AT JOHNS HOPKINS,
IT WASN'T LIMITED TO JOHNS
HOPKINS.
IT'S TRUE EARLY CAREER
RESEARCHERS ARE INTERESTED IN
PRE-PRINTING BUT IT IS ALSO TRUE
THAT SOME ARE INTERESTED IN
PUBLISHING IN HIGH-IMPACT
JOURNALS.
I HAVE TALKED TO FACULTY WHO
TELL ME STUDENTS ASK THEM TO
PUBLISH, TO SEND MANUSCRIPT TO
"CELL" OR "NATURE," SO THEIR
CAREER STRUCTURE, WE RAN THE
SURVEY TO REALLY BUST SOME
MYTHS.
WE SHOWED THERE ARE PEOPLE THAT
DIDN'T HAVE "SCIENCE" OR
"NATURE" ARTICLES AND STILL GOT
FACULTY POSITIONS.
WE ALSO FOUND 7% INCREASED
CHANCE OF GETTING FACULTY
POSITION IF YOU HAVE THOSE
ARTICLES.
IT'S A HETEROGENOUS COMMUNITY OF
RESEARCHERS, SOME DO PRE-PRINT A
LOT.
THEY GO FOR THAT, FOR THE
FACULTY JOB WITH THAT ONLY ON
THEIR C.V.
I WOULD BE VERY INTERESTED IN A
WORKSHOP LIKE THIS FOR IMPROVING
FACULTY HIRING.
THERE ARE A LOT OF ISSUES IN
FACULTY HIRING.
THERE ARE DISCUSSIONS ON THE
FUTURE P.I. COMMUNITY WHERE
POSTDOCS MENTION THEIR
COLLABORATORS OR P.I.s ARE THE
BARRIER FOR PRE-PRINTING.
THAT'S WHY I THINK -- AND THIS
IS FIELD DEPENDENT TOO.
IN ROBOTICS, PHYSICS,
MATHEMATICS, IT'S VERY
ESTABLISHED IN THE COMMUNITY
PEOPLE COME IN THE MORNING AND
READ ARCHIVE.
THIS HAS NOT HAPPENED IN
BIOMEDICAL FIELD.
I THINK IF YOU LOOK AT THE DATA
FROM FOUNDERS OF bioRxiv AND
NET ARCHIVE, IT'S RISING BUT A
STEEPENED PERCENT AFTER ALMOST
TEN YEARS, ONLY 10% OF ALL THE
SCIENTIFIC OUTPUT WHICH IS OVER
A MILLION A YEAR, SCIENTIFIC, IT
GOES TO BIOMEDICAL PRE-PRINT
SERVER.
I THINK IT'S TAKING FAR SLOWER
TO GET THERE IN BIOMEDICAL
SCIENCES AS OPPOSED TO PHYSICIST
AND MATHEMATICIANS.
I THINK THERE ARE PHYSICISTS AND
MAT MATHEMATICIANS THAT SUBMIT
BUT COMPARING DIFFERENCES WOULD
BE VERY HELPFUL IN INCREASING
THE UPTAKE OF PRE-PRINTS.
I ALSO THINK THERE'S MORE OF
MAYBE THIS CONVERSATIONS IN
TERMS OF WORKSHOPS OR
CONFERENCES, ON INTRODUCING
PRE-PRINTS MAY HELP P.I.,
FACULTY, WHO HAVE THE CONTROL OF
THE CORRESPONDING AUTHOR TO
UNDERSTAND IT'S OKAY TO
PRE-PRINT AND YOU DON'T
REALLY -- ALMOST ALL JOURNALS
ACCEPT PRE-PRINTS, THERE'S NO
BARRIER TO PUBLISHING IN A
JOURNAL WHEN YOU PRE-PRINT.
I THINK TO THE POINT THAT YOU
ALSO MENTIONED, MANDATES WOULD
BE HELPFUL IN PRE-PRINTING SO WE
GET THERE.
MICHAEL J. FOX FOUNDATION
MANDATED PRE-PRINTS, IT'S A
SMALL FOUNDATION BUT I THINK NIH
OR WELLCOME TRUST MANDATES
PRE-PRINTS, INSTEAD OF JUST
ENCOURAGING IT, WOULD BE REALLY
HELPFUL.
JUST AS MUCH LEVERAGE IN SOMEONE
ELSE'S LACKS FOR DECISIONS IN
PUBLISHING, THEY OBVIOUSLY, YOU
KNOW, VOICE INTEREST IN
PRE-PRINTING BUT AT THE END DAY
THE DECISION BY FACULTY MAY BE
SOMETHING DIFFERENT.
SO, HAVING MORE DISCUSSIONS, YOU
KNOW, REALLY COMMUNITY-WIDE
DISCUSSIONS WOULD BE HELPFUL.
>> LET ME JUMP IN, A QUESTION ON
THE PRE-PRINT FRONT.
PEOPLE ARE CONCERNED ABOUT
PUTTING OUT THEIR WORK BEFORE
FORMAL PUBLICATION BECAUSE THEY
ARE AFRAID SOMEONE ELSE IS GOING
TO SCOOP THEIR IDEA AND BEAT
THEM TO THE PUBLICATION FASTER?
>> I WAS GOING TO ASK THAT,
RELATED TO THAT.
>> YES, IT'S DEFINITELY A THING.
PEOPLE HAVE THIS CONCERN.
I THINK A LOT OF EDUCATION IS A
BIG PIECE OF THIS.
A LOT OF PEOPLE HAVE FEAR OF
PRE-PRINTS THAT DON'T TOTALLY
UNDERSTAND WHAT THEY ARE OR
WHAT'S POSSIBLE OR WHAT IT MEANS
OR I THINK OFTEN THE ARGUMENT
I'VE MOST BEEN ABLE TO COMBAT
THIS IDEA WITH IS THAT I THINK
IT'S PRETTY WELL ACCEPTED AND
PEOPLE HAVE NO PROBLEMS SHARING
THEIR EARLY RESEARCH AT
CONFERENCES WITH THEIR LIKE
TIGHTEST STRONGEST COMPETITORS,
AND THAT WHERE THERE'S NO SORT
OF CLAIM OF PRIORITY, THERE'S NO
TIME STAMP THERE THAT COULD
SIGNAL PRIORITY, SO IN MANY
CASES THE SAFER THING TO DO THAN
WHAT THEY ARE ALREADY DOING IS
TO PRE-PRINT THEIR WORK SO BUT
IN MANY CASES THEY JUST DON'T
KNOW WHAT IT MEANS AND SO A BIG
BARRIER FOR TRYING TO GET
ADOPTION THIS WAY, SOMETIMES IT
FEELS LIKE EDUCATION AND HOLDING
SOMEONE'S HAND SO THEY CAN
UNDERSTAND WHAT DO THE LICENSES
MEAN, ALL OF THE ASPECTS THAT GO
INTO THIS AND WHAT ARE THE
DOWNSTREAM IMPLICATIONS, IT'S
TOTALLY NOT MUTUALLY EXCLUSIVE
WITH JOURNALS AND ALL THESE
THINGS, IT'S LIKE OPEN
QUESTIONS, PEOPLE ARE NEW AND
DON'T REALLY KNOW.
RESOURCES, INFINITE COLLECTION
OF RESOURCES ABOUT ALL VARIOUS
ASPECTS SO I THINK THAT'S A BIG
PIECE.
>> SCOOPING IS ONE OF THE
BIGGEST PROBLEMS PEOPLE SAY
STOPS THEM FROM PUBLISHING, THEY
MIGHT PUBLISH A HYPOTHESIS
BEFORE PROTOCOL BEFORE THEY
PUBLISH RESULTS.
ACTUALLY IT'S THE OPPOSITE.
IF YOU'RE HOLDING ON OR TALKING
ABOUT IT AT CONFERENCES WITHOUT
HAVING PUBLISHED, YOU COULD
POTENTIALLY GET SCOOPED.
ONCE IT'S THERE IT'S LIKE A
PATENT OFFICE WITH YOUR NAME ON
THE DATA, YOU CAN CITE IF
SOMEBODY COLLECTS DATA AND
PROVES YOU WERE RIGHT, THE YOUR
IDEA.
>> I THINK THE GOOD FEATURE OF
THESE COMPLICATED PAPERS, IF
SOMEONE WANTS TO SCOOP YOU AND
SPEND THREE YEARS DOING THAT,
YOU KNOW, GO FOR IT.
BUT THERE ARE CASES WHETHER
OBVIOUSLY IT'S VERY QUICK AND
YOU SHOULDN'T SCREW AROUND WITH
IT.
>> IT HAS TO DO MORE WITH THE
JOURNAL'S POSITION, HOW IN
DIFFERENT FIELDS YOU CREDIT WHO
IS THE ONE WHO IS CREDITED FOR
ORIGINATING THE IDEA, WHO GOT IT
PUBLISHED IN A OFFICIAL JOURNAL
OR GOT IT FIRST IN A PRE-PRINT?
>> PRE-PRINTS ARE NOW IN
PubMed.
SOMEONE WHO IS AN HONEST
RESEARCHER PROPERLY CITING CAN
CERTAINLY ACKNOWLEDGE.
WE'RE SEEING AS A PRE-PRINT
COMES OUT GOOD DATA
AVAILABILITY, PROVENANCE OF
FINDINGS, ANOTHER PRE-PRINT
COMES OUT, QUESTIONING IT, WE
ANALYZE AND FIND SOMETHING ELSE,
HOW SCIENCE SHOULD BE.
YES AND KNOW, TIME STAMP FOR
COMPLEX STUDIES, 2 1/2 YEARS TO
PUBLISH.
YOUNG INVESTIGATORS AND FOR THE
LEADING P.I.s.
THEY REALLY DON'T WANT TO RISK
JUST GOING DARK AND JUST REALLY
BEING SCOOPED BY SOMEBODY.
>> OKAY.
THE SCOOP WAS MY QUESTION.
I'LL ASK ANOTHER ONE.
YEAH, I AGREE WITH THE BENEFITS
OF THE PRE-PRINTS, I HAVE TO
ADMIT I DIDN'T KNOW THE
DIFFERENT PLATFORM, WHETHER THEY
ARE ALL ON PubMed, FOR
INSTANCE.
I KNOW SOME ARE PEER REVIEWED,
SOME ARE NOT.
IT DOESN'T GUARANTEE IT'S A
VALUED STUDY BUT IT'S A
CHECKPOINT.
PAPERS ARE REVIEWED, REVISED,
REJECTED.
THE WORK HAS BEEN REVIEWED.
IN THE CASE OF bioRxiv IT'S
NOT.
SO HOW DO WE SORT OF PREVENT
MISUSE OF THIS CONCEPT?
IT CAN BE USED, YOU KNOW, YOU
WANT TO AMPLIFY MY C.V., AND I
LOOK GOOD FOR AN APPLICATION.
DO WE KNOW FOR INSTANCE WHAT
PERCENTAGE OF THE PAPERS THAT
SHALL SUBMITTED TO bioRxiv
TRANSITION INTO A PEER-REVIEWED
JOURNAL.
>> 70%.
>> AND 70% THAT DON'T, WHAT IS
THE MAIN RECENT THEY DON'T MAKE
IT?
>> FROM WHAT I'VE READ I THINK
THERE WAS A PRE-PRINT ON
bioRxiv ON THE FIRST FIVE
YEARS OF THE SYSTEM, BY THE
FOUNDERS, AND THEY FOUND 30% OF
PRE-PRINTS DON'T TRANSITION TO
AN ARTICLE, THEIR REASONS WERE
THAT THEY MAY HAVE BEEN JUST
SMALLER STUDIES OR THINGS IN
PROGRESS OR STUDENT MAY HAVE
GRADUATED AND WORK WAS NOT
COMPLETED OR THAT THE GROUP JUST
DECIDED THEY DIDN'T WANT TO
PURSUE IT TO A JOURNAL BECAUSE A
LOT OF TIMES JOURNALS WANT A
STORY.
SOME SORT OF FRAMEWORK THAT
DESCRIBES EVERYTHING ABOUT THE
FINDINGS, AND IF THE RESEARCH
GROUP IS NOT YET THERE TO PUT IT
OUT THERE THEY ALSO HAVEN'T DONE
A VERY LONG-TERM STUDY I THINK
TO SEE IF THEY EVENTUALLY GET
PUBLISHED.
THEY ALSO DIDN'T DO A STUDY TO
SEE IF IT WAS PUBLISHED AS PART
OF ANOTHER STUDY.
A LOT OF THESE THINGS CAN'T BE
TRACKED IN AN AUTOMATED WAY
UNFORTUNATELY.
ALSO A LOT OF THINGS THAT YOU
THINK ARE VERY KNEW OR MAY NOT
BE READY TO BE PUBLISHED, AT
SOME POINT GET PUBLISHED IN
JOURNALS ANYWAY.
SO I THINK PRE-PRINTS AT LEAST
GET MORE EYES FOR PEER REVIEW AS
OPPOSED TO JOURNAL REVIEW WHERE
WE EVEN DON'T KNOW WHO IS
REVIEWING PAPER, TWO TO THREE
PEOPLE THAT ARE DOING IT AT
JOURNALS.
>> THAT'S A GREAT POINT.
I WANT TO MAKE A PLUG FOR MY
FAVORITE PRE-PRINT EVER THAT
WILL NOT APPEAR IN A JOURNAL,
EPIC TAKEDOWN OF IMPACT FACTOR,
THE AUTHOR IS HERE.
YEAH, STRONGLY CAUTION AGAINST
MAKING ASSUMPTIONS ABOUT THINGS
THAT DON'T MAKE IT INTO A
JOURNAL IT COULD BE ANY NUMBER
OF REASONS, AND IF WE TAKE THIS
SERIOUSLY THAT IT'S ON US TO
MAKE SURE THAT'S PUBLIC, IF WE
GET PUBLIC FUNDING, AND
SUGGESTIONS FROM EARLIER THAT
STUDENTS SHOULD HAVE TO, YOU
KNOW, MAKE THEIR WORK PUBLIC,
WHETHER OR NOT ALL THE TIME WHAT
WOULD HAPPEN.
PEOPLE DO OTHER THINGS AND MAYBE
NO ONE ELSE IS THERE TO FINISH
IT.
THERE'S ALL SORTS OF REASONS
THAT HAVE NOTHING TO DO WITH THE
QUALITY OF THE WORK SO THAT'S
IMPORTANT TO REMEMBER THERE'S A
LOT OF REASONS CERTAIN THINGS
DON'T MAKE IT THROUGH AND WE
DON'T KNOW ULTIMATE OUTCOMES BUT
I HOPE MORE TAKE THE PATH THAT
--
>> MY POINT IS HOW DO WE KEEP
OURSELVES HONEST AND USE
PLATFORMS FOR WHAT THEY ARE AND
DON'T MISUSE THEM.
THAT WAS THE POINT.
>> I THINK YOU SHOULD NOT TAKE
ANYTHING THAT'S PUBLISHED IN A
JOURNAL FOR GRANTED ON QUALITY
EITHER.
ANYBODY WHO IS LOOKING DOWN THE
LIST OF PUBLICATIONS AND GOING,
OH, ONE, TWO, THREE, FOUR, FIVE,
YEP, WHATEVER, NOT LOOKING AT
THOSE PUBLICATIONS IS NOT DOING
THEIR JOB PROPERLY HIRING AND
PROMOTION.
>> WE'RE AT TIME FOR THIS
SESSION.
THANK YOU SO MUCH.
WE HAVE ANOTHER SHORT BREAK AND
WILL MEET BACK AT 4:30.
>> IT'S 4:30, WE WANT TO GET
STARTED ON TIME.
IT'S END OF DAY, I UNDERSTAND IF
YOU'RE FATIGUED.
THIS WILL HAVE VERY FEW SLIDES.
MOSTLY OPEN CONVERSATION, EACH
PERSON WILL TALK ABOUT THEIR
EXPERTISE AREA AND WE'LL HAVE A
VERY OPEN DISCUSSION LIKE WE'VE
BEEN HAVING ALL DAY.
THIS AFTERNOON, THIS LAST ONE,
WE'RE GOING TO TALK ABOUT
PROBLEMS TODAY AND INSTITUTIONAL
EFFORTS TO INCENTIVIZE REPORTING
OF NULL STUDIES, TALKING MORE
ABOUT INSTITUTIONS THIS
AFTERNOON, THE PROGRESS, THE
BARRIERS, I'M THE MODERATOR,
GINA POE, PROFESSOR AT UCLA,
NEUROSCIENTIST AND JUST BECAME
TWO WEEKS AGO DIRECTOR OF BRAIN
RESEARCH INSTITUTE.
THANK YOU.
I WANT TO BRING TO THE JOB NEW
STANDARDS OF RIGOR AND
REPRODUCIBILITY AND OPEN
PUBLICATIONS.
I'M REALLY GOING TO TAKE A LOT
OF THINGS I'VE LEARNED TOGETHER
AND I HOPE TO LEARN TOMORROW
BACK TO UCLA AND VERY EXCITED
ABOUT THIS.
ALL RIGHT.
SO OUR PANELISTS TODAY ARE
ROBERT NOBLES, AT EMORY
UNIVERSITY.
HE WAS AT TIMOTHY -- TENNESSEE
STUDIES AT RISK YOUTH.
AND HE'S A VICE PRESIDENT FOR
RESEARCH AT EMORY, A TOP 20
UNIVERSITY IN THE NATION, HE
WILL TALKING ABOUT INSTITUTIONAL
INCENTIVE.
RACHEL MILES, VIRGINIA TECH FOR
SIX YEARS, SHE IS GOING TO TALK
ABOUT AND HAS BEEN STUDYING
RESEARCH IMPACT AND ASSESSMENTS,
SHE'S A LIBRARY SCIENCES
PERSON, RESEARCHING SCHOLARLY
COMMUNICATIONS, DR. RENE
BERNARD, NOT WITH US TODAY
BECAUSE THEY CANCELED HIS FLIGHT
FROM EUROPE, SO HE'S ON HIS WAY,
IN FLIGHT RIGHT NOW, HE'S
PROBABLY -- HE LANDED AT 4:30,
HE'S RUSHING HERE AND PROBABLY
WON'T MAKE IT BY THE END OF THE
MESSAGE.
WE HAVE A RECORDED MESSAGE
BECAUSE PLANNED HEAD, AT A
UNIVERSITY IN BERLIN, AND
CHARITE, TALKING NOT LIVE, BUT
YOU CAN ALSO TALK TO HIM OVER
DINNER OR TOMORROW.
WE HAVE ERIN MCKIERNAN, FROM
NATIONAL ACADEMY UNIVERSITY,
BIOMEDICAL PHYSICS, PROMOTION
AND TENURED DOCUMENTS IN THE
U.S. AND CANADA, WHAT'S IN THESE
DOCUMENTS.
WE HAVE SRIRAMA RAO, VIRGINIA
COMMONWEALTH UNIVERSITY, A
LEADER AT MULTIPLE INSTITUTIONS,
HIGH LEVEL PERSPECTIVE ABOUT
INSTITUTIONAL BARRIERS, HONING
DOWN ON FACULTY FEEDBACK AND
REALITY OF THE PUBLICATION
WORLD.
EACH SPEAKER HAS FIVE MINUTES,
ONE HAS SLIDES, ONE WILL BE A
RECORDING, AND WE'LL OPEN IT UP
FOR QUESTIONS.
THANK YOU VERY MUCH.
>> I'LL MAKE SURE I STAY UNDER
FIVE MINUTES, WITH MY PHONE
WATCH.
ROBERT NOBLE, ENVIRONMENT FOR
RESEARCH, AGAIN AT EMORY.
I'LL START WITH A QUICK STORY.
I'VE BEEN LISTENING TO YOU ALL
MORNING, WHICH IS -- REALLY
SITTING IN THE BACK TRYING TO BE
QUIET.
WHICH ISN'T REALLY EASY.
BUT LET ME TELL YOU MY MAIN
TAKEAWAY FROM TODAY.
MANY TIMES I DON'T STAND IN
FRONT OF AN AUDIENCE THAT WANT
TO DO THE RIGHT THING.
ON THE OTHER SIDE, I INVESTIGATE
AND I MEDIATE PROBLEMS THAT
HAPPEN AT THE INSTITUTIONN MY
ROLE AS VICE PRESIDENT FOR
RESEARCH.
I PROMOTE AND REALLY SERVE AS
CONSCIOUSNESS OF WHAT IS RIGHT,
BUT MY AUDIENCES TYPICALLY DON'T
KNOW OR WOULD RATHER GO REALLY
FAST.
SO THIS PROSPECT OF -- .
>> OR CHOOSE NOT TO KNOW.
>> OR CHOOSE NOT TO KNOW, VERY
TRUE.
VERY TRUE.
SO, THIS PROSPECT OF PUBLICATION
BIAS, IT MADE ME REALLY THINK
AND ONE OF THE RULES OF THIS
SESSION WHICH IS VERY DIFFICULT,
DON'T JUMP INTO SOLUTIONS
BECAUSE SOLUTIONS ARE FOR
TOMORROW, ONLY TALK ABOUT THE
PROBLEMS.
THAT'S NOT VERY TYPICAL.
SO LET ME HIGHLIGHT, I THINK
WHAT I WANT TO DO FOR YOU IN THE
FIVE MINUTES I HAVE IS REALLY
HELP YOU UNDERSTAND THE
COMPLEXITIES OF THE INSTITUTION
AND HOW WE INCENTIVIZE OUR
FACULTY.
BUT THERE'S SOMETHING THAT I
HEARD EARLIER, EVEN WHEN
LAWRENCE WAS GIVING THE KEYNOTE,
AND HE ASKED THE QUESTION, I
THOUGHT IT WAS INTERESTING.
HE ASKED DOES PUBLISHING NULL
RESULTS IN THE NEW ENGLAND
JOURNAL OF MEDICINE COUNT FOR
TENURE PROMOTION?
I TOLD MYSELF TO BE QUIET.
THE ANSWER IS YES.
TO MAKE SURE WE'RE CLEAR, WHAT
WE DO AT INSTITUTIONS IS A
NUMBER OF THINGS TO INCENTIVIZE
THE CONDUCT OF RESEARCH, THE
SEEKING OF RESEARCH.
I'LL WALK YOU THROUGH THAT.
WHAT WE MEASURE, WHAT WE MEASURE
IN TERMS OF FACULTY
PRODUCTIVITY, ESPECIALLY FROM
RESEARCH OFFICE, IS TWO THINGS.
NUMBER OF AWARDS, NUMBER OF
PUBLICATIONS.
IF IT'S PUBLISHED IN NEW ENGLAND
JOURNAL, IT COUNTS AS A
PUBLICATION, HIGH-IMPACT
JOURNAL, PERIOD.
AND WHEN IT BECOMES REVIEWED WE
DON'T DIG -- TRUST ME, I DO NOT
READ EVERY ARTICLE THEY CITE.
WE CONFIRM IT WAS PUBLISH AND
CITED.
AND SO WE DON'T DIG INTO IF IT
WAS NULL RESULTS, POSITIVE
RESULTS, NEGATIVE RESULTS.
WE TRUST THIS PACT WITH THE
COMMUNITY AND PROCESS OF PEER
REVIEW.
I THINK ABOUT PUBLICATION BIAS,
AS DEFINED, BECAUSE I THINK WHEN
I STARTED TEACHING EPIDEMIOLOGY
A LONG TIME AGO TO GRADUATE
STUDENTS, I ALWAYS START WITH
THE DEFINITION, WHAT WE'VE BEEN
TALKING ABOUT IS THIS SELECTIVE
PUBLICATION OF RESEARCH STUDIES
BASED ON THE RESULTS FOR WHICH
WE HAVE FOUND, POSITIVE OR
NEGATIVE.
IT'S FAIR.
ONE OF THE THINGS THAT HAPPENED
TODAY IN SOME OF THE DISCUSSIONS
WAS THAT WE MADE THIS THIS
REALLY SYNONYMOUS JUMP THAT
PUBLICATION BIAS EQUALS
RESEARCHER COLLECTIVE REPORTING.
AND I NEED TO PUSH BACK, I THINK
WE'VE PUT A LOT OF STUCK THAT
IT'S UP TO THE RESEARCHER AND
INVESTIGATOR, ON IF THE
COMMUNITY CARES ABOUT THE
RESULTS FOR WHICH THEY FIND.
I DON'T REALLY THINK, IT'S LESS
CONSCIOUS THAN THAT.
I THINK HOW WE'VE BEEN
CONDITIONED OVER TIME DICTATES
WHAT WE END UP SHARING WITH THE
WORLD, NOT THER ON THE -- NOT
THE OTHER WAY AROUND.
I WONDER IF I SHOULD HIDE THESE
RESULTS IF THERE WAS INCENTIVE
TO PROVIDE RESULTS IT WOULD BE
PROVIDED, GUARANTEED.
AND SO I THINK -- OH, YEAH, I'M
GOING OVER MY ONE MINUTE, I'M
GOING TO LET YOU KNOW.
COUNT THIS AS SEVEN OR EIGHT
MINUTES, SOMEBODY MISSED THEIR
FLIGHT.
REALLY QUICK.
WHAT DO ACADEMIC INSTITUTIONS DO
TO INCENTIVIZE RESEARCH BY
GIVING FACULTY MEMBERS TIME.
AND DO UNDERSTAND THE WAY WE
CALCULATE TEACHING IN CLASS IS
25% OF THE TIME, AT SOME
INSTITUTIONS THE FACULTY LOAD IS
TWO-TWO.
OR THREE-THREE.
AND THERE'S SOME INSTITUTIONS
OUT OF THE 4,000 INSTITUTIONS
IN THE U.S. THAT HAVE A
FOUR-FOUR TEACHING LOAD WHICH
THEY DON'T INCENTIVIZE THROUGH
TIME, THE OPPORTUNITY FOR
FACULTY TO CARRY OUT RESEARCH.
THEY HAVE TO USE EXTRA TIME FOR
THAT.
WE IS INCENTIVIZE BY PROVIDES
INFRASTRUCTURE, CARRYING OUT
RESEARCH, BUT THE PROSPECT OF
DOING THAT WHEN WE GO A STEP
FURTHER, WE INCENTIVIZE THE
OPPORTUNITY TO SEEK FUNDING.
WHICH MEANS WHAT?
IT MEANS THAT YOU HAVE TO PITCH
AN IDEA AND SOMEBODY CARES
ENOUGH TO GIVE YOU RESOURCES SO
YOU CAN THEN CARRY OUT YOUR
IDEAS, INNOVATION AND DRIVE THE
CREATION OF NEW IDEAS.
SO WE PROVIDE SEED FUNDING.
WE OFTENTIMES SUPPORT PILOT
PROJECTS, WE PROVIDE FUNDING FOR
THAT WITH PROSPECT THEY WILL
SEEK EXTERNAL FUNDING LATER.
WE FUND DATABASES WHERE THEY CAN
USE TO SEEK FUNDING AND FUND
RESEARCH DEVELOPMENT OFFICERS AT
OUR INSTITUTION TO HELP NEW
SOURCE, WHERE TO GO FISHING IN
THE POND.
WE INCENTIVIZE ETHICAL CONDUCT
OF RESEARCH.
WE HAVE OVERSIGHT COMMITTEES
THAT I SUPPORT, YOU CAN GO DOWN
THE LIST, BUT WE INCENTIVIZE
THROUGH COMPLIANCE.
PEOPLE ASK IF IT'S CARROTS OR
STICKS.
COMPLIANCE IS AN INCENTIVE
BECAUSE NON-COMPLIANCE IS A
DISINCENTIVE.
SO WE HELP OUR FACULTY BY
HELPING THEM UNDERSTAND HOW TO
FIND FUNDING, WHY TO DO
RESEARCH, HOW TO DO IT
APPROPRIATELY.
WE INCENTIVIZE THE PUBLICATION
OF RESULTS.
AND THE PUBLICATION OF RESULTS
HAPPENS FOR A COUPLE OF REASONS.
ONE IS THAT THERE'S A MANDATE TO
SHARE RESULTS, FOR SOME OF OUR
SPONSORS, AND FOR OTHERS IT'S
NOT.
IT'S ENCOURAGED.
BUT THERE'S AN EXPECTATION.
IMPLICIT AND EXPLICIT.
THAT RESULTS ARE SHARED.
I THINK THE CHALLENGE IS RESULTS
YOU SHARE HAVE TO BE ACCEPTED BY
YOUR PEERS.
AND THEN THAT'S A DIFFERENT --
THAT'S THE INCENTIVE STRUCTURE
FOR THE UNIVERSITY IS THERE BUT
BARRIERS WHICH WE WERE ASKED
ABOUT, THE BARRIERS REALLY FLOW
INTO THIS OTHER COMPONENT OF
HAVING A RELATIONSHIP WITH THE
COMMUNITY FOR WHICH THAT WE ARE
SCIENTISTS WITHIN.
SO THEN WE HAVE PEER INCENTIVES.
DO THEY CARE ABOUT WHAT YOU'RE
WRITING?
IF THEY DON'T, WHOEVER REVIEWS
IT, THEY WON'T PUBLISH IT.
THEY WON'T TALK BIT.
THEY WON'T CITE IT.
AND SO WHAT WE'VE BEEN
CONDITIONED TO DO IS FIND VALUE
IN POSITIVE RESULTS.
COLLECTIVELY.
SPONSOR INCENTIVES, POSITIVE
RESULTS LEAD TO MORE GRANT
FUNDING.
NULL RESULTS DON'T.
WE'VE BEEN CONDITIONED TO FIND
POSITIVE RESULTS.
JOURNAL INCENTIVES.
WHEN THEY REVIEW AND SEE IT IT'S
THE IMPACT SAME THING.
TELL US YOUR STORY OF HOW YOU'RE
MARKETING THE INFORMATION AND
WHY WE SHOULD CARE.
ONCE YOU MARKET IT, THEN WE'LL
PUBLISH IT.
BECAUSE OUR READERS WILL CARE.
SCIENTISTS ARE NOT TRAINED HOW
TO MARKET THEIR RESEARCH OUTPUT.
SO, SOME DO WELL.
SOME DO NOT.
AND SO THIS PACT WITH COMMUNITY
SETS THE STAGE OF OUR FACULTY,
OUR RESEARCHERS, TRYING TO
FIGURE OUT HOW TO NAVIGATE AND
BE SUCCESSFUL, GO THROUGH TENURE
PROCESS.
OVER TIME WHEN GRADUATE STUDENTS
ASK THE QUESTION HOW DO I GET MY
FACULTY AND ADVISORS TO SUPPORT
ME PUBLISHING NULL RESULTS, WE
HAVE TO CHANGE AN ENTIRE
ENVIRONMENT BECAUSE THE SHORTCUT
WHEN WE'RE BUSY AND WE'RE MOVING
FAST IS DON'T FOCUS ON THAT,
FOCUS ON SOMETHING THAT YOU'RE
GOING TO GET NOTORIETY AND
VISIBILITY.
BUT THAT'S THE WRONG ANSWER
TRUTHFULLY.
IT'S THE WRONG ANSWER.
BUT OVER TIME WE'VE BEEN
CONDITIONED, THE ENVIRONMENT HAS
CONDITIONED US, ONLY LOOK FOR
POSITIVE THINGS.
SO WHEN WE TALK ABOUT
PUBLICATION BIAS, I THINK IT
GOES INTO THE NOTION OF THE WAY
THAT WE SEEK INFORMATION.
WE REALLY ONLY WANT TO BE TOLD
THAT WHAT YOU SOUGHT IS WHAT YOU
FOUND.
IN SOMEBODY TELLS YOU
ALTERNATIVE, YOU'RE NOT
NECESSARILY HAPPY.
AND I CAN ONLY THINK OF ONE
REASON, THIS IS THE CYNICAL
COMPONENT, THAT INDIVIDUALS
WOULD INTENTIONALLY PUSH
THEMSELVES TO FIND NULL RESULTS
OF THINGS PUBLISHED IN A
POSITIVE FASHION.
AND THAT IS IF THE FACULTY
MEMBER OR RESEARCHER SETS
THEMSELVES OUT TO BE A MYTH
BUSTER THAT ALL THESE PEOPLE ARE
CITING X, Y,Z, AND I'M GOING TO
PROVE THEM WRONG BECAUSE I CAN'T
DO THE SAME THING.
AND AS THE FORMER POSTDOC WHO
WENT OVER TO U.K., CAME FROM
STANFORD, AS HE STATED, IT WOULD
TAKE A LOT OF COURAGE TO STAND
UP TO GIANTS IN YOUR FIELD WHEN
YOU'RE JUST STARTING OUT.
SO THEN THE ENVIRONMENT THEN
AGAIN CONDITIONS US TO BE QUIET
ABOUT THE THINGS FOR WHICH THAT
WE CAN'T PUSH DIALOGUE ON
BECAUSE IT WOULD NOT BE WELCOMED
IN OUR PEER GROUPS, AND THE
COMMUNITY.
>> THANK YOU.
I'M RACHEL, I'M THE -- WORK IN
UNIVERSITY LIBRARIES AT VIRGINIA
TECH, RESEARCH IMPACT
COORDINATOR.
AND I'M GOING TO TALK ABOUT SOME
EFFORTS THAT WE HAVE THAT I'VE
STARTED AT MY OWN INSTITUTION
AND THESE GO BACK FIVE YEARS TO
WHEN I FIRST STARTED OUT AT THE
INSTITUTION, I CAME FROM KANSAS
STATE UNIVERSITY BEFORE I'VE
BEEN WORKING ON SCHOLARLY
COMMUNICATION FOR PAST NINE
YEARS OR SO.
AND SO STARTING WHEN I FIRST
CAME IN, I STARTED IN 2018, THEN
IN 2019 THERE WAS A SUBCOMMITTEE
FORMED UNDER THE FACULTY SENATE
SO I'VE BEEN INVOLVED WITH
FACULTY SENATE SINCE I STARTED
AT VIRGINIA TECH.
AND THAT SUBCOMMITTEE WAS
FOCUSED ON REALLY CONDUCTING A
SURVEY AND ALSO TRYING TO MAKE
RECOMMENDATIONS BASED ON THE
RESULTS OF THAT SURVEY TO LOOK
AT SALARIES BUT ALSO LOOK AT
EVALUATION OF FACULTY RESEARCH,
SO PERCEPTION ON THE FACULTY
LEVEL.
WE HAD A HIGH SURVEY RESPONSE
BECAUSE OF THAT SALARY COMPONENT
IN THERE.
SO FACULTY ARE ALWAYS GOING TO
LOOK FOR AN EXCUSE TO EXPLAIN
ABOUT THEIR SALARIES, BUT WE
HAVE A HIGH SURVEY RESPONSE
RATE, AROUND 10%, AND
REPRESENTATION FROM EACH COLLEGE
INCLUDING UNIVERSITY LIBRARIES,
WE WANTED TO ASSESS PERCEPTIONS
AND ALSO WANTED TO PRESENT
OUTCOMES TO THE PRESIDENT OF THE
FACULTY SENATE WOULD PRESENT
OUTCOMES TO THE BOARD OF
VISITORS, OUR GOVERNING BOARD,
BY THE REPORT WHICH WAS DUE JUNE
3.
SO THIS IS GOING BACK ALMOST
EXACTLY FIVE YEARS.
SO, THIS IS JUST SOME OF THE
RESULTS I WANT TO SHOW YOU, THAT
WE'RE LOOKING AT THE METRICS
THAT THEY ARE USING, WHY WE'RE
USING THEM.
WE ASKED THEM WHAT DO YOU USE
AND WHY DO YOU WANT TO USE IT.
WE WOULD SAY, FOR EXAMPLE, MOST
OF THEM WERE USING NUMBER OF
PUBLICATIONS, JOURNAL METRICS,
BUT THEY WERE MORE INCLINED TO
USE NUMBER OF PUBLICATIONS
JOURNAL METRICS, JOURNAL
ACCEPTANCE RATES, WANTING TO USE
METRICS, ATTENDANCE NUMBERS,
SURPRISINGLY GRANT AWARD AMOUNT
AND AWARDS RECOGNITIONS, HONORS,
JOURNAL REPUTATION, MORE TO DO
WITH WHAT IS REPUTATION IN YOUR
FIELD, IS IT WELL KNOWN IN YOUR
FIELD, AND WE FOUND THAT WAS
MORE SO FOR PERSONAL REASONS.
THEY WANTED TO USE THAT METRIC.
WE ASKED THEM HOW THEY PERCEIVED
THE FAIRNESS OF RESEARCH
ASSESSMENT AT THE DEPARTMENT,
THE COLLEGE OR ADMINISTRATIVE
UNIT AND UNIVERSITY LEVEL.
OVERALL WE FOUND THAT IN GENERAL
THEY PERCEIVED THE DEPARTMENT
LEVEL TO BE FAIRER THAN THE
COLLEGE LEVEL AND THE COLLEGE
LEVEL TO BE FAIRER THAN THE
UNIVERSITY LEVEL.
HIGHER UP ASSESSMENT LEVEL WENT,
LESS THEY TRUSTED ASSESSMENT.
THE LESS THEY TRUSTED THAT, AS
IT WENT UP.
WE HAVE QUALITATIVE RESULTS,
ESPECIALLY FROM FACULTY IN
LIBERAL ARTS AND HUMAN SCIENCES,
COLLEGE OF LIBERAL ARTS AND
HUMAN SCIENCES WERE FRUSTRATED
BECAUSE THEY NEEDED MORE
COUNTING, THAT'S WHAT'S VALUED
MORE AND THEY GOT FRUSTRATED,
THERE WAS A LOT OF PLEASE STOP
COUNTING COMMENTS AND REMARKS.
THERE WAS ALSO -- WE ASKED ABOUT
PERCEPTION ON TIME COMMITMENT,
WENT OVER 100%, LET THEM
SELF-SELECT.
REPORTED ACTUAL, IT WAS
SIGNIFICANT HOW MUCH THEY WERE
CONTRIBUTING TOWARDS SERVICE AND
THERE WAS A LOT OF QUALITATIVE
RESPONSES RELATED TO HOW MUCH
THEY WERE NOT ABLE TO CONTRIBUTE
TO TEACHING, TOWARDS RESEARCH
BECAUSE OF ALL THE WORKLOAD THEY
HAD TOWARD AS -- TOWARD SERVICE
AND ADMINISTRATIVE DUTIES.
WE ALSO ASKED THEM SENTIMENT AND
CONCERNS FOR USE OF BUDGET
MODEL, THAT WE HAD JUST SWITCHED
TO IN 2018.
AND ASKED ABOUT WHAT ARE YOUR
SENTIMENTS TOWARDS WHAT IF
PUBLICATIONS AND CITATIONS ARE
INCLUDED, AND OVERALL IT WAS
NEGATIVE SENTIMENT OR IT WAS
MIXED OR NOT ENOUGH KNOWLEDGE TO
EVALUATE TO BE INCLUDED IN THE
PIBB MODEL.
IT WAS OVERALL NEGATIVE, PLEASE
DON'T DO THIS, PLEASE DON'T DO
THIS.
WE HAD A LOT OF THAT.
SO WE HAD A LOT OF
RECOMMENDATIONS BASED ON ALL OF
THIS, AND ONE HIGHLIGHTED IS THE
ONE THAT WE ENDED UP COMING TO
THE CONCLUSION OF DOING BECAUSE
WE HAVE TO START SOMEWHERE.
THIS IS WHERE WE STARTED.
THIS IS WHERE OTHER INSTITUTIONS
START.
AND MANY OF YOU MIGHT BE
FAMILIAR WITH THIS TYPE OF
RECOMMENDATION THAT YOU DEVELOP
A BRIEF -- I SAY BRIEF BUT IT'S
NOT AS BRIEF AS IT COULD BE.
BUT DEPARTMENT LEVEL DRIVEN BUT
UNIVERSITY-WIDE INCLUSIVE AND
CAREFULLY WRITTEN RESPONSIBLE
RESEARCH ASSESSMENT STATEMENT OF
PRINCIPLES TO SUPPORT AND DRIVE
RESEARCH PRODUCTION.
SO THIS IS WHAT WE FOCUSED ON.
THE UNIVERSITY SHOULD PROVIDE
RESOURCES TO REDUCE TEACHING
SERVICE AND EXPECTATIONS, WE'RE
FOCUSING ON, I'M ONE OF THE
SUPPORTS FOR THE ELEMENT SYSTEM,
RESEARCH INFORMATION MANAGEMENT
SYSTEM, TRUST ME I'M THE KEY
PERP THAT GETS TO HEAR ALL THE
COMPLAINT ABOUT THAT SYSTEM.
I GET TO ALSO TRY TO PROVIDE
MORE SUPPORT FOR THAT SYSTEM AS
WELL.
AND SO THAT'S ALSO SOMETHING
WE'RE WORKING ON.
WE'RE ALSO CONSIDERING FACULTY
RESEARCH LIAISONS BETWEEN
ADMINISTRATION AND FACULTY.
I'M THINKING WE NEED TO REVISIT
THESE THINGS, SO IT'S AMAZING TO
LOOK BACK AT OUR WORK FROM FIVE
YEARS AGO.
SO, YEAH, A LOT OF THESE THINGS
AS YOU CAN SEE UP HERE ARE VERY
MUCH WHERE WE SHOULD BE FOCUSING
BUT WE HAVE TO FOCUS ON ONE
THING AT A TIME BECAUSE I DO
BELIEVE IN QUALITY OVER
QUANTITY.
WITH THE DEPARTMENT LEVEL
RECOMMENDATIONS, THEY ARE DOING
THINGS AT FACULTY AFFAIRS.
ALMOST DONE.
MAKING THINGS CLEAR IN THE
GUIDELINES, OUTSIDE MY AREA BUT
THEY ARE DOING THINGS.
AVOID PUTTING OVERLY BURDENSOME
AND UNREALISTIC EXPECTATIONS ON
FACULTY.
WHAT DOES THIS HAVE TO DO WITH
PUBLICATION BIAS?
THIS IS OUR PROGRESS AND
TIMELINE.
WE HAD ANOTHER SUBCOMMITTEE THAT
NORMANDY TO -- FORMED TO
ADDRESS ALL THE AREAS TO ADDRESS
THIS PROBLEM WITH TIME
ALLOCATIONS, WE ADDRESSED
TEACHING SERVICE AND RESEARCH TO
MAKE FURTHER RECOMMENDATIONS.
ANOTHER TWO YEARS TO FORM A TASK
FORCE BECAUSE I HAD TO BE, LIKE,
WELL, IS ANYTHING HAPPENING?
NO, NOTHING WAS HAPPENING.
I SAID LET'S MAKE SOMETHING
HAPPEN.
IN 2022 FORMED A TASK FORCE TO
DRAFT OUR OWN STATEMENT.
WE DID DRAFT OUR OWN STATEMENT,
2023, THE STATEMENT WAS VOTED
ON, ENDORSED BY FACULTY SENATE.
THIS YEAR WE'VE CRAFTED A
RESOLUTION THROUGH UNIVERSITY
GOVERNANCE, JUST ENDORSED BY
UNIVERSITY COUNCIL AND PRESIDENT
MAY 6, IT'S AVAILABLE ON THE
INTERNET.
YOU CAN GO TO THAT STATEMENT AND
READ IT.
BASEDDED ON THE LIEDEN
MANIFESTO.
WE CAN'T JUST HAVE IT DORMANT ON
THE INTERNET.
WE NEED TO COMMUNICATE TO
UNIVERSITY LEADERS, DEANS GROUP,
DEPARTMENT HEADS, ASSOCIATE
DEANS FOR RESEARCH.
THE STATEMENT IS NOT POLICY.
IT'S ASPIRATIONAL, EDUCATIONAL
GUIDANCE TOOL, DOES NOT
SPECIFICALLY MENTION NULL
STUDIES.
IT WAS MENTIONED IN THE REPORT.
WHEN I WAS WRITING THE SLIDES WE
DIDN'T MENTION IT IN THE NOTES.
WE HAD A HUGE FOCUS ON
INTERDISCIPLINARY AND
TRANSDISCIPLINARY RESEARCH, A
HUGE PROJECT WITH INCENTIVIZING
NULL STUDIES AND INCENTIVIZING
RESEARCH AS WELL, A PROBLEM AT
OUR INSTITUTION AS WELL.
THE PIBB MODEL INCORPORATION IT.
IT'S A SMALL PORTION OF THE
BUDGET MODEL BUT STILL, TO ME, A
PROBLEM AND OUR INCENTIVE BASED
MODEL.
THE UNIVERSITY IS TRYING TO
BECOME A MEMBER OF AAU, SOME
DOES GET AWAY FROM THAT BUT I
THINK IF WE WORK IN CONJUNCTION
WITH COMMUNICATING AND WORKING
WITH UNIVERSITY LEADERS THROUGH
WHAT'S CALLED GLOBAL DISTINCTION
INITIATIVE I THINK THAT WE CAN
KIND OF BLEND BOTH OF THE GOALS,
THE STRATEGIES TOGETHER TO HAVE
THE BEST OUTCOME.
SO IT'S REALLY IMPORTANT TO HAVE
FACULTY VOICES ON KEY COMMITTEES
WHILE WE MOVE THESE FORWARD.
SO THEY ARE CLOSELY RELATED TO
PUBLICATION BIAS, I WON'T GO
OVER THIS SLIDE IN TOTAL BUT I
REALLY WANT TO EMPHASIZE THAT
THIS IS -- THE INCENTIVE SYSTEM
IS RELATED TO THIS AND WE DID
HAVE A NULL STUDIES HIGHLIGHTED
IN OUR REPORT AND EDUCATING THE
COMMUNITY ON THIS STATEMENT OF
PRINCIPLES IS A KEY FIRST STEP
MOVING TOWARDS A RESPONSIBLE
RESEARCH CULTURE.
THANK YOU.
>> HELLO, I'M RENE BERNARD,
BERLIN INSTITUTE OF HEALTH.
I'M AFFILIATED WITH THE CENTER
FOR RESPONSIBLE RESEARCH, BERLIN
INTUITION FOR HEALTH,
COORDINATOR AT NEUROCURE, MY
AFFILIATIONS ARE PART OF THE
CHARITE MEDICINE BERLIN, ONE OF
THE LARGEST MEDICAL CAMPUSES IN
EUROPE.
I'M SORRY I CAN'T BE THERE IN
PERSON BECAUSE MY FLIGHT WAS
CANCELED, I SHOULD BE THERE IN
PERSON TONIGHT TO MEET YOU ALL.
LET ME BRIEFLY TELL YOU WHAT MY
INSTITUTION HAS DONE TO INCREASE
AWARENESS FOR NULL PUBLICATION
DATA.
NUMBER ONE, SINCE 2017 THE QUEST
CENTER OFFERS 1,000 URONULL
PRIZE TO FIRST, LAST,
CORRESPONDING AUTHORS FROM OUR
INSTITUTION, NO BUDGET LIMIT,
ONCE APPROVED AWARDEES RECEIVED
A LETTER, NULL PUBLICATION
DISPLAYED ON THE QUEST WEBSITE.
WITH A CLEAR AND DEFINED PROCESS
FOR THIS, FOR INSTANCE
HYPOTHESIS NEEDS TO BE CLEARLY
DEFINED AND NULL FINDING TO BE
REPORTED IN ABSTRACT AND
RESULTS.
SO FAR WE'VE GIVEN OUT 46
AWARDS, 8 APPLICATIONS DENIED,
THE REASON WAS OFTEN IT WAS
PURELY EXPLORATORY OR MIX OF
POSITIVE AND NEGATIVE DATA, AND
BEFORE WE SEND OUT THESE DENIALS
WE ALSO HAVE TO APPLY
PRINCIPLES, SOMEBODY LOOKS OVER
THIS AGAIN.
WE HAVE AN AWARENESS TOOL FOR
FILE DRAWER DATA, IT'S CALLED
FIDDLE.
THAT GOES BEYOND JUST NULL DATA,
BUT THE TOOL REPRESENTS
ALTERNATIVE OUTLETS AND
STRATEGIES TO MAKE UNPUBLISHED
DATA PUBLISHABLE WITH A PROPER
RECOGNITION.
WE ALSO PUBLISHED AN ACCOMPANY
PAPER, HIGHLIGHT QUALITY
PREREQUISITES AND WHAT ENOUGH OF
INFORMATION SHOULD BE INCLUDED
WHEN PUBLISHING NEGATIVE DATA
ALSO TO PROVIDE STUDIES OF
QUESTIONABLE QUALITY TO SURFACE.
SO IT WAS NICE TO SEE THAT THE
AWARDS WE STARTED GOT ALSO
ADOPTED BY OTHERS, FOR INSTANCE
BY ASAP BIO, NEGATIVE RESULTS,
BY PRE-CLINICAL DATA FORUM WHICH
GIVES OUT NEGATIVE DATA PRIZE IN
PRE-CLINICAL NEUROSCIENCE.
THE FIRST THING WE STARTED IS TO
INCLUDE EXPECTATIONS TO PUBLISH
NEGATIVE RESULTS IN OUR
INTRAMURAL FUNDING AGREEMENTS
AND APPLICANTS FOR INSTANCE IN
THE COST OF EXCELLENCE NEUROCURE
WHERE I WORK, WE COMBINE WITH
OFFER TO INTERACT WITH
APPLICANTS AND TO COUNSEL THEM.
OF COURSE, THERE ARE STILL
BARRIERS, ONE OF THEM IS LIMITED
VISIBILITY.
I WOULD SAY PROBABLY NOT
EVERYONE, OUR INSTITUTIONS,
KNOWS ABOUT THESE THREE THINGS
THAT I JUST TALKED ABOUT, AND
OUTSIDE OF INSTITUTION FROM
BERLIN, SO ALSO OUR CALL SERVICE
25 P.I.s IS SMALL SO
VISIBILITY NEEDS TO BE
INCREASED.
SECOND, NULL STUDIES ARE NOT
LABELED AS SUCH, WHICH IS GOOD
BECAUSE THEY ARE STILL TRUE
RESEARCH RESULTS, BUT AT THE
SAME TIME IT MAKES IT REALLY
HARD TO QUANTIFY THEM OR
VISUALIZE TRENDS AND THAT'S
EXTREMELY DIFFICULT TO FOLLOW UP
AND UNTIL WE HAVE SOMETHING WE
WILL NEVER FIND OUT WHAT THE
TRUE NUMBER OF NULL PUBLICATIONS
IS.
STUDY PRE-REGISTRATION IS LIKE A
REMEDY FOR OUTCOME SWITCHING, IT
IS EXTREMELY UNPOPULAR.
WE HAVE A PRIZE FOR THIS, IT'S
ALMOST NEVER, EVER REQUESTED.
SO, WE NEED TO FIND A WAY TO
MAKE SCIENTISTS TO ADOPT THIS
AND MAYBE THEN WE GET ALSO
CLOSER TO THE REAL NUMBER OF
NEGATIVE DATA PUBLISHED
WORLDWIDE.
AND THE FOURTH POINT IS OF
COURSE COMMUNITY NEEDS TO DO
MORE THAN JUST HANDING OUT
MONEY.
WE NEED ANNUAL AWARENESS DAYS OR
WEEKS, HACKATHONS WHERE PEOPLE
COME TOGETHER AND PUBLIC
NEGATIVE DATA OR BRING THEM TO
THE NEXT STEP, EXPERIENCE
SHARING OF PEOPLE THAT HAVE
TROUBLE GETTING THEIR DATA
PUBLISHED AND SHARED WITHIN THE
COMMUNITY AND ALSO THE TOPIC OF
INTEGRATED IN BIOMEDICAL
CURRICULA.
THANK YOU FOR THE INVITATION.
I'M EXCITED TO BE PART OF THIS.
WHAT WE TRY TO ACHIEVE IS
NOTHING SHORT OF GRAND CHANGE,
SCIENTISTS HAVE A LOT TO SAY
ABOUT THIS, THE GREAT TOOLS
THAT'S BEEN DEVELOPED, THE
BEHAVIORAL CHANGE WHILE, MATCH
SOURCES OF BEHAVIOR WITH
POSSIBLE INTERVENTIONS AND
POLICIES AND MAYBE TOMORROW WE
COULD ADOPT THIS TO CREATE
BEHAVIORAL CHANGE WHEEL FOR
PREVENTING PUBLICATION BIAS.
ALL RIGHT.
THAT'S IT.
AND THANK YOU VERY MUCH AGAIN.
BYE.
>> HI THERE.
IT'S LATE, I'VE GIVEN GINA
PERMISSION TO KICK ME IF I GO
OVER TIME.
WE WERE GIVEN THE INSTRUCTION TO
FOCUS TODAY ON PROBLEMS, RATHER
THAN SOLUTIONS.
WHICH IS PERFECT FOR ME BECAUSE
ALL I HAVE ARE DATA THAT SHOW
THE PROBLEMS.
AND GINA SET ME UP NICELY WHEN
SHE SAID THERE ARE MYTHS AROUND
PROMOTION AND TEN TENURE.
THAT'S VERY TRUE.
SOMETHING WE REALIZE AROUND
2016, 2017, MY COLLEAGUES AND I
WHEN TALKING ABOUT INCENTIVES
AND IMPROVING INCENTIVES, WE
REALIZED WE DIDN'T REALLY HAVE A
GOOD IDEA OF WHAT WAS IN
PROMOTION AND TENURE DOCUMENTS.
A LOT OF EVALUATIONS HAPPEN
BEHIND CLOSE DOORS, THOSE ARE
PRIVILEGED CONVERSATIONS.
AND WE THINK WE HAVE A LOT OF
IDEAS ABOUT WHAT IS GETTING
EVALUATED AND HOW.
BUT WE WANTED TO KNOW WHAT WAS
GOING ON.
SO, WE DECIDED TO COLLECT.
IT WAS A REPRESENTATIVE SAMPLE
FROM 129 UNIVERSITIES, OVER 300
ACADEMIC UNITS IN THE U.S. AND
CANADA.
WE COLLECTED 850 DOCUMENTS THAT
HAD TO DO WITH PROMOTION AND
TENURE PROCESS.
WE WENT THROUGH AND ANALYZED
THOSE.
I DON'T THINK ANY OF THE NUMBERS
THAT I'M GOING TO GIVE TODAY
WILL SURPRISE ANYONE BUT I THINK
IT'S GOOD TO HAVE THOSE NUMBERS.
AND I'LL TALK IN A MOMENT ABOUT
HOW IT RELATES TO PUBLICATION
BIAS.
SO, AGAIN, THIS WILL NOT
SURPRISE ANYBODY, WE FOUND A
PROMINENT USE OF BIBLIOMETRICS
LIKE JOURNAL IMPACT FACTOR AND
CITATION COUNTS.
IF WE BROKE IT DOWN BY, FOR
EXAMPLE, RESEARCH INTENSIVE
R-TYPE INSTITUTIONS, AS MANY AS
40% WERE USING JOURNAL IMPACT
FACTOR IN PROMOTION AND TENURE
GUIDELINES, PROBABLY AND
UNDERESTIMATION, LOOKING FOR
HIGHLY RECOGNIZED JOURNAL, SO
40% USING THAT JOURNAL IMPACT
FACTOR.
MORE WORRYING IS A LOT OF THOSE
INSTITUTIONS WERE USING JOURNAL
IMPACT FACTOR AS PROXY MEASURE
FOR QUALITY EVEN THOUGH WE HAVE
EVIDENCE TO THE CONTRARY JOURNAL
IMPACT FACTOR MEASURES QUALITY.
A LOT OF THEM WERE FOCUSING ON
PUBLICATION COUNTS, HOW MANY
ARTICLES, HOW MANY PRODUCTS.
MY DEPARTMENT IN PARTICULAR HAS
PUT A VERY CLEAR MARK ON IF YOU
WANT THIS PROMOTION LEVEL YOU
NEED TEN, IF YOU WANT THIS ONE
YOU NEED TWENTY SO WE'RE GOING
BACK, IN MY VIEW.
ANOTHER THING WE DID WAS WE
SURVEYED FACULTY, WE DIDN'T JUST
WANT TO KNOW HOW OFTEN THINGS
WERE MENTIONED BUT WHAT DID THE
FACT AT THE SAME INSTITUTIONS
THINK ABOUT HOW THEY WERE BEING
USED.
IT WAS MENTIONED EARLIER TODAY
THE DIFFERENCE BETWEEN WHAT YOU
THINK USUAL COLLEAGUES ARE DOING
AND WHAT YOU'RE ACTUALLY DOING.
WHEN WE SURVEYED PEOPLE THEY
SAID, WELL, I'M WORRYING ABOUT
READERSHIP BUT MY COLLEAGUES ARE
WORRIED ABOUT JOURNAL IMPACT
FACTOR AND PRESTIGE.
NON-TENURED WERE MOST WORRIED.
THEY ARE WORRIED ABOUT JOURNAL
IMPACT FACTOR AND PRESTIGE,
GOING INTO THEIR EVALUATION.
WE LOOK A LOOK AT
NON-TRADITIONAL OUTPUTS.
WHAT ELSE IS IN THERE?
AND UNFORTUNATELY WE FOUND THAT
UP TO 95% OF THESE INSTITUTIONS
ARE MENTIONING TRADITION
OUTPUTS, VERY FEW MENTIONING
NON-TRADITIONAL.
FOR PRE-PRINTS FOR R-TYPE IT'S
20%.
WHEN YOU LOOK AT M TYPE AND
B-TYPE, IT'S AS LOW AS 5%
MENTIONING PRE-PRINTS IN WORKING
PAPERS.
WHY IS THAT IMPORTANT?
WHAT DOES HAVE IT TO DO WITH
PUBLICATION BIAS?
THESE THINGS ARE PUSHING FACULTY
TOWARDS JOURNALS THAT ARE LESS
LIKELY TO ACCEPT NULL RESULTS,
AND TWO, SOME OF MIGHT HAVE OWN
RESULTS ENDED UP AND STATE IN A
PRE-PRINT.
MINE COULDN'T GET PUBLISHED, I
PREFERRED SHARING RATHER THAN
FILE DRAWER IT.
IF IT DOESN'T GET RECOGNIZED, I
DON'T GET CREDIT.
AND WE KNOW THIS IS HAPPENING.
WE ACTUALLY HAVE DATA NOW
SHOWING HOW THIS SYSTEM IS
PUSHING RESEARCHERS AWAY FROM
PUBLISHING NULL RESULTS AND
TOWARDS PRESTIGE AND JOURNAL
IMPACT FACTOR, ALL THESE OTHER
THINGS THAT ARE NOT GOOD
INDICATOR AND NOT EMBODYING
VALUES.
ACCELERATING KNOWLEDGE, PUBLIC
GOOD, OUR CRITERIA ARE IN DIRECT
CONTRAST TO THOSE.
THAT'S THE MESSAGE THAT I WANT
TO PUT OUT THERE.
THOSE ARE THE BARRIERS.
HOW DO WE GET PAST THAT?
WE NEED TO REFOCUS, GET AWAY
FROM PRODUCTS AND COUNTS AND GO
BACK TO WHAT ARE THE ACTUAL
VALUES, WHAT ARE THE THINGS WE
WANT TO SEE, WHAT ARE THE
BEHAVIORS WE WANT TO SEE FROM
OUR FACULTY BECAUSE UNTIL WE DO
THAT WE'RE GOING TO CONTINUE
FOCUSING ON BIBLIOMETRICS AND
COUNTS, AND ALL OF THAT IS
GOING TO BE INCENTIVIZED.
>> CAN YOU HEAR ME IF I SPEAK
FROM HERE?
OKAY.
JUST TO SHAKE IT UP.
SO, I WANT TO THANK THE FOLKS
FOR INVITING ME TO BE PART OF
THIS VITAL DISCUSSION.
AND I WILL BE PROVIDING A
PERSPECTIVE OF MIGHT HAVE ROLE
AS AN INSTITUTIONAL OFFICIAL
WHICH IS THAT OF A VICE
PRESIDENT FOR RESEARCH AT
VIRGINIA EQUALITY UNIVERSITY, AS
ROBERT VERY ELEGANTLY TALKED, A
LARGE PART OF OUR ROLE IN THIS
CAPACITY IS TO REALLY FACILITATE
AND PROMOTE THE RESEARCH THAT IS
DONE BY OUR FACULTY AND STUDENTS
AND STAFF ACROSS VARIOUS
DISCIPLINES.
BUT IN MY COMMENTS TODAY
PROBABLY I WILL PEPPER SOME OF
MY STATEMENTS WITH MY PAST
EXPERIENCES AS BEING A
RESEARCHER AS WELL OVER 30 YEARS
PREVIOUSLY AT UNIVERSITY OF
MINNESOTA, PRIOR TO THAT IN
CALIFORNIA, IN SAN DIEGO.
SO WHEN WE TALK ABOUT WHAT IS IT
WE WANT TO PROMOTE AT THE
HIGHEST LEVEL, OBVIOUSLY WE WANT
TO SUPPORT THE MOST HIGHEST
QUALITY RESEARCH THAT HAS THE
MOST IMPACT BECAUSE AFTER ALL
SEVERAL OF US ARE PUBLICLY
FUNDED AND STATE FUNDED
INSTITUTIONS, SO THIS IS PUBLIC
TRUST THAT IS IMPORTANT.
BY THE WAY WE HAVEN'T TALKED
ABOUT IT, IN THE UNITED STATES
AND ELSEWHERE THE PUBLIC
SENTIMENT AND HIGHER ED IS NOT
WHERE IT NEEDS TO BE.
AND I THINK ONE REDEEMING
FEATURE IS THAT INSTITUTIONS OF
HIGHER ED THAT DO RESEARCH HAVE
A GREATER VALUE IN THE PUBLIC
SPACE.
SO, I THINK IT IS INCREDIBLY
IMPORTANT WE NOT ONLY FACILITATE
AND CONDUCT HIGHEST QUALITY
RESEARCH THAT'S MOST IMPACTFUL
BUT HAS BEEN STATED SEVERAL
TIMES TODAY NEEDS TO BE DONE
WITH RIGOR AND REPRODUCIBILITY.
AND WITH TRANSPARENCY.
AND ETHICALLY DONE, REALLY VITAL
ELEMENTS TO IT.
NOW, IF YOU ASK WHAT WE AS
RESEARCH ADMINISTRATORS ARE
DOING TO PROMOTE THIS WHOLE
CONCEPT OF THE HYPOTHESIS TODAY
THAT NULL STUDIES, WE HAVE
CHALLENGES, BARRIERS, BUT WE'RE
MAKING PROGRESS, BOTH OF US WILL
BE THE FIRST TO ADMIT AT AN
INSTITUTIONAL LEVEL THAT'S NOT
THE FIRST THING COMING TO OUR
ROLES AND RESPONSIBILITIES TO
PUSH AND PROMOTE.
FOR ALL THE REASONS ERIN TALKED
ABOUT.
IT'S NOT TRUE WE'RE NOT DOING
THINGS ADMINISTRATIVELY IN
MULTIPLE DIFFERENT LEVELS AT OUR
INSTITUTION, FOR EXAMPLE, WE
HAVE THIS DATA SCIENCE LAB THAT
PROMOTES ALL THE THINGS THAT YOU
GUYS TALKED ABOUT,
PRE-REGISTRATIONS, REGISTERED
REPORTS, VARIETY OF TRAINING,
RESPONSIBLE CONDUCT OF RESEARCH,
GRANT ACADEMIES, THERE ARE ALL
THESE THINGS TAUGHT AND WE'RE
PLEASED WITH THE NINDS BECAUSE
OUR UNIVERSITY WAS PART OF THE
INAUGURAL GROUP OF THE FIRST
FIVE RIGOR CHAMPIONSHIP PRIZE,
THANK YOU.
SO IT'S GOING TO BE AN IMPORTANT
INAUGURAL GROUP TO LOOK AT THIS.
BUT IT'S NOT JUST THAT IT'S
THESE FIVE INSTITUTIONS.
IT'S ALL OF US WHO CARE ABOUT
AND I THINK FUNDAMENTALLY ALL OF
US UNDERSTAND AND APPRECIATEED
BENEFITS OF BROADENING, I
SUSPECT THE STANDARDS OF WHAT
SHOULD QUALIFY IN TERMS OF WHAT
SHOULD BE PUBLISHED.
WE UNDERSTAND BY NOT PUBLISHING
NULL STUDIES, WE CREATE
SIGNIFICANT GAP IN KNOWLEDGE
BECAUSE THAT NEVER GETS THERE.
THAT'S BEEN STATED MULTIPLE
TIMES.
INSTITUTIONALLY AND FEDERALLY,
THE AMOUNT OF RESOURCES WE'RE
WASTING BY REPEATING STUDIES
THAT ONLY COULD HAVE DONE US A
LOT OF GOOD HAVE THEY BEEN
PUBLISHED.
YOU'RE A YOUNG RESEARCHER
STARTING ON A NEW RESEARCH LINE,
WOULDN'T YOU BENEFIT IF MUCH OF
THE INFORMATION HAD BEEN IN THE
PUBLIC DOMAIN ALREADY?
AND THEN YOU COULD HAVE CHARTED
YOUR COURSE ELSEWHERE.
IN TALKING TO FACULTY AND IN
TALKING TO A NUMBER OF PEOPLE IN
TERMS OF WHERE WE ARE AND I
THINK CULTURALLY WE'RE NOT WHERE
WE NEED TO BE, AND WHAT ARE THE
CHALLENGES, AND I THINK YOU ALL
HAVE SUMMARIZED FOUR
FUNDAMENTALS, I WANT TO REPEAT
THAT.
ONE IS THE PUBLICATION BIAS AND
YOU'VE TALKED ABOUT IT SO I
WON'T SAY ANYTHING MORE ABOUT
IT.
I THINK THE LAST OF INCENTIVES
IS SOMETHING THAT IS STRIKING,
AT THE LEVEL OF FUNDING AGENCIES
AND ORGANIZATIONS, INCLUDING
NIH, I'LL SAY THAT HERE.
AND OTHER AGENCIES.
AND I THINK THAT NEEDS TO
CHANGE, HOW THE REVIEW PROCESS
IS AND ALL OF THAT.
AND I WOULD SAY THE
METHODOLOGICAL CHALLENGES THAT
SOME OF YOU TALKED ABOUT, THOSE
ARE IMPORTANT.
ARE THESE NULL STUDIES TRULY
NULL STUDIES OR ARE THEY ALSO AN
OUTCOME OF POTENTIALLY ERRORS?
THERE ARE CONSEQUENCES TO SOME
OF THAT.
FINALLY I THINK WHAT YOU WERE
REFERRING TO, IF YOU DON'T
PUBLISH YOU PERISH, HOW DO YOU
DEAL WITH THAT?
SO, A LOT WAS ASKED.
IN FACT WHEN THE NIH AND OTHERS
WERE ASKING ROBERT AND I TO
JOIN, WHAT IS IT THAT WE AS
RESEARCH ADMINISTRATORS ARE
DOING TO HELP AND FACILITATE
WITH THIS PROCESS, LUCKILY THIS
IS ABOUT BARRIERS, WE WON'T TALK
ABOUT SOLUTIONS RIGHT NOW.
THAT'S FOR TOMORROW.
BUT WE NEED TO BE HONEST ABOUT
IT.
IT'S NOT THAT THAT'S THE FIRST
THING THAT COMES TO MIND WHEN WE
WAKE UP EACH MORNING, HOUR JOBS
HAVE OTHER ELEMENTS TO IT BUT
IT'S NOT AT ALL LOST ON US.
BUT I THINK FUND MENTAL AS ERIN
WAS TALKING AS MUCH AS WE VALUE
IMPORTANCE OF ALL OF THIS, IF
JOURNALS ARE NOT ACTUALLY
PUBLISHING NULL STUDIES THESE
IMPACTFUL JOURNALS THAT MAKE A
DIFFERENCE IN THE PROMOTION AND
TENURE OF FACULTY, IT IS GOING
TO BE A CONTINUING CHALLENGE FOR
US AS INSTITUTIONS TO DO
SOMETHING ABOUT IT.
THAT HAS TO CHANGE.
I THINK I WOULD CONFESS AND SAY
THAT THERE IS -- BY NOT
ENCOURAGING PUBLICATION OF NULL
RESULTS, WE ARE DISINCENTIVIZING
OUR SELF WITHOUT REALIZING IT.
THERE NEEDS TO BE IN THAT SENSE
A COMPLETE OVERHAUL HOW WE LOOK
AT HOW RESEARCH IS REWARDED AND
VALUED.
AND I THINK IT IS REALLY
FUNDAMENTAL TO A LOT OF WHAT WE
DO, AND SO I WOULD SAY THAT THE
CHALLENGES THAT WE HAVE,
BARRIERS THAT WE HAVE, ARE TRULY
AT A SYSTEMIC LEVEL.
I MEAN, FIRST AND FOREMOST, THE
AGENCIES AND ORGANIZATIONS THAT
ARE FUNDING, THAT HAS TO CHANGE
WHERE THAT IS VALUED.
IF THAT DOESN'T CHANGE WE'RE
STILL AT THE SAME PLACE.
THE JOURNALS, A LOT WAS TALKED
ABOUT THE OPPORTUNITIES ARE
THERE BUT AS HAS BEEN SAID THE
IMPACT JOURNAL CONCEPT HAS NOT
GONE AWAY.
AND THEN OF COURSE AT
INSTITUTIONAL LEVEL, HOW ARE WE
GOING TO RECOGNIZE, SUPPORT,
REWORD ALL RESEARCH NO MATTER
WHERE THESE JOURNALS ARE AND AS
LONG AS THEY ARE DONE ETHICALLY
AND THEY CONTINUE TO ADD VALUE.
IN CLOSING, I WANT TO SAY I
DON'T THINK THERE'S ANYBODY HERE
WHO FUNDAMENTALLY DISAGREES WE
NEED TO DO MORE TO PROMOTE HOW
FACULTY CAN PROMOTE NULL STUDIES
BUT THIS CONCEPT OF PROMOTION
AND TENURE AND EMPHASIS ON HIGH
IMPACT JOURNALS IS A HUGE
BARRIER NOBEL LAWYERATES WILL
TELL YOU SEVERAL FEEL IF THEY
WERE IN THE SAME CLIMATE TODAY
THEY WOULD NOT HAVE THE SAME
TENURE.
WE HAVE TO THINK ABOUT THIS.
THE LAST THING I'LL SAY, WE HAVE
TO BE CAREFUL ABOUT IMPACT OF
NULL FINDINGS.
I'M NOT SAYING WHERE ERRORS ARE
DONE BUT NULL STUDIES PUBLISHED
BASED ON CERTAINLY FUNDAMENTALS,
AGENCIES CAN FEEL THERE'S NO
REASON TO SUPPORT THIS.
JOURNALS TAKE THE ATTITUDE
THERE'S NO REASON TO PUBLISH THE
RESULTS, LATER MORE IN DEPTH
STUDIES BY SOMEONE OPENED UP, IT
OPENS UP A NEW AREA OF WORKING
YOU ABOUT PEOPLE HAVE LOST TIME,
CAREERS JEOPARDIZED, PEOPLE
FORCED TO DO OTHER THINGS.
WE NEED TO DO MORE, SO LOOKING
FORWARD TO YOUR QUESTIONS AND
DISCUSSION TOMORROW WHERE WE
TALK MORE ABOUT SOLUTIONS.
THANK YOU.
>> WE SEE HIGH LEVEL
ADMINISTRATIVE PEOPLE CAN REALLY
TALK.
WE'VE DONE A GOOD JOB PAINTING
THE PROBLEMS, I'D LOVE TO HEAR
YOUR QUESTIONS AND NOT.
LET'S BEGIN.
>> ARE INDIVIDUAL NULL NEGATIVE
STUDIES CITED.
JOURNAL IMPACT FACTOR,
DISCUSSION FOR 20 YEARS IS THAT
ARTICLE CITED IN THE FIELD
CONFECTION, MORE NUANCED WAY OF
LOOKING AT IT.
I DIDN'T SEE THAT IN YOUR
SLIDES.
>> IN TERMS OF TENURE AND
PROMOTION DOCUMENTS?
YEAH, THEY DO FOCUS ON CITATION
COUNTS, NOT JUST JOURNAL LEVEL
METRICS.
I SUPPOSE THAT WOULD BE INCLUDED
IN THAT.
THERE WAS A QUESTION ABOUT IF
YOU GET A NULL STUDY PUBLISHED
DOES IT COUNT.
IT'S A PUBLICATION BUT ARTICLE
LEVEL METRICS, NOT CITED, YOU
MIGHT GET --
>> IF IT DOESN'T GET CITED.
>> YES.
>> I DON'T KNOW WHETHER PEOPLE
CITE NEGATIVE STUDIES.
>> SOMEBODY MENTIONED THERE
MIGHT BE SOME STUDIES SHOWING
THEY GET CITED LESS.
>> ESPECIALLY IN CLINICAL TRIALS
NEGATIVE STUDIES ARE OFTEN
CITED.
>> UC SANTA CRUZ.
I'M INTERESTED IN THIS QUESTION
ABOUT INCENTIVIZING AT THE
INSTITUTIONAL LEVEL, BECAUSE I'M
A SCIENTIST WHO GOES THROUGH
TENURE AND PROMOTION, I'M ALSO A
FACULTY EQUITY ADVOCATE AT
UNIVERSITY OF CALIFORNIA SANTA
SANTA CRUZ, TRYING TO MANGE
REVIEW MORE EQUITABLE AFTER
INITIALLY FOCUSING ON HIRING.
IN THIS ANALYSIS, ANALYSIS ABOUT
INDIVIDUALS RESPONDED TO THEIR
ASSESSMENT DURING TENURE AND
PROMOTION YOU MENTIONED SOME
PEOPLE SAW IT AS LESS THAN FAIR,
MORE UNFAIR, MORE UNFAIR AS IT
WENT UP THE ACADEMIC LATTER.
WAS THERE ANY -- YOU MENTIONED
WOMEN WERE DOING MORE SERVICE,
WAS THERE DEMOGRAPHIC DATA, MORE
UNFAIR OR MORE FAIR DURING THAT
PROCESS, BECAUSE ONE THING THAT
WE HAVE TO THINK ABOUT WHEN
TALKING ABOUT HOW TO ASSESS NULL
RESULTS AND SCIENCE IN GENERAL
IS THAT OFTEN THAT HAPPENS AT
HAPPEN HAZARD LEVEL, OFTEN
INEQUITABLY WHERE PEOPLE FROM
HISTORICALLY MARGINALIZED GROUPS
ARE BEARING THE BRUNT OF BEING
INNOVATORS AND NOT BEING
RECOGNIZED FOR THAT INNOVATION
AND SO I'D BE INTERESTED IF ANY
OF THE WORK THAT WAS REFERENCED
HERE TOOK INTO ACCOUNT FACULTY
DEMOGRAPHICS.
>> THAT WAS UNIVERSITY LEVEL, WE
CAN GENERALIZE ACROSS GENERAL
POPULATION.
BIGGEST GENDER DIFFERENCE WAS
SERVICE DIFFERENTIAL THERE, AND
WE DIDN'T SEE A SIGNIFICANT
DIFFERENCE BETWEEN THE PERCEIVED
I GUESS DIFFERENCES BETWEEN WHAT
WAS DEEMED FAIR BETWEEN
DEPARTMENT REVEL, UNIVERSITY
LEVEL, COLLEGE LEVEL OF
ASSESSMENT.
WE ENDED UP NOT REPORTING.
WE DIDN'T FOCUS ON DISPARITIES
BETWEEN GENDER AND/OR RACE,
ESPECIALLY WITH RACE, WE HAD SO
FEW RESPONDENTS, WE'RE A
PREDOMINANTLY WHITE INSTITUTION,
UNFORTUNATELY, WE COULDN'T BRING
OUT THAT DATA WITHOUT BEING, YOU
KNOW, PRIVACY CONCERNS AND
THINGS OF THAT NATURE.
SO I WILL SAY THAT FROM A
GENERALIZED, ERIN MAY HAVE MORE
INFORMATION, THERE'S A REALLY --
THERE ARE SOME GOOD PAPERS ABOUT
THIS, WHERE THERE'S A LOT OF
ISSUES EQUITY IN SCIENTIFIC
PAPERS, I THINK IT'S CASSIDY'S
WORK, SHE DOES A LOT OF GOOD
PAPERS, I DON'T KNOW IF YOU
HEARD OF HER WORK.
SHE DOES A LOT OF GOOD RESEARCH
HOW EVEN IF YOU FIND THERE'S
EQUITY IN SCIENCE THERE'S
STILL -- PARITY IN SCIENCE,
STILL PROBLEMS WITH EQUITY.
JUST BECAUSE YOU MAYBE HAVEN'T
MET THAT THRESHOLD DOESN'T MEAN
THERE'S ACTUAL EQUITY, AND WE
STILL HAVEN'T GOTTEN INTO PARITY
YET.
THAT'S GOING TO TAKE
GENERATIONS.
DO YOU HAVE ANYTHING TO ADD TO
THAT?
>> MAYBE A CUSPEL RESULTS
BECAUSE WE DID DO SOME ANALYSIS,
PARTICULARLY IN SURVEY STUDY.
WE WERE ONLY LOOKING AT
DECISIONS IN TERMS OF HOW THEY
DECIDE WHERE TO PUBLISH.
I BELIEVE A SIGNIFICANT EFFECT
IN TERMS OF WOMEN PUBLISHING
LESS ACROSS THE STUDY.
THAT COULD BE BECAUSE THEY HAD A
LARGER SERVICE COMMITMENT AND
THERE ARE QUITE A FEW STUDIES AS
YOU SAY DO SHOW THAT SERVICE
BURDEN COMES MORE HEAVILY ON
WOMEN AND UNDERREPRESENTED
GROUPS.
RELATED TO THAT IN ANOTHER ONE
OF OUR STUDIES WE FOUND SERVICE
IS WHERE THE PUBLIC DIMENSIONS
OF THE WORK WAS GETTING DUMPED,
IT'S ALSO THE MOST UPDATE VALUED
CATEGORY IN PROMOTION AND
TENURE.
YOU KNOW, THAT CATEGORY WHERE
YOU'RE PUTTING ALL THE THINGS
THAT SUPPOSEDLY ARE HOW YOUR
INSTITUTION EMBODIES VALUE AND
MEETS MISSION STATEMENTS ABOUT
DIVERSITY, INCLUSIVITY, PUBLIC
OUTREACH GETTING DUMPED IN THE
LEAST VALUED CATEGORY, WHERE YOU
ALSO HAVE PRIMARILY WOMEN AND
MINORITIES TAKING THE BRUNT.
>> THIS IS INTERESTING.
BEING AT A DISCUSSION WORKSHOP
THAT WE TALK ABOUT UNPUBLISHED
RESULTS, I'VE BEEN AT A NUMBER
OF INSTITUTIONS, WE LOOK AT THIS
DATA.
WE DON'T PUBLISH THE DATA.
ALL RIGHT.
SO WE DO NOT PUBLISH THE DATA.
WHAT I WOULD SAY IS THAT IT IS
VERY TRUE, AND I WROTE AN
EDITORIAL TO VICE PRESIDENTS OF
RESEARCH THAT WE NEED TO BE VERY
CAUTIOUS, IN THE WAY WE SOLICIT
INDIVIDUALS TO PERFORM INTERNAL
SERVICE OBLIGATIONS EVEN FOR
RESEARCH BECAUSE WE FIND
OURSELVES HEAVILY WEIGHTED WOMEN
WITH D.E.I. INITIATIVES IN
GENERAL, HEAVILY WEIGHTED ON
MINORITIES.
IT'S INTERESTING THAT FULL
PROFESSORS TYPICALLY HAVE A
BIGGER VOICE SO WE'RE HEAVILY
WEIGHTED IN THE ASSISTANT AND
ASSOCIATE PROFESSORS IN SERVICE
OBLIGATIONS FOR INSTITUTION,
THOSE ARE SERVICE OBLIGATIONS
THAT HAVE LESSER VALUE AND WE
PROBABLY SHOULD THINK
HOLISTICALLY ABOUT THAT.
AND WHEN WE DO, WHEN I HAVE IT
AT THE INSTITUTIONS THAT I'VE
BEEN AT, WE'VE CHANGE THE THE
DYNAMIC, THE AGREEMENT WAS WITH
THE DEPARTMENT, HAVING SOME
SENIOR VICE CHAIRS OF RESEARCH
DEANS, PERFORM ACTIVITIES SO
THEIR DEPARTMENT CAN CONTINUE
INSTEAD OF PUTTING THE WEIGHT OF
RUNNING AN INSTITUTION ON THE
BACKS OF INDIVIDUALS THAT NEED
TO GET PROMOTED THROUGH THE
INSTITUTION.
SO IT'S A VERY REAL PHENOMENON.
THERE'S NO INTERNAL REPORTS TO
SHARE BUT I CAN TELL YOU IT'S
TRUE.
>> YEAH, ONE QUICK THING TOO.
A LOT OF TIME WE'RE GETTING --
IN MY CASE, I SUSPECT OTHER
FOLKS, WHEN YOU'RE DOING
RESEARCH WITH UNDERGRADS YOU GET
NO RESULTS, DUMPED IN THE
MENTORING CATEGORY, WHICH GETS
DUMPED IN SERVICE, UNDERVALUED.
IF WE LINK BACK TO PUBLICATION
BIAS IS LOT IS FALLING INTO THAT
UNDERVALUED GROUP AND SERVICE AS
WELL.
>> UNIVERSITY OF FLORIDA.
A FEW THINGS ABOUT THE DISPARITY
IN SORT OF NUMBER OF
PUBLICATIONS FOR MEN, FOR WOMEN,
WE ALL KNOW DURING THE PANDEMIC
WOMEN AND PEOPLE OF COLOR AND
CAREGIVERS FOR CHILDREN AND
PARENTS WERE DISPROPORTIONATELY
AFFECTED.
EVERYBODY GOT A YEAR OF COVID
FOR TENURE DELAY SOME OTHER
THINGS WERE NOT CONSIDERED, AND
THAT'S A BIG PROBLEM.
ONE BIG PROBLEM INSTITUTIONS
HAVE, I WAS AT EMORY FOR TEN
YEARS, U.C. SOUTHWESTERN,
UNIVERSITY OF FLORIDA FOR FIVE
YEARS, EVERYTHING REALLY DOES
REVOLVE AROUND NUMBERS.
I GOT A SPACE SURVEY TO MAKE
SURE THE SPACE I WAS SITTING IN,
THE HOW MANY PAPERS HAVE YOU
PUBLISHED IN THE LAST THREE
YEARS, HOW MANY GRANTS DO YOU
HAVE, HOW MANY GRANTS HAVE YOU
SUBMITTED THAT GOT FUNDED, HOW
MANY GRANTS HAVE YOU SUBMITTED
THAT DIDN'T GET FUNDED.
NO QUESTIONS ABOUT HOW MANY
PEOPLE DO YOU HAVE SITTING IN
THAT SPACE, NO REAL QUESTIONS
ABOUT TEACHING, OR SERVICE.
I GUESS YOUR SPACE ALLOCATION
HAS NOTHING TO DO WITH THAT, AND
IT SHOULD.
THAT'S A HUGE PROBLEM.
THE OTHER PROBLEM IS THE H-INDEX
THAT IS USED FOR PROMOTION AND
TENURE, I SAT ON THE TENURE AND
PROMOTION COMMITTEE AT EMORY AND
YEARS AT FLORIDA, THAT IS REALLY
ARTIFICIAL.
BECAUSE THERE ARE A LOT OF
PEOPLE WHO GET ON A PUBLICATION,
ONE THAT COMES TO MIND IS
AUTOPHAGY METHODS, THOUSANDS OF
AUTHORS, CITED ALMOST EVERY DAY,
YEAH, A THOUSAND TIMES, THE
IMPACT FACTOR IS INFLATED FOR
THE THOUSAND PEOPLE ON THERE FOR
THEIR CONTRIBUTION.
RIGHT?
THE OTHER THING, H-INDEX IS
ABOUT TIME, RIGHT?
THERE HAS TO BE ENOUGH TIME TO
SOMEONE TO CITE YOUR PAPERS.
IF YOU PUT THEM IN THE WRONG
JOURNAL WHERE YOUR READERSHIP IS
NOT NO ONE IS GOING TO FIND
THEM.
bioRxiv AND OTHER PRE-PRINTS
THEY MAY COUNT FOR A POSTDOC
GOING OUT FOR A JOB INTERVIEW
BECAUSE IT'S A DELIVERABLE, BUT
NOBODY COUNTS THEM FOR ACTUAL
PEER-REVIEWED PUBLICATIONS,
RIGHT?
THAT IS ALSO A BIG PROBLEM.
BECAUSE THAT'S A DISINCENTIVE TO
PUT THEM OUT THERE.
I SUSPECT THE REGISTERED, YOU
KNOW, PRE-REGISTRY WILL BE THE
SAME ALTHOUGH I LOVE IT, AND I'M
GOING TO PUSH FOR NATURE TO
START CONSIDERING IT.
>> THANK YOU, MALA.
>> I AGREE WITH YOU.
I WAS GOING TO SAY I'M ELOQUENT
IN DESCRIBING WHAT THE
CHALLENGES ARE, THIS WHOLE ISSUE
OF SPACE AND HOW IT'S QUANTIFIED
AND QUANTITATED, AGAIN, IF YOU
LOOK AT A MORE COMPREHENSIVE
UNIVERSITY SYSTEM THAT HAS ARTS
AND SOCIAL SCIENCE VERSUS
BIOMEDICAL SCIENCES, YOU HAVE
DIFFERENT YARDSTICKS, IN TERMS
OF HOW WE MEASURE RESEARCH AND
SCHOLARSHIP.
WHAT IS DONE, EXPECTED IN SOCIAL
SCIENCES, IS ENTIRELY DIFFERENT
THAN THEY WOULD CRINGE IF YOU
USED THE H-INDEX AS A VALUE
FACTOR THERE, RIGHT?
SO IN THE BIOMEDICAL SCIENCES
AND MEDICAL SCHOOLS AND SUCH,
YES, ABSOLUTELY, THEY ARE ALL --
IT'S A VERY NUMBERS-DRIVEN GAME.
IT'S NOT THAT WE NEED TO DO
AWAY, BUT I THINK WE NEED TO
ALSO LOOK AT SOME OF THE FACTORS
THAT YOU'RE TALKING ABOUT.
>> I JUST WANT TO -- WITH A
GROUP OF COLLEAGUES WE DID A
STUDY, GLOBAL STUDY OF RESEARCH
ASSESSMENT ALSO.
WE FOUND THAT THE EVALUATION
BETWEEN THESE DISCIPLINES WAS
THE SAME.
IT DIDN'T CHANGE AT ALL.
>> NUMBER OF PAPERS?
>> SPACE OR PAPERS?
YOU'RE TALKING LABS OR PAPERS?
>> IT'S EVALUATION FOR
PROMOTION.
>> SHE WAS TALKING ABOUT LAB
STAYS EVALUATION, I WAS
COMMENTING ON THAT PART, BUT
YOU'RE RIGHT.
>> WE DID ANALYZE DIFFERENT
ACADEMIC DISCIPLINES, BROKE IT
DOWN BY LIFE SCIENCES OR
PHYSICAL SCIENCE, MATHEMATICAL
SCIENCE DEPARTMENTS,
MULTI-DISCIPLINARY UNITS.
THERE WERE DIFFERENCES BUT IT
DEPENDED WHAT WE LOOKED AT.
SO FOR EXAMPLE WHEN WE WERE
LOOKING AT THE NON-TRADITIONAL
OUTPUTS, SOME IN THE HUMANITIES,
THEY WOULD MENTION ARTS THAT
DIDN'T COME UP ANYWHERE IN THE
LIFE SCIENCES, LIFE SCIENCE HAD
A STRONGER FOCUS ON PARTICULAR
PUBLICATION COUNTS OR THINGS, SO
WE DID SEE SOME DIFFERENCE IN
TERMS OF HOW THEY WERE BREAKING
DOWN OR WHAT ADDITIONAL
NON-TRADITIONAL OUTPUTS THEY
INCLUDED IN THEIR PROMOTION AND
TENURE DOCUMENTS.
>> REAL QUICK, LET ME RESPOND TO
THIS.
YOU'VE BEEN AT RESEARCH
INTENSIVE INSTITUTIONS.
THE CHALLENGE IS THAT WE MEASURE
OUTPUTS AND IMPACT.
COMMON DENOMINATOR COMES FROM
MULTIPLE DATA SOURCES.
SO, THE WAY CONCEPTUALLY THAT
INSTITUTIONS ARE THINKING ABOUT
OUR FACULTY FOR RESEARCH
INTENSIVE INSTITUTIONS, ARE LIKE
MANY NON-PROFITS.
YOU'RE THE EXECUTIVE DIRECTOR OF
YOUR OWN NON-PROFIT ORGANIZATION
WITHIN OUR INSTITUTION, AND WE
VALUE -- THE INSTITUTIONS VALET
WARDS.
SO YOU'RE PAYING FOR SPACE
THROUGH YOUR WORK.
IF YOU DON'T GENERATE, IT'S
UNFORTUNATE, BUT IT'S TRUE.
IN THESE INSTITUTIONS IF YOU'RE
NOT GENERATING RESOURCES THAT WE
USE TO JUSTIFY THE SPACE, THEY
GIVE THE SPACE AWAY TO PEOPLE
WHO GENERATE MORE RESOURCES.
BUT IF YOU GO TO A LIBERAL ARTS
INSTITUTION OR SOME EMERGING
RESEARCH INSTITUTIONS THEY DON'T
THINK ABOUT IT THE SAME WAY.
AND SO THE ISSUE, PARADOX THAT
WE'RE IN, INSTITUTIONS THAT
YOU'VE BEEN AT ALL OVER 500
MILLION, CLOSE TO ONE BILLION IN
RESEARCH AWARDS ON ANNUAL BASIS,
THEY HAVE CONDITIONED THEMSELF
TO OVERVALUE THAT.
AND NOT VALUE JUST THE HEART OF
THE SCIENCE OR OTHER ACTIVITIES.
SO THEY PROBABLY HAVE THAT SAME
DEBATE ON THOSE CAMPUSES FOR
ARTS AND HUMANITIES AND OTHER
GROUPS NOT GETTING THE SPACE
THEY NEED.
>> BUT THE IMPACT CAN BE
MEASURED IN DIFFERENT WAYS.
WHERE ARE MY TRAINEES, ARE THEY
MAKING AN IMPACT?
WHAT BOARDS DO I SIT ON?
NOBODY ASKS THOSE QUESTIONS.
>> YEAH, I WANT TO MAKE ONE
COMMENT ABOUT THAT.
WHICH IS WHAT WOULD HAPPEN TO
ALL OF THESE INSTITUTIONS, R 1
INSTITUTIONS, IF INDIRECT COST
RETURN WAS 10%?
15%?
WOULD WE BE CARRYING AS MUCH
ABOUT THESE GRANTS IF IT WAS
SUCH A SMALL AMOUNT, WHERE OUR
WEIGHT OF PUBLICATIONS, WOULD IT
BE DIFFERENT IF IT WERE THAT?
THERE IS SCARILY OR EXCITINGLY A
CALL FOR MUCH LESS
INSTITUTIONAL -- IBC, YEAH,
I.C. -- INCORRECT COST, ICR,
YEAH.
IT MIGHT -- THAT MIGHT BE THE
REVOLUTION.
>> WAIT, WAIT, WE'RE NOT
ADVOCATING FOR THAT.
>> I THINK SOMETHING SHE'S
TRYING SAY, WHAT WOULD HAPPEN IF
WE STARTED MEASURING
INSTITUTIONS IN TERMS OF WHETHER
THEY ARE MEETING THEIR MISSION
STATEMENTS BECAUSE WE DON'T
MEASURE THAT.
ARE YOU ACTUALLY GENERATING
PUBLIC GOOD?
ARE YOU MEETING YOUR LOCAL
COMMUNITY'S NEEDS?
WE DON'T MEASURE THAT.
I THINK THAT'S THE POINT, LIKE
WHAT IS YOUR MEASURE OF IMPACT?
HOW DOES THAT MATCH UP?
THERE ARE SOME INSTITUTIONS WHO
WILL REMAIN NAMELESS THAT DON'T
ALLOW INVESTIGATORS TO APPLY FOR
GRANTS THAT CARRY NO INDIRECTS.
>> ONE THING I WOULD SAY AS A
SOLUTION, THAT'S FOR TOMORROW, I
WOULD SAY THIS.
INSTITUTIONS CYCLE THROUGH
LEADERSHIP.
WHAT THEY DON'T HAVE IS IS A
PLAYBOOK ON HOW TO THINK ABOUT
MEASURING IMPACT.
IF YOU WANT TO MAKE A DIFFERENCE
IT NEEDS TO BE A COLLECTIVE,
WRITE A PAPER, AND INFORMATION
THAT GOES TO ADMINISTRATION ON
HOW TO MEASURE RESEARCH
APPROPRIATELY, AND THEN THEY
WILL ADOPT IT.
WE'RE
>> WE'RE GOING TO THE NEXT
QUESTION.
>> I THINK .
>> WE'RE GOING TO THE NEXT
QUESTION, WE DON'T HAVE MUCH
MORE TIME.
>> THANK YOU.
I HAVE A QUESTION MOSTLY FOR
RACHEL AND ERIN, BOTH HAVE DONE
SURVEYS ABOUT FACULTY HOW THEY
FEEL ABOUT THE TENURE PROMOTION
CRITERIA, RIGHT?
TO WHAT EXTENT DO YOU FEEL LIKE
CONCERNS RAISED ARE UNIVERSAL,
HUMAN CONCERNS ABOUT HOW THEY
ARE BEING JUDGED AND SPECIFIC TO
YOUR UNIVERSITY OR TO YOUR
GEOGRAPHIC REGION?
>> IT'S VERY UNIVERSAL.
ERIN CAN SPEAK MORE, BEING
GENERALIZED, 800 DOCUMENTS
ACROSS U.S. AND CANADA.
I DIDN'T MENTION WE DID A
NATIONAL STUDY FOCUSED ON
ACADEMIC LIBRARIANS, AND WE
ASKED THEM LOTS OF QUESTIONS BUT
IT WAS FOCUSED ON RESEARCH
METRICS AND ASSESSMENT AND WE
ASKED THEM TWO QUESTIONS, ABOUT
CONSULTATIONS WITH FACULTY, SO
WE ASKED THEM, YOU KNOW, WHAT
METRICS DO YOU DISCUSS WITH
FACULTY AND HOW OFTEN DO YOU
DISCUSS THOSE ON A LIKERT SCALE.
SO WHEN WE WERE ASKING THEM HOW
OFTEN DO YOU DISCUSS THESE
METRICS ON A LIKERT SCALE, WHEN
IT COMES TO PUBLISHING ISSUES,
YOU KNOW, THERE WAS NOT THAT
MUCH INTEREST.
WHEN WE ASKED HOW OFT OFTEN DO
YOU DISCUSS THESE WITH TENURE
AND GRANTS, IT'S A 4 OR 5 ON THE
LIKERT SCALE WITH STATISTICAL
DIFFERENCE BETWEEN THE ANSWERS,
AND I WASN'T EVEN LOOKING,
REALIZED THERE WAS A DIFFERENCE
WHEN DOING A POSTER.
THIS IS UNIVERSAL.
THAT WAS A NATIONAL STUDY.
THIS HAS BEEN FOUND IN OTHER
SURVEY RESULTS AS WELL.
I THINK THAT THE PRESSURES BEING
PUT ON ACADEMICS ARE COMING FROM
EXTERNAL FACTORS, NOT REALLY --
THIS IS NOT AN INTERNAL DRIVING
THING, THEY ARE NOT SAYING, OH,
YEAH, I CARE ABOUT THE JOURNAL
IMPACT FACTOR.
THEY ARE REALLY CARING I
THINK -- I THINK THEY ARE
TRAINED TO FIND -- TRAINED AS
SCIENTISTS AND THEN HAVING
EXTERNAL PRESSURES TO BE
SUPERSTARS IN THEIR FIELD AND
PRESENT THEMSELVES IN A WAY AND
BEHAVIOR CHANGES AS A RESULT.
>> YEAH, SO I'LL PUT A TAYLOR
SWIFT REFERENCE IN HERE.
WE WERE THE PROBLEM.
IT'S US.
SO, EVERYBODY'S TALKING ABOUT,
WELL, I VALUE THIS BUT THEY
VALUE.
WHO IS "THEY"?
THEY ARE YOUR COLLEAGUES, RIGHT?
WE NEED TO BE TALKING MORE TO
ONE ANOTHER BECAUSE IT'S THIS
GROUP OF FACULTY AND WE ALL
THINK WE'RE -- WE ALL WANT TO
VALUE, YOU KNOW, READERSHIP AND
AUDIENCE AND ARE WE GETTING TO
THE RIGHT PEOPLE BUT WE THINK
THE OTHER FOLKS -- THAT HAPPENS
WHEN YOU HAVE THESE EVALUATIONS
HAPPEN BEHIND CLOSED DOORS,
WHERE NOBODY WANTS TO TALK ABOUT
WHAT'S GOING ON, HOW PEOPLE ARE
EVALUATED.
WE NEED TO OPEN THAT UP.
THAT FEELING IS UNIVERSAL,
RIGHT?
I'LL SAY WE ANALYZE THESE
DOCUMENTS, SURVEYED FACULTY
ACROSS DIFFERENT INSTITUTIONS,
IT WASN'T JUST THE R TYPES, IT
WAS M TYPE AND B TYPE
INSTITUTIONS WHERE SUPPOSEDLY
THE FOCUS IS DIFFERENT, TEACHING
LOAD, RESEARCH INTENSIVE IS
DIFFERENT, STILL SAYING MY
COLLEAGUES ARE TOO FOCUSED ON
PRESTIGE AND TOO FOCUSED ON
JOURNAL LABEL AND JOURNAL
PACKAGE AND THINGS LIKE THAT.
I THINK THOSE THINGS DID GO
ACROSS TYPE WHEN MISSIONS AND
RESEARCH INSTITUTIONS WERE
DIFFERENT.
>> LAURA SMITH.
I THINK WE'RE GETTING INTO THIS
NOW.
AS A PERSON GOING THROUGH TENURE
APPLICATION PROCESS I THOUGHT I
WOULD BRING UP THINGS I WAS
ASKED SPECIFICALLY ABOUT MY
IMPACTS ON THE FIELD.
HOW HAVE I MOVED MY FIELD, YOU
KNOW, ON A NATIONAL,
INTERNATIONAL LEVEL, AND THINGS
YOU CAN USE AS EVIDENCE FOR
THAT, H-INDEX OF COURSE, MOMENTA
SCORES, YOU CAN -- SOCIAL MEDIA,
GET iCITE, I GUESS I'M JUST
WONDERING, IF I'M SUCCESSFUL
PUBLISHING NULL PUBLICATIONS,
THAT'S NOT GOING TO CHANGE MY
IMPACT ON THE FIELD, I DOUBT.
SHOULD WE BE ASKING ABOUT
IMPACTS AND IF WE SHOULD, THEN
WHAT SHOULD IMPACT BE?
>> IT'S A VERY INTERESTING
QUESTION.
AGAIN, AS WE TALK IMPACT IN
DIFFERENT FIELDS, GOING TO BE
EXTRAORDINARILY DIFFERENT, IF
YOU'RE A CLINICIAN IMPACT IS ONE
THING.
IF YOU'RE AN ARTS RESEARCHER,
IT'S VERY DIFFERENT.
IF YOU'RE IN EDUCATION, IMPACT
IS DIFFERENT.
EACH OF THOSE HAS TO BE
EVALUATED ON THEIR OWN MERITS
AND HAS TO BE FACTOR IN WHERE
WE'RE LOOKING AT WHAT DOES
COMMUNITY IMPACT LOOK LIKE, NO
MATTER WHERE WE'RE DOING.
SO, I THINK INCREASINGLY WE ARE
ASKING THESE SORTS OF QUESTIONS
HOW DO WE MEASURE IMPACT AND ONE
WAY WE FEEL THAT WE
ADMINISTRATIVELY HAVE A LEVER
WHERE WE CAN ASK THESE RIGHT
KINDS OF QUESTIONS, THE INTERNAL
FUNDING WE PROVIDE OUR FACULTY.
WE'RE NOT JUST SAYING IT ONLY
LEADS TO EXTRAMURAL FUNDING,
YES, IT'S AN IMPORTANT CRITERION
FOR ALL THE REASONS OF INDIRECT
COST BUT WE'RE ALSO SAYING IF WE
ARE TO SUPPORT RESEARCH IN
MULTIPLE AREAS WE TRULY HAVE TO
MEASURE IMPACT OF WHAT IT MEANS
IN EACH OF THE DISCIPLINES AND
NOT USE JUST H-INDEX AND JUST
THOSE KIND OF CRITERIA ACROSS
THE BOARD BECAUSE IF WE DO THAT
WE ENDS UP WITH THESE
SITUATIONS.
>> THANK YOU.
I WONDER IF YOU REPRESENT
SIGNATORIES OF DORA?
>> DECLARATION OF RESEARCH
ASSESSMENT.
WE DECIDED TO GO IN A DIFFERENT
DIRECTION DID OUR OWN STATEMENT
ON RESPONSIBLE RESEARCH METRICS.
>> MINE IS NOT.
>> I HAVE TO LOOK IT UP.
>> THAT'S PART OF MY QUESTION.
I'M A POSTDOC RIGHT NOW.
I JUST MOVED TO THE U.S. FROM
EUROPE.
AND IN ALMOST EVERY POSTDOC TALK
OR CONFERENCE I WENT IN EUROPE
THEY KIND OF WERE AWARE OF DORA,
WHICH WAS CREATED HERE IN THE
U.S.
BUT SINCE I'VE COME HERE TO THE
U.S. VERY FEW PEOPLE KNOW ABOUT
DORA.
I WONDER WHY AND HOW CAN WE TALK
THIS?
>> IT'S A GOOD QUESTION.
OH, RIGHT THERE.
YOU'VE GOT THE PERFECT PERSON
RIGHT BEHIND YOU TO ANSWER THAT.
I'LL JUST SAY A LOT OF
INSTITUTIONS IN LATIN AMERICA
HAVE NOT SIGNED O THERE'S A
GENERAL MOVEMENT TOWARDS THAT.
IN THE EVALUATION SYSTEMS WE'RE
SEEING FOLKS GOING BACKWARDS,
ADDING IN IMPACT FACTOR.
NO, WE'RE SUPPOSED TO GO THE
OTHER WAY.
THE OTHER PROBLEM, THIS IS NOT
TO DISCREDIT ANY DECLARATION,
BUT I THINK IF YOUR INSTITUTION
SIGNS THAT NEEDS TO BE BACKED UP
WITH HERE ARE ACTIONS WE'RE
GOING TO TAKE FOLLOWING OUR
SIGNATURE, RIGHT?
BECAUSE WE HAVE SEEN EXAMPLES
WHERE INSTITUTIONS HAVE SIGNED
AND THEN THEIR CRITERIA CONTINUE
TO INCLUDE IMPACT FACTOR OR DO
THINGS IN DIRECT CONTRADICTION.
STEPHEN, I'LL HAND IT TO YOU.
>> I'LL CHIP IN BRIEFLY.
THERE ARE HUGE CULTURAL
DIFFERENCES BETWEEN EUROPE AND
THE UNITED STATES.
AND AS FORMER CHAIR OF DORA I
DON'T CARE WHETHER PEOPLE SIGN
IT OR NOT.
WHAT I CARE ABOUT IS HOW THEY GO
ABOUT RESEARCH ASSESSMENT AND SO
IF YOU HAVE YOUR OWN STATEMENT
AND YOU THINK IT'S EVERY BIT AS
GOOD AS DORAS, BULLY FOR YOU.
THAT'S FINE BY ME AS LONG AS YOU
LIVE UP TO THE IDEAS.
THAT'S A WHOLE OTHER
CONVERSATION.
>> I AGREE, TRYING TO, IT'S NOT
ACTIVELY TALKED AND DISCUSSED
AND SO I THINK WE NEED TO
ELEVATE THAT.
>> THE LAST TIME I LOOKED, IN
THE U.K. THERE WERE 84 I THINK
INSTITUTIONS THAT SIGNED ON TO
DORA.
IN THE GREAT UNITED STATES, TWO.
>> UPenn.
I HAVE TWO QUICK THINGS.
FIRST, JUST A COMMENT ON THE --
WHAT YOU BROUGHT UP IN YOUR TALK
ABOUT SERVICE, AND DIFFERENCES,
BETWEEN MEN AND WOMEN, AND
PERCEIVED AMONG SERVICE DONE,
THERE'S ACTUALLY A LOT OF DATA
ON THIS FROM UNIVERSITY OF
PITTSBURGH, IT'S SOMETHING THAT
SHE HAS CALLED NON-PROMOTABLE
WORK AND THERE'S A SEVERE GENDER
DIFFERENCE BETWEEN MEN AND
WOMEN, WOMEN TAKING ON MORE
NON-PROMOTABLE WORK, MOSTLY
SERVICE TYPE WORK.
AND SO FOR THOSE INTERESTED AND
I THINK IT'S IMPORTANT JUST TO
BE AWARE OF THAT DATA, THAT IT
EXISTS, SHE LOOKS AT CAUSES OF
THIS TOO.
SO JUST SOMETHING TO KEEP IN
MIND.
THE OTHER THING IS WE'VE BEEN
TALKING ABOUT FACULTY BUT THE
ONES CONDUCTING THE RESEARCH ARE
TRAINEES MOST OF THE TIME.
AND SO I THINK IT'S IMPORTANT TO
THINK ABOUT THAT AND LIKE HOW
ARE THESE EXPECTATIONS BEING PUT
ON FACULTY, TRICKLING DOWN TO
TRAINEES AND HOW WILL THAT
AFFECT THEM GOING THROUGH
TRAINING GETTING DOCTORATES,
BEING POSTDOC, FINDING A FACULTY
POSITION LATER.
>> EXCELLENT POINT.
IT'S SOMETHING WE NEED TO TRAIN
THE NEXT GENERATION OR ACADEMIA
PERCEIVES ONE FUNERAL AT A TIME.
IT'S NOT THE CASE.
THE PROBLEM IS IF THE FACULTY
MEMBERS ARE TRAINING STUDENTS TO
THINK ABOUT PRESTIGE AND IMPACT
FACTOR AND PUBLICATION AND NOT
PUBLISHING NULL RESULTS, YOU'RE
GOING TO END UP WITH THAT
PROBLEM PROPAGATING AND
CONTINUING TO PROPAGATE THROUGH
THE SYSTEM.
THEY ARE BEING TRAINED BY THE
VERY PEOPLE THAT THINK THAT BAY.
IF WE'RE GIVING PROMOTION AND
TENURE TO THOSE PEOPLE WE'RE
FILTERING OUT FOLKS THAT MIGHT
THINK DIFFERENTLY, YOU KNOW,
VERY EARLY ON IN THE PROCESS
URGES.
>> A COMMENTER EARLY ON, THE
YOUNGER POSTDOCS, PEOPLE
PROMOTING OPEN SCIENCE, THIS IS
GOING TO TAKE SOME TIME.
>> EVEN LONGER IF OUR
INSTITUTIONS ARE USING H-INDEXES
IN PROMOTION CRITERIA.
THAT WILL GRIND OUT THE IDEALISM
AND NOBILITY WE FIND IN THE
YOUNGER GENERATION BECAUSE THEY
WILL GET SUCKED INTO THE SYSTEM
AND HERE IN 20 YEARS TIME HAVING
THE SAME CONVERSATION WE'RE
HAVING TODAY.
IF YOU'RE AN INSTITUTION USING
H-INDEX OR PLAY THE UNIVERSITY
RANKING GAME I HOPE YOU'RE NOT
CONSIDERING YOURSELF FIT TO
TEACH YOUR STUDENTS ABOUT
CRITICAL THINKING.
YOU NEED TO STEP UP TO THE PLATE
AND LOOK AT HOW FLAWED THESE
MECHANISMS ARE.
>> I THINK THE OBSESSION WITH
INSTITUTIONAL RANKINGS IS A BIG
PART OF THE PROBLEM.
>> THEN SECEDE, PULL OUT.
>> ALL THESE WORLD LEADING
INSTITUTIONS, SELF-PROCLAIMED,
PLAY THE RANKING GAMES.
>> THEY ARE ON THE REVIEW
COMMITTEE TOO.
>> H-INDEX ARE NOT THE SAME.
LET ME SAY THIS.
AT OUR LEVEL WE DO NOT ESTABLISH
THE TENURE PROMOTION GUIDELINES
FOR OUR INSTITUTION.
THOSE HAPPEN WITHIN THE
DEPARTMENT BY PEERS WITHIN THAT
DEPARTMENT WHERE THEY COME UP
WITH CRITERIA AND SUBMIT IT
THROUGH INSTITUTION.
SO, IT IS NOT US WHO ARE
PROMOTING H-INDEXES FROM
INSTITUTIONAL LEVEL.
THIS GOES BACK TO WHAT I SAID.
PEERS, EVEN WITHIN YOUR
DEPARTMENT, AND THE DEPARTMENT
CHAIR, INCENTIVIZE WHAT THEY
DETERMINE TO BE IMPORTANT BASED
ON WHAT'S GOING ON IN THE
ENVIRONMENT.
IT IS NOT -- I MEAN, I DON'T
THINK YOU ESTABLISH PROMOTION.
WE DON'T DO THAT.
IT HAPPENS SOMEWHERE ELSE, SO
THE FOCUS, THE FOCUS, AND THE
DIRECTION, IS REALLY TO HELP
DEPARTMENT AND PEERS UNDERSTAND
HOW TO JUDGE THEMSELVES.
>> BUT CAN INFLUENCE
DEPARTMENTS, CAN'T THEY?
I DON'T KNOW.
>> I CAN PUSH BACK.
THIS IS PERFECT TIMING.
WHEN WE RUN WORKSHOPS WITH
ACADEMICS OR WITH UNIVERSITY
LEADERSHIP, THE UNIVERSITY
LEADERSHIP SAYS IT'S THE FACULTY
THAT DECIDES.
THEY HAVE TO DO THE REFORM.
FACULTY SAY I CAN'T DO THIS
WITHOUT SUPPORT FROM LEADERSHIP.
SOMETHING IS FAILING IN THE
COMMUNICATION THERE.
LIKE AGAIN WE HAVE TO TALK TO
ONE ANOTHER.
WE HAVE TO GET THOSE PEOPLE IN
THE SAME ROOM BECAUSE I REALIZE
IT DOESN'T HAPPEN AT THE
DEPARTMENT LEVEL BECAUSE WE DO
HAVE DISCIPLINARY NORMS AND
STANDARDS BUT THERE NEEDS TO BE
AGREEMENT FROM LEADERSHIP TO SAY
TO FACULTY I'M EMPOWERING YOU TO
GO FORWARD WITH THIS REFORM.
WE NEED BOTH.
>> SHE'S RIGHT.
AND SO ROBERT AND I IN OUR ROLES
AS VPs FOR RESEARCH DON'T
MANAGE ACADEMIC AFFAIRS.
WE CAN'T NECESSARILY TALK TO THE
SPECIFICS BUT OUR PROVOST AND
ACADEMIC OFFICERS DO, AT VCU
THIS ISSUE HAS COME UP.
SO LED BY THE PROVOST'S OFFICE,
THEY HAVE DONE AN ENTIRE
ACADEMIC TRANSFORMATION WHICH
WAS LED THROUGH A FACULTY AND
ADMINISTRATIVE INTEGRATION THAT
NEEDS TO BE DEPLOYED.
MAYBE I'LL BE BACK A YEAR FROM
NOW AND REPORT HOW THAT'S
LOOKING BUT THESE ARE EXACTLY
THE ISSUES, YEAH.
>> SO, THANKS FOR SETTING ME UP.
MIKE DOUGHERTY, DEPARTMENT
CHAIR, MID-LEVEL MANAGEMENT,
MATTER TO NOBODY BUT EVERYBODY
SIMULTANEOUSLY.
REFORMING ACADEMIC SYSTEM,
INCENTIVE SYSTEM WITHIN MY
DEPARTMENT, THIS HAS BEEN FROM
HIRING UP THROUGH TENURE
REVIEWS, PROMOTION, ONE THING I
LEARNED EARLY ON FACULTY THOUGHT
THEY COULDN'T DO THINGS THEY
REALLY HAVE A LOT OF POWER OVER.
SO I WOULD GO TO MY FACULTY AND
SAY WE NEED TO REFORM THESE
SYSTEMS.
THEY SAID WELL, ARE WE ALLOWED?
I JUST TALKED TO ASSOCIATE DEAN
FOR FACULTY AFFAIRS, YEAH, WE'RE
THE ONES WHO DECIDE.
WE DID THINGS LIKE NOW HAVE
PROMOTION AND TENURE GUIDELINES,
RESEARCH QUALITY AND RIGOR.
WE ELIMINATED ALL THOSE METRICS
PEOPLE ARE COMPLAINING ABOUT, WE
NO LONGER LOOK AT IMPACT FACTOR
CITATION COUNTS, NOW EXCLUDED,
SPECIFICALLY MENTIONED WE DO NOT
USE THESE BECAUSE THERE'S NO
DATA TO SUPPORT THEIR USE.
IT'S INCREDIBLY LIBERATING FROM
THE FACULTY SIDE TO KNOW, HEY, I
DON'T HAVE TO PURSUE NATURE,
DON'T HAVE TO PURSUE SCIENCE, I
CAN PUBLISH WITH MY SCIENCE BEST
FITS, DIFFERENT DEPENDING ON
SUBDISCIPLINE.
NOT EVERYBODY IS DOING WORK
THAT'S USEFUL FOR THE GENERAL
SCIENCE OR NATURE AUDIENCE,
MIGHT NEED TO GO TO A MORE
SPECIALIZED JOURNAL.
THAT'S OKAY.
THERE'S A DIVERSITY OF DIFFERENT
TIMES OF RESEARCH PEOPLE DO THAT
NEED TO GO TO OUTLETS AND WE
SHOULD BE EMPOWERING PEOPLE TO
PUT THE WORK WHERE IT'S GOING TO
HAVE VISIBILITY FOR COMMUNITIES
THEY SERVE, WHETHER IT'S, YOU
KNOW, VERY SPECIFIC COMMUNITY
THAT NEEDS A SPECIFIC SOLUTION
THAT'S GOOD FOR THEM BUT MAY NOT
APPLY TO ANYBODY ELSE OR MAYBE A
BASIC SCIENCE SOLUTION.
THE OTHER THING I'VE LEARNED
FROM EARLY CAREER PERSPECTIVE
THIS HAS BECOME A RECRUITMENT
TOOL.
EVERYBODY COMES INTO MY OFFICE,
RECRUITING FACULTY, THEY SAY SO
I HEAR YOU'VE REFORMED YOUR
TENURE AND PROMOTION, TELL ME
ABOUT THIS.
AND THEY ARE ON BOARD WITH A LOT
OF THESE THINGS WE'VE BEEN
TALKING ABOUT HERE, REALLY WANT
TO SEE REFORM EFFORTS HAPPEN.
THIS IS A GREAT WORKSHOP TO HELP
PUSH THROUGH.
IT'S BIGGER THAN PUBLICATION
BIAS, THAT'S A HUGE PART OF IT,
AND SO I THINK THIS IS THE RIGHT
FORUM TO BE ADDRESSING ISSUES.
TACKLING AT SYSTEMIC LEVEL, BUT
ONE THING THAT COMES UP WHEN I
PRESENT ON THIS IS ARE YOU
PUTTING YOUR GRADUATE STUDENTS
AT DISADVANTAGE?
BECAUSE THEY LOOK AT THE SYSTEM.
WE'RE TRAINING STUDENTS, PUTTING
THEM IN A SYSTEM WHERE WE'RE
INCENTIVIZING THEM TO DO THE
WORK THAT MATTERS TO THEM,
PUBLISH IN A PLACE THAT MATTERS
TO THEM, AND THEN THE QUESTION
IS, WELL, DOES THAT PUT THEM AT
DISADVANTAGE?
TO ME THAT'S A PERCEPTION
PROBLEM ABOUT THE SYSTEM.
THE SYSTEM THEY ARE GOING TO BE
IN, AND SOME OF THEM SEE THAT
SYSTEM AND THINK I DON'T WANT TO
BE PART OF IT.
THEY OPT OUT.
THAT'S A HUGE PROBLEM.
WE HAVE PUBLICATION BIAS,
SELECTING CERTAIN THINGS TO BE
PUBLISHED, SELECTING CERTAIN
PEOPLE TO BE PART OF THAT
SYSTEM.
>> THANK YOU, I'M NOT GOING TO
ASK THE PANEL TO RESPOND BECAUSE
WE'RE HAVE FOUR MINUTES.
>> I JUST WANT TO PUT A PLUG IN.
I JUST PUBLISHED A PAPER, WE'RE
DOING EXACTLY WHAT YOU
SUGGESTED, WE MEASURE FORGET
WHERE YOU PUBLISHED, HOW MANY
CITATIONS IN ABSOLUTE SENSE, BUT
WE'D LIKE TO TABULATE WHEREVER
YOU PUBLISHED, SEE HOW WELL YOU
ARE FARING RELATIVE TO OTHER
PEOPLE IN THE SAME JOURNAL, SAME
YEAR, AND METRIC CALLED THE
P-INDEX, PROVIDES A VERY
DIFFERENT WAY OF LOOKING AT WHAT
WE CALL SOFT LEADERSHIP, SO THAT
YOU STILL WANT TO BE ABLE TO
MEASURE SOME KIND OF QUALITY
INDEX BUT STILL RETAIN THE
FLEXIBILITY, THE FREEDOM FOR
RESEARCHER TO NOT BE BOUND BY
H-INDEX OR WHATEVER THE JOURNAL
THAT MAXIMIZES THAT CITATION.
IN OUR DATASET, H-INDEX IS THE
CORRELATION WITH THE NUMBER OF
PUBLICATION, .93, NUMBER OF
CITATIONS, NUMBER OF
PUBLICATIONS, . 89, NOT ADDING
INFORMATION OVER AND ABOVE
PRODUCTIVITY, METRICS WE HAVE
PROVIDE POTENTIAL COMPLEMENT.
THE OTHER POINT THAT WE HAVEN'T
TALKED ABOUT GOING BACK TO THE
NULL EFFECT, I DO THINK THAT WE
NEED TO BE PRECISE.
THERE'S A WHOLE RANGE OF THINGS
THAT COULD PRODUCE NULL RESULTS.
SOME NULL RESULTS HAVE MORE
INFORMATION THAN OTHERS.
IF YOU PUBLISH A SILLY STUDY,
THAT NOBODY CARESB I -- CARES
ABOUT, I DON'T THINK WE'RE
HELPING THE FIELD.
EARLIER A PAPER DEBUNKED A
COMMON INTERPRETATION OF
MOLECULAR EFFECT, IF YOU CAN --
WE HAVE TO BE MORE PRECISE.
>> THANK YOU.
WE'RE GOING TO TRY ONE MORE
QUESTION.
>> THIS IS A QUESTION FOR ERIN
MAYBE.
I'M WONDERING, I LOVE YOUR
ANALYSES OF WHAT GOES INTO THESE
DOCUMENTS AND EVERYTHING.
THIS WHOLE THING IS LIKE A GAME
OF PROXY AND METRICS, METRIC
BECOMES A GOAL, WHETHER OR NOT
THERE ARE ASPECTS IF YOU HAVE
DATA ON ASPECT FOR WHICH THERE'S
MORE DISCRETION, YOU COUNT ON
EXTERNAL REVIEWERS TO WEIGH IN
ON THOSE THINGS BUT SELECT THOSE
EXTERNAL REVIEWERS AND DEPENDING
WHO YOU SELECT THERE'S
DISCRETION THERE, WHO YOU SELECT
AND WHAT THEY VALUE, WHAT THEY
SAY IS IMPACTFUL, GOING TO HAVE
A BIG EFFECT HERE.
CAN THERE BE SOME -- ESPECIALLY
IT'S ALWAYS SOMEBODY ELSE'S
FAULT, IT'S UP TO SOMEONE TO
DECIDE WHO TO ASK, PEOPLE THAT
MIGHT VALUE ONE THING OR
ANOTHER, DO YOU HAVE ANY DATA ON
LIKE THE STANDARDIZATION OF LIKE
PRACTICES FOR EXTERNAL
REVIEWERS?
>> YES.
IT'S A GOOD QUESTION.
I MEAN, WE DIDN'T LOOK
SPECIFICALLY AT EXTERNAL
REVIEWER COMPONENT LIKE WHAT THE
TENURE AND PROMOTION DOCUMENTS
WERE SAYING ABOUT EXTERNAL
REVIEWERS.
I WILL SAY TOO THAT I THINK
THERE'S AN ASSUMPTION ALL
UNIVERSITIES HAVE EXTERNAL
REVIEW, MY UNIVERSITY DOESN'T.
SO IT'S NOT PART OF THE
PROMOTION AND TENURE PROCESS.
I THINK THAT'S ANOTHER THING.
I THINK YOUR QUESTION IS
PROBABLY MORE BROADER ABOUT LIKE
HOW MUCH WIGGLE ROOM IS ACTUALLY
IN THERE.
IT'S TRUE THERE'S WHAT'S WRONG,
HOW IT'S INTERPRETED
INTERPRETED AND CLOSED DOOR
DISCUSSIONS, NOT ONE-TO-ONE.
WE DID SEE INTENTIONAL VAGUENESS
MEANT TO PROVIDE EXACTLY THAT.
WE DID ANALYSIS, THE TERM
COLLEGIALITY WHICH I HATE,
PRECISELY BECAUSE IT IS USED IN
THAT IF YOU'RE GETTING ALONG, IT
CAN BE USED IN YOUR FAVOR AND
ALSO AGAINST YOU IN REALLY
BIASED WAYS.
THERE'S A LOT OF INTENTIONAL
VAGUENESS IN THE DOCUMENTS.
WE DON'T HAVE THE COMPLETE
PICTURE BY LOOKING AT THE DOCS
BUT IT GETS US CLOSER.
>> IT'S PAST 6:00.
>> WE'LL WRAP THIS DISCUSSION.
THANK YOU TO THE MODERATOR AND 12493 07:38:14,166 PANELISTS.