I'D LIKE TO WELCOME YOU TO THIS
WORKSHOP WHICH WE'RE LOOKING
FORWARD TO BE REALLY EXCITING
AND GREAT THINGS WILL HAPPEN.
AND THIS WORKSHOP WOULDN'T HAVE
HAPPENED WITHOUT THE SUPPORT OF
OUR LEADER, WALTER KOROSHETZ WHO
HAS BEEN VERY EXCITED ABOUT
TRYING TO DO SOMETHING ABOUT
CHANGING THE CULTURE OF SCIENCE
FOR QUITE SOME TIME.
AND I'LL INVITE HIM TO SAY SOME
OPENWORDS.
>> THANKS, FOLKS.
THANK YOU, VERY MUCH STEVE AND
VERONIQUE FOR CHAIRING THIS VERY
IMPORTANT SESSION AND THANKS TO
DEVAN AND THE TEAM FOR WORKING
THROUGH COVID OF UP THE UPS AND
DOWNS AND THE THUNDERSTORMS AT
THE AIRPORT YESTERDAY TRYING TO
GET PEOPLE HERE.
I CAN'T OVER ESTIMATE HOW
IMPORTANT THIS CONFERENCE IS I
WANT TO STEP BACK AND TALK TO
YOU FROM THE NIH STANDPOINT.
WE'RE A GRANTING AGENCY.
WE'RE NOT REALLY EMPLOYEEING
SCIENTISTS, WE'RE GIVING GRANTS
OUT AND THESE GRANTS ARE FOR
PEOPLE WHO DO WORK TO TURN IN A
PRODUCT.
THE PRODUCT IS NOT JUST THE
RESULT IT'S THE WAY SCIENCE IS
DONE IT'S THE FOUNDATION FOR
SCIENCE IN THE U.S.
BUT WHAT IS IT THAT WE ARE
PAYING FOR?
IF YOU HAVE TO TALK TO CONGRESS
AND SAY IR PUTTING $41 BILLION
INTO NIH BECAUSE YOU ARE
GETTING, WHAT?
YOU'RE GETTING EXCELLENT
SCIENCE.
THAT'S WHAT WE TELL CONGRESS AND
THE TAXPAYERS WHO ARE PAYING THE
TAXES AND PUTTING THE MONEY UP
THAT COMES INTO NIH.
WHAT DOES EXCELLENT SCIENCE
MEAN?
I THINK YOU'LL BE DISCUSSING
THAT VARIOUS VARIANTS OF THAT
TERM, "EXCELLENT SCIENCE."
BUT I THINK IF YOU BOIL IT DOWN,
JUST FROM MY POINT OF VIEW, IT'S
SOMETHING YOU CAN TRUST.
IT'S TRUSTWORTHY RESULTS.
THAT'S REALLY IMPORTANT BECAUSE
THAT'S THE BASIS OF SCIENCE
MOVING FORWARD THAT WE CAN TRUST
WHAT SOMEONE HAS PUBLISHED OR
POST.
IF THAT TRUST IS DIMINISHED AND
IF THAT TRUST IS DIMINISHED THE
TAXPAYER SAYS RIGHTLY WHY AM I
GIVING MONEY TO SCIENCE
RESEARCHERS IF I CAN'T TRUST THE
RESULTS.
IF THAT GETS TO CONGRESS YOU CAN
BET OUR BUDGET AND SCIENCE IN
THE U.S. WOULD BE IN CRITICAL
STATES VERY QUICKLY.
SO I THINK THAT'S WHAT I'D ASK
YOU TO THINK ABOUT TODAY IN THE
BIG PICTURE.
HOW DO WE HELP SCIENCE PRODUCE
WHAT WE CAN REALLY TRUST.
THERE'S LOTS OF PIECES TO THIS
BUT I WOULD SAY A PIECE THAT
WE'RE GOING TO BE TALKING ABOUT
MOST DURING THE SESSIONS IS HOW
IS THE SCIENCE EXECUTED
ACCORDING TO THE BEST POSSIBLE
PRACTICES?
SO IF YOU PUT A RESULT OUT AND
IT'S WRONG BUT YOU DID
EVERYTHING YOU COULD AT THAT
TIME IN HISTORY TO EXAMINE IT
AND IT LOOKED LIKE ALL RIGHT AND
TURNS OUT WRONG, THAT'S FINE.
DID IT ACCORDING TO BEST
PRACTICES BUT WHAT DOES IT MEAN
TO HAVE BEST PRACTICES IN
SCIENCE?
WHO DETERMINES WHAT THE BEST
PRACTICES ARE?
THE GOVERNMENT AND LIKELY THE
INSTITUTIONS?
I DON'T THINK SO.
WHEN I STARTED OUT IN AN AREA OF
SCIENCE IT WAS A SMALL AREA.
THE BEST PRACTICES WERE
REINFORCED BY A SCIENTIST'S
REPUTATION AND YOU WENT INTO A
CONFERENCE WITH 50 PEOPLE
EVERYBODY KNEW THE REPUTATION OF
EVERY SCIENTIST IN THE ROOM AND
THEY KNEW WHO THEY WOULD TRUST
AND WHO THEY COULDN'T TRUST.
THAT'S THE WAY IT WAS IN THE
'70s.
NOW, THAT'S NOT TRUE ANYMORE.
SCIENCE IS TOO BIG.
THE REPUTATIONS DON'T EXPAND OUT
BUT IN YOUR UNIVERSITY AND I CAN
TELL YOU FROM MY UNIVERSITY
TIME, WE KNEW WHO THE SLOPPY
SCIENTISTS WERE AND WE KNEW WHO
THE RIGOROUS SCIENTISTS WERE BUT
NO ONE ELSE KNEW OUTSIDE OUR
UNIVERSITY.
SO THE REPUTATION SIDE OF THINGS
THAT WERE DRIVING TRUSTWORTHY
SCIENCE HAS BEEN DIMINISHED AS
WE EXPANDED SCIENCE.
SO WHAT'S THERE TO TAKE ITS
PLACE?
THINK WE'RE IN A VOID AND WHAT
WE'RE ASKING YOU TO THINK ABOUT.
HOW DO WE FILL THAT VOID TO
ESTABLISH BEST PRACTICES
THROUGHOUT OUR RESEARCH FIELD SO
THAT OUR WORK CAN BE CONCEIVED
AS TRUSTWORTHY?
NOT JUST BY THE PUBLIC BUT BY
THE SCIENTISTS AS WELL.
THE DANGER OF SCIENCE GOING
FORWARD THAT IS NOT TRUSTWORTHY
HAS EFFECTS EVEN ON THE SCIENCE
COMMUNITY.
YOUNG PEOPLE DO NOT WANT TO BE
PART OF SOMETHING THEY CAN'T
TRUST.
SO THEY'RE IN THE LAB AND THEY
KNOW WHAT'S GOING ON AND UNLESS
THE WORK IS BEING PERFORMED AT
THE HIGHEST LEVEL OF QUALITY,
THEY'RE GOING TO BE TURNED AWAY
FROM SCIENCE AS WELL.
IT'S NOT JUST THE TAXPAYERS AND
CONGRESS BUT IF WE DON'T SEE
WHAT WE DO AS TRUSTWORTHY WE
CAN'T EXPECT CONGRESS OR THE
PUBLIC TO THINK SO.
IF WE DON'T FILL THAT VOID, WHAT
HAPPENS?
WE ALL KNOW THERE ARE INCREDIBLE
INCENTIVES TO MOVE SCIENCE
FORWARD TO MAKE IT LOOK GOOD
THOUGH IT MAY NOT BE CORRECT.
THOSE ARE HOW DO YOU GET GRANT
MONEY AND YOU HAVE TO CONVINCE
PEOPLE AND THERE ARE SOME PEOPLE
REALLY GOOD AT WRITING GRANTS
AND I'M NOT CONVINCED EVERY ONE
OF THEM IS A RIGOROUS SCIENTIST
BUT THEY'LL GET THEIR GRANTS
BECAUSE THEY CAN WORK THE
SYSTEM.
I DON'T HAVE A WAY, DR. KRETSKY
SAID YOU HAVE TO STOP FUNDING
THE CROOKS AND A SAID HOW DO YOU
I KNOW WHO THE CROOKS ARE AND HE
SAID YOU KNOW THEM AND I SAID
NOT ALL OF THEM.
WE NEED A BETTER SYSTEM THAN
WHAT WE HAVE NOW BECAUSE
FOLLOWING THE PREVAILING
NARRATIVE ARE THE INCENTIVES
PUSHING SCIENCE IN SOMEWHAT
PERVERSE DIRECTIONS AND THIS
WILL LEAD TO FAILURE.
I CAN TELL YOU I WAS IN THE
STROKE FIELD IN THE LATE '90s,
EVERYONE IN THE STOKE FIELD WAS
TOTALLY CONVINCED WE COULD GIVE
A NEUROPROTECTIVE AGENT TO A
STROKE PATIENT AND PREVENT THEIR
STROKE FROM GETTING WORSE.
THAT'S BECAUSE ALL THE ANIMAL
EXPERIMENTS SHOWED AND THAT
EXPERIMENT INS CULTURE I WORKED
ON THAT LOOKED LIKE THAT WOULD
BE TRUE.
THE WHOLE THING WAS A SHAM.
IT DIDN'T TURN OUT -- INDUSTRY
CAME IN WITH WILL YONZ OF
DOLLARS TO FUND STROKE STUDIES
AND DEVELOP NEUROPROTECTIVE
AGENTS AND THEY ALL FAILED.
THE GROUP SEEING THE FAILURE
WENT BACK AND STARTED THIS
ORGANIZATION.
IT'S STILL RUNNING NOW CALLED
THE STARE COMMITTEE AND THEY
ESTABLISHED RIGOROUS PRACTICES
FOR STROKE, NEUROPROTECTIVE
RESEARCH.
A WHOLE LIST OF THINGS THEY
ASKED FOR TO BE DONE BEFORE
INDUSTRY GETS INVOLVED AND
BRINGS SOMETHING TO PATIENTS.
IT WAS BORN OUT OF FAILURE.
LOTS OF THINGS YOU TALK ABOUT
NOW ARE BASICALLY A LOT OF
THINGS THAT CAME OUT OF THAT
STROKE FIELD IN THE LATE '90s,
EARLY 2000s BUT THE DAMAGE DONE
IS UNBELIEVABLE.
RIGHT KNOW STROKE IS THE THIRD
LEADING CAUSE OF DEATH IN THE
UNITED STATES AND THERE'S NO
PHARMACEUTICAL COMPANY IN THE
U.S., EUROPE, MAJOR
PHARMACEUTICAL COMPANY
INTERESTED IN DEVELOPING ANY
TREATMENT FOR STROKE.
THAT'S NOW 22 YEARS AFTER.
THE PENALTY YOU PAY FOR GOING
DOWN THIS PATH OF LESS RIGOROUS
OR NON-RIGOROUS RESEARCH IS
REALLY QUITE EXTREME AND SO
WE'RE TALKING TO YOU TODAY
BECAUSE WE WANT YOU TO FILL THIS
VOID TO HELP PEOPLE AROUND THE
COUNTRY AND WORLD TO BUILD TRUST
IN THE SCIENCE WE DO.
THE PENALTIES ARE HUGE IF WE
DON'T MEET THIS CHALLENGE.
IF WE DO MEET THIS CHALLENGE
THEN I THINK SCIENCE WILL
CLEARLY ADVANCE AT A MORE RAPID
PACE AND NINDS IS REALLY
COMMITTED AND SO WE STARTED THIS
OFFICE FOR RESEARCH QUALITY WITH
CHEY AND DEVAN AND WE'LL HAVE
CENTER TO VALIDATE MATERIALS BUT
THIS CAN'T BE DONE BY A
GOVERNMENT OFFICE IT REQUIRES
THE PEOPLE IN THE TRENCHES TO
PICK UP THE TORCH AND CHANGE THE
SYSTEM FOR THE BETTER.
AND IT'S NOT EASY BECAUSE THE
DISINCENTIVES ARE PRETTY STRONG
THAT WE MENTIONED AND YOU'LL BE
TALKING ABOUT THEM BUT I THINK
PEOPLE WANT TO DO THE RIGHT
THING AND SO IN THE END I THINK
PEOPLE WILL DO THE RIGHT THING.
WE JUST HAVE TO KEEPER SIFTING
AND HELPING DEVELOP BEST
PRACTICES, GET THEM OUT THERE.
WHEN PEOPLE DON'T FOLLOW BEST
PRACTICES, THERE WILL BE A
PENALTY AND THEY WILL SEE THAT
IT'S NOT ADVANTAGEOUS TO THEM AT
LEAST IN THE LONG RUN.
I'M VERY HOPEFUL THAT THIS KIND
OF ACTIVITY WILL SUCCEED AND I'M
REALLY -- A LOT OF GRATITUDE TO
THE PEOPLE HERE BECAUSE I THINK
YOU'RE GOING TO HOPEFULLY BE THE
ONES WHO LEAVE THIS MEETING WITH
THE AMMUNITION, THE IDEAS AND
COMMUNITY SUPPORT AMONG
YOURSELVES TO MAKE THIS HAPPEN
IN YOUR INSTITUTIONS AND
CERTAINLY AT NINDS WE'RE LEER --
HERE TO BACK YOU.
THANK YOU AGAIN FOR COMING AGAIN
AND THANK YOU STEVE AND LOOK
FORWARD TO A GOOD MEETING.
>> I FORGOT TO REMIND EVERYONE
WE'RE REQUIRED TO REMAIN WITH
SIX FEET OF DISTANCING AND WEAR
MASKS THE ENTIRE TIME.
THAT'S THE CHANGE IN THE LEVEL
OF COVID IN THE AREA SO PLEASE
TRY TO MAINTAIN DISTANCE.
THANKS A LOT.
>> GOOD MORNING.
THAT'S SUPPOSED TO WAKE YOU UP.
BETWEEN THE RISING LOCAL COVID
RATES AND THE DROPPING
BAROMETERS WHICH CANCELED A FEW
FLIGHTS LATE LAST NIGHT I CAN'T
TELL YOU WHAT A RELIEF IT IS TO
SEE YOU ALL HERE AT LEAST THOSE
WHO MADE IT.
I'M STEVE GOODMAN A PROFESSOR AT
STANFORD OF EPIDEMIOLOGY AND
MEDICINE AND CO-CHAIRING THIS
AND WITH OUR CO-CONSPIRATORS
FROM THE OFFICE OF RESEARCH
QUALITY AND DEVAN CRAWFORD, I
WANT TO BOTH WELCOME YOU AND
PROFOUNDLY THANK YOU FOR
DEVOTING SERIOUS TRAVEL AND TWO
IN-PERSON DAYS FOR THIS MEETING
AND FOR SOME MY BE THE FIRST
SCIENTIFIC MEETING YOU'VE BEEN
TO IN A WHILE.
WE TRIED VERY HARD TO DESIGN IT
IN A WAY TO MAKE IT WORTH YOUR
WHILE.
I'LL EXPLAIN WHY.
I'VE BEEN INVOLVED WITH MANY
RESEARCH REFORM EFFORTS OVER THE
LAST 30 YEARS INCLUDING AT THE
CREATION OF THE CARBON
COLLABORATION WHICH MEANS I HAVE
TO BE CARBON DATED IT'S THE
FIRST WITH THIS EXPLICIT FOCUS
WHICH IS HOW TO CHANGE THE
RESEARCH INSTITUTIONS WHERE MANY
OF US LIVE NOT JUST HOW WE DO
OUR SCIENCE.
ON A PERSONAL LEVEL THIS ALIGNS
WITH MY OWN EVOLUTION.
I'VE BEEN WRITING AND LECTURING
FOR DECADES HOW TO DO BIOMEDICAL
RESEARCH DIFFERENTLY AND I
DECIDED I WANTED TO TRY AND MAKE
THAT HAPPEN WHERE I LIVE AND
HAVE AN ANSWER TO THE QUESTION,
OH, WHAT'S HAPPENING AT STANFORD
AND I'D HAVE TO SAY, ARE YOU
DOING THAT AND I'D HAVE TO SAY
NOT MUCH.
WE CREATE THE STANFORD PROGRAM
ON RESEARCH RIGOR AND
RESPONSIBILITY, SORRY,
REPRODUCIBILITY.
IT'S PARTLY FUNDED THROUGH THE
STANFORD CTSA.
THAT'S AN IDEA FOR THOSE AT CTSA
INSTITUTIONS.
IT'S ATTEMPTING TO DO MANY OF
THE THINGS SEVERAL OTHER
ENTITIES AND MORE IN EUROPE ARE
ALREADY DOING OR TRYING TO DO.
MANY REPRESENTATIVES OF THOSE
GROUPS ARE HERE AND I'M EAGER TO
LEARN FROM THEM AND OTHERS WHO
ARE STRUGGLING WITH THE SAME
ISSUES WE ARE WITH SOME WE
HAVEN'T YET ENCOUNTERED AND
LEARNING FROM THOSE WAY AHEAD OF
US.
THIS MEETING IS STRUCTURED IN A
WAY SO EACH CAN MAKE A
DIFFERENCE.
THERE'S NO PASSIVE ATTENDEES
HERE.
THOUGH HALF OF YOU ARE NAMED ON
THE AGENDA, EACH OF YOU WAS
INVITED FOR A REASON.
WE DEVOTED MORE THAN HALF THE
WORKSHOP TO HEAR YOU FROM AND
TIME FOR NETWORKING.
WE CONSIDER YOU ALL SPEAKERS
YOU'LL SEE HOW IN A MINUTE.
FIRST I'D LIKE TO ENCOURAGE YOU
TO LOOK AROUND THE AUDIENCE AT
THE ATTENDEE LIST.
MY GUESS IS FOR THOSE WHO SPENT
TIME IN REFORM AND METASCIENCE
MEETINGS THERE'LL BE QUITE A FEW
FOLKS YOU DON'T KNOW BY DESIGN.
MANY ARE DOING TRANSFORMATIVE
WORK IN THEIR OWN DISCIPLINES OR
HOMES BUT NOT IN OTHERS SO THEY
DON'T ATTEND THE COLLECTIVE
MEETINGS.
EACH OF THEM AND YOU IS A
CHAMPION FOR GOOD SCIENCE AND
HENCE THE WORD IN THE CONFERENCE
TITLE.
MAUI NOT SEE YOURSELF AS A
CHAMPION.
IT'S JUST WHAT YOU DO.
AND WHAT YOU'RE DRIVEN TO DO.
YOU EACH COME FROM A WIDE ARRAY
OF DISCIPLINES AND HAVE AN
OPPORTUNITY TO TEACH OTHERS.
I ENCOURAGE YOU TO IDENTIFY YOUR
DISCIPLINARY OR METH OMETHODOLO
HOME.
SCIENCE REFORM MEETINGS I
OBSERVED MANY PEOPLE USE THE
WORD SCIENCE QUITE GENERICALLY
THOUGH THE PROBLEMS AND
SOLUTIONS IN THEIR OWN FIELD
DON'T NECESSARILY APPLY TO ALL.
ACKNOWLEDGING SCIENTIFIC
DIVERSITY BECOMES IMPORTANT
WHERE THE EXPERIMENTAL
METHODOLOGY AND NORMS ARE
DIFFERENT FIELD FROM FIELD SO I
THINK WE'LL LEARN THE MOST TO
EXPLAIN WHAT SCIENTIFIC CULTURES
WE KNOW BEST AND A SIGN OF
RESPECT FOR OTHERS.
LET'S TALK ABOUT THE CONFERENCE
THEME AND THAT THEME IN A SINGLE
WORD THERE'S A LOT OF WORDS OUT
THERE IS CHANGE AND CHANGE AND
COMPLEX SYSTEMS IS NOT SOMETHING
THAT HAPPENS EASILY OR NATURALLY
AND SMART PEOPLE WHO STUDY
ORGANIZATION HAVE A VARIETY OF
THEORIES ABOUT HOW TO AFFECT
CHANGE.
WE'VE CHOSEN ONE POPULARIZED
VERSION TO STRUCTURE THE MEETING
DESCRIBED IN A BOOK, SWITCH, HOW
TO CHANGE THINGS WHEN CHANGE IS
HARD AND THIS IS AN AREA WHERE
CHANGE IS VERY HARD.
IT KEEPS TELLING KNOW CONNECT TO
THE NETWORK.
THE FIRST FRAME IN THE SESSION
IS HOW TO MOTIVATE OTHERS TO
ACKNOWLEDGE THE PROBLEM AND THAT
SOMETHING NEEDS TO BE DONE AND
EVEN BETTER TO JOIN YOU.
IN OUR CASE IN ACADEMIC RESEARCH
THOSE PEOPLE ARE INSTITUTIONAL
LEADERS, FACULTY, STAFF,
STUDENTS AND MEMBERS OF
COMMUNITY.
THE SECOND SESSION IS HOW TO
TEACH PEOPLE THE THINGS THEY
NEED TO KNOW ALONG WITH THEIR
DESIRED BEHAVIORS TO CHANGE HOW
SCIENCE IS DONE.
THE THIRD IS HOW TO REMOVE THE
BARRIERS TO THAT CHANGE OR TO
MAKE IT EASIER THROUGH TOOL
CREATION, NORMING OF THE RIGHT
BEHAVIORS AND INCENTIVES
REMOVING NEGATIVE ONES AND
CREATING POSITIVE ONES.
THE FINAL MEETING SESSION WHICH
IS ACTUALLY NOT FORMALLY PART OF
THEIR FRAMEWORK IS HOW OTHER
COMPONENTS OF THE SCIENTIFIC ECO
SYSTEM WHICH INCLUDES OTHER
INSTITUTIONS, FUNDERS,
REGULATORS, GOVERNMENT, JOURNALS
AND PROFESSIONAL SOCIETIES AND
THE NON-PROFIT OR FOR-PROFIT
PRIVATE SECTOR CAN WORK TOGETHER
TO FACILITATE CHANGE.
THIS CHANGE FRAMEWORK IS NOT THE
ONLY ONE NOR WILL I IT CAPTURE
EVERY ELEMENT IN THE FIRST TWO
DAYS.
THE FIRST SPEAKER, DAMON CENTOLA
HAS A NEWER BOOK CALLED CHANGE,
HOW TO MAKE BIG THINGS HAPPEN.
HE HAS A SOMEWHAT DIFFERENT
THEORETICAL APPROACH FOR
PROGRAMMATIC PURPOSE TO BREAK
DOWN THE COMPLEX TASK WE HAVE
BEFORE US INTO DISCREET
PERFORMANCE AND HAVE A PLACE TO
PARK VARIOUS ISSUES OTHERWISE
AND EVERYONE HERE HAS SEEN THIS,
THERE'S A TENDENCY TO TALK ABOUT
ALL ASPECTS OF THE PROBLEM AT
ONCE WHICH IS BOTH OVERWHELMING
AND SELF-DEFEATING BECAUSE WE
CAN'T SOLVE IT AS A WHOLE UNTIL
WE BREAK IT DOWN AND HAVE THE
SAME CONVERSATION IN EVERY
SESSION, WE GET MORE AND MORE
TIRED, LITTLE PROGRESS IS MADE
EXCEPT FOR A RESOLUTION TO HAVE
ANOTHER MEETING BECAUSE IT S SO
LITTLE.
WE PROVIDED CERTAIN PLACES TO
HAVE CONVERSATIONS TO HOPEFULLY
HAVE PROGRESS ON EACH AND THE
MODERATORS WILL BE ON ALERT TO
DEFER OR DEPART OTHER TOPICS TO
OTHER SECTIONS TO BE MORE FULLY
DISCUSSED.
PLEASE BE RESPECTFUL OF THAT.
IF IT'S A TOPIC THAT DOESN'T FIT
IN YOUR FRAME WORK WE'LL TAKE
MOTE AND THE MODERATOR CAN USE
THEIR DISCRETION TO MODERATE IT.
THE STRUCTURE OF THE THEME.
EACH SPEAKER WILL GIVE 10 TO 15
MINUTE PRESENTATIONS AND HAVE
EVERYONE A SPEAKER.
ONE IS A PLENARY CONVERSATION
WITH A FULL HOUR DEVOTED TO IT
AS MUCH AS THE SESSION ITSELF
AND USE IT ON MOTIVATION AND
WORKING TOGETHER THE FIRST AND
LAST SESSION.
THE TWO MIDDLE SESSIONS WE'LL
USE A VERSION OF THE WORLD CAFE
METHOD WHICH GETS EVERYBODY'S
INPUT ON CRITICAL SUBTOPICS AND
FOR EACH WORLD CAFE SESSION WE
HAVE A TOPIC IN THE AGENDA AND
IT WILL ROTATE AND THEY'LL BRAIN
STORM SOLUTIONS TO THE THEME AND
WRITE THEM ON STICKY NOTES ON A
BOARD AND ORGANIZED BY A
MODERATOR.
THE SECOND GROUP WILL HEAR WHAT
THE FIRST GROUP DID THROUGH THE
MODERATOR AND THEN BUILD ON WHAT
THE FIRST GROUP DID.
THE THIRD GROUP WILL ORGANIZE
AND PRIORITIZE SUGGESTIONS OF
THE PREVIOUS GROUP ADDING
ANYTHING THEY THINK WAS MISSED.
WE'LL TAKE A PICTURE OF THOSE
BOARDS AND WHEN IT'S ALL OVER
AND WRITE THEM UP.
THIS GIVES EACH OF YOU A CHANCE
TO WEIGH IN ON MULTIPLE TOPICS
IN THE SMALL GROUPS TO HAVE YOUR
SAY BECAUSE THEY ARE SMALL,
EXTRACT YOUR COLLECTIVE WISDOM
ELIMINATES THE NEEDS FOR
REPORT-OUTS COLLECTIVELY WHICH
ARE ALWAYS A BEAR AND NOT SURE
WE GET A LOT OUT OF THOSE.
BEFORE I FINISH, A FEW WORDS ON
TERMINOLOGY BECAUSE WE HAVE A
FRIGHTENING ARRAY OF OUR WORDS
THAT WE'LL BE USING, RIGOR,
REPRODUCIBILITY AND
REPEATABILITY, I CAN GO ON.
BUT THIS MEETING WILL TILT
TOWARDS THE RIGOR SIDE WHICH A
LITTLE LESS ON THE REPRODU
REPRODUCIBILITY SIDE AND THE
LATTER APPLIES TO ALL SCIENCE
BUT THE REQUIREMENTS OF RIGOR
ARE MORE DISCIPLINE SPECIFIC I
WOULD SAY THAN OPEN SCIENCE AND
CAN BE A HARDER SIDE SO IT'S
UNCLEAR WHAT RIGOR AS WE MEAN IT
MEANS TO MANY COLLEAGUES OAR
WHAT RIGOR REFORM DOES UNLESS
THE TRAINING WE TALK ABOUT
DIFFERS FROM WHAT'S ALREADY
OFFERED.
THIS IS A TRICKY PROBLEM
POLITICALLY AND SCIENTIFICALLY.
THE NIA DOESN'T HELP TOO MUCH IN
DEFINE RIGOR BECAUSE THEY USE
RIGOR TWICE IN THE DEFINITION OF
RIGOR.
I'M SYMPATHETIC IT'S LIKE TRYING
TO DEFINE CAUSE WITHOUT THE WORD
CAUSE.
I'LL OFFER A CONCEPTUAL
DEFINITION HERE WHICH IS THAT
IT'S ABOUT GETTING THE
UNCERTAINTY RIGHT.
IT HAS TWO PARTS.
I WOULD SUGGEST A RIGOROUS IS
WHERE WE LOOK AT THE EPIDEMIC
UNCERTAINTY AND MINIMIZED IT
USING WHAT WE CALL BEST
PRACTICES AND WHAT REMAINS IS
PROPERLY REFLECTS IN THE
KNOWLEDGE CLAIM.
THE FIT BETWEEN THE METHODOLOGY
AND STOCHASTIC AND GETTING THE
DESIGN AND SCRUPULOUS
MEASUREMENT TO ALL THE SOURCES
OF VARIATION INCLUDING THE
SOURCES HARD TO QUANTIFY LIKE
MODEL CHOICE.
SO A NON-RIGOROUS STUDY IS WHERE
THE INVESTIGATORS CLAIM MORE
CERTAINTY THAN DESERVED.
PARTICULARLY IF THAT INFLATED
UNCERTAINTY IS DUE TO NEGLECT OF
BASIC PRINCIPLES OF SIGN,
ANALYSIS OR INFERENCE.
IT'S PARTICULARLY IMPORTANT IN A
MEETING LIKE THIS TO NOT HAVE A
MORE NORMATIVE DEFINITION OF
RIGOR BECAUSE THE DESIGN AND
DATA SOURCES AND METHODS AND
STANDARDS OF PROOF VARY SO
WIDELY ACROSS FIELD AND IF WE
START ARGUING ABOUT THE
DEFINITION OF RIGOR IN A
NORMATIVE SENSE IT CAN SLIDE
INTO UNHELPFUL IMPLICATIONS FOR
EXAMPLE ONE CAN'T USE
RANDOMIZATION OR THE OTHER
DOESN'T UNDERSTAND MECHANISMS
AND ISN'T DETERMINISTIC.
EVERYBODY HERE IS DOING THE BEST
THEY CAN WITH THE RESOURCES THEY
HAVE.
REPRODUCIBILITY HAS MANY
MEANINGS WHICH DIFFER ACROSS
CULTURES.
IT CAN MEAN COMPLETE AND
TRANSPARENT REPORTING AND
METHODOLOGY AND RESULTS.
IT CAN MEAN A SIMILAR RESULT
UPON REANALYSIS OR SIMILAR
RESULT IN A REPEATED EXPERIMENT.
I'M NOT GOING TO PIN US DOWN TO
ONE BUT ASK IF YOU USE THAT WORD
BE CLEAR WHAT YOU MEAN.
WE CAN'T ASSUME UNIVERSALS IN A
MEETING LIKE THIS.
SO WHAT'S OUR GOAL?
WELL, IT'S TO PROVIDE IDEAS AND
SUPPORT TO THOSE TRYING TO
STRENGTHEN THE SCIENCE IN THEIR
INSTITUTIONS AND DISCIPLINES.
WE'LL DO THAT THROUGH THE NEXT
TWO DAYS OF CONVERSATIONS
SHARING THE PRODUCTS OF THIS
MEETING AND POTENTIALLY CREATING
AN ONGOING INFRASTRUCTURE SO
THIS CAN CONTINUE GOING FORWARD
WHETHER THIS IS THE FIRST
MEETING OR FIRST ANNUAL MEETING
I'M NOT SURE, WE'LL SEE.
IF WE HAVE A SECOND I'M GOING TO
SUGGEST ADJUSTING THE TITLE.
A LITTLE CLUTCHY BUT IT CAPTURES
THE ESSENCE OF WHAT WE'RE TRYING
TO DO.
WHAT IT MEANS CAN BE DECIDED BY
ALL OF US RECOGNIZING THE TIME
COURSE OF THESE MEETINGS WILL BE
LONG.
IT WON'T HAPPEN IN THE NEXT YEAR
TWO YEARS IF WE UNDERSTAND THIS
IS AN EVOLUTION I THINK WE'LL
PUT THE RIGHT THINGS IN PLACE.
THANK YOU AGAIN FOR TRUSTING US
ENOUGH TO MAKE THIS TREK AND
SPEND THE TWO DAYS HERE AND I
HOPE THE TRUST IS VINDICATED BY
THE END OF TOMORROW.
I'LL PASS THE BATON TO DAVID
CENTOLA AT THE UNIVERSITY OF
PENNSYLVANIA WHERE HE'S THE
LEADER OF THE NETWORKS DYNAMICS
GROUP AND WON A WIDE ARRAY OF
NICE RECOGNITIONS AND PRIZES
INCLUDING MY FAVORITE, A PRIZE I
DIDN'T KNOW EXISTED UNTIL I SAW
HE WON IT THE GOODMAN AWARD IN
SOCIOLOGY.
NOW I HAVE TO LEARN A LITTLE BIT
ABOUT SOCIOLOGY.
I ASSUME PROFESSOR GOODMAN KNEW
SOMETHING ABOUT THE FIELD.
HE'S CONSULTED FOR A WIDE
VARIETY OF COMPANIES OR ENTITIES
WE INTERACT WITH EVERY DAY AND
MY PARTICULAR FAVORITES ARE BEN
AND JERRY'S AND BNBA.
I WON'T BORE YOU BY TALKING
ABOUT THE WARRIORS AND THEIR
GAME LAST NIGHT.
HE ALSO WRITTEN HOW BEHAVIOR
SPREAD, THE SCIENCE OF COMPLEX
CONTAGIONS WRITTEN BEFORE COVID.
SO HE'LL GIVE US A BRIEF TASTE
OF THIS WORK TO HELP US DO OURS
BETTER AND MAYBE HE'LL ENTICE
SOME OF TO YOU SEND ROYALTIES
HIS WAY WHEN HE'S DONE.
THANK YOU VERY MUCH AND I'LL
HAND IT OVER TO YOU, DAMON.
>>THANKS VERY MUCH AND I
APPRECIATE THE OPPORTUNITY TO
SHARE SOME OF THIS WORK WITHHE
GROUP HERE.
I'M GOING TALK ABOUT IDEAS FROM
MY RECENT BOOK, CHANGE,
PARTICULARLY HOW IT'S RELEVANT
TO CHANGES IN SCIENTIFIC
PRACTICES.
WHEN WE LOOK AT INDIVIDUALS
TRYING TO MAKE A BEHAVIOR CHANGE
WE TEND TO THINK HOW EACH
INDIVIDUAL WOULD ACT TO MEASURE
LIKE THE DETAILS, THE F MRI AND
COMPLETE PHYSIOLOGY OF THE FISH
BUT WHEN YOU PUT 10,000 OF THEM
TOGETHER ANTICIPATE HOW THEY'LL
BEHAVE COLLECTIVELY AND THE
ANSWER IS NO AND TO UNDERSTAND
THE COLLECTIVE BEHAVIOR OF A
COMPLEX SET OF ORGANISMS YOU
NEED TO UNDERSTAND THE
COLLECTIVE AS A WHOLE AND HOW
THEY INTERACT TO PREDICT THE
COMPLEX, BEAUTIFUL AND OFTEN
ADAPTIVE BEHAVIORS AND SAME WITH
PEOPLE.
INDIVIDUAL PSYCHOLOGY IT DOESN'T
TELL US A LOT HOW SOCIAL NORMS
TAKE HOLD A LARGE POPULATION NOR
HOW TO SHIFT THE NORMS AND WE
LOOK AT SCIENTIFIC PRACTICES AND
NOT JUST A BELIEF IN RIGOR BUT
DAILY ROUTINES OF RIGOR THAT
TAKE HOLD WITHIN THE ACADEMIC
CENTERS AND UNIVERSITIES.
BASICALLY WE TALK ABOUT THESE
KINDS OF LARGE-SCALE CHANGES IN
BEHAVIOR IN GENERAL IN
ORGANIZATIONS AND SOCIETY IS
TYPICALLY THROUGH SOME KIND OF
SOCIAL NETWORK.
WE SAY MAYBE THERE ARE DIFFERENT
TYPES OF PEOPLE IN THE NETWORK
SOME IN THE CENTER MASSIVELY
CONNECTED AND THEN SOME IN THE
PERIPHERY MORE MODESTLY
CONNECTED AND IT'S NOT JUST THE
NUMBER OF CONNECTIONS THEY HAVE
BUT THEY TEND TO HAVE
CONNECTIONS SHOOTING OUT IN ALL
DIRECTIONS FROM THE CENTER WE
CALL WEAK TIES AND ON THE
PERIPHERY HAVE TIES AND IT
ALTERS HOW THEIR POSITION IN THE
NETWORK AFFECTS THEIR CAPACITY
TO INFLUENCE THE REST OF THE
POPULATION.
SO ONE OF THE CLASSIC IDEAS I
HAVE OF CHANGE AND ONE I'LL
DISABUSE YOU OF TODAY IS WHEN WE
LOOK AT CHANGE PROCESSES WE TEND
TO IMAGINE SOME HIGHLY CONNECTED
IN A SENSIBLE CENTER OF THE
NETWORK, ADOPTION INNOVATION
TAKES UP THE BANNER OF THE NEW
IDEA FOR INSTITUTIONAL CHANGE
AND SPREADS THAT AROUND AND
PEOPLE BECOME MEMBERS OF THE
BANDWAGON AND SO ON AND SO FORTH
AND SEE THIS CASCADING CHANGE IN
IDEAS AND PRACTICES IS A POPULAR
MOTION OF CHANGE FROM THE 1940s.
WHAT'S INTERESTING TODAY IS WE
HAVE DATA.
WE HAVE DATA FROM ORGANIZATIONAL
CHANGE AND DATA IN CHANGES IN
SCIENTIFIC PRACTICES.
WHATE TEND TO SEE IS WHEN
CHANGE TAKES HOLD, INTERESTINGLY
IT TENDS TO TAKE HOLD OUT IN THE
PERIPHERY OF THE NETWORK AND
WHEN IT SPREAD IT STARS STARTS A
CASCADE TO PERCOLATE AROUND THE
NETWORK AND DOESN'T GET TO THOSE
IN THE CENTER WHICH IS
INTERESTING SCIENTIFICALLY
BECAUSE IT FRAMES A CLEAR PUZZLE
WHERE OUR BEST THEORIES SAY
CHANGE, INNOVATION AND DIFFUSION
AND SCIENTIFIC PRACTICES SPREAD
FROM THE INSIDE-OUT AND THE DATA
SHOWS IT GOES FROM THE PERIPHERY
TOWARDS THE CENTER.
WHAT'S GOING ON IS A SENTIMENTAL
DISCONNECT BETWEEN OUR
INTUITIONS AND WHEN THE DATA
SAYS IS HAPPENING.
INTUITION IS THE NETWORKS WE'RE
SHOWN AND THE ONE ON THE LEFT
LOOKS FAST.
EVERYTHING SEEMS LIKE IT WILL
ACCELERATE A CHANGE AND THE
PERSON IN THE CENTER HAS LOTS OF
TIES AND THEY'RE RADIATING OUT
IN DIFFERENT DIRECTIONS.
AND THERE'S CERTAIN KINDS OF
CONTAGIONS LIKE COVID-19
EFFECTIVE FOR DIFFUSION.
WE'RE LOOKING AT THE BEHAVIOR
CHANGE THAT REQUIRES SWITCHING
FROM ONE KIND OF BEHAVIOR THAT
YOU UNDERSTAND IS FAMILIAR TO
NEW BEHAVIOR THAT REQUIRES
PUTTING AT RISK YOUR COMPETENCY
OR HOW YOU ENGAGE EVERY DAY.
A DIFFERENT NETWORK WILL BE
IMPORTANT TO GETTING THOSE
CHANGES OFF THE GROUND.
THE NETWORK ON THE RIGHT IS
BETTER SUITED FOR ACCELERATING
CHANGE WHEN IT COMES TO
SCIENTIFIC PRACTICES.
SO IT'S WORTHWHILE ASKING WHY
ARE INTUITIONS WRONG?
IF WE HAVE THEM WE SHOULD HAVE
THEM FOR A REASON.
WE'RE USING OUR INTUITION SAYING
IF SOMEONE HAS A VIRAL INFECTION
OR COVID-19 THEN THE MORE PEOPLE
THEY COME INTO CONTACT WITH THE
MORE LIKELY THEY'LL GET COVID-19
AND SO ON AND SO FORTH.
BASICALLY THE IDEA OF SUPER
SPREADER SOMEONE HIGHLY
CONNECTED IS THE LINCHPIN IN
THINKING HOW ACADEMICS TAKE
HOLD.
WHEN WE MOVE TO SOCIAL THINKING
WHERE IT'S NOT JUST A MATTER OF
MERE EXPOSURE TRANSLATING TO
TRANSMISSION BUT RATHER THAT
PEOPLE HAVE A SOCIAL CONTEXT.
THEY LOOK AT PEERS ADOPTING AND
THAT FEEDBACK HAS A COMPLEX
INTERACTION TO IT.
ALL OF A SUDDEN THE STRUCTURE OF
THE NEIGHBORHOOD PEOPLE
INTERACTING AND THE WAYS IN
WHICH THE NEIGHBORHOODS INTERACT
PLAY AN ENORMOUS ROLE IN THE WAY
BEHAVIOR SPREAD THAT DOESN'T
SHOW UP FOR EPIDEMIOLOGICAL
SPREAD.
WE'RE THINKING ABOUT THE V
VIRALITY AND SIMPLE CONTAGIONS
LIKE THE SPREAD OF INFLAMMATION
AND DISEASE AND COMPLEX SPREAD
LIKE BEHAVIORS AND PRACTICES
WHICH REQUIRE US TO SWITCH FROM
SOMETHING WE'RE DOING NOW TO
SOMETHING NEW.
THIS BRINGS TO THE FOREGROUND
SEVERAL DIFFUSION I'LL TALK
ABOUT BRIEFLY AND IMPLICATIONS
TO COME FROM THIS WITH
METHODOLOGIC TIPPING POINTS IN
OUR SCIENTIFIC COMMUNITIES AND A
NICE THING THAT COMES OUT FROM
THIS IS THE BEHAVIOR OF
ECONOMICS TURN AND THE SCIENCE
IS PUSHING AWAY FROM RATIONAL
CHOICE TOWARDS COGNITIVE MODEL
OF BEHAVIOR AND STILL
INDIVIDUALISTIC AND SAYING THERE
ARE THINGS YOU CAN DO AND THE
RATIONALE CHOICE MODEL IS GIVEN
INFORMATION AND ANOTHER IS
GIVING STIMULI SAYING THINGS YOU
CAN DO INDIVIDUALLY TO CHANGE
THAT AND I WANT TO TAKE YOU AWAY
FROM THAT THAT RATIONALE CHOICE
ASSUMING INDIVIDUALS ACT ALONE.
FUNDAMENTALLY WE'RE LOOKING AT
LARGE-SCALE PRACTICE AS NORMS
AND THE QUESTION IS HOW DO WE
SHIFT NORMS OF THE POPULATION
AND NOT JUST ADDRESS THEM ONE
INDIVIDUAL AT A TIME?
TALKING ABOUT INFLUENCERS AND
LAND WITH HOW TO BUILD A
CONTAGION INFRASTRUCTURE AND
WHAT THIS MEANS FOR TIPPING
POINTS.
I'LL TALK ABOUT CONTAGIOUS NESS.
THAT'S A HUGE PART IN THINKING
HOW PRACTICES TAKE HOLD AND
SPREAD.
WHAT'S THE DIFFERENCE BETWEEN
SPREADING SOMETHING LIKE NEW
SCIENTIFIC PRACTICE AND
BEHAVIORAL ROUTINE VERSUS SIMPLY
SPREADING INFORMATION ABOUT IT?
SO IT COMES DOWN TO THE CLASSIC
IDEA OF HOW A VIRUS SPREAD.
FOR A SIMPLE CONTAGION SOMEONE
IS MORE ACTIVATED IN THE
NEIGHBORHOOD.
THAT PERSON COMES INTO CONTACT
WITH THE PERSON IN THE MIDDLE
AND THAT PERSON GETS IT.
AND THAT PERSON CAN THEN TURN
AROUND AND TELL EVERYBODY ELSE.
YOU HAVE WHAT WE THINK OF AS THE
CLASSIC KIND OF NETWORK
DIFFUSION PROCESS.
THIS PROCESS DESCRIBES
EVERYTHING FROM THE SPREAD OF
COVID-19 AND OTHERS AND THERE'S
SEVERAL ASSUMPTIONS BUILT IN.
ONE IS EVERY CONTACT IS
POTENTIAL TRANSMISSION AND
WHATEVER SPREADING IS EASY AND
FAMILIAR.
AND WHAT'S DIFFERENT ABOUT
SPREADING THE SCIENTIFIC
PATHOGENS HERE AGAIN YOU HAVE
SOMEONE IN A NEIGHBORHOOD AND
TRYING TO TALK TO THEM ABOUT A
NEW SCIENTIFIC PRACTICE THE
PERSON IN THE MIDDLE ALSO SEES
EVERYONE ELSE THEY'RE CONNECTED
TO SHOWN IN GRAY FOR THE SPREAD
OF A NEW PIECE OF INFORMATION OR
SPREAD OF A VIRUS BUT
NON-ADOPTERS ARE IRRELEVANT
BECAUSE IT'S A NOVEL THING
ENTERING THE COMMUNITY.
FOR A NEW PRACTICE THAT
REQUIRING CHANGING YOUR BEHAVIOR
THE PERSON IN THE CENTER HAS TO
LOOK AT EVERYONE ELSE THEY KNOW
AND SAY IF THEY'RE STILL
EXHIBITING THE SAME BEHAVIOR AND
PRACTICES, THIS ONE PERSON IS
ACTUALLY ILLEGITIMATE AND GOING
AGAINST THE NORMS OF COMMUNITY.
MAYBE IT TAKES A SECOND PERSON
TO SHOW UP AND THE WEIGHTED
EVIDENCE IN THEIR COMMUNITY ARE
WHAT THE NON-ADOPTERS AND THEY
DON'T HAVE TO ANYTHING EXCEPT
NOT ADOPT AND RISK THE NEW IDEA
AND INNOVATION AND THE THIRD
PERSON ADOPTS IT MAY SHIFT THE
COMMUNITY.
THIS IS TRUE WHENEVER INNOVATION
OR NEW PRACTICE OR IDEA
ENCOUNTERS RESISTANCE AND
NOTHING ENCOUNTERS MORE
RESISTANCE THAN A CHANGE
CAMPAIGN THAT TRIES TO SHIFT OUT
OUT OF A BEHAVIORAL ROUTINE WE
FEEL COMFORTABLE AND MAKES US
FEEL CONFIDENT AND REINFORCES
AND THAT REQUIRES MULTIPLE
CONTACTS.
NOT JUST MULTIPLE CONTACTS BUT
WE NEED IT FOR PEOPLE WHO ARE
ADOPTERS TO OUTWEIGH THE
INFLUENCES FROM THAT.
THIS IS RESISTANCE TO SOMETHING
WE DON'T KNOW HOW TO DO AND
THAT'S TRUE FOR ANYTHING
DIFFICULT AND FAMILIAR AND
COSTLY.
THIS RAISES SEVERAL RISKS.
THE MYTH OF VIRALITY THAT
EVERYTHING SPREAD LIKE A VIRUS
IS NOT TRUE FOR COMPLEX C
CONTAGIONS AND IF YOU MAKE IT
MORE FUN AND SELLABLE EVERYONE
WILL WANT IT.
THAT'S BUILT ON THE LOGIC OF
VIRUSES AND INFECTIOUSNESS BUT
DOESN'T WORK FOR BEHAVIORS THAT
REQUIRE A SOCIAL LEGITIMACY AND
THE LOGIC OF INFLUENCERS WHICH
IS CREATING TREMENDOUS EXPOSURE
FOR SOMETHING IT WILL SPREAD.
I'LL GLANCE OVER THE SECOND TWO.
I LIKE TO LOOK AT THIS EXAMPLE
BECAUSE IT GROUNDS WHAT I'M
TALKING ABOUT IN A CONCRETE WAY.
IN THE 1960s THERE WAS A MASSIVE
DEMOGRAPHIC TRANSITION TOWARDS
MODERNIZATION IN SOUTH KOREA,
INDONESIA AND PAKISTAN AND OTHER
COUNTRIES WITHIN A PERIOD OF 10
YEARS HAD INCREASED FOOD SUPPLY.
THEY HAD VACCINATION INCREASED,
SANITATION AND WATER SUPPLY AND
RADICALLY LOWERED INFANT
MORTALITY.
IN THESE CULTURES IT WAS NORMAL
FOR PEOPLE TO HAVE FIVE TO SIX
CHILDREN PER FAMILY AND THAT WAS
A STATUS SYMBOL SHOWING YOU WERE
DOING A GOOD JOB AS PARENTS AND
ALL OF A SUDDEN WHEN HAVE YOU
MODERN ADVANCES ENTERING, THEN
HAVING FIVE TO SIX CHILDREN
GENERATION AFTER GENERATION THE
MATH KICKS IN FAST IS EXTREMELY
DANGEROUS.
THE U.S. AND GREAT BRITAIN FROM
THE 1890s HAD A LONG TIME TO
ADJUST THE CULTURE OF FAMILY
PLANNING LITHE BIRTH -- LIKE
BIRTH CONTROL.
IN THESE CASES THE COUNTRIES HAD
TO ADAPT WITHIN A DECADE WHICH
IS TOO SHORT A TIME TO CHANGE
THE NATIONWIDE CULTURE AND HAD
MASS CAMPAIGNS TO SHIFT THE
FAMILY CULTURE ACROSS THE
NATION.
ALMOST EVERY NATION FAILED,
PAKISTAN AND KOREA SUCCEEDED IN
CHANGING THE ENTIRE COUNTRY'S
ACTIVITY TO CONTRACEPTION WITHIN
ONLY 20 YEARS -- SHIFTING AN
ENTIRE NATION'S CULTURE IN 20
YEARS IS AN INCREDIBLE
ACHIEVEMENT.
WHAT THEY DID THAT WAS UNIQUE
WAS THEY USED SOCIAL NETWORKS IN
THE VILLAGE COMMUNITIES INSTEAD
OF OPPOSITION POINTS THEY USED
THEM AS THE ENTRY POINTS.
THERE WERE THREE KINDS OF
VILLAGES THEY DISCOVERED.
SOME HAD BROKERAGE STRUCTURES
SMALL COMMUNITIES CONNECTED BY
WEAK TIES TO OTHER GROUPS AND WE
FIND ESSENTIALLY YOU WOULD GET
SPREADING WITHIN ONE OF THEIR
CLUSTERS AND EVERYONE WOULD
CONVERGE ON THE CONTRACEPTION
AND FOUND IN URBAN COMMUNITIES
THEY COULDN'T GET THE PEOPLE
THEY ADOPTED ADDS CHANGE AGENTS
TO HAVE ANY EFFECT ON ANYBODY
ELSE BECAUSE THEY WERE TOO
WIDELY DISTRIBUTED.
BY CONTRAST THEY FOUND THE
SCRIMMAGES WERE CHARACTERIZED
AND SMALL COMMUNITY CLUSTERS AND
THEY HAD MULTIPLE OVERLAPPING
TIES TO NEARBY COMMUNITIES WE'RE
ABLE TO GENERATE A CHANGE
PROCESS IN WHICH EACH CLUSTER
WOULD ADOPT CONTRACEPTION AND
SPILL OVER TO A NEARBY CLUSTER
AND UNTIL YOU GET ENTIRE
ADOPTION WITHIN VILLAGES.
THIS DIDN'T SPREAD IN A VIRAL
WAY OR EVERYWHERE AT ONCE LIKE
COVID-19 JUMPING ACROSS
AIRPLANES.
IT SPREAD IN THIS SPECIFIC
ALMOST SLOW LOCAL REGIONAL
FASHION AND CREATED
REINFORCEMENT AND THEN SPILLED
OVER TO A NEARBY GROUP.
AND WHAT'S GOING ON HERE WE CAN
THINK ABOUT THIS NOT JUST IN THE
CONTEXT OF COMMUNITIES IN
GENERAL BUT WITHIN ORGANIZATIONS
ONE OF THE CLASSIC WAYS OF
THINKING ABOUT SPREADING SCIENCE
IN THEIR PRACTICES IS TO SAY YOU
HAVE DIFFERENT SCIENTIFIC GROUPS
AND COMMUNITIES SILOED.
IF YOU CAN HAVE SOMEONE WHO ACTS
AS A BROKER YOU CAN CREATE TIES
TO LIKE ELIZABETH IN GROUP B AND
ANOTHER TO GROUP C AND SPREAD IT
FROM GROUP TO GROUP.
THE THING TO KNOW IS IT'S TRUE
PEOPLE LIKE ROBERT HAVE BEEN
HERALD THE IN THE COMMUNITY
SINCE THE 1990s AS THE LINCHPIN
FOR GETTING NEW SCIENTIFIC
PRACTICES ADOPTED AND
ORGANIZATIONAL CHANGE TO
SUCCEED.
THE THING IS ALL THEY'RE TALKING
ABOUT IN THE CHANGES IS SPREAD
NOT ACTUAL ADOPTION.
ROBERTS TIE IS A WEAK TIE, ONE
SINGLE LINK AND HE CAN SPREAD
INFORMATION TO ELIZABETH BUT HER
GROUP AND THE WAY THEY DO THINGS
AND ROUTINES AND PRACTICES
AREN'T GOING TO BE INFLUENCED BY
ONE PERSON SAYING EVERYONE
SHOULD CHANGE THEIR PRACTICES TO
DO THIS NEW THING.
IN FACT THEY'LL RECOGNIZE AS
ROBERT GAINED STATUS AND
PRESTIGE AS THE BROKER AND WILL
CONVINCE PEOPLE.
IT'S NOT THAT ROBERT'S
UNTRUSTWORTHY AS A PERSON BUT
HIS ADVANTAGE COMES FROM
SPREADING THIS AND THERE'S
SCEPTICISM FROM WHICH HIS IDEAS
ARE GREETED.
THE NETWORK STRUCTURE CHANGES
WITH NOT JUST ONE LINK OR NARROW
BRIDGE BUT INTERACTING TIES AND
A WIDE BRIDGE IS A TECHNICAL
CONCEPT AND THERE'S A SERIES OF
PAPERS AND IN THE BOOK AND WHAT
IT BASICALLY MEANS IS THERE'S
MULTIPLE INTERACTING TIES FROM
MEMBERS OF GROUP A AND THEY CAN
LOOK AND SEE HOW OTHER MEMBERS
OF GROUP B ARE USING INNOVATION
AND HOW IT WORKS FOR THEM AND
AFFECTS THS THEIR DAILY PRACTIC
AND COMMUNICATE HOW TO INTEGRATE
IT.
THE STRUCTURE OF WIDE BRIDGES
WITHIN AN ORGANIZATION IS THE
KEY INFRASTRUCTURAL ELEMENT THAT
ALLOWS A NEW SET OF SCIENTIFIC
PRACTICES TO SPREAD FROM ONE
GROUP TO ANOTHER.
WHAT'S INTERESTING IS ONCE YOU
HAVE THIS KIND OF STRUCTURE, IT
CREATES KIND OF AN ADOPTIVE
INFRASTRUCTURE SO IT'S NOTE JUST
SPREADING ONE INNOVATION OR
ANOTHER BUT CREATES THE CAPACITY
FOR LOTS OF DIFFERENT
INNOVATIONS TO SPREAD AND IN THE
BOOK I TALK ABOUT A LOT OF
DIFFERENT EXAMPLES BUT ONE MORE
STRIKING AND A CASE WE'RE
DISCUSSING IS THE HUMAN GENOME
PROJECT.
YOU HAVE DOZENS OF DIFFERENT
LABS AND CENTERS AROUND THE
WORLD ALL OF WHOM WERE COMPETE
FORG A NOBEL PRIZE EACH YEAR AND
TRYING TO COORDINATE ON METHODS
AND PROTOCOLS FOR DOING THEIR
WORK ON THE GENE SEQUENCING.
THEY WERE ALL USING DIFFERENT
PRACTICES AND PROTOCOLS AND HAD
DIFFERENT I.P. STANDARDS AND HAD
TO FIGURE OUT HOW EACH GROUPS
PART COULD BE SHARED AND
REPLICABLE AND SHARE IT WITH
ANOTHER GROUP AND ONE OF THE
VERY SMART THINGS PEOPLE DID
WITH THE HUMAN GENOME PROJECT
INSTEAD OF SETTING IT UP AS A
TOP-DOWN ORGANIZATION LIKE THE
DIFFERENT CENTERS HAD WIDE
BRIDGES AND GROUPS SHOWED THEY'D
EXCHANGE EXCURSIONS AND THEN
MEET WITH THE GROUPS FROM THE
OTHER LAB AND DO DAILY TOURS AND
SEE HOW THEY WORKED AS A TEAM
AND THEIR PROTOCOLS FUNCTIONED
AND BRING THAT BACK HOME.
IT TURNED OUT THE SAME LOGIC WAS
VALID FOR THE FIELD OF NETWORK
SCIENCE.
THERE ARE OTHER CASES
INTERESTING LIKE THE SOFTWARE
INITIATIVE AND BLACK LIVES
MATTER BUT THE NETWORK
INFRASTRUCTURE PROVIDES A
GENERAL MODEL FOR THINKING HOW
ORGANIZATIONS THAT WOULDN'T BE
CONNECTED CAN CREATE THE
INTERCONNECTIONS IN A WAY THAT'S
FUNCTIONAL FOR ADOPTING NEW
TECHNIQUES AND PRACTICES.
SO ONE OF THE THINGS WE CAN
THINK ABOUT ONCE WE SEE THIS IS
WE CAN IMAGINE TWO ORGANIZATIONS
OR TWO SCIENTIFIC ECOLOGIES
WHERE THEY BOTH HAVE THE SAME
STRUCTURE.
YOU CAN SAY THAT'S GREAT AND
THAT'S WHERE WE'LL SUCCESS.
IT'S NOT JUST THE INFRASTRUCTURE
ALONE BUT HOW THE INFRASTRUCTURE
IS TARGETED.
WHAT ARE OUR ENTRY POINTS INTO
THE ORGANIZATION TO GENERATE THE
MOST IMPACT FOR SOCIAL CHANGE.
WELL, WE CAN SEE HERE FOR A
STRATEGY LIKE WHERE DO WE PLACE
OUR CHANGE AGENTS?
THE STRATEGIES TO PLACE THE
CHANGE AGENTS AS WIDELY
DISPERSED THE LOGIC IS VIRAL
LOGIC.
IF THEY'RE SURROUNDED BY LOTS OF
SUSCEPTIBLE PEOPLE THEY WANT
THEM DISPERSED AS MUCH AS
POSSIBLE TO SPREAD YOUR IDEA.
THAT'S TRUE IF WHAT YOU'RE
SPREADING SAY VIRAL CONTAGION
BUT YOU CAN GET A STRONG BACK
FIRE EFFECT.
ESSENTIALLY EVERYONE WHO IS A
CHANGE AGENT BECAUSE THEY'RE
SURROUNDED BY PEOPLE IS ALSO
SURROUNDED BY A SEA OF
COUNTERING INFLUENCES.
PEOPLE PUSHING BACK AGAINST
THEM.
NOT ONLY ARE THEY INEFFECTIVE AT
CONVINCING PEOPLE THEMSELVES BUT
IN SHORT ORDER THEY'LL BEGIN TO
EXHIBIT HIGH LEVELS OF
RECIDIVISM BACK TO OLD BEHAVIOR
BECAUSE EVERYONE ELSE AROUND
THEM IS DOING IT AND ALTERNATIVE
STRATEGIES IS TO TAKE THE SAME
NUMBER OF CHANGES AND FROM THE
COST PERSPECTIVE IT'S THE SAME
CAMPAIGN BUT INSTEAD OF
DISTRIBUTING THEM ALL OVER THE
NETWORKS YOU GROUPED THEM IN ONE
CLUSTER.
THIS SEEMS COUNTERINTUITIVE FROM
MY CAMPAIGN AND HAVING
TELEMARKETERS TELL EACH OTHER
WHY WOULD YOU CONNECT YOUR
CHANGE AGENTS?
YOU CAN REINFORCE THIS PRACTICE
AND THESE ARE THE STANDARDS OF
RIGOR AND THE RIGHT ONES TO
ADOPT AND CAN CONVINCE PEOPLE IN
THEIR CLUSTER AND THAT CAN SPILL
OVER TO PEOPLE IN NEARBY
CLUSTERS AND WIDE BRIDGES CARRY
THIS INNOVATION FROM CLUSTER TO
CLUS
CLUSTER.
WHAT IS SPREADING IS A REAL
CHANGE IN BEHAVIOR AND SOCIAL
NORMS WHICH HAS A CHARACTERISTIC
TO IT.
I'LL QUICKLY WRAP UP WITH A
COUPLE NOTES WITH THE
INFLUENCER.
WE LOOK BACK AT THE NETWORK
STRUCTURE AND SAY SOME PEOPLE IN
CENTER ARE INFLUENTIAL WHY DON'T
WE GET THOSE AND WHEN WE
DISCOVER AN IDEA THEY CAN SPREAD
IT TO LOTS OF PEOPLE.
IT'S WHERE THE IDEA OF A SUPER
SPREADER COMES FROM.
THE PROBLEM COMES WITH THE
COMBUST CONTAGION EVERY TIE YOU
HAVE TO A NON-ADOPTER ISN'T JUST
AN OUTGOING TIDE BUT INCOMING
TIDE FROM PEOPLE MAINTAINING AN
EXISTING SOCIAL NORM.
THE MORE CONNECTED SOMEONE IS
THE MORE THE INFLUENCES THEY
ENCOUNTER WHEN THINKING OF
ADOPTING SOMETHING NO ONE ELSE
IS USING EXCEPT FOR ONE PERSON
IN THE PERIPHERY.
THE REASON WHY THE PEOPLE MOST
HIGHLY CONNECTED OR RELUCTANT
EARLY ON TO SHIFT THEIR
STANDARDS OF PRACTICE IN A WAY
PUBLIC BECAUSE YOU HAVE PEOPLE
PRIMARILY RESISTING WHEREAS ON
THE RIGHT SIDE IT ONLY TAKES A
SMALL NUMBER OF ACTORS TO
GENERATE THE CRITICAL PASS OF
INTEREST FOR PEOPLE IN THAT
NETWORK AND THAT CAN SPREAD AND
CASCADE IN THE WAYS I SHOWED
YOU.
THIS IS WELL KNOWN.
FOR DIFFICULT OR ORGANIZATIONAL
CHANGES THAT REQUIRE SHIFTING
BEHAVIORS FROM ONE NORM TO
ANOTHER NORM THAT THE KIND OF
EDGE ACTORS PRO ACTORS TEND TO
BE BETTER SUITED TO INITIATE
THAT KIND OF CHANGE SINCE 1970
AND IN THE LAST DECADE WE HAD
GOOD DATA ON HOW WE CHANGE
SPREAD AND CONSISTENTLY FIND THE
HIGHLY CONNECTED INFLUENCERS
TEND TO BE THE LAST ADOPTERS NOT
THE FIRST.
WHAT THIS MEANS FOR TARGETING
NETWORKS THERE'S MORE TO SAY
ABOUT TIPPING POINTS BUT FOR NOW
THE BEST WAY TO TALK ABOUT THIS
IS BY SAYING IF WE HAVE A LARGE
CONNECTED SCIENTIFIC COMMUNITY
WOULDN'T OUR BEST STRATEGY FOR
INITIATING A CHANGE IN PRACTICES
JUST BE TO FIND THE SOCIAL STAR
AND ACTIVATE THEM?
LIKE THEY MAY BE RELUCTANT
INITIALLY BUT IF WE CAN
INCENTIVIZE THEM OR COERCE THEM
WE SHOULD BE GET CASCADING.
THERE'S A REAL DANGER AND THIS
IS ONE OF THE REASONS WHY I
WROTE THE BOOK IS BECAUSE A LOT
OF PEOPLE THINK THE KITCHEN
THINK APPROACH CAN'T DO HARM.
THERE MAY BE A DIFFERENCE BUT
SINCE WE HAVE TECHNOLOGY AND
MARKETING KNOW-HOW BUILD AROUND
CON TAKE -- CONTAGIONS AND IF
YOU TAKE SOCIAL NORMS HEAD ON
AND TREAT THEM LIKE A VIRUS,
THEY ACTUALLY CREATING A VERY
STRONG BACKFIRE AGAINST
INITIATIVE WHICH WILL SALT THE
EARTH FOR FUTURE ATTEMPTS TO
ADVANCE YOUR INITIATIVE.
SO A GREAT EXAMPLE IS SOMETHING
LIKE GOOGLE GLASS WHERE MOST OF
US REMEMBER AND IT WAS A STICKY
PRODUCT AND IT IS STILL BECAUSE
WE STILL TALK ABOUT IT THEY GOT
ELITES AND INFLUENCERS AND GOT
THEM TO BUY THESE SUNGLASSES AS
A SIGN OF THEIR SOPHISTICATION
AND THE IDEA WAS HIGH STATUS
PEOPLE WOULD WEAR THE GLASSES
YOU COULD USE TO SURF THE WEB
WHILE TALKING TO SOMEONE OR USE
TO SURVEY THE LANDSCAPE AND
RECORD IT WHILE TALKING.
WHAT THEY DIDN'T EXPECT IS WHILE
THE BEHAVIOR OF THE ELITE WHICH
WAS SUPPOSED TO CREATE
ASPIRATIONAL HEIGHTS AND THERE
WAS A NORM SURFING THE WEB WHILE
TALKING IS SOMEONE IS RUDE AND
RECORDING THE LANDSCAPING IS A
VIOLATION OF SOCIAL NORMS AND IT
BECAME SEEN AS WHAT ARE
GLASS-HOLES.
ON THE OTHER SIDE OF A SOCIAL
DIVIDE AND IT KIND OF TO SOME
EXTENT CREATED A CULTURE WAR
WITHIN THE COMMUNITY WHERE
BASICALLY SHOPS AND RESTAURANTS
PUT UP SIGNS GLASS-HOLES NOT
ALLOWED IN.
THAT'S A STRONG SOCIAL REACTION.
COMPLETELY UNANTICIPATED BY THE
COMPANY BUT HAD TWO DOWN SIDES.
ONE IS THE PRODUCT LINE WAS
ESSENTIALLY SHUT DOWN BUT THE
COMPANY THOUGHT LESS OF GOOGLE
BECAUSE THEY TRIED TO PROMOTE SO
AGGRESSIVELY A TECHNOLOGY
AGAINST SOCIAL CULTURE AND THEY
TRIED TO FLOOD THE MARKET AND
GOOGLE SAW THE VAST MAJORITY OF
PEOPLE THEY KNEW KNEW ABOUT
GOOGLE PLUS AND THERE WAS
MASSIVE AWARENESS BUT WEREN'T
USING IT AND PEOPLE STARTED
EXPLAINING TO THEMSELVES WHY
THEY'RE NOT USING IT.
THEY CREATED AN INADVERTENT WORD
OF MOUTH BUZZ.
THERE'S DANGER IN USING THE
VIERING VIRAL STRATEGIES BECAUSE
YOU CAN GENERATE BACKFIRE
EFFECTS.
THIS MEANS THAT IF YOU'RE IN
THIS STRUCTURE WHERE A NORMS OR
PRACTICES IN PLACE AND JUST
GETTING AN INFLUENCER TO ADOPT
IT WON'T JUST WORK AND EVERYONE
THEY'RE CONNECTED TO FACES THOSE
FORMS AND IT'S HARD TO GET A
CASCADE GOING.
THIS THESE ARE INTERESTING
RESULTS AND IF YOU CAN TARGET
OTHER PARTS OF THE SOCIAL
NETWORKS I REFER TO AS THE
COMPLEX CENTER IN THE CLASSIC
VISUALIZATION LAYOUT WITH
LOCATIONS WITH MANY WIDE BRIDGES
THAT ARE WELL POSITIONED TO
SPREAD SOCIAL NORMS THESE ARE
NOT GOOD FOR SPREADING SIMPLE
CONTAGIONS LIKE DISEASES BUT
GOOD FOR GENERATING SOCIAL
SUPPORT.
YOU CAN SEE HOW EFFECTIVE THEY
ARE.
WHAT HAPPENS IS YOU GET SOCIAL
REINFORCE MANY FOR THE NORMATIVE
CHANGE AND IT DOESN'T JUMP RIGHT
TO THE CENTER AND TAKE OVER THE
NETWORK AND SPREAD LIKE A VIRUS.
IT BUILDS A CRITICAL MASS AND
ONCE THE CRITICAL MASS IS
REACHED IT REACHES TO THE
NETWORK AND INITIATES CHANGE.
IT'S A DIFFERENT SPREADING
PATTERN BUT THE PATTERN THAT
EXPLAINS HOW SOCIAL NORMS AND
COMMUNITY PRACTICES CHANGE.
I'LL LEAVE YOU WITH A COUPLE
POINTS FOR THINKING ABOUT THE
IDEAS FOR THE REST OF THE
MEETING.
ONE IS THAT SOCIAL REINFORCEMENT
IS KEY.
WHEN YOU'RE CHANGING SCIENTIFIC
OR SOCIAL PRACTICES WHAT YOUR
FOCUSSED ON IS THE QUESTION OF
LEGITIMACY AND CREDIBILITY AND
WIDE BRIDGES FROM ONE GROUP TO
ANOTHER GROUP ARE ESSENTIAL FOR
SPREADING THE COMMITMENT ACROSS
THE GROUPS IS KEY FOR LARGE
SCALE CHANGE YOU'LL HAVE TO
EXPAND GROUPS AND THINK WHERE
THE BRIDGING INFRASTRUCTURE WILL
ALLOW TO YOU SPREAD THE
INNOVATION WITH REINFORCEMENT SO
THE LEGITIMACY CAN BE SHIFTED BY
THE BRIDGE AND THE REINFORCING
TIES ACROSS IT.
THE SECOND IS RELEVANCE.
OBVIOUSLY, IF PEERS PERCEIVE
DISTANCE OR SOCIAL OPPOSITION
THEN THOSE CONNECTIONS AREN'T
GOING TO BE AS MEANINGFUL.
WE SAW THIS WITH FACE MASKS.
IT'S CLEAR THE ADOPTION OF FACE
MASKS SPREAD IN A DIFFERENT WAY
THAN THE ADOPTION OF COVID-19.
COVID-19 WAS NOT AT ALL AFFECTS
AND THERE'S SOMETHING DIFFERENT
.
WHAT WE THINK ABOUT IN HOW WE DO
OUR SUPPORT FOR VACCINATION AND
FACE MASKS AND GENERATE
REINFORCE
REINFORCEMENT.
IT'S IMPORTANT SIMILARITIES AND
THERE ARE CASES WHERE DIVERSITYT
THAT'S DIFFERENCE BETWEEN
PEOPLE IS ALSO VERY EFFECTIVE IN
CREATING RELEVANCE AND WHEN WE
LOOK AT THE CHANGES AND
PRACTICES WE'RE NOT LOOKING AT
THE SPREAD OF INFLAMMATION.
IT'S NOT JUST TELLING PEOPLE
THIS IS WHAT YOU SHOULD DO OR IS
AVAILABLE.
THAT OFTEN BOUNCES OFF THE FACT
PEOPLE HAVE EXISTING PRACTICES
AND FALLS ON DEAF EARS AND
THERE'S FRUSTRATION ABOUT
CLIMATE CHANGE BECAUSE IT'S THE
RELEVANT PEER NETWORK AND NORMS
DOING THE WORK.
IN ORDER TO THINK ABOUT THE
CHANGE AND ALTER SCIENTIFIC
RIGOR AND ACCEPTANCE ON NEW
PRACTICES ENFORCING THE
STANDARDS OF RIGOR IT'S ABOUT
SHIFTING THE NORMS OF DAILY
PRACTICE FOR THE COMMUNITIES.
SO I CAN TALK MORE ABOUT THIS
AND DISCUSS CHANGE FOR THE
IMPLICATIONS FOR TIPPING POINTS.
THE WEB ADDRESS INCLUDES MY
RESEARCH AND THANK YOU VERY
MUCH.
>> THAT WAS FANTASTIC.
IT MAKES ME LOOK AT WHAT I'M
DOING AND PLAN TO DO AND THE
THINGS OTHERS HAVE SAID AND
WHERE THEY'VE SUCCEEDED AND
INTERESTED TO HEAR IN THE
DISCUSSIONS HOW IT AFFECTS HOW
WE MIGHT THINK.
OUR THIRD SESSION IS FOCUSSED ON
NORMS AND BOW -- BEHAVIORS AND
MOTIVATION WILL RELATE TO WHAT
YOU TALKED ABOUT.
DAMON SAID HE'LL BE LISTENING IN
AND MAYBE REFLECT ON ABOUT WHAT
HE'S HEARD IN THE SPEAKERS AND
ENSUING DISCUSSION BEFORE THE
BREAK.
HE'LL STILL BE LISTENING AND BE
ABLE TO SHARE WISDOM.
THANK YOU FOR THAT, DAMON AND
I'M SURE YOU'LL BE GETTING
SEVERAL DIMES OF ROYALTIES FROM
THAT TALK.
THIS IS NOT SOMETHING WE
NORMALLY HEAR IN THESE MEETINGS
FROM PROFESSIONAL SOCIOLOGISTS
WHO STUDY THIS.
I WANT TO INVITE THE FIRST
MODERATOR AND THREE SPEAKERS FOR
THE NEXT SESSION.
IT'S DAVID ELLISON.
HE'S A DEAN OF THE INDIANA
SCHOOL OF PUBLIC HEALTH A
STATISTICIAN ACTIVE IN THE
NATIONAL ACADEMIES ON ISSUES OF
RESEARCH, REPRODUCIBILITY AND
RIGOR AND WILL BE MODERATING THE
SESSION AND THE THREE SPEAKERS
WHO HE'LL INTRODUCE ARE MALCOLM
MACLEOD AND ANNA HATCH AND APRIL
CLYBURNE-SHERIN.
WE'RE A LITTLE BEHIND ON TIME
BUT BETWEEN THE BREAK AND LUNCH
WE'LL BE ABLE TO RECLAIM A
LITTLE BIT SO WE MAY HAVE A
SHORTER BREAK AND LUNCH BUT
WE'LL PLAY IT BY EAR.
>> THANK YOU STEVE AND NINDS AND
FOR INVITING ME HERE TO BE WITH
YOU.
I'M ALSO DEEPLY HONORED TO
INTRODUCE OUR SPEAKERS TODAY WHY
I'LL THEN MODERATE AFTER THE
BREAK.
I'LL KEEP MY INTRODUCTIONS
CHEAP.
YOU HEARD THEIR NAMES.
YOU SEE THEIR NAMES BEFORE YOU.
I'M GOING JUMP RIGHT IN SO WE
HAVE THE MAXIMUM TIME FOR
SPEAKERS TO REWARD US WITH THEIR
ACUITY AND I'LL BE LONG-WINDED
LATER.
THE FIRST SPEAKER IS APRIL
CLYBURNE-SHERIN FROM
REPRODUCIBILITY FROM EVERYONE.
WELCOME.
>> SO THANKS SO MUCH, DAVID.
I'M EXCITED TO BE HERE.
THIS IS MY FIRST TALK IN A FEW
YEARS WITHOUT MY DOG SITTING ON
MY LAP SO IT'S A LITTLE BIT OF A
MILESTONE FOR ME AND I'M GLAD
IT'S HERE WITH ALL OF YOU
BECAUSE IT'S SOMETHING I'VE
WORKING ON.
I'M THE EXECUTIVE DIRECTOR OF A
GROUP CALLED REPRODUCIBILITY FOR
EVERYONE.
AND SO WHAT WE ARE IS AN
EDUCATIONAL PROJECT.
WE'RE COMMUNITY LED AND WE FOCUS
ON REACHING RESEARCHERS AT
SCALE, REACHING RESEARCHERS
WHERE THEY'RE AT AND TRYING TO
INCREASE THE DISCOVERY OF TOOLS
AND METHODS.
TO SPEAK TO CONNECT ON WHAT WE
TALKED ABOUT AND WHAT I SEE OUR
PROJECT DOING IS FOCUSSING ON
THE PEER-TO-PEER TYPE OF
KNOWLEDGE SHARING TRYING TO FILL
SOME GAPS IN RESEARCHER TRAINING
AND ALLOWING RESEARCHERS TO HAVE
A PLATFORM TO SHARE THOSE THINGS
THAT WORKED FOR THEM TO IMPROVE
ORGANIZATION, DOCUMENTATION,
OPTIMIZATION AND DISSEMINATION
OF THEIR WORK.
IT'S RESEARCHERS TEACHING
RESEARCHERS WITHIN THEIR OWN
DISCIPLINES.
SO WE ARE VOLUNTEER RUN.
WE'VE RUN MANY WORKSHOPS ALL
OVER THE WORLD TO LARGE LARGE
GROUPS OF SCIENTISTS.
WE'RE OFTEN TIMES DOING THIS AT
CONFERENCES SO WE'RE MEETING
FOLKS WHEN THEY ALREADY ALLOTTED
TIME TO LEARN AND THEY MAY HAVE
TIME TO COME AND DROP IN ON
SOMETHING THEY ARE NOT ACTUALLY
THAT KNOWLEDGEABLE ABOUT YET.
THE THEME FOR THIS SECTION IS
MOTIVATION.
I'M NOT GOING TO TALK ABOUT THE
COMMUNITY ITSELF I'LL TALK ABOUT
THE MOTIVATIONAL SIDE AND WHAT
WE LEARNED ON DOING THE WORK THE
LAST FEW YEARS.
THE FIRST SECTION ON WHAT
MOTIVATED IS IN THE SLIDE DECK
AND YOU CAN READ THROUGH OUR
ORIGIN STORY AND YOU CAN TALK TO
ME ABOUT IT LATER BUT I'LL SKIP
IT FOR THE SAKE OF TIME AND I'LL
TALK ABOUT WHAT MOTIVATES
RESEARCHERS TO ATTEND THE
TRAINING BASED ON THE SURVEY
RESULTS AND CONVERSATIONS WE HAD
IN OUR COMMUNITY.
THE WAYS IN WHICH INTRODUCTORY
TRAINING MOTIVATES CHANGES IN
BEHAVIOR AS WELL AS WHAT
MOTIVATES SCIENTIST TO GO FROM
DECIDING TO BECOME A PARTICIPANT
TO A TRAINER WHICH IS WHAT WE'RE
TRYING DO AND COMMONLY WHAT
E OF OUR WORK SHOPS.E ATTENDS
SO THIS IN SUMMARY FROM DOING
MANY WORKSHOPS WE SAW A GAP WE
WERE LEAVING RESEARCHERS BEHIND.
THERE WAS A LOT PEOPLE EXCITED
BUT A SMALL GROUP COMING AND
DEDICATED THREE, FOUR HOURS
EVERY DAY TO LEARN NEW
PRACTICES.
AND MOST RESEARCHERS WERE NOT
COMING TO THESE LONGER FORMAT
WORK SHOPS.
WHAT WE FOCUS ON IS CHANGING THE
INTRODUCTORY REPRODUCIBILITY
WORKSHOP FOR RESEARCHERS AND
AIMING AT FUNNELING PEOPLE IN A
SHORT PERIOD OF TIME TO THE
OTHER AMAZING PROJECTS GOING ON.
MANY ARE RUNNING THE OTHER
PROJECTS WE POINT FOLKS TO
WITHIN OUR WORKSHOPS.
WE PLAN TO BE THE FIRST TOUCH
FOR FOLKS THAT DON'T EVEN KNOW
WHAT REPRODUCIBILITY MEANS AND
MAYBE THEY DON'T CARE AT ALL.
WE'RE THERE TO FILL GAPS IN
TRAINING AND GIVE THEM THE
THINGS THEY NEED TO START A
JOURNEY OF LIFE-LONG JOURNEY.
WE FOCUS ON INTRODUCTIONS TO
REACH MORE PEOPLE AND FIT IT IN
THEIR SCHEDULE.
30 TO 90 MINUTES IS A USUAL
WORKS
WORKSHOP.
WE'RE TARGETING CONFERENCES
BECAUSE THAT'S WHERE WE GET
TOOLS AND METHODS AND SOMETIMES
WE HAVE 300 RESEARCHERS AT A
TIME.
WE ALSO END UP WITH MORE DIVERSE
GROUP OF CAREER LEVELS IN THOSE
WORK SHOPS THAN WE DID WHEN WE
WOULD RUN THE WORKSHOPS ON
CAMPUS.
WE HAD SOME OF THE MORE SENIOR
RESEARCHERS IN THE ROOM AND MORE
P.I.s RATHER THAN BEING EARLY
CAREER RESEARCHERS AND STUDENTS.
SO IN TERMS OF WHAT MOTIVATES
SCIENTISTS TO ATTEND THESE
TRAININGS IS FROM OUR SURVEY
OVER THE LAST COUPLE YEARS.
THIS IS WHEN WE'RE DOING THE
VIRTUAL.
MOST PEOPLE ARE COMING BECAUSE
THIS IS SOMETHING THEY HAVE
EXPERIENCED.
82% OF THE PARTICIPANTS HAVE
TALKED ABOUT STRUGGLING WITH
REPRODUCING RESULTS.
72% TALK ABOUT HOW THEY'VE HAD A
HARD TIME REPRODUCING THE
RESULTS OF SOMEONE ELSE AND THAT
COULD BE A PUBLISHED PAPER OR
COLLEAGUE IN THEIR LAB AND HALF
REPORT STRUGGLING WITH
REPRODUCING THEIR OWN RESULTS.
THIS IS THE THING WE FOCUS ON IN
THE WORKSHOP.
WE SAY THE OVERLAP BETWEEN THOSE
THINGS YOU CAN DO TO IMPROVE
YOUR LIFE TO REDUCE THE PAIN AND
FRUSTRATION YOU FEEL WHEN YOU
CAN'T FIND THE MOST RECENT FILE
AND SOMEONE WALKED AWAY WITH
YOUR DATA.
THAT OVERLAPS SIGNIFICANTLY WITH
THE STEP TO REPRODUCE IN YOUR
DISCIPLINE.
WE FOCUS ON PUSHING PEOPLE
TOWARDS THOSE TOOLS AND TRICKS
THAT WILL BENEFIT THEM
PERSONALLY IN THE SHORT TERM.
SO IN OUR SURVEYS WE ASK AND
INVITE THE PARTICIPANTS TO SHARE
PERSONAL EXPERIENCES WITH
REPRODUCIBLE ISSUE.
OBVIOUSLY, YOU WILL NOT BE ABLE
TO READ THIS BUT WHAT WE SEE
OVER AND OVER AGAIN IS PEOPLE
BEING FRUSTRATED WITH NOT BEING
ABLE TO REPRODUCE WORK AFTER
MANY ATTEMPTS.
LOSING TIME MONTHS, SOMETIMES
PEOPLE SAY A YEAR OF THEIR Ph.D.
BEING TOLD THEY SHOULD DO THIS
METHOD AND THE METHOD DOESN'T
WORK AND DOESN'T WORK AND
SOMEONE ELSE TRIES IT AND IT
DOESN'T WORK.
AND WE'RE LEARNING THE HARD WAY
YOU DIDN'T DOCUMENT IT PROPERLY
AND HAVE TO DO SOME WORK AND
MISSING INFORMATION.
THERE'S NOT ENOUGH INFORMATION
IN THE METHOD SECTION THEY HAVE
TO CONTACT YOU HAVE TO CONTACT
THE RESEARCH TO FIND OUT MORE
AND SOMETIMES THEY'RE COMING
FRUSTRATED AND SOMETIMES THEY'RE
COMING WITH A SOLUTION.
WE LIKE TO MAKE SURE PEOPLE KNOW
FROM THE BEGINNING WE WANT THEM
TO COME, TAKE WHAT THEY LEARNED
THAT HARD KNOWLEDGE OF A YEAR OF
SOMETHING NOT WORKING WHAT
HELPED THEM OVERCOME AND THEY
CAN INCLUDE THAT IN A FUTURE
VERSION OF THE WORKSHOP.
SO TO REITERATE, THEIR COMMUNELY
LED AND CURATED AND THE
MATERIALS ARE OPEN AND USABLE
AND ADAPTABLE AND VERSION JABL
AND RESEARCHERS MAKE THEM THEIR
OWN SO NO TWO WORK SHOPS ARE
EXACTLY THE SAME AND IF YOU'RE
INTERESTED IN TAKING A LOOK AND
SEEING SOME OF OUR CONTENT IT'S
HERE.
THESE ARE SOME TOPICS WE COVER,
DATA MANAGEMENT IS A COMMON ONE
AND THERE'S STUFF IN THERE YOU'D
EXPECT EVERYBODY TO HAVE LEARNED
IN THEIR CORE RESEARCH TRAINING
BUT WE'VE BEEN SURPRISED OVER
AND OVER AGAIN HOW THE BASICS
HOW TO ORGANIZE MATERIALS AND
NAME YOUR FILES IS SOMETHING A
LOT OF PEOPLE ARE NOT ACTUALLY
EVER BEING TAUGHT HOW TO DO
PROPERLY.
FINALLY, THIS TRAINING IS
MOTIVATING NEW BEHAVIOR IN OUR
PARTICIPANTS.
86% IN OUR POST-WORKSHOP SURVEY
REPORTED THEY LEARNED SOMETHING
THEY'LL USE IN THEIR RESEARCH.
I THINK A TAKEAWAY FOR EVERYONE
HERE IS THIS IS JUST
PEER-TO-PEER LEARNING, THIS IS
GRASSROOTS TRAINING.
IT'S NOT SOPHISTICATED
CURRICULUM BUT ALREADY GOING A
LONG WAY TO REACH EIGHT OF
FOLKS.
IT WOULDN'T TAKE EVERYBODY ALL
THE WAY BUT IT GETS PEOPLE
STARTED.
SO WILL WE DID A CAMPAIGN
RECENTLY GETTING OUR
PARTICIPANTS TO SHARE WHAT THEY
LEARNED IN A WORKSHOP.
THIS IS AN EXAMPLE OF SOMEONE
TALKING ABOUT HOW THEY DIDN'T
KNOW ABOUT PROTOCOL SHARING AND
REPOSITORY AND THAT WOULD HAVE
SAVED THEM SIX MONTHS IN THEIR
PH.D.
AND THIS RESEARCHER THEN BECAME
AN INSTRUCTOR.
THIS IS ANOTHER RESEARCHER WHO
LEARNED ABOUT ELECTRONIC LAB
NOTE BOOKS WHO IMPLEMENTED THAT
FOR THEIR RESEARCH IN THE FUTURE
AND THEIR RESEARCHER ALSO BECAME
AN INSTRUCTOR AND AGAIN THE
IMPORTANCE OF RE-AGENT
REPOSITORIES WAS A NEW CONCEPT
FOR THEM AND THEY SAID HOW MUCH
NICER IT WOULD HAVE BEEN AND
THIS RESEARCHER ALSO BECAME AN
INSTRUCTOR AND PROTOCOL
REPOSITORIES ARE SOMETHING THAT
ALSO MANY PEOPLE LEARN ABOUT FOR
THE FIRST TIME IN A WORKSHOP.
FINALLY, THE OTHER MOTIVATION WE
SEE WHEN WE TALK ABOUT
RESEARCHERS TO DO THIS WORK AND
LABOR IS TO THINK ABOUT WHAT
MOTIVATES THEM TO KEEP DOING IT
AND WHAT DEMOTIVATES THEM.
SO BURN OUT WILL DEFINITELY
CAUSE PEOPLE TO LEAVE A
COMMUNITY.
EXCLUSIONS AND NOT BEING
DELIBERATE IN PEOPLE'S NEEDS AND
INCLUSION AND BELONGING.
WE HAVE A PROGRAM OF DIVERSITY
STIPENDS AND IF YOU WANT TO
LEARN MORE ABOUT THAT REACH OUT
AND WE'LL TALK TO YOU ABOUT IT.
IN SUMMARY, SCIENTIST SEEKING
REPRODUCIBILITY TRAINING BECAUSE
THEY'RE MOTIVATED BY THEIR OWN
CHALLENGES IN RESULTS.
THEY'RE INTERESTED IN TOOLS AND
METHODS AND SHORT DURATION
TRAININGS CAN MOTIVATE SCIEN
SCIENTISTS TO TRY MORE THINGS
AND HAVE TO AVOID BURNOUT AND
INCLUDE THE WHOLE COMMUNITY.
>> THANK YOU.
THAT WAS TERRIFIC.
THE NEXT SPEAKER IS ANNA HATCH
AND WE'RE PICKING UP TIME AND
WE'LL HAVE OUR QUESTIONS IN THE
PLENARY DISCUSSION AFTER ALL THE
SPEAKERS.
>> THANK YOU FOR THE KIND
INVITATION AND THE PROGRAM
DIRECTOR FOR RESEARCH ASSESSMENT
IT'S A LONG NAME SO WE CALL IT
DORA FOR SHORT AND WE FOCUS ON
RESEARCH ASSESSMENT.
THAT'S THE LENS WITH WHICH I'M
GOING TO TAKE MY DISCUSSION
TODAY BECAUSE THAT'S MOST OF
WHAT I'M FAMILIAR WITH THE
REWARD AND RECOGNITION SYSTEM.
TO SITUATE THIS TALK WHAT I WANT
TO DO IS TO FIRST ACKNOWLEDGE
THE EXISTING TENSION BETWEEN THE
REWARD AND THE SYSTEM AND
ACADEMIA RIGHT NOW THAT PUSHES
INSTITUTIONS TOWARDS PRESTIGE
AND RECOGNITION THINK UNIVERSITY
RANKING AND IMPACT FACTOR AND
HOW THAT'S INTENTION OR CAUSES
ATTENTION WITH PROMOTING
RESEARCH RIGOR AND INSTILLING
TRUST AND CONFIDENCE IN FINDINGS
OF OUR RESEARCH STUDY.
AND THAT BEGS THE NEXT QUESTION
HOW WE THEN SHIFT WHAT IS
VISIBLE IN ACADEMIA TO PROMOTE
RESEARCH RIGOR AND HOW DO
MOTIVATIONS INFLUENCE THIS AND
WE WERE TOLD TO REFLECT ON OUR
PERSONAL EXPERIENCES.
DORA IS A NON-PROFIT
INTERNATIONAL INITIATIVE TO
IMPROVE THE WAY RESEARCH IS
ASSESSED FOR DECISIONS THAT
AFFECT ACADEMIC CAREERS AND ALSO
GRANTS EVALUATION.
IN DOING THIS WE WORK WITH A LOT
OF ACADEMIC STAKEHOLDERS.
RESEARCH INSTITUTIONS AND
RESEARCH FUNDERS AND RESEARCHERS
THEMSELVES, PUBLISHERS AND SO
TODAY I'LL BE SHARING SOME
LEARNINGS I'VE PICKED UP FROM
INTERACTING WITH ALL THE
DIFFERENT STAKEHOLDER GROUPS.
SO FIRST OFF I THINK WHAT MAKES
ASSESSMENT PERFORM DIFFICULT TO
SOLVE AND BRINGING THIS BACK TO
MOTIVATIONS, THE SYSTEM'S
CHALLENGE THAT INVOLVE MULTIPLE
STAKEHOLDERS.
MANY OF THEM ARE THE ONES THAT I
JUST LISTED.
BUT THE TRICKY PART HERE IS THAT
EACH STAKEHOLDER HAS DIFFERENT
MOTIVATIONS AND SOMETIMES
THEY'RE CONFLICTING WHICH IS WHY
YOU'RE NOT GOING TO SEE SINGLE
INTERVENTIONS REALLY BE
EFFECTIVE FORM OF CULTURE CHANGE
BECAUSE INSTEAD IT RELIES ON
CHANGING UNDERLYING CULTURE AND
CONDITIONS AND INFRASTRUCTURE
AND REALLY REQUIRES A
COLLABORATIVE APPROACH.
SO I'M NOT GOING TO DWELL ON
THIS POINT BECAUSE IT DOES TIE
INTO MOTIVATIONS THOUGH NOT
DIRECTLY BUT WHEN TRYING TO
UNDERSTAND MOTIVATIONS IT'S ALSO
REALLY IMPORTANT TO EXAMINE THE
POWER DYNAMICS.
WHO HAS POWER WITHIN THE SYSTEM
AND WHO DOESN'T HAVE POWER
WITHIN THE SYSTEM AND HOW MIGHT
THAT BE PREVENTING CHANGE FROM
GOING FORWARD.
SO THIS IS ONE OF THE FIRST
STEPS WE TRY TO TAKE WHEN
THINKING ABOUT CREATING CHANGE.
SO NOW MORE TOWARDS THE MEAT OF
THE TALK WHICH IS WHAT INSPIRES
ORGANIZATIONS TO ACT ON
ASSESSMENT REFORM?
SO DORA STARTED OUT AS A
DECLARATION IN 2013 FOR
ORGANIZATIONS AND INDIVIDUALS
CONSIGN TO COMMIT TO PROMOTING
RESPONSIBLE RESEARCH ASSESSMENT.
I WAS HIRED IN 2017 AND THIS IS
A TIPPING POINT IN DORA WHERE WE
WENT FROM A DECLARATION TO
PROMOTING RESPONSIBLE RESEARCH
ASSESSMENT AND THAT WAS DONE
AFTER THE DECLARATION GATHERED A
LOT OF ATTENTION AND FAMOUS FOR
CALLING OUT A TOOL FOR RESEARCH
ASSESSMENT AND THERE WERE
SIGNATURES IT WASN'T LEADING TO
THE WIDESPREAD CHANGE THE
AUTHORS HOPED TO SEE.
THIS SWITCH WAS IN PART TO START
TO CHIP AWAY AT BUILDING THE
CULTURE OF RESPONSIVE RESEARCH
ASSESSMENT.
SO THE THREE BIG THINGS THAT I
LEARNED SINCE BEING WITH DORA
SINCE 2017 WAS THE IMPORTANCE OF
FACILITATING KNOWLEDGE SHARING
WITHIN THE COMMUNITY.
SO DIFFERENT WAYS WE DO THIS IS
PRIMARILY THROUGH THE WEBSITE
WHICH ACTS AS A HUB OF
INFORMATION.
WE HAVE OUR RESOURCE LABORATORY
AND HAVE A COLLECTION OF CASE
STUDIES I'LL DIVE INTO AND OUR
CASE STUDY REPOSITORY IS A
COLLABORATION WITH THE
UNIVERSITY ASSOCIATION AND SPARK
EUROPE AND WHEN WE CREATED THIS
THE MAIN IDEA WAS WE KNEW GOOD
PRACTICE EXISTED.
WE DIDN'T KNOW HOW IT CAME TO
BE.
SO THIS WAS REALLY TO
INTERROGATE KEY ASPECTS OF
INSTITUTIONAL CHANGE.
ONE OF THOSE BEING MOTIVATIONS
SO WHAT IS CAUSING INSTITUTIONS
TO ACT ON RESPONSIBLE RESEARCH
ASSESSMENT?
I'LL WALK THROUGH A COUPLE OF
EXAMPLES FROM THE MOTIVATION
SECTION.
SO EMBL OVER IN EUROPE, ONE
THING THAT PROMPTED THE RESEARCH
ASSESSMENT REFORM HAPPENING AT
THEIR INSTITUTION WAS A GROWING
RECOGNITION TO FULFILL THEIR
MISSION THEY NEEDED TO FOCUS ON
TRAINING AND LOOKING AT THAT IN
ASSESSMENT.
THEY RECOGNIZED THAT THEIR
MISSION WAS OUT OF ALIGNMENT
WITH HOW THEY WERE DOING
RESEARCH ASSESSMENT AND WANTING
TO PUT THAT BACK INTO THERE.
THE SECOND EXAMPLE I WANT TO
SHOW YOU FROM A UNIVERSITY IN
BELGIUM AND HERE WHAT THEY
NOTICED WAS THEIR MOTIVATION WAS
RECOGNITION OF THIS AND HOW THAT
WAS LIMITING THEIR ABILITY TO
LOOK AT NEW OPPORTUNITIES FOR
INTERDISCIPLINARY RESEARCH.
THIS LED TO A WIRED DEBATE ABOUT
THE EVALUATION OF THE RESEARCH.
THESE A FEW EXAMPLES AND
LESSONS LEARNED ARE MOTIVATIONS
COMING FROM PLACES WHERE
ACKNOWLEDGING SOMETHING'S NOT
QUITE RIGHT WITH THE SYSM AND
ALSO RECOGNIZING THAT WE WANT TO
REWARD SOMETHING NEW IN SOME
CASES A LOT OF IT IS AROUND OPEN
SCIENCE AND ALSO ABOUT HOW CAN
WE BRING PARITY FOR EVALUATING
TEACHING AND MENTORSHIP ALONG
WITH RESEARCH AND INNOVATION.
SO THESE ARE SOME OF THE
INSTITUTIONAL MOTIVATIONS WE'VE
SEEN AND NOW THE SECOND THING WE
WORKED ON SO FACILITATING
KNOWLEDGE SHARING BUT THEN
FINDING COMMON INTERESTS.
SO BRINGING GROUPS TOGETHER AND
REALLY PROMOTING RESEARCH
ASSESSMENT REFORM LIKE RIGOR IS
A WIDE TOPIC AND ARE THERE BITS
AND PIECES PEOPLE CAN AGREE ON
AND WORK TOGETHER.
AN EXAMPLE OF WHAT WE'VE DONE
WITH THIS AT DORA IS REALLY
USING THESE COMMON INTERESTS TO
INSPIRE ACTION.
SO ONE OF OUR COMMUNITIES OF
PRACTICE IS A GROUP OF RESEARCH
FUNDERS FROM AROUND THE WORLD
AND THEY MEET EVERY QUARTER TO
DISCUSS REFORMS THEY'VE BEEN
DOING AT THE ORGANIZATIONS AND
WHAT'S BEEN WORKING AND WHAT
HASN'T AND THROUGH THESE
DISCUSSIONS WE CAME TO HEAR THAT
THE APPROACH OF ASSESSMENT KEPT
COMING UP IN THE CONVERSATION.
THIS WAS A COMMUNITY-LED EFFORT
TO EXAMINE THE NARRATIVE AND WE
WERE ABLE TO MOTIVATE CHANGE AND
INSPIRE ACTION.
THE LAST THING IS BUILDING AND
SUPPORTING COMMUNITIES OF
PRACTICE.
FACILITATING KNOWLEDGE SHARING
AND ALL FIT INTO HOW CAN WE
SUPPORT COMMUNITIES OF PRACTICE.
ANOTHER WAY WE'RE TRYING TO DO
THIS IS THROUGH OUR COMMUNITY
ENGAGEMENT GRANT.
THESE ARE MICROGRANTS GIVEN TO
MEMBERS OF THE COMMUNITY TO SPUR
THEM TO DEVELOP WHAT WORKS FOR
THEM FOR RESPONSIBLE ASSESSMENT
PRACTICES AND SHARE THAT BACK
OUT WITH THE REST OF THE DORA
COMMUNITY.
AND THAT'S A LITTLE BIT OF WHAT
WE'VE BEEN DOING.
A LITTLE BIT ABOUT WHAT WE'VE
LEARNED FOR MOTIVATIONS.
THANK YOU AND I LOOK FORWARD TO
QUESTIONS.
>> THANK YOU, DR. HATCH.
THAT WAS OUTSTANDING.
OUR NEXT SPEAKER AND LAST
SPEAKER FOR THIS SESSION BEFORE
WE TAKE A BREAK AND THEN GO INTO
OUR PLENARY SESSION WHERE WE'RE
HOPING ALL OF YOU WILL ASK
CONCISIVE AND CHALLENGING AND
THOUGHT-PROVOKING QUESTIONS WILL
BE DR. MALCOLM MACLEOD.
WELCOME.
>> THANK YOU.
VERY MUCH.
I FEEL THIS MUST BE LIKE WHAT
FRANK SINATRA FELT LIKE WHEN HE
CAME OUT OF RETIREMENT.
IT'S GOOD TO BE BACK.
THANK YOU FOR HAVING ME.
SO MY BACKGROUND I'M A CLINICAL
NEUROLOGIST AND INTERESTED WITH
WALTER KOROSHETZ AND HOW WE
DEVELOP DRUGS FOR STROKE AND
ATTEMPT TO UNDERSTAND WHY IT
FAILED SO MISERABLY AND I'M THE
ACADEMIC LEAD FOR RESEARCH
INTEGRITY AT THE UNIVERSITY OF
EDINBURGH AND I SHOULD SAY
ANYTHING HOW I SAY ABOUT
INSTITUTIONS BEHAVE HAS NOT
NECESSARILY BEEN DRAWN FROM MY
OWN PERSONAL EXPERIENCE OF MY
OWN INSTITUTION AND MY SLIDES
ARE AVAILABLE IN THE OPEN
SCIENCE FRAMEWORK.
SO WHAT MIGHT BE DRIVING I THINK
WE ALL KNOW THERE'S A PROBLEM
BUT WHAT MIGHT BE DRIVING THE
PROBLEM?
FIRST I THINK THERE'S FOUR
CATEGORIES OF THINGS WE MIGHT
WORRY ABOUT.
FIRST A CLAIM MAY BE MADE ON A
TYPE 1 STATISTICAL ERROR AND
THERE'S A 5% CHANCE SWEET THE
FALSE POSITIVE AND WE MAY HAVE
MADE A CLAIM MADE UNDER WHICH
WAS VALID UNDER THE PARTICULAR
CIRCUMSTANCE UNDER WHICH IT WAS
TESTED BUT NOT OBSERVED IN THE
CIRCUMSTANCES AND THE
OBSERVATIONS MAY HAVE BEEN DUE
TO PEOPLE DOING WONKY STUDIES
AND MAY NOT HAVE BLIND IT AND
MAY HAVE EMERGED THE BIAS OR
PEOPLE HYPOTHESIZING AFTER THE
RESULTS ARE KNOWN.
THESE ARE QUESTIONABLE RESEARCH
PRACTICES BUT I THINK THEY HAVE
VARYING DEGREES OF RESEARCH AND
CULPABILITY AND MAY FOURTH THE
RESEARCH CHAI HAVE BEEN MADE UP
OR FALSIFIED AND WE HAVE
HISTORICALLY FOCUSSED ON THIS
AND THOUGHT HOW DO WE SORT THE
BAD GUYS BUT I DON'T THINK THOSE
ARE WHERE THE MAIN BENEFITS ARE
TO BE HAD BECAUSE THERE'S
INTEGRITY OF THE CLAIM AND ISSAY
QUESTION OF RESEARCHER INTEG
INTEGRITY.
SOME OBSERVATIONS TO PROGRESS
CHANGE WITHIN AN ORGANIZATION
THERE'S A HUGE AMOUNT OF
HETEROGENEITY AND I USED TO LOOK
AT THE WELCOME TRUST AT NIH
THEY'RE LIKE THAT BUT THEY'RE
NOT BECAUSE THEY'RE FILLED OF
INDIVIDUALS WITH A SLIGHTLY
DIFFERENT TAKES AND SOME TAKE
MAY BE EVERYTHING'S DANDY OR WE
CAN ALWAYS DO BETTER BUT THERE'S
NO GREAT PROBLEM.
THERE'S A HUGE PROBLEM BUT WE'RE
FIXING IT, TRUST US.
WE HAVE NO IDEA HOW TO FIX IT.
SOME PEOPLE MIGHT ARGUE WELL
MAYBE YOU SAY WE'VE GOT A
PROBLEM BUT OUR STAKEHOLDERS ARE
HAPPY AND THE GRANT INCOME'S
COMING IN, WHY SHOULD WE ROCK
THE BOAT OR KILL THE GOOSE
LAYING THE GOLDEN EGG OR IF YOU
THINK THINGS ARE BAD FROM WHERE
YOUR SITTING YOU SHOULD SEE IT
FROM THE INSIDE TRACK AND THAT'S
WHY I GET FRIGHTENED AND WE NEED
TO THI HOW TO ADDRESS THESE
BECAUSE WE HAVE A PERSUADING JOB
TO DO ABOUT THE CHANGE.
I WANT TO INTRODUCE YOU TO THE
ITALIAN COMMUNIST ACTIVE IN
MILLAN IN THE 1920s AND HE WAS
ARGUING HOW TO INTRODUCE
COMMUNISM AND THE INEVITABLE
COLLAPSE OF CAPITALISM AND WE'RE
TRYING DO MUCH THE SAME HERE
AREN'T WE AND TALKED ABOUT A WAR
OF POSITION BY ANTONIO GRAMSCI
AND THERE'S BASIS OF
SELF-ORGANIZED SIGNIFICANT
NETWORKS WITH A CLEAR VISION OF
CHANGE AND WAR OF MANEUVER IS
THE STRUGGLE OF FORCE, SLUGGING
IT OUT IN COMMITTEES AND FUNDING
PANELS AND ON THE BARRICADES AND
IT'S BETTER THAT IT NEVER COMES
TO THE USE OF FORCE BUT TO AVOID
THIS THE THREAT OF FORCE AND
COERCION NEEDS TO BE REAL AND
TANGIBLE AND FELT BY THOSE WHO
MIGHT SEEK TO OPPOSE CHANGE.
WE'RE LOOKING AT THE SCALE OF
THE PROBLEM AND POTENTIAL
CONSEQUENCES OF DOING SOMETHING
AND POTENTIAL BENEFITS OF DOING
SOMETHING.
AND THOSE ARE THE ARGUMENTS
WE'RE HAVING AND ESSENTIALLY
IT'S A RISK MANAGEMENT STRATEGY
FOR THE UNIVERSITY AND YOU NEED
TO DECIDE WHETHER YOU'RE
COMFORTABLE WITH THE RISK OR
SHOULD SEEK CHANGE.
SECOND OBJECTIVE, REASSURE
EVERYBODY ABOUT EVERYTHING ALL
THE TIME.
DON'T FRIGHTEN ANY HORSES.
NOTHING IS GOING TO HAPPEN HERE.
IT'S EVOLUTION NOT REVOLUTION.
WE'RE NOT TRYING TO RIP THE
HEART OUT OF OUR RESEARCH ECO
SYSTEMS AND BUILD THEM UP FROM
SCRATCH ACCORDING TO SOME
MANIFESTO.
IT'S A VALUES-LED PROCESS AND
THOSE VALUES ARE THE VALUES OF
OUR RESEARCH COMMUNITY.
WE'RE NOT IMPOSING SOMETHING
EXTERNALLY, WE'RE GIVING VOICE
TO WHAT THEY ASPIRE TO DO WITH
RESEARCH AND MANY HEARD OF THE
EXERCISE IN THE U.K. SOME IMPACT
CASE STUDIES WERE HIGHLY
EVALUATED, THANK YOU VERY MUCH
INDEED BUT THIS IS AN EXTERNAL
SET OF STANDARDS WITH LOCAL
DECISIONS ON THE PROCESS AND
OBJECTIVE AND HOW GRANULAR OUR
FOCUS WILL BE.
AND IN IMPORTANTLY, IF WE'RE
ABLE TO ARTICULATE AND DEVELOP
WAYS IN WHICH WE CAN INTRODUCE
AND IMPLEMENT INSTITUTIONAL
CHANGE WE HAVE MANY COLLEAGUES
ALSO INVOLVED IN EFFORTS TO
CHANGE STUCKSES WHO MIGHT WANT
TO JOIN WITH US IN ESTABLISHING
THOSE AND THIS IS ABOUT
IMPROVEMENT NOT ABOUT
ACCOUNTABILITY.
SEEK SUPPORT AND BUY-IN AND IF
YOU CAN GET BUY-IN MAKE THE TEST
SMALLER SO TRIVIAL NO ONE CAN
DISAGREE AND WE'RE TALKING ABOUT
RIGOR.
IF YOU THINK OF THE CHALLENGES
FOR INSTITUTIONS IN TERMS OF
EQUITY, DIVERSITY AND INCLUSION
IN TERMS OF THEIR RESEARCH
CULTURES MANY CHALLENGES THOSE
INDIVIDUALS FACE IN SEEKING TO
GET INSTITUTIONS TO CHANGE ARE
EXACTLY THE SAME AS OURS AND
IT'S IMPORTANT WE BUILD
ALLIANCES.
IF WE THINK THEN RESEARCHERS
MIGHT EXIST ALONG A SPECTRUM
WHAT I WOULD ARGUE IS WHAT
INSTITUTION HAVE SOUGHT TO DO
BEFORE IS IDENTIFY THOSE MOST
EGREGIOUS BEHAVIORS TO MARCH
INDIVIDUALS FROM THE PREMISES
FROM THEIR BELONGS PACKED IN A
CARDBOARD BOX PAUSING THEM TO
REMOVE THEIR CARD AND TELLING
THEM NEVER TO DARKEN OUR DOORS
AGAIN AND THE PROBLEM IS MY
INSTITUTION CHANGE EFFORTS
BECOME ASSOCIATED WITH BAD
THINGS AND WHEN THEY COME
KNOCKING ON THE DOOR IT'S NEVER
GOOD AND YOU WANT TO HIDE.
SO THINKING ABOUT THE
DISTRIBUTION OF BEHAVIORS,
THERE'S THAT APPROACH OF
ELIMINATING IT AND MAKES MORE
SENSE TO HAVE AN OBJECTIVE TO
HAVE EVERYONE GET A TINY A
LITTLE BIT BETTER LIKE IT WILL
DO MORE GOOD IF EVERYONE GETS 1%
BETTER THAN ELIMINATING THE MOST
10% OR 5% EGREGIOUS PRACTICES,
SMALL CHANGES AND MARGINAL GAINS
WITH THE CONFIDENCE OVER TIME
THE AGGREGATE EFFECT OF SEVERAL
MARGINAL GAINS WILL BE A BIGGER
CHANGE IN WHAT WE'RE DOING.
SO THIS IS ABOUT RESEARCH
IMPROVEMENT WHERE WE'RE FOCUSSED
ON SYSTEMS AND NOT INDIVIDUALS.
WHERE WHEN WE FIND FALLIBILITY
WE RECOGNIZE IT AND WE DON'T
HAVE THIS PERFECTION TOAV
POTENTIAL NOBEL PRIZE FINDERS
AND SOMETIMES HOW THINGS ARE
CHARACTERIZED AND WE RECOGNIZE
TEAM WORK AND IMPORTANTLY WHEN
WE IDENTIFY SHORT COMINGS WE SEE
THEM AS OPPORTUNITIES FOR
LEARNING AS AN INSTITUTION AND
OPPORTUNITIES TO IMPROVE
SYSTEMS, AS OPPORTUNITIES FOR
IMPROVEMENT NOT AS SHORT
COMINGS.
NOW, I WANT TO MOVE ON A LITTLE
BIT BECAUSE I'VE BEEN A BULLY
PULPIT AND HERE'S THE ISSUE.
WE HAVE INTERVENTION WE THINK
WILL IMPROVE THINGS.
IN MY ROLE AT EDINBURGH I'M
DESPERATE FOR INTERVENTIONS TO
IMPROVE THINGS BUT I CAN'T JUST
SAY LET'S DO THIS BECAUSE WE
NEED TO KNOW IF IT WORKS.
IF WE HAD A SYMPTOMOLOGY OF
KNOWING IF IT WORKED WOULD BE
GREAT.
HERE'S AN EXAMPLE THIS IS AN
AUDIT AND ESSENTIALLY THE BLUE
LINES YOU DON'T NEED THIS
PASSAGE AND I CAN'T REMEMBER THE
INTRAOCULAR IMPACT TEST.
THIS IS THE IMPACT OF ENFORCED
COMPLIANCE ON PUBLISHING THE
LIFE SCIENCES.
WE DID A RANDOMIZED CONTROL
TRIAL AND ASKED PEOPLE TO
COMPLETE A CHECKLIST MAKES NO
DIFFERENCE TO THE PROPORTION OF
CHECKLISTS COMPLETED.
IT'S COUNTERINTUITIVE BUT IT
WOULD BE HUGE AMOUNT OF WORK.
PEOPLE WERE DOING THE WRONG
THING BECAUSE WE ASSUMED THEIR
INTERVENTION -- OVER THE LAST
TWO YEARS THEY'VE BEEN TRYING TO
ARTICULATE WHERE INSTITUTIONS
CAN ADOPT AND THE COST AND
BENEFIT AND THE CERTAINTY WE
HAVE IN THOSE.
HERE WE KNOW HIGH COST AND LOW
BENEFIT.
THIS INTERVENTION IS POTENTIALLY
OF HIGH BENEFITLY LOW -- BENEFIT
POTENTIALLY C WITH LOWT AND
NEED TO KNOW IF WE NEED TO DO A
RANDOMIZED CONTROL TRIAL AND FOR
A HIGH-COST, HIGH-BENEFIT,
LOW-CERTAINTY INTERVENTION YOU'D
WANT TO DO THAT.
HERE'S THE TWO APPROACHES.
THERE'S IMPLEMENT WITH AUDIT AND
THIS IS FROM A PAPER WHERE A
COMMUNITY HAD WHAT THEY
CONSIDERED TO REPRESENT BEST
PRACTICE AND MEASURED THEIR
PERFORMANCE AND TRIED TO SHIFT
AND MEASURE AGAIN AND GO AROUND
IN CYCLE INCORPORATING THINGS
THAT WORK IN THEIR STANDARD
PROCEDURES AND WHERE THINGS
DON'T WORK CHANGING INTERVENTION
SYSTEMS AND TRYING AGAIN AND
HERE'S THE ALTERNATIVE, HIGH
COST AND HIGH POTENTIAL BENEFIT
RANDOMIZED THROUGH A TRIAL.
THESE ARE TOOLS WE USE EVERY DAY
IN OUR WORKING LIFE BUT THE
AMOUNT OF RESEARCH PRACTITIONERS
WITH THE SKILLS IN TRIALS AND
RESEARCH IMPROVEMENT AND THE
FUNDING FOR THEM IS PITYFULLY
SMALL AND SPEND YEAR AFTER YEAR
THROWING GOOD MONEY AFTER BAD
INSTEAD OF PAUSING, REFLECTING
AND DESIGNING STUDIES THAT TELL
US WHAT WE WANT.
THE FINAL OBJECTIVE IS TO
EMPOWER INSTITUTIONS OF
KNOWLEDGE.
THAT'S CHALLENGING WHEN THERE
ISN'T VERY GOOD EVIDENCE AROUND.
THERE'S A RECENT CALL AND TWO
WERE IMPORTANT IN THE SPACE AND
RESEARCH AND INNOVATION AND WHAT
EACH WOULD DO IS ESTABLISH A
MENU OF INTERVENTIONS IN THEIR
SPACE, GIVE A DESCRIPTION OF THE
INTERVENTION AND PURPOSE AND
IMPLEMENTATION AND EVIDENCE AND
COSTS AND THE FIRST IS LED BY
FOLKS LIKE ME WHO TAKE THE
BACKSTAGE AND THE TEXT OF THE
SUBMITTED APPLICATIONS IS
AVAILABLE ON THE OPEN SCIENCE
FRAMEWORK BEFORE ANY FUNDING
DECISION IS MADE BECAUSE WE'RE
UNLIKELY TO GET FUNDED 25%
CHANCE AND IF WE DON'T GET
FUNDED, WE'D LIKE SOMEONE ELSE
TO TAKE UP OUR GOOD IDEAS.
THE FINAL THING IS THE
GUARANTORS MAY LOOK AT THEIR
PRACTICES FOR DATA MANAGEMENT
AND HANDLING.
THANK YOU VERY MUCH.
I APPRECIATE IT AND ENJOYED THE
HISTORY LESSON.
>> GREAT TALK GUYS.
NOW IT'S TIME FOR GROUP
QUESTIONS.
WE WANT TO THINK ABOUT THE
INCENTIVES.
WE WANT TO THINK OF THE
PERSPECTIVES OF PARTNERSHIPS AND
HOW DIFFERENT ORGANIZATIONS CAN
MAKE A DIFFERENCE.
THIS IS MEANT TO BE A DIALOGUE.
WE HAVE A RELATIVELY SMALL GROUP
OF PEOPLE.
IT'S INTENTIONAL.
THERE ARE MICROPHONE.
WHEN YOU GET TO THE MICROPHONE
PLEASE STATE YOUR NAME AND THE
INSTITUTION YOU'RE AFFILIATED
WITH AND THEN ASK YOUR QUESTION.
TRY UNLIKE ME TO BE BRIEF AND
YET CONCISIVE.
I ASK OUR PANELISTS TO TRY TO BE
BRIEF IN THEIR ANSWERS.
THIS IS MEANT TO BE A FULL
EXCHANGE AND YOU IN THE AUDIENCE
CAN ARGUE WITH ME AND THE PANEL
AND FUN.
I'M REMINDED.
THE COMMENCEMENT SPEECH CALLED
WHAT IS WATER BY DAVID FOSTER
WALLACE AND IT'S A METAPHOR FOR
IF YOU'RE A FISH AND LIVE IN THE
WATER YOU DON'T KNOW WHAT IT IS
BECAUSE IT'S ALL YOU EXPERIENCE.
EACH OUR SPEAKER HAVE BEEN
CHOSEN IN PART BECAUSE LIVE IN
DIFFERENT BODIES OF WATER.
BRING THAT FORWARD.
WHAT DID THEY MISS?
WHAT DID THEY ASSUME THAT'S NOT
REALLY IN EVIDENCE AND WHAT
STATEMENTS DID THEY MAKE IF YOU
LIKE THE LATIN PHRASE FROM LAW.
ROUGHLY TRANSLATES TO WHAT I
MYSELF SAID IT IT'S TRUE BECAUSE
I SAID IT'S TRUE.
WHAT DID YOU HEAR TODAY THAT
SAYS I'M NOT VIEW IT'S ACTUALLY
TRUE.
COME TO THE MICROPHONE AND STATE
YOUR NAME.
>> I'M WITH QUALITY CONTROL AND
TRYING TO DEVELOP STANDARDS FOR
RIGOR IN NEURO IMAGING.
AND WE TALKED ABOUT
INSTITUTIONAL BUY-IN.
I THINK THERE IS AN ASSUMPTION
THAT INSTITUTIONS ARE INTERESTED
IN BUYING IN IT AND FROM MY
EXPERIENCE IN TRYING TO TALK TO
HIGHER UPS AND SO ESPECIALLY
MEDICAL SCHOOLS I DON'T THINK
THAT HAS CHANGED IN BUY-IN.
THERE ARE THINGS ON WEBSITES AND
SYSTEM REMAINS THE SAME AND HIGH
IMPACT PAPERS AND I WAS
WONDERING IS THAT ASSUMPTION
TESTED OR DO WE MAKE IT TRUE?
>> YOU NEED TO BE MORE OPERATIC.
>> I'M HESITANT IN MY RESPONSE
BECAUSE I'M NOT SURE ABOUT THE
RIGOR IN AND THE EXAMPLES FROM
THE CASE STUDIES I SHOWED YOU
WERE FROM ORGANIZATIONS THAT
WERE ACTIVELY INTERESTED IN
RESEARCH ASSESSMENT REFORM AND I
AGREE WITH YOU THAT IS A MAJOR
CHALLENGE IS.
AND THERE ARE TENSIONS I
MENTIONED AT THE BEGINNING OF
THE TALK DRIVING REWARDS AND
RECOGNITION TOWARDS PRESTIGE
SUCH AS WORLD RANKING AND I
THINK ONE BIT OF AGAIN WHY I'M
SLIGHTLY NERVOUS ABOUT THE
RIGOR, WHEN WE LOOK AT ORGANIZES
THAT SIGNED DORA THERE ARE
HUNDREDS OF UNIVERSITIES IN
EUROPE THAT HAVE DONE SO AND
THERE ARE FIVE ACADEMIC
INSTITUTIONS IN THE U.S. THAT
HAVE.
SO I THINK IT IS A FAIR
ASSUMPTION YOU'RE BRINGING UP.
I DON'T KNOW IF IT'S AS MUCH
INTEREST BUT CERTAINLY WE'RE NOT
SEEING AS MUCH INSTITUTIONS IN
THE UNITED STATES SIGNING DORA
NOT TO MEAN THEY'RE NOT
INSTITUTIONS WORKING ON
ASSESSMENT REFORM WE'RE SEEING
IN EUROPE.
>> I WOULD CHALLENGE THE
QUESTIONER AND SAY IS IT REALLY
IN DOUBT THAT ANY INDIVIDUAL
INSTITUTION HAS MORE THAN ONE
INTEREST THAT SOME AREN'T IN
FINANCIAL PROFIT BUT SOME IN
PRESTIGE AND EGO ENHANCEMENT AND
SOME ARE CERTAINLY FOR ACADEMIC
INSTITUTIONS IN RIGOR.
THE QUESTION IS ARE THEY
SUFFICIENTLY EFFECTIVE AND THEN
IF IT'S NOT SO MUCH INTERESTED
OR NOT AND WE HAVE IPSO FACTO
PRIMA FACIE EVIDENCE AND WHAT IS
YOUR QUESTION TO THAT?
>> THE WAY THEY EVALUATE FACULTY
IS DOWN TO GRANT MONEY --
>> WHAT'S YOUR EVIDENCE FOR
THAT, SIR?
>> I'M A NEW P.I. WHO JOINED A
YEAR AGO AND THE TRAINING AND
ORIENTATION OF FACULTY THEY TELL
YOU THIS IS WHAT YOU NEED TO
TARGET TO KEEP YOUR JOB AND PAY
YOUR SALARY AND PROGRESS.
I THINK THAT'S MOSTLY TRUE IN
MOST MEDICAL SCHOOLS.
>> IS THERE AN AN ACADEMIC
INTEGRITY STATEMENT IN YOUR
UNIVERSITY THAT SAYS IN ORDER TO
KEEP YOUR JOB YOU HAVE TO NOT
PLAGIARIZE, FALSIFY OR
FABRICATE.
>> YES.
>> SO THEY ALSO SAY THAT?
>> YES.
>> OKAY.
>> MY POINT IS THE WORK WE ARE
PURSUING TO ACHIEVE IMPROVING
RIGOR AND REPRODUCIBILITY IS
NOT -- BECAUSE THE KIND OF
GRANTS WE WRITE TO ACHIEVE THAT
ARE NOT BEING FUNDED.
IF WE DON'T GET DOLLARS I CAN'T
PAY MY SALARY AND THEN I'M OUT.
>> GOOD, THANK YOU, SIR.
>> I'M ALEXANDRA NELSON AND I'M
INVOLVED IN TRAINING IN
EXPERIMENTAL DESIGN FOR
NEUROSCIENCE Ph.D. STUDENTS.
I WAS REALLY INTRIGUED AND
EXCITED ABOUT THE COMMENT THAT
DR. MACLEOD MADE ABOUT MAKING
SMALL CHANGES AS A WAY TO GET
MORE BUY-IN AND I'M WONDERING IF
YOU CAN GIVE A COUPLE EXAMPLES
OF SOME OF THOSE SMALL CHANGES
WE MIGHT THINK ABOUT FOR OUR
INSTITUTIONS.
>> LET ME GIVE YOU ONE EXAMPLE
OF WHAT IS SMALL AND EFFECTIVE
AND I'M A NEUROOLOGIST AND IN
LOU GHERIG'S DISEASE IMAGINE WE
GET A TREATMENT THAT REDUCES
MORTALITY BY 10% IS ABOUT ONE
PATIENT A MONTH.
THAT IS BASED ON FOUR TREATMENTS
IN SERIES.
IT'S A GENEROUS ESTIMATE.
EVERY MONTH THAT A PUBLICATION'S
DELAYED IS ONE PERSON DYING OF
ALS THEY SHOULDN'T HAVE DIED.
WE AUDITED THE INTERVAL BETWEEN
US APPROVING IRB APPROVAL TO AN
ANIMAL STUDY AND BEING
PUBLISHED.
AND THE MEDIAN DELAY WAS 1,000
DAYS.
OF SHOWS 250 WERE BETWEEN THE
WORK BEING SUBMITTED FOR
PUBLICATION AND ACTUALLY BEING
PUBLISHED.
SO WE'VE TAKEN A FEW THAT
ACTUALLY IF OUR RESEARCHERS
SUBMITTED THEIR WORK TO BIO
ARCHIVE AT THE SAME TIME THEY
SUBMITTED IT TO THE JOURNALS, IT
WOULD BE THERE AND CITABLE AND
THERE FOR REUSE BY OTHER
RESEARCH MUCH QUICKER.
SO WE'RE WORKING WITH OUR
COMMUNITIES TO IDENTIFY THEIR
ATTITUDES TOWARDS BIO ARCHIVING
AND CONTINUING TO AUDIT THE
PERCEIVE THE PORTION THAT GO OUT
TO BIO ARCHIVE AND THAT'S A
PLAUSIBLE WAY TO LOOK AT BIO
ARCHIVES THAT MAY DO HARM AND
THAT'S A NICE PROJECT THAT IS
PORTABLE BECAUSE ANYONE IN THE
ROOM CAN DO THAT.
>> I WANT TO SEE IF OTHERS AGREE
WITH THIS IDEA.
IS IT BETTER TO EFFECT CHANGE OR
MORE LIKELY TO BE SUCCESSFUL TO
EFFECT CHANGE BY GETTING MANY
PARTIES IN MANY PROJECTS TO HAVE
A SMALL IMPROVEMENT OR WOULD WE
BE BETTER OFF HAVING A SMALLER
NUMBER OF PARTIES ON A SMALLER
NUMBER OF PROJECTS HAVE SOME
REMARKABLY LARGE NOTABLE,
PALPABLE IMPROVEMENT?
DOES SOMEBODY WANT TO TAKE THAT
ON?
I'LL CALL ON SOMEBODY.
>> WE DIDN'T HEAR ABOUT MOM WHO
IS ANOTHER WAY OF MOTIVATING AT
LEAST IN MY FAMILY AT WHAT
HAPPENED AND NIH IS MOM AND CAN
LAY DOWN THE LAW AND WONDER HOW
THAT AFFECTS WHETHER SOMETHING
SPREAD OR NOT?
ANY QUESTION IS ABOUT THE
TENSION BETWEEN TOP-DOWN VERSUS
BOTTOM-UP APPROACHES.
AND IT'S ALMOST A QUESTION OF
DEFINITIONS, WHEN IS RESEARCH
RIGOROUS?
IF WE CAN'T DEFINE IT HOW DO WE
KNOW IF WE'VE ACHIEVED IT.
I FEEL LIKE THAT'S THE PROBLEM
IN GETTING THE IDEA TO SPREAD
AND BUYING GOOGLE GLASS YOU
BOUGHT IT OR DIDN'T AND IT'S
CLEAR WHETHER THAT ACTION HAS
BEEN PERFORMED.
I'VE BEEN SURPRISED FOR GOOD OR
FOR WORSE ABOUT DATA STANDARDS
WITHIN NIH AND THEY HAVE THE
POWER TO PUT DOWN THE LAW BUT
THAT MAY NOT BE THE POWER YOU
WANT TO PUT DOWN IN ALL
CIRCUMSTANCES BUT IT'S A POWER
THAT SHOULDN'T BE IGNORED THAT
HAS POTENTIAL.
TO TRY TO SUMMARIZE MY MOM AND
MIKE HAVE DIFFERENT IDEAS THAN I
DO WHETHER MY ROOM IS CLEAN OR
NOT BUT IT WAS CLEAR THAT I
NEEDED TO REACT TO THAT AND DO
SOMETHING.
MY QUESTION IS HOW DO WE USE THE
COMBINATION OF TOP-DOWN AND
BOTTOM-UP APPROACHES TO FIGURE
OUT WHAT CHANGES AT THE
INCREMENTAL LEVEL ARE ENOUGH OR
NOT ENOUGH AND LETTING
COMMUNITIES DEFINE THOSE FOR
THEMSELVES AND MAKING SURE MOM
IS IN THE BACKGROUND WHEN
SOMEBODY'S NOT PLAYING NICELY.
TO HEAR THOUGHTS ON THAT AND
IT'S RELATED TO THAT.
>> ANNA, PLEASE.
>> WHAT WE'VE SEEN DOCUMENTED IN
THE CASE STUDIES IS IT'S KIND OF
A LITTLE BIT OF BOTH AT ACADEMIC
INSTITUTIONS WHEN WE'RE LOOKING
AT ONES THAT IMPLEMENTED CHANGES
THAT THERE'S AN ELEMENT OF
TOP-DOWN AND ELEMENT OF
BOTTOM-UP AND WORKING TOGETHER
IN DIFFERENT WAYS.
TYPICALLY CHANGE IS FROM THE
BOTTOM-UP AND PHYSICIAN WHO'S
FIND SOMEONE IN A PLACE OF POWER
OR DEAN OF RESEARCH THEY'RE ABLE
TO CONVINCE OR ALSO INTERESTED
IN THE SAME TOPIC.
WHEN WE'RE SEEING CHANGES FROM
THE TOP-DOWN THEN WE'RE SEEING
THEN WE'RE SEEING WORK BOTTOM-UP
BUY-IN CONVENING PEOPLE TO HAVE
DISCUSSIONS MUCH LIKE TODAY TO
BUILD CHAMPIONS IN THE
WORKFORCE.
>> I THINK IF ANYONE WAS SEEKING
TO ARTICULATE THE TOP-DOWN WOULD
BE THERE FOR A LONG TIME AND
MAKE A VERY GOOD JOB OF IT SO WE
SHOULD LEAVE THE DECISIONS ON
WHAT REPRESENTATIVES QUALITY TO
THE USERS, THE PRACTITIONERS OF
THAT RESEARCH.
WHAT I THINK THE TOP-DOWN
APPROACH COULD BE TO INSIST
INSTITUTIONS HAVE AND
DEMONSTRATE THE APPROPRIATE
FUNCTIONING OF SYSTEMS TO
IMPROVE AND MONITOR THE RESEARCH
SO IT'S QUALITY BY DESIGN.
DON'T SHOW US YOUR OUTPUT BUT
SHOW US YOU HAVE A PROCESS BY
WHICH YOU SEEK TO IDENTIFY WHAT
YOUR STANDARDS ARE AND WORK
TOWARDS ACHIEVING THOSE
STANDARDS BECAUSE MY BELIEF IS
ANY INSTITUTION WHO HAS THAT
WILL NOT BE DOING VERY MUCH
WRONG IF IT'S DOING IT PROPERLY.
LET'S GO TO THIS MICROPHONE.
>> MY NAME IS NICOLE NELSON I'M
A PROFESSOR AT UNIVERSITY
WISCONSIN-MADISON AND ONE AREA
OF RESEARCH IS HOW PEOPLE LOOK
TO COME TO PROBLEMS AS
REPRODUCIBILITY RIGOR PROBLEMS
AND THEN ACT ON THEM.
FOR APRIL I ENJOYED IN YOUR TALK
HEARING HOW MANY CAME TO THE
TRAINING SESSIONS WITH AN
INITIAL SELF-INTERESTED
MOTIVATION THEY WANTED TO FIX
ONE OF THEIR OWN PROBLEMS AND I
THINK IT'S IMPORTANT TO
HIGHLIGHT BECAUSE THE INCENTIVES
NARRATIVE SEEMS TO PRESUME THE
SYSTEM AS-IS IS WORKING WELL FOR
MOST AND THE DON'T WANT TO
CHANGE BUT SEEMS THERE'S A CHUNK
OF PEOPLE OR INSTANCES WHERE
IT'S NOT WORKING WELL AND
THERE'S INTEREST IN CHANGE.
WHAT I SEE IS THOSE GOING
THROUGH TRAINING IS TIED WITH
RESEARCH ETHICS AND OFFERED WITH
RESEARCH ETHICS TRAINING MODULES
ALSO REQUIRED WHICH GIVES A
SENSE IT'S NOT SOMETHING THAT
BENEFITS THEM BUT THE SCOLDY
THINGS YOU'RE DOING WRONG.
SO WHAT I'M INTERESTED IN
HEARING YOU REFLECT ON IS HOW
YOU MANAGED TO MAKE A CULTURE
WHERE SELF-INTEREST CAN CARRY
THE DAY AND WHERE THE NATURAL
INCENTIVE PEOPLE HAVE TO SAVE
THEMSELVES WASTED TIME IS ABLE
TO MOTIVATE THEM TO THEN ADOPT
THESE PRACTICES.
>> I THINK -- WE'RE ALL VERY
BUSY, RIGHT.
I THINK THE PEOPLE THAT COME TO
A TRAINING THAT'S CALLED LIKE
HOW TO IMPROVE THE
REPRODUCIBILITY OF YOUR WORK
THERE'S A LOT OF SELF-SELECTION
THAT HAPPENS THERE.
THOSE PEOPLE ARE INTERESTED IN
SOME WAY ALMOST LIKE A HOBBY IN
LIKE TAKING THAT ON AS PART OF
THEIR IDENTITY BUT FOR THE REST
OF RESEARCHERS THEY DO WANT
THEIR RESEARCH TO BE MORE
RIGOROUS.
THEY HAVE EXPERIENCES THAT HAVE
CAUSED THEM TO LOSE TIME TO BE
DEMOTIVATING TO FRUSTRATE THEM
AND THEY SEE THIS OPPORTUNITY TO
LEARN A LITTLE BIT MORE AND WHEN
YOU GIVE THEM PRACTICAL THINGS
TO START WITH IT SORT OF IS
ALMOST A PSYCHOLOGICAL SHIFT
WHERE REPRODUCIBILITY IS NO
LONGER THIS BIG THEORETICAL
THING THEY CAN'T DO
INDIVIDUALLY.
TAY HAVE THINGS THEY CAN START
DOING AND OUR PHILOSOPHY HAS
BEEN LIKE TO KEEP THINGS
POSITIVE TO FOCUS ON THE
ADVANTAGES AND BENEFITS AND ALL
THE REST OF IT IS POSSIBLE AFTER
THAT.
ONCE SOMEONE HAS STARTED THAT
LEARNING JOURNEY AND START TO
DIVE A LITTLE BIT DEEPER AND SEE
BENEFITS TO ADOPTING LIGHT WAIT
YOU CAN HAVE CONVERSATIONS
CONVERSATIONS LIKE A CARPENTRIES
WORKSHOP THEY DIDN'T KNOW ABOUT
BEFORE THEY CAME TO ARPA-H AND
THEY SEE WHY THEY SHOULD BE
AUTOMATING THEIR ANALYSIS AND
DOCUMENTING THEIR COMPUTATIONAL
STUFF AND NOW THEY KNOW HOW.
THEN THEY WILL DEDICATE THE
THREE DAYS WHEREAS THEY WOULDN'T
HAVE BEFORE.
WE NEED TO HAVE ALL THE
DIFFERENT MESSAGES AND NEED THE
STICKS AND CARROTS BUT WHEN YOU
HAVE YOUR FIRST CONVERSATION
WITH A RESEARCHER FOCUSSING ON
BENEFITS FIRST GETS TO A POINT
TO HAVING A BIGGER CONVERSATION
HOW THIS BENEFITS MORE PEOPLE
AND RESEARCH IN GENERAL.
>> KEEP YOUR HAND RAISED IF YOU
WORK AT A UNIVERSITY OR ACADEMIC
INSTITUTION.
RAISE YOUR HAND IF YOU WORK AT
THE NIH OR A GOVERNMENT RESEARCH
INSTITUTION.
THOSE TWO COMBINED IS ALMOST
EVERYBODY IN THE ROOM.
SO WHEN WE'RE TALKING ABOUT A
CONVERSATION AND COMMENT WE
HEARD, IT'S ABOUT A PARTICULAR
TYPE OF INDIVIDUAL AND A
PARTICULAR TYPE OF WORK SETTING.
THAT'S THE INDIVIDUAL ACADEMIC
CHOOSING HIS OR HER PROJECTS
WORKING LARGELY ON HIS OR HER
OWN WITH MINIMAL REGULATORY
OVERSIGHT OF THEIR PRODUCTS.
THAT'S ONE PART AND A VERY
IMPORTANT PART, THE PART I LIVE
IN, THE RESEARCH EDEAVOR BUT
IT'S ONLY ONE PART.
-- ENDEAVOR.
IT'S ONLY ONE PART.
I DON'T KNOW THE NUMBERS EXACTLY
BUT THE CNS BUDGET OF PFIZER
PHARMACEUTICALS PROBABLY RIVALS
THE NINDS' BUDGET AND INCENTIVES
ARE DIFFERENT AND PEOPLE PERFORM
DIFFERENTLY IN THEM AND THOSE
BECOME AN ILLUSTRATION WE MAY BE
ONLY SEEING THINGS FROM ONE
POINT OF VIEW AND WE TALK ABOUT
THE INCENTIVES.
HOW IS IT PHARMA OR ANOTHER
GROUP IS ABLE TO MOTIVATE PEOPLE
TO WORK VERY HARD AND VERY
RAPIDLY AND VERY RIGOROUSLY TO
SOLVE PROBLEMS LIKE PRODUCE
VACCINES?
SO I THINK WE NEED TO THINK
THERE ARE MULTIPLE MODELS OUT
THERE AND THERE'S A WHOLE BODY
OF RESEARCH THAT HAS VARYING
DEGREES OF TRUSTWORTHINESS AND
TRUST FOR DIFFERENT THINGS OTHER
THAN WHAT WE'RE TALKING ABOUT
AND I THINK DR. MACLEOD WANTS TO
COMMENT.
>> AND THERE'S BEEN SOME WHO
WORKED WITH US ON
REPRODUCIBILITY ISSUES AND TALK
ABOUT PHARMA AND RIGOROUS
RESEARCH.
THOMAS.
>> THOMAS, COME TO THE
MICROPHONE.
>> I'M THOMAS STIGLER FROM
JOHNSON AND IF YOU HAVE
UNRELIABLE DATA YOU COME TO
UNRELIABLE DECISIONS AND THAT OF
COURSE INVALIDATES PROJECTS YOU
SPENT MILLIONS AND MILLIONS ON
COMPOUNDS THAT DROP OUT IN THE
CLINIC EVENTUALLY BECAUSE OF
WRONG DECISIONS BASED ON SHADY
DATA.
FOR PHARMA IT'S VERY IMPORTANT
TO HAVE RELIABLE DATA YOU CAN
BASE DECISIONS ON.
WE HAVE COMBINED TOP-DOWN AND
BOTTOM-UP APPROACH.
WE HAVE SENIOR MANAGEMENT BUY-IN
AND HAVE BUILT A COMMUNITY OF
CHAMPIONS THAT WOULD AND WE HAVE
INTERNAL AUDIT TO SEE WHAT
PEOPLE DO AND TOOK US A LOT OF
TIME TO CONVINCE PEOPLE THIS IS
NOT A POLICE.
JUST WE WANT TO DETECT
WRONGDOING BUT HAVING
PARTNERSHIPS TO IMPROVE THE
CURRENT PRACTICES.
WE SEE LESS AND LESS ISSUES.
>> VERY GOOD.
THANK YOU, SIR.
>> I'VE BEEN MORE OF A
METASCIENTIST AND WE CAN USE THE
ACADEMIC MODEL FOR BETTER OR FOR
WORSE AND APRIL THIS MAY BE A
LITTLE BIT DIRECTED AT YOU AND
THERE'S EARLY CAREER RESEARCHERS
YOU CALLED OUT AND THEY'RE THE
ONES MOTIVATED MORE SO TO DO
RIGOROUS PRACTICE AND I'M MAKING
AN ASSERTION WITH NO EVIDENCE.
>> YOU'RE IN GOOD COMPANY.
>> THE MOM, THE LAB, IS THE ONE
THEY SIT IN.
I'M CURIOUS WITH REFLECTIONS IN
THIS GROUP WHICH IS HOW DOES
THAT GET EITHER COMPLIMENTED OR
RESISTED WHEN THE RESEARCHER
COMES BACK IN THE LAB
ENVIRONMENT THE INSTITUTION'S
PROBABLY FURTHER AWAY AND WE
HAVEN'T TALKED ABOUT THE MIDDLE
LAYER.
IT'S FUNDER INSTITUTION OR
RESEARCHER AT THE BOTTOM AND
THERE'S STRUCTURE IN THE
DIFFERENT SECTORS THAT PROBABLY
ARE MORE INFLUENCING FACTORS OF
THE NORMATIVE BEHAVIOR.
MAYBE WE CAN CHEW ON THAT.
>> APRIL?
>> HI, I COMPLETELY AGREE.
LABS RUN DIFFERENTLY AND SOME
MAY BE DOING THE SAME PROTOCOLS.
JUST A LITTLE BIT ABOUT WLAE WE
TALK ABOUT WE ASK SOMEONE TO RUN
WHAT THEY THINK THEY'RE LAB CAN
TALK ABOUT ADOPTING AN
ELECTRONIC LAB NOTEBOOK FOR
YEARS AND YEARS AND NO ONE DOES
IT.
SOMETIMES IT'S LIKE THE
CONVERSATIONS ARE THERE AND THEY
JUST NEED ONE PERSON IN THE LAB
TO PUSH IT INTO LIKE ACTIONS
THAT WE TRY AND GIVE THEM TINY
THINGS THAT LIKE WHAT MALCOLM
WAS SAYING UNLIKELY TO CAUSE A
LOT OF RESISTANCE.
THAT'S PART OF WHY WE DON'T HAVE
A LOT OF STUFF IN OUR CONTENT
THAT IS A BIG LIFT BECAUSE MY
PERSONAL EXPERIENCE IS THAT IT
DOESN'T REALLY MATTER.
IF YOUR P.I. IS NOT INTERESTED
IN ADOPTING THE METHOD IT'S NOT
HAPPENING.
THOSE ARE MY THOUGHTS.
IT'S A STICKY PROBLEM.
IT'S A HARD PROBLEM.
>> LET ME EMPHASIZE WHAT JASON
WAS SAYING ABOUT MOM AND I DON'T
KNOW IF EVERYBODY KNOWS BUT THE
THE T32 TRAINING REQUIREMENTS
WHICH HAVE HUGE IMPACT AND YOU
PROBABLY KNOW THIS THEY
RECOGNIZE THIS PROBLEM AND SAY
NOT ONLY SHOULD RIGOR AND
REPRODUCIBILITY BE TAUGHT TO ALL
TRAINEES UNDER THESE GRANTS
WHICH SUPPORT A HUGE PROPORTION
OF PRE-DOC AND POST-DOC
RESEARCHERS BUT YOU HAVE TO SHOW
IN YOUR APPLICATION AND AGAIN
WE'RE JUST STARTING -- THIS CAME
OUT IN MARCH OF 2020, YOU HAVE
TO SHOW THE FACULTY WHO ARE
GOING TO BE ROLE MODELLING THIS
IN YOUR LABS.
DON'T KNOW THE FACT THEY'RE
COMPLETELY PREPARED FOR THIS
REQUIREMENT AND IT MAKES GROUPS
LIKE MINE AND OTHERS MAKE OUR
JOBS EASIER BECAUSE WE CAN SAY
WE'RE PREPARING THE INSTITUTION
FOR THIS REQUIREMENT.
IT'S INTERESTING.
I T WAS REALLY
INTERESTING THE PRACTICE
REQUIREMENT AND AS I SAID, I
DON'T KNOW.
IT REMAINS TO BE SEEN TO WHAT
TENT IT'S WORDS ON PAPER AND
THEY'LL DO IT -- EXTENT IT'S
WORDS ON PAPER BECAUSE YOU'RE
RIGHT WE COULD BE RUNNING THEM
INTO A CONCRETE WALL WHERE WE
TRAIN AND THEY GO IN THE LAB AND
DON'T SEE THEM AND GET
UNEDUCATED OR UNTRAINED OR
CYNICAL.
>> IT'S AN IMPORTANT POINT AND
WE NEED TO RECOGNIZE THE
CANOPOLY OF THE POTENTIAL TOOLS
AVAILABLE TO US TO EFFECT CHANGE
AND NOT GET FIXATED ON ONE.
WHETHER THAT'S SOCIAL CHANGE OF
A CATALYZING MOMENT OF AN
EXEMPLAR.
THIS GOES BACK TO THE QUESTION
OF SHOULD WE HAVE MANY SMALL
CHANGES OR A FEW BIG ONES.
SOMETIMES BIG ONES CAN CATALYZE
IDEAS.
NO ONE COULD BREAK THE
FOUR-MINUTE MILE UNTIL ROGER
BANISTER BROKE THE FOUR-MINUTE
MILE AND THEN LOTS OF PEOPLE
BROKE THE FOUR-MINUTE MILE.
IS IT IMPORTANT TO HAVE THE ONE
CATALYZING MOMENT?
OTHERWISE IT'S A LAW.
VIRTUOUSO JAZZ CAME OUT BECAUSE
THEY TAXED AND SO LAWS PROMOTE
THINGS.
AND EDUCATION AND WHETHER IT'S
RULES OR THE LAWYERS SAY OR
VIRTUOSO JAZZ IS BORN OUT OF
RULES WE NEED TO LOOK AT THE
WHERE THE LEVERAGE POINTS.
>> I'M A PRACTICING LABORATORY
SCIENTISTS AND TEACH BIO
STATISTICS IN THE PROGRAM IN
BIOLOGY.
I APPRECIATED HEARING FROM STEVE
AND MALCOLM AND MAYBE OTHERS.
A RECOGNITION OF THE DIVERSITY
OF SCIENTIFIC PRACTICES AND
THERE'S MANY BIOLOGIES AND WHAT
KIND OF PROCEDURES THEY USE ARE
UNIQUE TO EACH DISCIPLINE AND
SUB DISCIPLINE THERE ISN'T GOING
TO BE A ONE-SIZE FITS ALL REFORM
AND THAT'S WHY IT'S SO IMPORTANT
THEY BUBBLE UP FROM WITHIN
COMMUNITIES AND NOT BE IMPOSED
ON THEM.
IN CONNECTION WITH THAT, I
WANTED TO TALK ABOUT THE
MOTIVATION TO GET THESE
COMMUNITIES TO TAKE THIS ON AND
I HAD THOUGHTS I WANTED TO SHARE
ABOUT THAT.
ONE HAD TO DO WITH WHAT WE HEARD
WITH REPUTATION WE WANT TO BE
VIEWED AS A FIELD THAT'S
EXCELLENT AND WHY DON'T WE STHOU
THAT BY DEVELOPING STANDARDS.
>> I HAVE THE GOOD FORTUNE AS
WELL AS THE SAME SIMULTANEOUSLY
OF WORKING IN THE FIELD OF
NUTRITION.
THAT'S ONE WHERE ACCORDING TO
THE CHARITABLE TRUST DATA TRUST
IN NUTRITION SCIENTISTS IS LOWER
THAN TRUST IN OTHER SCIENTISTS
AND LOWER THAN TRUST IN
NUTRITION PRACTITIONERS.
AND ABLE TO QUANTITATIVELY SCORE
THE TRUST OF DIFFERENT JOURNALS
AND INSTITUTIONS WOULD BE
USEFUL.
>> THE OTHER POINT IN MOTIVATION
IS IT'S DE-MOTIVATING TO A
COMMUNITY SO HAVE SOMEONE COME
AND SAY WE'RE GOING TO FIX YOU
AND WE'RE HAVING A CONVERSATION
HOW TO GET PEOPLE TO CHANGE YOUR
BEHAVIOR BECAUSE WE DECIDED IT'S
BAD AND WE'RE STRATEGIZING HOW
TO MANIPULATE YOU.
IT WOULD BE MORE MOTIVATING TO
HAVING THE RHETORIC OF ASSERTING
WHAT THEY TAKE TO BE THEIR
RIGOROUS GOOD PRACTICE.
IN THAT SELF-EXAMINATION THEY
MAY STUMBLE ON WE HAVE A PROBLEM
HERE AND IDENTIFY THINGS THAT
NEED TO BE CHANGED BUT IF THEY
IDENTIFY THINGS THAT NEED TO BE
CHANGED IT'S MORE MOTIVATING TO
SEEK OUT THE CHANGES.
I FEEL LIKE FRAMING IT WITHOUT
THE PRESUMPTION THERE'S A CHANGE
AND LATE THE CHANGE COME FROM
THE TEAM.
>> I VERY MUCH LIKE THAT AND ASK
PROFESSOR MACLEOD TO RESPOND
BRIEFLY AND THEN APRIL BRIEFLY
AND THEN I'LL TRY TO BE BRIEF.
>> I AGREE BUT I WOULDN'T USE
THE WORD PROBLEM BECAUSE PROBLEM
IMPLIES A NEGATIVE.
WHEN WE APPROACH COMMUNITIES
IT'S HOW DO YOU RECOGNIZE GOOD
RESEARCH IN YOUR AREA AND ARE
YOU AS GOOD AS AS YOU COULD EVER
EXPECT TO BE AND WHERE CAN YOU
GET CLOSER TO BEING PERFECT AND
WOULD TAKE PROBLEM OUT OUT.
>> I SEE THE COMMUNICATION FROM
THE SCIENCE REFORM MOVEMENT AND
I THINK IT GETS RESISTANCE
GOING.
SALT OF THE EARTH AS SOMEONE
SAID.
>> APRIL.
>> WHEN WE TALK ABOUT
REPRODUCIBILITY WE TALK ABOUT
WAYS OF CHANGING IT.
>> AND THERE'S A NARRATIVE IN
THE GOOD OLD DAYS WE NEW
SOMETHING AND THINGS ARE BAD NOW
AND WE'RE IN CRISIS AND WE MUST
CHANGE.
I WILL ASSERT WITHOUT HAVING
COMPLETE AND THOROUGH DATA
THOUGH I CAN BRING SOME DATA TO
BEAR ON THIS IS THE GOOD OLD
DAYS WEREN'T THAT GOOD.
I CAN SHOW YOU FROM MENDEL AND
PASTUR AND OTHERS OBFUSCATED
DATA, ETCETERA, BY MANY
INDICATORS, SCIENCE IS MORE
RIGOROUS AND REPRODUCIBILITY AND
TRANSPARENT TODAY THAN IT HAS
EVER BEEN BUT WE STILL HAVE A
LONG WAY TO GO.
AND THAT'S SAYING THINGS ARE
WORSE AND WE'RE IN CRISIS BUT
THINGS ARE BETTER THAN THEN BUT
COULD BE BETTER STILL.
>> I REPRESENT THE HEALTH
RESEARCH ALLIANCE A GROUP THAT
DR. ALISON MISSED WHEN WE WERE
DOING THE HAND RAISING SO YOU
CAN IMAGINE THE LEVERAGE PEOPLE
LIKE THE AMERICAN HEART SOCIETY
AND OTHER WEALTHIER
ORGANIZATIONS HAVE IN THE SPACE
AND THIS WASN'T MY QUESTION BUT
IT CAME OUT EARLIER.
WE HAVE THE LEVERAGE.
YOU ASK ON AN APPLICATION DO YOU
USE PRE PRINTS YOU CAN IMAGINE
THE APPLICANT GOING WHAT ARE YOU
TALKING ABOUT?
OR ASKING WHERE IS YOUR DATA
SHARED?
YOU DON'T WANT TO LEAVE THINGS
BLANK ON AN APPLICATION SO BY
FOLLOWING THE NIH AND THE NINDS
NOW THE RIGOR THE GUIDELINES,
NON-PROFIT FUNDERS CAN ALSO MOVE
BY NUDGING BEHAVIOR.
MY QUESTION IS LET'S GO BACK TO
DAMON'S TALK.
WE BELIEVE IN THE NON-POVERTY
FUNDER WORLD THAT COHORTS CAN
MOVE AND MAKE A DIFFERENCE
BECAUSE YOU CERTAINLY DON'T WANT
TO BE THE FIRST.
YOU WANT TO JOIN A GROUP BUT YOU
SURE AS HECK DON'T WANT TO BE
THE LAST BUT WHAT DOES THAT
COHORT HAVE TO LOOK LIKE?
I'D LOVE TO HEAR DAMON AND
EVERYBODY ELSE'S EXPERIENCE WITH
HOW COHESIVE THAT COHORT HAS TO
BE.
IF YOU'RE GOING TO MOVE SOMEBODY
IN THE DIRECTION, DO ALL HAVE TO
BE AT ZERO AND THEN OR IN THAT
NODE IN THAT COHORT BECAUSE I
THINK WE'RE TRYING TO GET RIGOR
WHICH IS NOT ONE PERSON BUT A
GROUP OF PEOPLE THAT CAN HAVE
THESE WIDE BRIDGES BUT HOW CAN
WE BEST --
>> OKAY.
LET'S GET DAMON UP TO ANSWER
YOUR QUESTION.
IS HE HERE?
>> USING THE COHORT MODEL DOES
IT HAVE TO BE HOMOGENOUS.
>> LET'S GET A RESPONSE.
DOES SOMEBODY WANT TO RESPOND?
>> DR. KAREN SULT A DEPUTY
DIRECTOR FROM THE LARGEST FUNDER
IN RESEARCH INNOVATION IN THE
U.K. AND MY POLICY IS AREA IS
FROM INTEGRITY TO OPEN DATA AND
SCIENCE AND RESEARCH AND
EQUALITY, DIVERSITY AND
INCLUSION AND MITIGATING
INTERNATIONAL RISK AND HAVE A
LOT TO SAY ABOUT RIGOR AS YOU
CAN IMAGINE AND TO GET TO THE
QUESTION, U.K. FUNDED FUTURE
LEADERS FELLOWSHIPS AND IT'S ONE
OF THE MORE BELALATENT EXAMPLESD
CREATING SETS OF SUPPORTS AND
WHAT DO THEY WANT TO DO THAT
WILL BENEFIT THE WIDER WORLD AND
ENABLING THEM TO DO THAT OPPOSED
TO YOU ARE NOW CARRYING THE
WEIGHT OF BEING A CHAMPION
THEREFORE GO OUT AND BE
BRILLIANT AND IF YOU'RE NOT
YOU'RE GOING TO SUCK.
NO ONE IS PUTTING THAT PRESSURE
ON THEM.
THERE'S A LOT OF DATA.
>> COHESIVE SELF-INTEREST AND
MOTIVATION AND GOALS ARE THE
MOST IMPORTANT THING ABOUT
CREATING THAT.
>> AND BEING VERY INTENTIONAL
AND STRATEGIC.
THERE'S A LOT OF QUESTIONS
AROUND I THINK COHORT CREATIONS
IN TERMS OF WHAT DISINCENTIVES
CAN YOU CREATE TO CREATE THEM IN
THE FIRST PLACE BUT HOW DO YOU
CUT AGAINST THAT AND THINK
CAREFULLY ABOUT THE CRITERIA AND
HOW YOU THINK ABOUT THE PANELS
OF HOW YOU SET THINGS UP AND WE
CAN TALK ABOUT THAT LATER.
I WANTED TO TALK TWO QUICK
THINGS AND ONE THING TO TALK
ABOUT IS SYSTEMS.
WE'VE BEEN SAYING THE WORD
SYSTEM BUT I DON'T THINK PEOPLE
HAVE BEEN TALKING ABOUT SYSTEMS
WE'RE JUST TALKING ABOUT THINGS
IN THE SYSTEM AND THE SECOND
PART IS SCALE.
THIS GOES BACK TO MALCOLM'S
POINT ABOUT SCALE.
ON THE SYSTEMS POINT I'VE NOT
ALWAYS BEEN WORKING FOR A FUNDER
AND BEING MORE OF A POLICY MAKER
SIDE AND STRATEGIST.
I CAME INTO THE ORGANIZATION AS
AN ACADEMIC AND RESEARCHER FROM
A LARGE RESEARCH TEAM BUT I'M A
SYSTEM'S ANALYST AND RESEARCHER
BY BACKGROUND.
I'M REALLY INTERESTED IN WHAT IS
THE COMPLEXITY OF UNDERSTANDING
A SYSTEM AND I THINK WE DO A
DISSERVICE AND DEFINITELY IN THE
U.K. AND WITHIN THE WIDER WORLD
OF TALKING ABOUT SYSTEMS BUT
WE'RE NOT TAKING SYSTEMS
ANALYSIS TO IT IN ANY REAL WAY.
WHETHER IT'S FROM ECOLOGIST OR
BIOLOGICAL SYSTEMS.
THE BIT ABOUT NETWORKS BUT THAT
TYPE OF WORK IS A VERY VERY
SPECIFIC AND YOU CAN SEE TALKING
ABOUT GOOGLE GLASS AND THINKING
ABOUT INFLUENCERS COMING OUT AND
PEOPLE AND THE TYPE OF CREATION,
IT HAS MODELS AGAINST THIS SORT
OF P.I. SOLE PERSON AND WHAT
THEY'RE DOING.
IT'S GOT SOME SIMILARITIES BUT
NOT THE SAME WHEN WE'RE ACTUALLY
TALKING ABOUT WHAT TRANS FORMS
IN YOUR FIELD OVER A PERIOD OF
TIME, RIGHT, SO THE RIGOR ISN'T
THE ONE SINGULAR MOMENT BUT A
PROCESS OF ADAPTATION AND HOW
DOES IT CHANGE OVER SCALE FROM
YOU WORKING ON AN INDIVIDUAL
PROJECT TO NOW WORKING ON
MULTIPLE PROJECTS
SIMULTANEOUSLY?
OR YOU'RE A PART OF 23
INSTITUTIONS AND NOW HAVE
DIFFERENT NATIONAL STANDARDS
YOU'VE GOT TO GRAPPLE WITH SAY
OVER THINGS LIKE DATA.
TO GIVE YOU A GREAT EXAMPLE WE
HAVE DATA SOVEREIGNTY ISSUES OR
CONCERNS RAISED BY INDIGENOUS
PEOPLE'S AROUND THE WORLD AND
THINK OF CHAMPIONING OPEN DATA
IT'S NOT EVEN PART OF THEIR
THINKING.
THEY'RE NOT THINKING WHAT IT
MEANS FOR A COLLECTIVE TO OWN
THE DATA.
NOT A COMMUNITY OR CITY BUT
THERE IS NO INDIVIDUAL PERSONAL
DATA NOW.
IT'S COLLECTIVELY OWNED.
FOR ME THE TYPE OF WORK I DO IS
WHERE SYSTEMS WORK IS
TRANSFORMED.
>> IS THAT A COMMENT OR
QUESTION?
>> A CHALLENGE ACTUALLY.
>> RESTATE THE CHALLENGE,
PLEASE.
>> ARE WE TALKING ABOUT SYSTEMS
REALLY THINKING ABOUT SYSTEMS
AND A POINT YOU RAISED, DAVID,
ABOUT THINKING ABOUT THE NARROW
SPLIT PLACE AND COMING UP WITH
INTERVENTIONS AND SOLUTIONS
MYOPIC OPPOSED TO THINKING ABOUT
THE LARGER STRUCTURE AND MY
OTHER POINT WAS SCALE.
ARE WE ACTUALLY THINKING ABOUT
SCALE?
I APPRECIATE MALCOLM'S POINT
ABOUT THINKING ABOUT SMALL AND
BIG BUT WE HAVE TO AGGREGATE ANT
WE'RE ALSONKING ABOUT THE
RIGOR INTEGRITY OF INSTITUTIONS
BECAUSE THEY HAVE TO DO THAT ALL
THE TIME IN DIFFERENT WAYS.
THERE'S THE FINANCIAL THINGS AND
VARIOUS PRESSURES.
HAVE YOU SOMEBODY LOOKING AT
YOUR BOOKS BUT IT'S NOT JUST
ABOUT RESEARCH INTEGRITY.
HOW DO WE BRING THESE IN THE
CONVERSATION MORE?
>> RESPONSES FROM THE AUDIENCE
AS WELL?
SOMEBODY WANT TO RESPOND TO
THAT?
MALCOLM AND LETS BE SUCCINCT.
>> WE NEED TO BE LOOKED AFTER
BUT I'D LIKE TO MAKE AN
DISTINCTION IN AUDIT FOR
IMPROVEMENT VERSUS
ACCOUNTABILITY.
SYSTEMS, ABSOLUTELY.
BUT MY INSTITUTION PRESUMABLY
HAS SYSTEMS FOR MAKING DECISIONS
BUT THAT'S ENTIRELY PRIVILEGED
INFORMATION ABOUT HOW YOU
ACTUALLY GET ANY CHANGE MADE AND
NOBODY EVER WRITES IT DOWN.
IF THEY WROTE IT DOWN AND
SOMEONE MIGHT READ IT AND WORK
OUT -- AND WE DID A RESEARCH
CULTURE SURVEY AND A KEY THEME
IS PEOPLE DIDN'T UNDERSTAND HOW
THE INSTITUTION REACHED
DECISIONS AND HAVING GREATER
TRANSPARENCY IN THOSE SYSTEMS
WILL GO A LONG WAY.
>> VERY GOOD.
>> I WANT TO ASK A QUESTION SO
WE'RE ALL I THINK COMMITTED TO
DATABASE THINKING HERE AND IT
DOESN'T MEAN WE ALWAYS HAVE DATA
OR ALWAYS DO IT, WHAT PERCENTAGE
OF PEOPLE, WHAT PERCENT OF
PEOPLE THAT DO SCIENCE ARE
TENURE TRACK AT UNIVERSITIES AND
INSTITUTIONS?
ANYBODY KNOW?
SCIENCE OVERALL.
I'LL LEAVE IT TO YOU TO DEFINE.
IF YOU LOOK AT Ph.D. SCIENTISTS
IT'S SUBSTANTIALLY UNDER 50%.
AND SOME MAY BE UNIVERSITIES BUT
NOT TENURE TRACK.
WHEN WE'RE THINKING ABOUT THE
INCENTIVES WE TALKED ABOUT AND
THE PARTNERSHIPS AND NEED TO
THINK BROADLY TO THE ELEMENTS OF
THE SYSTEM.
>> IT'S GOING TO SAVE TIME AND
TROUBLE.
>> AND LOOKING AT RIGOR IN THEIR
INSTITUTION.
IS THERE AN EQUIVALENT THIS IS
GOING TO BE GOOD FOR ME THIS
YEAR.
>> I WILL NOT SPEND MY TIME
TALKING WITH THE RESEARCH
INTEGRITY OFFICE WRINGING MY
HANDS THIS PERSON WHO MAY HAVE
SHOT THEMSELVES IN THE FOOT AND
MAY ENDED THEIR CAREER BUT I I
CARE ABOUT THE INTEGRITY OF
SCIENCE AND A WILL NOT LOSE MY
INVESTMENT ON THIS PERSON I'VE
HIRED WHO'S CAREER MAY BE OVER
OR MANAGE THE SCANDAL
POTENTIALLY OF THE MISFIRE.
SO THERE'S SO MUCH BENEFIT TO
HAVING RIGOR.
>> I CAN QUESTION IT BECAUSE
YOU'RE FOCUSSING ON BREACHES OF
RESEARCH INTEGRITY.
THE FOURTH CATEGORY THAT MALCOLM
THAT IS EGREGIOUS BREACHES.
RIGOR IS THE BOTTOM OF THE
ICEBERG AND SO NOBODY AT YOUR
INSTITUTION IS PROBABLY GOING TO
LOSE THEIR REPUTATION OVER DOING
SLOPPY WORK.
THAT DOESN'T REACH THAT LEVEL OF
PREOCCUPATION OR THAT LEVEL OF
IMMEDIATE MOTIVATION.
>> IT MAY REACH MORE THAN YOU
GIVE IT CREDIT FOR.
>> IF THE NEWSPAPERS DRIVE YOUR
FACULTY NAMES THROUGH THE MUD,
IF FUNDERS DON'T TRUST US, IT
DOES HAVE IMPLICATIONS BUT IT'S
NOT ONE TO ONE.
IT'S NOT A PERFECT THING OF IF
THIS THEN THIS HAPPENS.
>> IN MY OPINION I'M INCON
CONVENIENCE -- INCONVENIENCED TO
SAY WHAT DRIVES THE INSTITUTIONS
IS THE FUNDING AND WHEN YOU RUSH
TO STUDIES FOR PUBLICATION THAT
COULD HAVE BENEFIT FROM
ADDITIONAL CONTROLS OR A FEW
MORE SAMPLES, ETCETERA, THEN
THERE'S NO MOTIVATION ON THE
PART OF LEADERSHIP AND
INSTITUTIONS TO BRING ABOUT
CHANGE BECAUSE THEY WANT THE
MONEY TO COME IN AS LONG AS THE
MONEY'S COMING IN, THAT'S WHAT
THEY WILL DO.
THEY DEAL WITH PEOPLE AT THE TOP
AND THAT'S WHAT DRIVES IT.
I'VE SPOKEN TO RESEARCH PROVOSTS
AND THEY SAY WHY DO YOU WANT ME
TO CHANGE?
I JUST WANT TO LOOK AT THE
BOTTOM LINE.
>> I THINK THAT YOU'RE PARTIALLY
AGHT I THINK WE'D BE NAIVE
TO SAY OTHERWISE.
I DON'T THINK ANYBODY ELSE WOULD
SAY MONEY DOESN'T DRIVE THESE
BUT WE ALSO HAVEN'T PRESENTED
EVIDENCE THAT'S THE SOLE OR
OVERWHELMING DRIVER.
IF WE LOOK AT THE FIELD OF
BEHAVIORAL ECONOMICS AND BOOKS
LIKE DANIEL CONNAMON PEOPLE ARE
NOT PURELY RATIONALE BEINGS AND
RESPOND TO OTHER THINGS THEN
OTHERS AND THE JOAN OF ARCS THAT
WITH ALLOW THEMSELVES TO BE
BURNED AT THE STAKE NOT BECAUSE
THEY GOT PAID OFF.
I THINK IT'S REASONABLE TO SAY
THERE'S OTHER SYSTEMS.
I CAN GIVE YOU ANECDOTES OF
PEOPLE WHO TAKE TREMENDOUS PRIDE
IN T AIR WORK SAY I WOULD
NEVER PUT OUT ANYTHING I DON'T
BELIEVE IN AND WE HAVE TO LOOK
FOR DATA BECAUSE IT'S IMPORTANT
AS WE LOOK FOR CHANGE.
IS CHANGE GOING TO BE ABOUT
DRIVING CULTURES AND IDENTITY?
IF IT'S ALL ABOUT MONEY THEN ALL
THE COURSES WE OFFER DON'T
MATTER.
IF IT'S NOT ONLY ABOUT MATTER
THEN ALL THE THINGS STEVE IS
TALKING ABOUT GETTING LAB
LEADERS TO WORK WITH PEOPLE IN
THE LAB HAVE EXPERIENCES MATTER.
NIH SEEMS TO THINK EXPERIENCE
MATTERS BECAUSE THEY'RE ASKING
STEVE TO PUT IT IN.
I JUST WANT TO CHALLENGE THE
NARRATIVES AND KEEP US FOCUSES
ON BEING DATA SCIENTISTS.
WHERE'S THE DATA FOR THESE
THINGS.
I SEE FOUR PEOPLE.
WE'LL HAVE FOUR PEOPLE STATE
YOUR QUESTION OR COMMENT BRIEFLY
AND THEN WE'LL DECIDE WHERE WE
GO WITH IT.
>> I'M A PEDIATRIC NEUROLOGIST,
M.D., Ph.D. AND HAVE A RESEARCH
LAB AND STARTING TO DO CLINICAL
RESEARCH.
SO MY COMMENT IS THAT IT SEEMS
MORE INSIDIOUS.
THESE AREN'T THINGS THAT WILL
RISE TO THE LEVEL OF A DEAN.
THESE ARE THINGS THAT ARE
HAPPENING AT A LOW LEVEL ON
GRANT APPLICATIONS USING THE
WRONG STATISTICS.
IT'S NOT ABOUT THE INTEGRATE OF
THE PERSON IT'S JUST THAT
THEY'RE MAYBE MISEDUCATED ON
WHAT TO DO.
MY QUESTION WAS ABOUT
CLINICALTRIALS.gov.
THAT'S AN EXPERIMENT WE DID
COLLECTIVELY AND I'M WONDERING
WHAT THE RESULTS OF THAT
EXPERIMENT HAVE BEEN.
HAS IT IMPROVED THE
TRUSTWORTHINESS OF CLINIC
RESEARCH.
>> SUBSTANTIALLY.
THERE WAS A PAPER OUT AND I
THINK THE AUTHOR IS LAMBORINQU
OVE OR SOMETHING LIKE THAT IF
SOMEONE CAN HELP ME AND
SUBSTANTIAL UNDER THE OF
CLINICAL TRIALS REGISTERED AND
DECREASE IN PRIMARY ENTRIES
POST-DOC.
SO WE HAVEN'T HIT AUTOPIA YET.
>> I'M THE REPRODUCIBILITY
LIBRARIAN AT UNIVERSITY OF
FLORIDA AND WE LOOK TO PROMOTE
OPEN AND -- OPENNESS AND
THERE'S MORE TO RESEARCH RIGOR
AND OPEN SCIENCE AND
REPRODUCIBILITY THAN WHAT IS
JUST EASY TO DO AND THERE'S A
CONFLICT BETWEEN THE EARLY
CAREER RESEARCHERS WHO WANT TO
BE OPEN AND MOSTTIVATED TO DO
RIGOROUS SCIENCE AND WHAT
THEY'RE DOING WITH THEIR P.I.s.
I'D LIKE TO PUT THE CHALLENGE
OUT THERE IT'S A PROBLEM TO WORK
ON, HOW DO WE SUPPORT THOSE
INDIVIDUALS WHAT WANT TO DO
BETTER BUT CONSTRAINED BY THE
FACT THERE'S A PRESSURE IN
ACADEMIA.
WE'RE SEEING MORE PRECARITY IN
FUNDING AND WE NEED TO ADDRESS
THIS ALONGSIDE PRODUCING RIGOR
AND REPRODUCIBILITY MORE
BROADLY.
>> I'M TRAINED IN OBSTETRICS AND
CURRENTLY A METASCIENTIST.
WE HELD A VIRTUAL BRAIN STORMING
EVENT LAST FALL WHERE WE INVITED
BACK PARTICIPANTS WHO HAD TAKEN
BACK A MAJOR REPRODUCIBILITY
WORKSHOP A YEAR TO THREE YEARS
AGO TO TALK ABOUT WHAT HAPPENED
WHEN THEY WENT BACK TO THE LAB,
WHAT WENT WELL AND WHAT WERE
THEY PREPARED FOR AND WHAT DID
NOT GO WELL AND HOW DID THEY
NAVIGATE THAT PROCESS.
WE SYNTHESIZED THOSE DISCUSSIONS
INTO A PAPER AVAILABLE AS A PRE
PRINT AND HOPE ONE DAY WE'LL GET
PEER REVIEWS BACK BUT IN A VERY
SKILL AGNOSTIC WAY OF WALK
THROUGH THE OF HOW DO YOU DECIDE
WHAT TO IMPLEMENT FOR YOUR
PROJECT AND HOW DO YOU GO ABOUT
TALKING TO YOUR LAB TO DECIDE
WHO YOU CAN IMPLEMENT AND HOW DO
YOU GET THE RESOURCES AND
TROUBLE SHOOT AND ADAPT THE PLAN
AND SHARE YOUR SKILL AND
TRAINING OTHERS MOVING FORWARD
AND LOOK FOR FUTURE POSITIONS
ALIGNED WITH THE PRIORITIES
YOU'VE ESTABLISHED BASED ON YOUR
PAST WORK.
I WOULD ENCOURAGE THOSE OFFERING
EDUCATION TO MAKE SPACE, IF
YOU'RE NOT DOING APPLIED
APPLICATION TO TALK ABOUT WHAT
HAPPENS AFTERWARDS BECAUSE IF
YOU'RE APPLICATION IS NOT
APPLIED, YOU'RE LEAVING THAT ON
YOUR STUDENTS MOSTLY EARLY
CAREER RESEARCHERS TO FIGURE IT
OUT ONCE YOU ARE GONE AND
THEY'RE BACK IN THEIR LABS AND
IT'S A VERY COMPLICATED SPACE TO
NAVIGATE SO THAT MAY BE A
STARTING RESOURCE HELPFUL TO YOU
AND INCLUDES INFORMATION
DIRECTLY FROM WORKSHOP
PARTICIPANTS ABOUT THINGS THEY
WOULD HAVE LIKED TO SEE OR FOUND
HELPFUL.
THAT'S ALL.
>> THANK YOU.
>> MIKE DOHERTY, UNIVERSITY OF
MARYLAND I'M A DEPARTMENT CHAIR
AND HAD THE PLEASURE OF LEADING
CHANGE IN OUR DEPARTMENT WHERE
WE IMPLEMENTED THE ANNUAL
PROCESSES AND TENURE PROMOTION
PROCESSES IMPLEMENTED IN THE
LAST MONTH OR TWO.
THE DEPARTMENT OF PSYCHOLOGY
WHERE WE NOW INCENTIVIZE OPEN
SCIENCE AND OPEN METHODS AND
REPRODUCIBILITY AND SPECIFIC
ACTIONS PEOPLE CAN TAKE.
THE PRINCIPLE BEHIND THIS IS
IMPACT AS ACCESS.
REDEFINING IMPACT AS A WAY WE
CAN MEASURE IT INDEPENDENT OF
THINGS LIKE CITATION AND IMPACT
FACTORS AND THAT BUSINESS WE
KNOW ARE PROBLEMATIC.
ONE OF THE THINGS I WANTED TO
BRING UP THOUGH IS THAT MUCH OF
WHAT WE WERE ABLE TO ACCOMPLISH
WAS FRAMED FROM THE PERSPECTIVE
OF VALUES.
SO WHEN I ASKED MY FACULTY OR
ASKED MY YOUNG FACULTY IN
PARTICULAR YOUNG FACULTY SAYING
I WANT TO DO RESEARCH IN A
PARTICULAR WAY AND WANT TO DO IT
IN A WAY TRUE TO THEIR VALUES
AND THEN THEY GET IN THE SYSTEM
AND REALIZE HOLY CRAP I CAN'T DO
IT LIKE THAT.
THE SYSTEM DOESN'T SUPPORT IT.
WHEN WE TOOK THE PROBLEM ON IT
WAS FRAMED FROM THE PERSPECTIVE
OF WE WANT TO GIVE OUR YOUNG
FACULTY AGENCY TO DO THE WORK
THEY NEED TO DO AND DO IT HOW
THEY WANT TO IT SO THEY CAN DO
THE BEST QUALITY WORK.
THAT'S MORE OF A COMMENT NOT A
QUESTION BUT THE ISSUE IS COMING
BACK TO THE VALUES AND
EMPHASIZING THAT AS A WAY TO
GIVE THAT TO AN EARLY CAREER
RESEARCHER.
>> THAT'S A GOOD NOTE TO GO TO
LUNCHEON AND TO CHEW ON AND
ARGUE WITH EACH OTHER AGAIN IN
THE BEST SENSE OF ARGUE.
I THINK THE IDENTITY AS A
SCIENTIST IS A BEAUTIFUL
IDENTITY.
AND WHEN SOMEONE ASKS ME HOW DID
YOU CHOOSE TO BE BECOME A
SCIENTIST I'M SURE MANY OF YOU
GET THAT SCIENTIST TOO, I OFTEN
SAY I DIDN'T CHOOSE, THIS IS
WHAT I AM.
AND THAT MEANS A LOT TO ME.
AND I THINK ABOUT THAT HISTORY
OF GALILEO BECAUSE HE LIKED TO
ASK HARD QUESTIONS ABOUT
ARISTOTLE AND THE WORD MEANS
WRANGLE AND WRANGLE WITH DOES IT
NEED TO BE MORE RIGOROUS.
STEVE TALKED ABOUT ALL THESE "R"
WORDS AND TALKED ABOUT FOCUSSING
ON RIGOR.
WE HEARD ABOUT IDENTITY.
IS THAT IDENTITY REALLY ALWAYS
ABOUT BEING RIGOROUS OR IS PART
OF IT ABOUT BEING CLEAR ON
RIGOR.
IS IT OKAY TO SOMETIMES DO
CHEAP, QUICK, SLOPPY SCIENCE
BECAUSE WE WANT TO TRY SOMETHING
OUT QUICKLY OR SOMETIMES LIKE
WHEN WE PUT A DRUG TO THE FDA OR
PUT TIRES ON A REAL CAR THAT
WILL SAVE OR COST LIVES WE WANT
TO BE RIGOROUS.
MAYBE SOMETIMES WE WANT TO PLAY
BUT CAN WE BE OPEN.
THERE'S THE IDENTITY OF THE
SCIENTIST AS THE UNCOMPROMISING
TRUTH TELLER WHO SAY HERE I WAS
VERY RIGOROUS AND HERE I WAS
NOT.
I WAS JUST DOING QUICK ANALYSES
TO TAKE A LOOK AND HAVE SOME FUN
AND YOU CAN MAKE OF IT WHAT YOU
WILL.
AS LONG AS I'M TRANSPARENT THEN
I HAVE LATITUDE WITH RIGOR.
FOOD FOR THOUGHT AS YOU HAVE
SOME FOOD TO REPLACE THE
CHEMICALS THAT YOU HAVE ALREADY
BURNED UP IN YOUR BODIES THIS
MORNING.
THANK YOU.
HI, MY NAME IS JASON WILLIAMS
FROM COLD SPRING HARBOR
LABORATORY, NEW YORK.
MY BACKGROUND IS IN BIO
INFORMATICS AND HELPING BRING
TOOLS INTO RESEARCH AND
TRAINING.
I'M DELIGHTED TO KICK OFF
SESSION 3, A CLEAR PATH, NEEDS,
BEHAVIORS, KNOWLEDGE SKILLS AND
COMPETENCIES.
I GUESS THIS GETS INTO SOME
SPECIFICS BECAUSE IT'S WHAT DO
WE ACTUALLY WANT RESEARCHER
GROUPS TO DO AND BEHAVIORS TO
ADOPT AND HOW WE PRIORITIZE
RIGOR AND TRANSPARENCY AND WHAT
SUPPORTS THESE BEHAVIORS.
I'M GLAD TO BE JOINED BY A
FANTASTIC PANEL.
I BELIEVE IT TO THEMSELVES TO
INTRODUCE THEMSELVES BECAUSE
THEY'LL DO A FAR BETTER JOB BUT
WE HAVE A NUMBER OF SPEAKERS
INCLUDING ONE VIRTUAL AND WE
HAVE GUNDULA BOSCH AND CARI
JORDAN JOINING US AS THE DEPUTY
DWRISHTH FROM THE CARPENTRIES
AND WITH WEATHER AND TRAVEL, AND
RUSSELL POLDRACK FROM STANFORD
HAD MANY CHALLENGES TO MAKE IT
TODAY AND WE HAVE TRACEY
WEISBERGER FROM BERLIN.
I THINK IT WILL BE SIMILAR TO
BEFORE AND EVERYONE WILL GIVE A
BRIEF PRESENTATION TO START
STIMULATING IDEAS AND HAVE A
ROUND OF QUESTIONS AND
DISCUS
DISCUSSION.
WE'RE GOING TO DO THIS.
I'LL EXPLAIN THE WORLD CAFE
APPROACH TO THIS BECAUSE WE HAD
BREAKOUT AND DISCUSSIONS THAT
ARE GOING TO BE PART OF THAT SO
I THINK WE NEED TO GET THIS
PROJECTED TO THE SCREEN.
>> THANK YOU, JASON.
IT'S A PLEASURE TO TELL YOU A
BIT OF WHAT WE'VE BEEN UP TO AT
JOHNS HOPKINS SCHOOL OF PUBLIC
HEALTH.
MY NAME IS GUNDULA BOSCH AND THE
NAME ARE THE CENTER SAYS WHAT
WE'RE DOING.
WE'RE DEVOTED TO RIGOR,
REPRODUCIBILITY AND
REPRODUCIBILITY AND WANTED TO
GIVE A LITTLE INSIGHT IN OR WORK
TO TRY TO CATALYZE RIGOR WITH
OUR TRAINING.
WE'RE TRYING TO GET INSPIRED BY
OUR HEROES IN SCIENCE.
AND MANY HAVE SHOWN THEY'RE
BROADLY THINKING DISCIPLINARY
SCIENTISTS AND WE THINK WE
SHOULD GET INSPIRED THROUGH
THEIR EXAMPLE BY LOOKING OUTSIDE
THE BOX AND NOT BEING SO
SPECIALIZED IN HOW WE TRAIN OUR
STUDENTS BUT ALLOW OUR STUDENTS
TO REALLY HAVE AN IDEA OF WHAT
IS OUTSIDE THE NARROW FIELD AND
THINK BROADLY AND THINK ABOUT
THE PHILOSOPHY OF SCIENCE.
ESSENTIALLY WE HAVE STARTED ON I
THINK AROUND 2016 THINKING HOW
TO BRING THE P.H. BACK INTO
Ph.D. SOMEBODY ELSE COINED FOR
US AND LOOKED INTO HOW CAN WE
BRING MORE OF THE PHILOSOPHICAL
FOUNDATION AND THE CRITICAL
THINK OF SCIENTIFIC RIGOR INTO
HOW WE TRAIN GRADUATE STUDENTS
IN OUR CASE IN THE BIOSCIENCES.
IN THE MEANTIME WE HAVE EXPANDED
TO OTHER DISCIPLINES AS WELL.
THIS FULL SLIDE IS SOMETHING I
CANNOT TALK THROUGH ALL THE
DIFFERENT ASPECTS IN MY SHORT
TALK TODAY BUT WHAT WE FOUND ARE
COMPETENCIES AND SOME CAUSES
HAVE BEEN SIMILAR TO THE
COMPETENCIES.
WHAT YOU SEE OUT THERE ARE
APPLIED FORMS OF THE PHILOSOPHY
OF SCIENCE.
YOU FIND ELEMENTS OF KNOWLEDGE,
COMMUNICATION AND SOCIAL
JUSTICE.
THIS IS WHAT WE TRY TO BRING
INTO THE WAY WE TRAIN GRADUATE
STUDENTS.
AND I'M HAPPY TO ELABORATE MORE
IN THE BREAKS OVER THE NEXT TWO
DAYS BUT I WANTED TO SHOW YOU
BEFORE WE HAVE GROWN SINCE WE
STARTED EARLY CONCEPTUALIZING
AND THINKING ABOUT IT IN 2015
AND 2016.
A WORLDWIDE NETWORK OF PARTNER
INSTITUTIONS JOINING US IN THIS
ENDEAVOR OR HARDLY IMPLEMENTING
WHAT WE ARE DOING.
WE'RE HAPPY TO MAKE THIS NETWORK
OPEN TO THE COMMUNITY HERE AND
ALSO FIND WAYS TO SYNERGIZE.
WE HAVE IN THE MINE TIME SERVED
OVER 3,000 STUDENTS AT JOHNS
HOPKINS ACROSS 10 DIVISIONS AND
FOUR PROGRAMS.
IT DOESN'T MEAN WE HAVE HUGE
ENROLLMENTS WE HAVE SOME
REQUIRED BY BIG CLASSES.
MOST ARE SMALLER BUT ESSENTIALLY
WE HAVE COURSES THAT ARE
AVAILABLE TO ALL OF OUR STUDENTS
AT JOHNS HOPKINS AVAILABLE AS
ELECTIVES AND AS PART OF
REQUIRED PROGRAMS.
IT'S AVAILABLE TO EVERYBODY ALSO
MEANS EVERYBODY HERE IN THE ROOM
IF THEY WANT TO JOIN OUR
INSTITUTIONAL PARTNERSHIPS AND
WHAT IS MOST IMPORTANT FOE
PEOPLE IN THE ROOM IS WE HAVE
NOW THROUGH GENEROUS FUNDING
BOTH THROUGH OUR FOUNDATION AS
WELL AS NSF AND NIH A GROWING
COLLECTION OF OPEN EDUCATIONAL
RESOURCES WE ARE HAPPY TO MAKE
AVAILABLE TO EVERYBODY
INTERESTED.
BASICALLY OUR TRAINING LOOK AT
SCIENTIFIC RIGOR THROUGH
MULTIPLE LENSES.
YOU TALKED ABOUT THIS AT LUNCH
AND YOU BROUGHT UP SOMETHING
EARLIER THAT WE CANNOT JUST
TRAINING RIGOR AS WELL AS
REPRODUCIBILITY AND
RESPONSIBILITY THROUGH ONE
COURSE AND CHECK IT OFF.
WE TRY TO INCLUDE OUR TRAINING
IN SCIENTIFIC RIGOR IN MANY
COURSES AND WORKSHOPS AND
EDUCATIONAL EXPERIENCES.
THE SLIDE SUMS UP OUR
EDUCATIONAL PHILOSOPHY AND THE
THREE Rs WERE IMPLEMENTED IN
TRAINING EXPERIMENTS IN SOUND
TRAINING AND STATISTICS AND FORM
OF TRAINING IN LOGIC AND
RESEARCH DESIGN OF COURSE WE
TALKED ABOUT THIS EARLIER AS
WELL AS ERROR ANALYSIS.
IS SOMETHING WE ENGINEER ALL THE
TIME AND DON'T DO ENOUGH I
THINK.
THIS EXCERPT OF THE CURRICULUM
IS SOMETHING THAT Ph.D. STUDENTS
TAKE OUR CLASS REQUIRED AND
PARTNER PROGRAMS IMPLEMENT ALL
THE COMPONENTS.
THEY'RE AVAILABLE IN SEPARATE
SMALLER MODULES BUT CAN BE TAIL
MADE TO FIT INDIVIDUAL PROGRAMS
IN INDIVIDUAL CURRICULUM.
WE STARTED WITH SOMETHING WE
CALL THEORY OF HISTORY AND
PRACTICE IN SCIENCE AND TWO
COURSES THAT HAVE TO DO WITH
ERROR ANALYSIS.
THE SEQUENCE IS NOT REQUIRES.
IT ALLOWS THE STUDENTS TO GET A
ROUGH IDEA OF HOW THE PHILOSOPHY
OF SCIENCE IS IMPORTANT TO
SCIENTIFIC PRACTICE.
TALK ABOUT WHAT IS SCIENCE AND
HOW DO SCIENTIFIC METHODS
EVOLVE?
WHAT COUNTS AS EVIDENCE ACROSS
THE DISCIPLINES AND HOW DO WE
EVALUATE IT.
WHAT CAN WE LEARN FROM HISTORY
OF SCIENCE WHICH IS PARTICULARLY
POPULAR AS YOU CAN IMAGINE.
WHAT CAN WE LEARN ABOUT
SCIENTIFIC REVOLUTIONS.
TO GIVE YOU AN IDEA OF HOW THIS
LOOKS LIKE, I WANTED TO DRAW
YOUR ATTENTION TO A PUBLICLY
AVAILABLE LECTURE THAT ALL OF
YOU CAN WATCH.
THE PERSON YOU SEE ON THE LOWER
RIGHT-HAND SIDE IS THE CHAIR OF
OUR DEPARTMENT WHO IS ACTUALLY A
FOUNDER OF OUR PROGRAM.
HE GAVE A PUBLICLY AVAILABLE
GRAND ROUNDS LECTURE ABOUT THE
HISTORY OF THIS THERAPY AND THE
ROLE OF COVID-19 RESEARCH.
THAT'S A FASCINATE BEING LECTURE
WHERE HE LISTS ALL THE
PRINCIPLES I JUST TALKED TO YOU
ABOUT BEFORE.
I THOUGHT THIS WOULD BE
SOMETHING PARTICULARLY OUR
COLLEAGUES WOULD ENJOY BECAUSE
OF THE HIGHLIGHTS.
WE HAVE A COURSE WE CALL ANATOMY
OF SCIENTIFIC ERA.
WE BORROWED A PRINCIPLE OUR
ENGINEERING COLLEAGUES DO ALL
THE TIME AND LOOK INTO HOW
SCIENCE CAN GO WRONG AND HOW CAN
WE DO IT BETTER.
SO BASICALLY WHAT WE DO IS NOT
ONLY TEACH A COURSE THAT ALLOWS
US TO LOOK INTO ERROR ANALYSIS
BUT ALSO AND THE REASON I'M
HIGHLIGHTING A CLASSICAL CONDUCT
OF RESEARCH AND TRANSPLANTING
THIS INTO THE RESPONSIBLE
CONDUCT OF RESEARCH COURSE TO
HAVE AN ETHICS PERSPECTIVE TO
THE ERROR ANALYSIS AND THE
MULTI-FACTORIAL WAY OF LOOKING
AT RIGOR AND REPRODUCIBILITY AN
THIS FUNDED PROJECT THROUGH THE
NSF IS SOMETHING THAT MEANS
SOMETHING TO THOSE WHO DO THE
SCIENCE OF SCIENCE OR
METASCIENCE AND WE'RE TEACHING
MATERIAL AND A COUPLE MONTHS AGO
IN THE INSTITUTION AND WE USED
NEW MATERIAL TO TEACH OUR
STUDENTS WHAT CAN GO WRONG IN
SCIENCE AND HOW TO DO IT BETTER.
HOW CAN WE HAVE METHODS
IMPLEMENTED IN ORDER TO AVOID
AND REDUCE ERRORS.
WE ALSO LEAVE THE ROOM FOR THE
ROLE OF SERENDIPITY AND WE
CANNOT ONLY TEACH NEGATIVITY BUT
SOMETHING THE STUDENTS ENJOY.
I THINK I'M RUNG OUT OF TIME I
WANTED TO SHOW ANOTHER PROBLEM
SPEAKING OF RIGOR AND RESEARCH
REPRODUCIBILITY WE NOTICED OUR
STUDENTS MAKE A LOT OF MISTAKES
WHEN IT COMES TO PROBABILITY AND
STATISTICS IN THE LAB.
WE HAVE A DEAR COLLEAGUE AND
WORLD CLASS DEPARTMENT OF
STATISTICS AND OUR DEAR
COLLEAGUE TEACHES A WONDERFUL
COURSE IN STATISTICS FOR THE
PUBLIC HEALTH SCIENCES THE
PROBLEM IS OUR OWN STUDENTS
ACTUALLY HAVE AN ISSUE APPLYING
THE STATISTICS THAT IS TAUGHT
FOR THE PUBLIC HEALTH LEVEL TO
THE LAB AND MAKE A LOT OF
MISTAKES.
WE INTRODUCED ANOTHER
FUNDAMENTAL COURSE THAT HAS
EVOLVED MUCH LOWER THAN THE
STATISTICS COURSE THAT IS TAUGHT
BY OUR COLLEAGUES.
SO TO HAVE AN IDEA OF WHAT
BASICALLY IS THE SENSE MAKING
FROM A STATISTICAL DATA AND HOW
DO THEY APPLY TO THE LAB?
WE WORK WITH VISUALIZING AND GET
PEOPLE TO TALK ABOUT IT AND GET
MORE IN THE PUBLIC HEALTH LEVEL
OR SCALE OF BIOSTATISTICS AND
LOOKED AT RESEARCH THROUGH THE
LENS OF HOW CAN WE APPLY METHODS
AND RELATED STATISTICS IN THE
LEVEL OF CAUSATION RESEARCH.
THE ONLY THING I WANTED TO SHOW
YOU HERE AND I'M RUNNING OUT OF
TIME IS THAT CAUSATION HELP OUR
STUDENTS LOOK AT THIS ACROSS THE
AND THERE'S HOW THIS CAN BE
APPLIED THE BIOMEDICAL AND IT'S
A COLLABORATION ACROSS THE
DIFFERENT DIVISIONS IN OUR
SCHOOL HEARD THE HISTORY ASPECT
AGAIN AND WHAT LIMITATIONS TO BE
FACED WHEN WE TRIED TO ESTABLISH
CAUSALITY.
WE HAVE DIFFERENT VIEWS ON RIGOR
TRAINING THROUGH DIFFERENT
COURSES AT DIFFERENT EDUCATIONAL
EXPERIENCES THAT HELPED OUR
STUDENTS MAKE CONNECTIONS THAT
HAVE A MORE HOLISTIC VIEW OF
SCIENTIFIC RESEARCH TRAINING AND
VARIOUS ASPECTS.
I DON'T NEED TO GO TO THE NEXT
SLIDE BECAUSE WE'RE RUNNING OUT
OF TIME BUT IF ANYBODY WANTS TO
KNOW HOW WE EVALUATE THIS I CAN
TALK ABOUT THIS IN THE BREAKS.
THANK YOU VERY MUCH.
>> WE DON'T HAVE TO CLOSE THE
TABS BECAUSE KARI'S TALK IS
THERE.
>> LET ME PRACTICE, FORWARD,
BACK.
OKAY.
HI, I'M KARI JORDAN EXECUTIVE
DIRECTOR FOR THE CARPENTRY.
TODAY I'LL BE SHARING THE NINE
CORE VALUES OF THE CARPENTRY'S
ORGANIZATION AND DRAWING PAIR
LEVELS BETWEEN THEM AND THE
SKILLS AND BEHAVIORS THAT I
BELIEVE PROMOTE RESEARCH RIGOR.
FIRST I WOULD LIKE TO GIVE YOU A
BRIEF OVERVIEW OF THE CARPENTRY
IN CASE YOU HAD NOT HEARD OF OUR
PROJECT.
IT BUILDS GLOBAL CAPACITY AND
ESSENTIAL DATA AND COMPUTATIONAL
SKILLS FOR CONDUCTING EFFICIENT,
OPEN AND REPRODUCIBLE RESEARCH.
WE TRY TO FOSTER AN INCLUSIVE
AND DIVERSE COMMUNITY OF
INSTRUCTORS AND LEARNERS THAT
PROMOTES AND MODEL THE
IMPORTANCE OF SOFTWARE AND DATA
IN RESEARCH.
WE COLLABORATIVE DEVELOP
OPENINGLY AVAILABLE LESSONS AND
USE EVIDENCE-BASED TEACHING
APPROACHES AND WE WANT TO BE THE
INCLUSIVE COMMUNITY TEACHING
DATA AND CODING SKILLS.
WE HAVE A RICH HISTORY GOING
BACK TO 1998 WHEN SOFTWARE CON
PEN -- CARPENTRY.
DATA CARPENTRY WAS FOUNDED IN
2014 WITH THE MISSION OF
BUILDING COMMUNITIES TEACHING
UNIVERSAL DATA LITERACY AND IN
2014 LIBRARY CARPENTRY WAS
FOUNDED WITH THE MISSION OF
TEACHING DATA SKILLS TO THOSE
WORKING IN LIBRARY AND
SCIENCE-RELATED ROLES.
AND THEN PROJECTS MERGED TO FORM
A NEW PROJECT CALLED THE
CARPENTRY UNDER THE FISCAL
SPONSORSHIP OF COMMUNITY
INITIATES.
WE HAVE SOME INSTRUCTORS IN THE
ROOM IF YOU WANT TO WAVE YOUR
HAND.
I APPRECIATE YOU SO SO MUCH.
SINCE 2012 AND AS OF THE END OF
2021, WE'VE RUN ABOUT 3,000 WORK
SHOPS IN 64 COUNTRIES.
WE'VE TRAINED 3,419 VOLUNTEER
INSTRUCTORS TO DELIVER OUR 45
COLLABORATELY OPEN-SOURCED
LESSONS TO LEARNERS ACROSS THE
WORLD.
THE WAY OUR ORGANIZATION
SUSTAINS ITSELF THROUGH PEOPLE.
WE CREATE OPPORTUNITIES FOR
INDIVIDUAL TO CONTRIBUTE AND
GROW AND PROVIDE OPPORTUNITIES
FOR THEM TO TEACH, TO BUILD
COMMUNITY AND TO DEVELOP
LESSONS.
WE'RE ALWAYS LOOKING FOR WAYS TO
INCENTIVIZE THEIR CONTRIBUTIONS
AND ALSO AVERAGE RESOURCES
AVAILABLE TO GLOBAL COMMUNITIES
AND DISCIPLINE-SPECIFIC
COMMUNITIES.
I HOPE THIS OVERVIEW HAS GIVEN
YOU A CONCISE OVERVIEW OF THE
CARPENTRY AND WHO OUR AUDIENCE
IS.
WE BELIEVE ALL PEOPLE AT ALL
LEVELS ABLE TO SHARE WHAT THEY
KNOW, THE MORE EFFICIENTLY WE
CAN DISTRIBUTE KNOWLEDGE AS
LEARNERS PROGRESS THEY OFTEN
BECOME INVOLVED IN THE COMMUNITY
IN DIFFERENT WAYS.
TEACHING WORKSHOPS, DEVELOPING
LESSONS, BUILDING LOCAL
COMMUNITIES.
IN THE LONG-TERM, WE HOPE TO
CREATE A COMMUNITY THAT CAN
CREATE WIDE-SPREAD SUPPORT AND I
BELIEVE THAT TYPE OF MIND SET IS
IMPORTANT TO CHAMPIONING
RESEARCH RIGOR.
GOING A BIT DEEPER, I WOULD LIKE
TO SHARE WITH YOU THE NINE CORE
VALUES OF THE CARPENTRIES AS I
BELIEVE THEY DIRECTLY ALIGN WITH
THE REQUIRED BEHAVIORS,
KNOWLEDGE, SKILLS AND
COMPETENCIES FOR RESEARCH RIGOR.
THE CARPENTRY SERVES OUR
COMMUNITY AND MAKES EVERY
DECISION WITH THESE NINE COUR
VALUES IN MIND AND THEY ARE ACT
OP
OPENLY, EMPOWER ONE ANOTHER,
VALUE ALL CONTRIBUTIONS, ALWAYS
LEARNING, INCLUSIVE EVOLVE,
PEOPLE FIRST, ACCESS FOR ALL,
AND COMMUNITY COLLABORATION,
STRENGTH THROUGH DIVERSITY.
I'LL WALK YOU THROUGH EACH
VALUES AS THEY RELATE TO
BEHAVIORS I FEEL PROMOTE
RESEARCH RIGOR.
STARTING WITH MOTIVATION.
WHY ARE WE AT THIS MEETING AND
WHY DO YOU THINK IT'S IMPORTANT
TO CHAMPION RESEARCH RIGOR?
THESE TWO CARPENTRY'S VALUES
COME TO MIND.
ALWAYS LEARNING.
WE VALUE A GROWTH MIND SET IN
ALL THAT WE DO AND STRIVE FOR
CONTINUOUS IMPROVEMENT AND
EVOLVING OURSELVES AND
METHODOLOGIES AND BEING
RESPONSIVE AND CURIOUS AND
RECEPTIVE TO FEEDBACK AND EAGER
TO LEARN.
STRENGTH THROUGH DIVERSITY.
APPRECIATING IDENTITIES ARE
COMPLEX AND INDIVIDUAL.
WE BELIEVE IN EMPOWERING A
DIVERSE GROUP OF PEOPLE TO WORK
WITH DATA AND CODE AND ANSWER
THE QUESTIONS IMPORTANT TO THEM
TO ADDRESS CHALLENGES IN SCIENCE
AND SOCIETY.
WHAT ARE YOUR GOALS WHEN
PROMOTING RESEARCH RIGOR?
I'D LIKE TO POINT TO A FEW OF
THE CARPENTRIES CORE VALUES.
ACT OPENLY.
WE BELIEVE TRANSPARENCY, HONESTY
AND FAIRNESS ARE KEYS TO
FOSTERING TRUST WITHIN AN OPEN
COMMUNITY.
EMPOWER ONE ANOTHER.
WE HELP PEOPLE BUILD KNOWLEDGE
BY CREATE CONDUCIVE ENVIRONMENT
FOR THE EXCHANGE OF SKILLS,
PERSPECTIVES AND EXPERIENCES
THAT EMPOWER PEOPLE AND ENABLE
THEM TO REACH THEIR FULL
POTENTIAL.
INCLUSIVE OF ALL.
WE WELCOME AND EXTEND EMPATHY
AND KINDNESS TO LEVERAGE
CONTRIBUTIONS FROM ALL COMMUNITY
MEMBERS REGARDLESS OF IDENTITY
AND EXPRESSION AND THIS IS ONE
OF MY FAVORITE AND I LOVE ALL
NINE BUT PEOPLE FIRST IS HOW I
LIVE MY ROLES.
WE MAKE EXPRESSION THAT LOWER
BARRIERS FOR PARTICIPATION.
LASTLY, BEST PRACTICES.
THIS IS FUNNY BECAUSE DURING
LUNCH I LEARNED FROM JASON THAT
WE SHOULD CHANGE BEST PRACTICES,
WHAT DID YOU SAY WE SHOULD
CHANGE IT TO?
CONSENSUS PRACTICES BECAUSE
WHO'S CONSENSUS IS THIS?
IT DEPENDS.
SOME PRACTICES I BELIEVE WOULD
BE SOLID IDEALS FOR PROMOTING
RESEARCH RIGOR AND THESE ARE THE
LAST THREE CARPENTRIES FOUR
VALUES.
VALUE ALL CONTRIBUTIONS.
WE VALUE INPUT THROUGH CODE,
LESSONS AND THE BROADER ECO
SYSTEM AS LONG AS THEY ADHERE TO
OUR CODE OF CONDUCT OF COURSE
ACCESS FOR ALL.
WE VALUE ACCESSIBILITY AS CORE
AND CREATE MULTIPLE AVENUES FOR
PARTICIPATION WHERE ALL PEOPLE
CAN LEARN AND CONTRIBUTE.
AND LASTLY, COMMUNITY
COLLABORATION.
OUR CURRICULA AND PROGRAMS ARE
DEVELOPED BY AND WITH OUR
COMMUNITY MEMBERS.
WE SEE COLLABORATION ACROSS
BORDERS, DOMAINS AND INITIATIVES
AS A PATHWAY TO EMPOWER PEOPLE
AND REALIZE SHARED GOALS.
I'LL END WITH A FEW QUESTIONS
FOR TO YOU CONSIDER AS WE
CONTINUE ON WITH THE SESSION.
WHICH OF THE CARPENTRIES VALUES
AND STRENGTH THROUGH DIVERSITY
ALIGNS WITH YOUR PERSONAL
MOTIVATION OR CHAMPIONING
RESEARCH RIGOR?
WHAT DOES SUCCESS LOOK LIKE FOR
RESEARCH RIGOR CONSIDERING THESE
VALUES, ACT OPENLY, EMPOWER ONE
ANOTHER, INCLUSIVE OF ALL AND
PEOPLE FIRST.
AND LASTLY, WHAT DO THESE
VALUES, VALUE ALL CONTRIBUTIONS,
ACCESS TO FOR ALL AND COMMUNITY
COLLABORATION.
DO YOU FEEL EQUIPPED TO
IMMEDIATELY APPLY OR ADVOCATE
FOR AS A RESEARCH RIGOR CHAMPION
AND WHICH WOULD YOU NEED MORE
RESOURCES OR SUPPORT FOR?
THANK YOU SO MUCH AND I LOOK
FORWARD TO OUR DISCUSSION.
>> RUSS IS JOINING US VIA ZOOM.
>> SORRY I CAN'T BE THERE IN
PERSON.
>> I'M GOING FOCUS ON A FIELD
RATHER THAN OTHER WAYS OF
THINKING ABOUT OUR ORGANIZATIONS
AND HIGHLIGHT LESSONS WE'VE
LEARNED WITHIN THE FIELD OF
NEUROIMAGING ONE OF THE RESEARCH
AREAS I WORK IN AND I'LL START
BY HIGHLIGHTING THE FACT THAT
PART OF THE REASON THAT A LOT OF
PEOPLE CARE ABOUT RIGOR AND
REPRODUCIBILITY IN OUR FIELD IS
WE'VE SEEN A NUMBER OF
CONTROVERSIES OVER THE LAST
COUPLE DECADES.
ONE YOU MAY HAVE HEARD OF WAS A
WELL-KNOWN PAPER SHOWING IF YOU
TAKE COMMONLY USED PRACTICES YOU
CAN PUT A DEAD FISH IN AN MRI
SCANNER YOU CAN FIND BRAIN AB
-- ACTIVITY AS IT EXHIBITS
THEORY OF MIND.
THAT WAS PROBLEMATIC AND ANOTHER
PAPER HIGHLIGHTED THAT PEOPLE
WERE REPORTING BASICALLY
UNREASONABLY LARGE CORRELATIONS
BETWEEN BRAIN ACTIVITY AND
BEHAVIOR WHICH LARGELY CAME FROM
THE FACT THAT THEY WERE DOING
CIRCULAR DOUBLE-DIPPING SORTS OF
ANALYSES.
MORE RECENTLY A GROUP OF
RESEARCH SHOWED THAT THE
TECHNIQUES FOR MULTIPLE
COMPARISON QUESTION WE THOUGHT
WERE DOING A GOOD JOB FOR A
COUPLE OF DECADES TURNED OUT TO
NOT BE DOING SO WELL AND THEY
ARGUE COULD HAVE A LARGE IMPACT
A NUMBER OF RESULTS ON STUDIES
AND A CONTRIBUTION LAST YEAR
BASICALLY GAVE OUT THE SAME DATA
SET TO 70 DIFFERENT RESEARCH
GROUPS AND ASKED THEM TO ALL ASK
THE SAME QUESTION WITH THE DATA
AND WE SAW REMARKABLE
VARIABILITY IN THE ANSWERS THEY
GAVE US.
SO ALL THESE HIGHLIGHT THE FACT
THAT THE COMPLEX DATA ANALYSIS
THAT GO INTO BRAIN IMAGING USING
FUNCTIONAL MRI WHICH IS THE
TECHNIQUE WE USED LEAVE OPEN THE
SUBSTANTIAL REPRODUCIBILITY
PROBLEMS.
WHAT WE'VE SEEN IN OUR FIELD
PARTICULARLY IN THE LAST DECADE
IS A STRONG MOVE TOWARDS OPEN
AND REPRODUCIBLE SCIENCE.
I WANT TO HIGHLIGHT TODAY WHAT I
THINK HAS DRIVEN THAT AND WHAT
ARE LESSONS TO BE LEARNED HOUN
WELL OUR FIELD HAS DONE.
I'LL HIGHLIGHT THE CONTRIBUTIONS
OUR GROUP HAS MADE AND WE'RE ONE
OF MANY.
THERE'S OPEN DATA THAT SHARE
DIFFERENT TYPES OF DATA.
ANOTHER IS OPEN TOOLS THAT FOCUS
ON RIGOR AND REPRODUCIBILITY BE
THAT FOR DATA CONTROL OR
PROCESSING OR STUDY DESIGN.
AND THEN THIS IS ALL UNDER
PINNED BY A SET OF OPEN
STANDARDS FOR THE ORGANIZATION
OF DATA.
DEVELOPING DATA STANDARDS IS ONE
OF THE MOST PORE THINGS ONE CAN
DO BUT TURNED OUT TO BE AMAZE
LIG IMPACTFUL IN OUR FIELD TO
SHARE DATA IN A HIGHLY EFFECTIVE
WAY.
THE FIELD HAD A FALSE START FOR
DATA SHARING A COUPLE DECADES
AGO.
THERE WAS A PIONEERING GROUP
THAT STARTED THIS PROJECT CALLED
THE FMRI AND REQUIRED SUBMISSION
OF DATA AFTER PUBLICATION.
AND THIS LED TO A FIRE STORM IN
THE FIELD.
THIS IS AN OPEN LETTER SENT
AROUND BASICALLY SAYING WE'RE
CONCERNED OF ALL AUTHORS EXCEPT
FOR PUBLICATION TO SUBMIT ALL
DATA TO THE DATA CENTER.
THE FIELD WASN'T READY FOR DATA
SHARING AT THAT POINT.
BUT OVER THE LAST TWO DECADES
THAT'S CHANGED COMPLETELY.
IN IN ADDITION TO THE PROJECTS
MANY HAVE SHARED DATA OF MANY
DIFFERENT TYPES AND SCALES AND
THE GROUP I'M INVOLVED IN THE
PROFESSIONAL SOCIETY THAT IS ONE
IN THIS AREA CALLED THE
ORGANIZATION FOR HUMAN BRAIN
HAPPENING HAS BEEN CENTER OF
DEVELOPMENT OF OPEN SCIENCE
PRACTICES AND I THINK IT
HIGHLIGHTS THE IMPORTANCE OF
THINKING ABOUT PROFESSIONAL
SOCIETIES BECAUSE THEY CAN
CHANGE CULTURE WITH LOCAL
ORGANIZATIONS.
A COUPLE YEARS AGO AN ANONYMOUS
RESEARCHER SAID IT HAD BEEN
TAKEN OVER BY THE ZEALOTS AND
NOW THEY HAVE BECOME AN OPEN
SCIENCE ZEALOT THEMSELVES AND
DOING THEIR WORK OPEN SOURCE.
THE ATTENTION HAS COME TO
PROJECTS SPECIFICALLY TO SHARED
DATA.
WE CALL IT PROSPECTIVE DATA
SHARING.
IT WAS FUNDED TO SHARE DATA ON
1200 ADULTS AND MY GUESS IS IT'S
OVER 600 PUBLICATIONS AT THIS
POINT.
ONE ONGOING STUDY THE ABCD STUDY
IS FOLLOWING ABOUT 10,000
CHILDREN OVER THE COURSE OF A
DECADE AND SCANNING THEM
REGULARLY TO UNDERSTAND BRAIN
DEVELOPMENT AND HOW IT RELATES
TO VARIOUS OUTCOMES PARTICULARLY
IN DRUG ABUSE.
MY GROUP'S BEEN INTERESTED IN
SHARING DATA AFTER THE DATA HAD
BEEN COLLECTED AND SO WE'VE
DEVELOPED A SET OF TOOLS THAT
BASICALLY ALLOW PEOPLE TO SHARE
DIFFERENT TYPES OF DATA.
SOME ARE LIKE OPEN NEURO ALLOW
TO SHARE FULLY VAU DATA AND
SHARE STATISTICAL RESULTS AND AT
THE OTHER END THERE'S THE
RESULTS THAT COME FROM FOUND IN
PAPE
PAPERS AND I'VE SEEN TOOLS FOR
OPEN SCIENCE AND I'LL COME BACK
TO HOW I THINKNFRASTRUCTURE IS
KEY TO THIS.
THIS SHOWS THE GROWTH OF THE
OPEN DATABASE WE'VE RUN AND
SHOWING CONTINUOUS GROWTH IN
BOTH THE NUMBER OF DATA SETS AND
NUMBER OF SUBJECTS AND EACH IS
AN INDIVIDUAL PERSON THAT WAS
SCANNED AND HAVE 00 PLUS STUDIES
OVER A 20,000 INDIVIDUAL SUBJECT
IS MUCH HIGHER THAN THIS AT THIS
POINT.
THESE ARE STATISTICS FROM LAST
YEAR.
SO I WANT TO SAY A LITTLE BIT
ABOUT WHAT I THINK THE KEYS TO
SUCCESS PARTICULARLY OF DATA
SHARING BUT I THINK IT ALSO
EXTENDS TO ALL ASPECTS OF OPEN
SCIENCE PRACTICE IN THE FIELD OF
NEUROIMAGING.
WE'RE LUCKY IN SOME SENSE.
ONE WAY WE'RE LUCKY IS FOR THE
MOST PART OUR DATA OUR DIGITAL
END TO END.
THEY COME OFF AN MRI SCANNER AS
DIGITAL IMAGES AND GET PROCESSED
FROM THERE ON AND REDUCES MANUAL
STEPS IN THE PROCESS.
THERE'S SOMETIMES MANUAL STEPS
PARTICULARLY IN DIFFERENT
ASPECTS OF DATA ANALYSIS BUT
WE'RE ABLE TO MINIMIZE THOSE AND
THAT HELPS MAKES IT EASIER TO
SHARE THINGS.
WE HAVE A SET OF STANDARDIZED
FILE FORMATS AND DATA STANDARDS.
WE'RE LUCKY A GROUP CAME
TOGETHER BACK AROUND 2000 AND
WERE LIVING IN THE TOWER OF
BABBLE WHERE EVERY SOFTWARE
PACKAGE USED A DIFFERENT FORMAT
FOR MRI IMAGES AND A GROUP SAID
WE HAVE TO SOLVE IT HIT AND
DEVELOPED A THING CALLED NIFTY
AND EVERY SOFTWARE PACKAGE NOW
USES NIFTY OR SEVERAL
ADAPTATIONS FOR DIFFERENT TYPES
OF DATA.
BASICALLY MUCH ANY SOFTWARE
PACKAGE CAN READ ANY DATA FILE
COLLECTED IN OUR FIELD.
THE DATA ARE IMMEDIATELY USABLE
BY ANYONE.
I MENTIONED UP FRONT THE DATA
STANDARD WE DEVELOPED THE BRAIN
IMAGING DATA STRUCTURE WE
STARTED DEVELOPING IN 2016 AND
HAS BECOME A LARGE STEERING
EFFORT AND THERE'S A STEERING
COMMITTEE AND INTERNATIONAL
EFFORT AT THIS POINT AND ONE
THING WE FOCUSSED ON IN THAT
PARTICULAR EFFORT WAS PROVIDING
TOOLS TO ALLOW PEOPLE TO
AUTOMATICALLY VALIDATE THEIR
DATA.
THE IDEA IS IF I HAVE A DATA
SET, DOES IT ACCORD WITH
STANDARD?
I CAN GO TO A WEB PAGE AND POINT
TO MY DATA ON MY COMPUTER AND
WITHIN SECONDS IT TELLS ME YES,
YOUR DATA SET MEETS THE STANDARD
OR DOESN'T, HERE'S THE THINGS
YOU NEED TO FIX IN ORDER FOR IT
TO WORK WITH THE STANDARD.
THAT HAS GREATLY REDUCED THE
BURDEN OF CURATION FOR THE
ARCHIVES WHO SHARE THE DATA.
IT PUTS A LITTLE BIT MORE WORK
ON THE OWNER OF THE DATA SET TO
GET IT IN THE STANDARD BUT
THERE'S A NUMBER OF STOOLS TO
MAKE IT AS EASY AS POSSIBLE.
WE HAVE A LOT OF PEOPLE IN OUR
FIELD DEVELOPING INFRASTRUCTURE
SOLELY FOR THE PURPOSE OF MAKING
IT EASIER TO SHARE DATA.
ANOTHER THING THINK IS OFTEN
FORGOTTEN IS THE IMPORTANCE OF
DEMONSTRATED SCIENTIFIC UTILITY.
EVERYBODY AGREES THAT OPEN
SCIENCE AND REPRODUCIBILITY AND
DATA SHARING ARE ALL GREAT
THINGS BUT KNOWING THEY'RE GREAT
THINGS IS NOT ENOUGH TO GET
PEOPLE TO DO IT WE'VE HAD A
NUMBER OF SUCCESS STORY AND A
POST-DOC IS NOW A FACULTY MEMBER
AT SIDNEY AND HE PUBLISHED
PAPERS IN HIGH IMPACT JOURNALS
BASED ON PON -- OPENLY SHARED
DATA AND HE'S ONE OF THOSE
CANONICAL DATA PARASITES WE
HEARD ABOUT BUT HAS MADE
SCIENTIFIC CONTRIBUTIONS BASED
ON THE SHARED DATA.
I MENTION THE IMPORTANCE OF
INFRASTRUCTURE.
YOU NEED INFRASTRUCTURE TO ALLOW
TO DO THINGS AS EASILY AS
POSSIBLE AND I HAVE A FIGURE
FROM A PROJECT CALLED A BURN AN
EARLY DATA SHARING PROJECT
PROBABLY 15 YEARS AGO THAT WENT
NOWHERE AND IF YOU LOOK AT THE
ARCHITECT YOU'RE YOU HAD TO
LEARN DIFFERENT ACRONYMS AND
COMPUTER SCIENCE CONCEPTS.
IT WAS A TOOL REALLY CUTTING
EDGE OF THE TECHNOLOGY BUT SO
COMPLEX NO ONE COULD GET THEIR
HEAD AROUND IT SO IT WENT
NOWHERE.
DOING THINGS THAT MAY NOT BE
TECHNICALLY THE COOLEST OR BEST
THING BUT WORKING SCIENTISTS CAN
GET THEIR HEADS AROUND AND CLOSE
TO WHAT THEY ALREADY DO IS
IMPORTANT.
THE OTHER THING IS WE CAN'T JUST
SAY YOU CAN SHARE YOUR DATA BUT
IF YOU SHARE YOUR DATA, THEN YOU
CAN TAKE ADVANTAGE OF OTHER NEW
THINGS OR FOR EXAMPLE WHEN WE
DEVELOP THE STANDARD ALONG THAT
WE DEVELOPED BIDS APPS WHICH ARE
TOOLS THEY CAN APPLY TO THE DATA
SET AND THEY WOULD PROVEVIOUSLY
REQUIRE YOU TO CHANGE THE
STRUCTURE OF THE DATA AND WE
MADE THEM AWARE OF THEM AND RUN
THEM WITHOUT THE NEED FOR
REFORMATTING.
THAT WAS A VALUE-ADD AND DOING
THE THINGS THAT ALSO HELPED
SHARE THE DATA.
WE SAW COMMUNITY BUY-IN IS
ESSENTIAL AND IN GENERAL
MANDATES BUT UNLESS YOU HAVE
OVERWHELMINGLY POWERFUL
ADVOCATES AS IN GENOMICS,
MANDATES ARE PROBABLY NOT THE
WAY TO GO ABOUT CHANGING
BEHAVIOR.
THE OPEN SCIENCE ADVOCATES ARE
NOT PEOPLE FROM THE OUTSIDE
COMING IN SAYING YOU NEED TO DO
OPEN SCIENCE.
IT'S BEEN MEMBERS OF THE
COMMUNITY LIKE ME AND IN SILICON
VALLEY WE HAVE A TERM CALLED DOG
FOODING WHICH MEANS USE YOUR OWN
PRODUCTS.
THOSE WHO HAVE BEEN THE
ADVOCATES ALSO EAT OUR OWN DOG
FOOD.
I THINK THAT'S ALLOWED US TO
MOTIVATE OTHERS TO MOVE IN THAT
DIRECTION.
AND FINALLY I THINK WE HAVE TO
AT LEAST PART OF THE SUCCESS OF
OUR PROJECT HAS BEEN BECAUSE WE
HAVE ADHERED STRICTLY TO THE
PAREDO PRINCIPLE.
NOT LETTING THE PERFECT BEING
THE ENEMY OF THE GOOD AND
REALIZING OFTEN A RELATIVELY
SMALL AMOUNT OF THE EFFORT WILL
COVER A RELATIVELY LARGE NUMBER
OF USE CASES AND THERE'S ALWAYS
GOING TO BE PEOPLE WHO FOCUS ON
HOW YOU CAN'T ADDRESS SOME
PARTICULAR EDGE CASE AND WE HAD
TO LEARN TO LISTEN TO THEM AND
THEN DECIDE IT'S IMPORTANT
ENOUGH TO WORRY ABOUT.
IT'S IMPORTANT TO NOT GET STUCK
ON COVERING EVERY POSSIBLE USE
CASE.
THANK YOU VERY MUCH.
>> THANK YOU FOR THE ORGANIZERS
FOR SETTING UP THE SESSION AND
INVITING ME TO SPEAK.
AS WE ALL KNOW THE THEME OF
TODAY'S SESSION IS CREATE PATH
AND WHEN WE SEE A NICE CLEAR
PATH IT PROBABLY DOESN'T START
THAT WAY AND WOULD LIKE TO SHARE
LESSONS LEARNED IN BUSHWACKING
THROUGH THE WILDERNESS I LOOK AT
PRACTICES AND WHAT HAS LED TO
JOURNAL POLICY CHANGE AND OVER
THE PAST FEW YEARS I'VE THOUGHT
ABOUT THE WAY WE EDUCATE AND
TRAIN YOUNG AS WELL AS OLDER
RESEARCHERS AND WE HAVE A LOT OF
EDUCATIONAL FORMATS FOCUSSED ON
STRAIGHT LECTURES AND SHARE WITH
STUDENTS ON HERE'S A BUNCH OF
STUFF YOU NEED TO DO AND GOOD
LUCK.
I WANTED TO RETHINK THE WAY WE
WERE OFFERING THIS FOR
REPRODUCIBLE RESEARCH PRACTICES
TO FOCUS ON THINGS THAT WERE
APPLIED THAT FOCUSSED ON GETTING
STUDENTS TO IMPLEMENT THIS IN
THEIR OWN RESEARCH RIGHT AWAY TO
REMOVE SOME BARRIERS PEOPLE
ENCOUNTERING.
I IT'S A COURSE IN METARESEARCH
WHERE PARTICIPANTS ARE FROM ALL
OVER THE WORLD OR FROM THE FOUR
UNIVERSITIES AND DIFFERENT FIELD
AND LEARN BY META AND WORK TO
DESIGN, CONDUCT AND HAVE A ME
METARESEARCH PROJECT AND WE HAVE
EIGHT SESSIONS IN ANOTHER COURSE
DURING THE FIRST HALF WE TALK
ABOUT A NEW PRACTICE AND IN THE
SECOND HALF STUDENTS WORK ON
IMPLEMENTING IN IN THEIR OWN
RESEARCH AND THE THIRD ONE IS
CALLED PUBLIC MY PROTOCOL WHERE
STUDENTS BRING A PROTOCOL AND
WORK ON PRO -- PROTOCOL AND
SUBMIT.
I'LL BE THINKING ABOUT TWO
QUESTIONS TODAY.
ONE, WHAT PRACTSES SHOULD WE
THINK ABOUT AND SECOND HOW CAN
WE BEST TEACH PEOPLE ABOUT THE
PRACTICES.
THE FIRST I WANT TO EMPHASIZE IS
I THINK IT WOULD BE HEL --
HELPFUL TO INCENTIVIZE OPEN DATA
AND CODE ON PAR WITH
PUBLICATION.
THAT'S OCCURRENCE FOR ACADEMIC
CREDIT.
THEY'RE PRIMARILY FOCUSSED ON
RESULTS AND WHEN WE THINK BIT
RESULT OFTEN THE LEAST USEFUL
PART OF OUR WORK AND MAY BE THE
LEAST REPRODUCIBILITY PART OF
OUR OUTPUT AND THEY UNDER VALUE
THE METHODS DEVELOPMENT WORK OR
OFTEN CUTTING OUT INFORMATION
METHODS TO LEAVE MORE SPACE FOR
THE RESULTS AND THE DISCUSSION.
AND METHODS ARE ONE OF THE
THINGS WE DEVELOPED THAT IS MOST
VALUABLE TO OUR COLLEAGUES IN
THE SCIENTIFIC COMMUNITY.
SO UNTIL OPEN METHODS AND OPEN
DATA AND CODE ARE ON PAR WITH
PUBLICATIONS, RESEARCHERS WHO
CREATE AND SHARE THESE MATERIALS
ARE DOING MUCH MORE WORK FOR THE
SAME UNIT OF CREDIT.
AND WE HEAR THIS ALL THE TIME
FROM PARTICIPANTS IN OUR COURSES
WHO SOME BACK AND SAY I'D LIKE
TO IMPLEMENT THESE THINGS BUT MY
SUPERVISOR SAYS IT'S NOT WORK
INVESTING THE TIME IN.
THERE'S ADDITIONAL PRACTICES AND
THE FIRST IS USING RESEARCH
RESOURCE IDENTIFIERSO SPES TIE
WHAT RESOURCES WILL BE USED --
SPECIFY WHAT RESOURCES WILL BE
USED AND THERE'S CLEAR EVIDENCE
THAT WHEN WE USE THESE
IDENTIFIERS WE GET RESOURCES
THAT CANNOT BE IDENTIFIED AND
RESOURCE IDENTIFIERS HAVE
WARNINGS ON CONTAMINATED OR
MISDIAGNOSED AND THE PROPORTION
OF PUBLICATIONS WITH THE CELL
LINE IS REDUCED.
THE NEXT IS OPEN METHODS.
THERE'S HUGE COMMUNITIES
SURROUNDING OPEN DATA AND OPEN
CODE BUT LITTLE FOR OPEN
METHODS.
IN PARTICULAR I THINK PROTOCOL
REPOSITORIES ARE IMPORTANT AND
WITH A PROTOCOL REPOSITORY
THRESHOLD FOR POSTING IS LOWER
AND YOU HAVE SOMETHING OTHER
PEOPLE CAN MAKE WORK AND IT
MAKES IT EASIER TO POST LIVING
PROTOCOLS AND TRACK THE
EVOLUTION ACROSS TIME AND
LABORATORIES.
DATA VISUALIZATION.
IT'S ESSENTIAL BECAUSE THE WAY
WE VISUALIZE DATA IS CENTRAL TO
OUR SCIENTIFIC KNOWLEDGE.
OFTEN THE FIGURES ARE WHERE WE
SHOW THE MOST IMPORTANT FINDINGS
AND IF WE DO A BAD JOB OF USING
THOSE FINDINGS AND USE DATA THAT
OBFUSCATE WE IMPAIR THE WAY TO
INTERPRET IT FOR THE FUTURE AND
OUR GROUP HAS DONE WORK ON THE
NEED TO REPLACE BAR GRAPHS WITH
CONTINUOUS DATA AND ANOTHER
OPPORTUNITY I MIGHT HIGHLIGHT IN
DATA VISUALIZATION IS THE NEED
TO USE FLOW CHARTS ACROSS STUDY
DESIGNS.
THEY'RE EFFECTIVE WAYS OF
REPORTING ATTRITION ESSENTIAL
AND THERE'S MANY WHO HAVE
EXPERTISE AND I THINK WE CAN
SPEND TIME ON IT BUT IT'S
ESSENTIAL TO REMEMBER OUR
STUDENTS ARE NOT BLACK SLATES.
THEY HAVE EXPECTATIONS HOW THIS
IS BEING DONE AND WHAT THEY
EXPECT TO SEE WE NEED TO
IDENTIFY RISK CON SENSE AND
PROBLEMATIC PRACTICES AND FIX
THESE THINGS.
TALK ABOUT WHEN STATISTICS ARE
NEEDED AND NOT AND THE
IMPORTANCE OF ESTABLISHING CRICK
COLUMN COLLABORATIVELY WITH
SCIENTISTS AS WELL STATISTICIAN
TO ENSURE IT MEETS THE NEEDS OF
STUDENTS AND WE PUBLISHED A
PAPER IN 2016 WHICH SHOWS ALMOST
ALL PHYSIOLOGY PAPERS HAVE
STATISTICS HOWEVER, STATISTICS
EDUCATION WAS NOT ALWAYS
REQUIRED TO GET A Ph.D. IN
PHYSIOLOGY AND RELATED FIELD
AMONG TOP-FUNDED PHYSIOLOGY
DEPARTMENTS.
WHAT HAVE WE LEARNED FROM OUR
EDUCATIONAL EXPERIENCES WITH THE
APPLIED COURSES WE'RE RUNNING?
THE FIRST THING WE'RE FOCUSSED
ON IS REPLACING STRAIGHT
LECTURES WITH APPLIED TRAINING
WE WANT PARTICIPANTS TO
IMPLEMENT PRACTICES IN THEIR OWN
RESEARCH AND INSTEAD OF SAYING I
HAVE TAKEN A COURSE OR DOES
ASSIGNMENTS OR PASSED A TEST WE
WANT TO SHOW WE HAVE IMPLEMENTED
THE PRACTICES.
HERE'S WHAT I DID TO IMPLEMENT
THE USEFUL RESEARCH IN MY OWN
RESEARCH IN MY LAST PAPER.
THE OTHER THING WE MAKE USE IS
GROUP MENTORING.
WHEN WE HAVE STUDENTS
IMPLEMENTING THINGS AND THEY'RE
DO THAT IN THE COURSE THEY START
TALKING TO EACH OTHER AND SAYING
I'M HAVING THIS PROBLEM AND
SOMEONE ELSE SAYS I HAD THAT
PROBLEM TOO, HERE'S HOW I SOLVED
IT.
WE'RE HERE TO HELP THEM GET OVER
EARLY ROAD BLOCKS THEY ENCOU
ENCOUNTERED AND THAT REDUCES THE
BURDEN THEY HAVE WHEN THEY GO
BACK TO THEIR OWN LAB AND HAVE
NO ONE AVAILABLE TO SUPPORT TO
FIGURE OUT HOW TO DO THIS ON
THEIR OWN.
TARGETING MISCONCEPTIONS.
TRAINING RESEARCH GROUPS INSTEAD
OF INDIVIDUALS.
COURSES ARE OFTEN DELIVERED TO
WHOEVER SIGNS UP WHEREAS WE KNOW
THE RESEARCH ENVIRONMENT IS
IMPORTANT FOR DETERMINING WHAT
CAN BE IMPLEMENTED.
WITH WE'RE ABLE TO GET MORE TO
TRAIN TOGETHER OR TRAIN THROUGH
AN ENTIRE RESEARCH GROUP WE CAN
GET EVERYONE TALKING ABOUT THE
ISSUES AND DEVELOPING AND
IMPLEMENTATION STRATEGY FOR AN
ENTIRE RESEARCH GROUP AT THE
SAME TIME WHICH IS MUCH FASTER
AND ALSO REALLY REDUCES THE
BURDEN ON OUR EARLY CAREER
RESEARCHERS WHO ARE TYPICALLY
TAKING THE COURSE BECAUSE
THERE'S AN INSTRUCTOR WITH
EXPERIENCE AND ABLE TO HANDLE
THE QUESTIONS THEY MIGHT GET OR
WHO MAY HAVE MORE CREDIBILITY
THAN THEY AS AN EARLY CAREER
RESEARCHER MAY BE AROUND MORE
SENIOR MEMBERS OF THEIR LAB
GROUP.
IMPLEMENT SYSTEM TO CHANGE
PRACTICES ACROSS CAREER STAGES.
THIS IS CRITICAL.
EVERY SESSION WE RUN WE THEY SAY
THIS SOUNDS AMAZING BUT I'LL
NEVER CONVINCE MY SUPERVISOR,
HOW CAN I CONVINCE THEM?
WE NEED TO THINK OF SHIFTING
ENTIRE INCENTIVE SYSTEMS SO
WE'RE ALSO TALKING TO THE MORE
SENIOR RESEARCHERS TO GIVE THEM
INCENTIVES.
I ALSO THINK IT'S IMPORTANT TO
TAKE ADVANTAGE OF UNDER UTILIZED
OPPORTUNITIES.
WE RECENTLY HELD A VIRTUAL BRAIN
STORMING EVENT WITH RESEARCHERS
AROUND GERMANY LOOKING AT MAKING
REPRODUCIBILITY TRAINING THE
NORM AND PEOPLE WERE DOING
DIFFERENT THINGS.
I WANT TO HIGHLIGHT SOME THAT
HAVE POTENTIAL TO BE USED MORE.
ONE IS REQUIRING OPEN SCIENCE
PRACTICES AND WE HAD SOME
DEPARTMENTS ABLE TO PUT THEM
INTO SYSTEM REQUIREMENTS AND
THEY FOUND WHEN THEY DID THAT,
STUDENTS WOULD GO TO THE
SUPERVISOR SAYING I'M
RESPONSIBLE FOR DOING THIS IN MY
THESIS.
HOW DO WE DO IT AND THEN THE
SUPERVISORS HAD ANSWERS.
IT HELPED GET BEYOND THE EARLY
CAREER AUDIENCE AND ADDRESS
ISSUES WITH THE MORE SENIOR
PEOPLE.
THE SECOND IS TO INTEGRATEUC
REPROD RESEARCH PRACTICES
AND HAVING QUESTIONS ON HOW TO
MAKE PROTOCOL REPRODUCIBILITY
AND IMPLEMENTING THAT.
AND THEY HAVE EXPECTATIONS AND
SKILLS BEFORE THEY ENTER
GRADUATE SCHOOL AND START THEIR
OWN RESEARCH AND LASTLY THERE
WAS A SMALL GROUP WHERE
PARTICIPANTS WORKED TOGETHER TO
PERFORM METARESEARCH STUDIES OR
REPLICATION STUDIES.
NOT ONLY DO STUDENTS LEARN TO DO
THE RESEARCH DISCUSSED BUT YOU
CAN OPEN RESEARCH WHILE
CONDUCTING THAT WITH STUDENTS
AND THE OUTPUT OF THE COURSE IS
NOT AN EXAMPLE OR TEST RESULT OR
GRADE BUT VALUE TO THE
SCIENTIFIC COMMUNITY.
NATION ALL I HAVE FOR TODAY AND
THANK YOU VERY MUCH.
THANK YOU FOR ALL THE WONDERFUL
TALKS.
WE HAVE A LITTLE BIT OF TIME FOR
DISCUSSION AND HAVE THE WORLD
CAFE AFTER?
FIRST OF ALL, I'M GOING TO TALK
ABOUT AND RECAP WHAT WE HEARD
AND IF YOU HAVE QUESTIONSES --
QUESTIONS YOU CAN COME TO THE
MICROPHONE AND AS AN INSTINCTIVE
TEACHER I HAVE A WAY OF FIGURING
OUT WHO IS AWAKE AND ASLEEP AND
MOST ARE AWAKE SO WE DID GOOD
BUT I WANTED TO RECAP SOME
THINGS I WAS HEARING BECAUSE WE
WENT FROM THE PHILOSOPHICAL
UNDER PINNINGS OF WHAT A GOOD
EDUCATION OR BASIS FOR EDUCATION
IN RESEARCH RIGOR COULD LOOK
LIKE IN OUR FIRST TALK WITH
GUNDULA AND THEN HEARD ABOUT
COMMUNITY AND VALUE AS KEY
ELEMENTS HOW IF YOU'RE GOING TO
CREATE CULTURE CHANGE PEOPLE
NEED TO BE ALIGNED WITH VALUES
AND IN FACT IT'S A FANTASTIC
ORGANIZATION FOR SUSTAINABILITY.
IT'S AN ORGANIZATION WHICH HAS
AFFECTED THOUSAND OF PEOPLE
THROUGH TREMENDOUS EFFORT.
THEN WE HEARD OF INFRASTRUCTURE
AS SOMETHING THAT ENABLED IT TO
TAKE PLACE AND THE LAST TALK
WITH TRACY TAKES US CLOSE TO
WHERE WE'LL BE 40 MINUTES FROM
NOW WITH DIFFERENT IDEAS AND
RESULTS WE CAN ACCOMPLISH AND
HOW INCENTIVES ARE LINKED TO
BEHAVIOR CHANGE.
YOU NEED TO MAKE THINGS POSSIBLE
FIRST BEFORE PEOPLE DO THEM AND
THEN YOU NEED TO INCENTIVIZE
THEM AND NEED DIFFERENT LAYERS.
THAT'S OUR TASK.
WE'RE IN DISCUSSION AND WE WANT
TO HEAR QUESTION AND COMMENTS
FROM THE AUDIENCE AND THE WHOLE
PURPOSE AND WE'LL NO LONGER BE
PEOPLE SITTING IN SEATS.
WE'LL BE PEOPLE SITTING IN SEATS
IN DIFFERENT ROOMS THAT ARE
SMALLER BUT WILL FORCE YOU TO
TALK AND TO ENGAGE.
THE THAFK TASK FOR THE SESSION
IS WHAT ARE THE ACTUAL BEHAVIORS
WE NEED TO IMPLEMENT RIGOR AND
WHAT DO YOU THINK THEY LOOK
LIKE, THAT'S WHAT WE'RE TRYING
TO GET TO NOW.
WE'RE ANXIOUS FOR PEOPLE TO TELL
US ABOUT WHAT THEIR COMMENTS AND
IDEAS ARE AND START WITH TALKING
TO THE FOLKS HERE.
ONE OF THE QUESTIONS I WOULD
GIVE FIRST TO TRACY AND ASK
OTHERS TO THINK ABOUT YOU
MENTIONED SOMETHING WHICH I'M
INTERESTED IN WHICH IS THE IDEA
OF CAREER SPANNING.
WE'VE BEEN TALKING ABOUT AND ALL
THE TIMES WE TALK ABOUT THINGS,
IT'S THOSE PEOPLE NEED TO LEARN
THIS OR THOSE PEOPLE NEED TO
LEARN THAT BUT IT'S MANY LAYERS
OF PEOPLE.
TRACY, YOU MENTION IN YOUR TALK
THE SPECIFIC SCENARIO OF WELL,
THE GRADUATE STUDENTS OR THE
POST-DOCS WANT TO DO THIS BUT
THE OLD FUDDY-DUDDIES IF ANYONE
KNOWS THAT TERM, THE MORE SENIOR
PEOPLE MAY NOT BE AS RECEPTIVE
AND WE HEARD IN THE TALK WITH
RUSS, WHEN PEOPLE COME IN AND
SAY OPEN SCIENCE IS INVATING OUR
FIELD, IT SEEMS THEY'RE
MOTIVATION AT HIGHER LEVELS YOU
OR RUSS WHO MAY STILL BE ONLINE,
CAN ANY OF YOU SPEAK TO
TARGETING OUR EFFORTS NOT JUST
AT ONE LAYER OF THE COMMUNITY,
THE GRADUATE STUDENTS WHO ARE
ALWAYS THE HOPE FOR THE FUTURE
BUT HOW DO WE CREATE
OPPORTUNITIES FOR LEARNING FOR
DIFFERENT LAYERS OF COMMUNITY
BOTH IN THEIR SENIORITY AND THE
DIFFERENT POWER OR POSITIONS
THAT PEOPLE PLAY?
I SEE A HAND FROM RUSS NOW AND
THEN I'LL GO KARI NEXT.
MAYBE WE CAN PROJECT SO OTHERS
CAN HEAR.
>> SO ONE THING I'VE SEEN AS
INTERESTING IS THAT EXACTLY AS
WAS MENTIONED THE STUDENTS WANT
TO DO THIS STUFF AND THEIR P.I.
SAYS NO YOU CAN'T OR NO, IT'S
NOT WORTH IT.
AND I'VE SEEN THE BEHAVIOR WHERE
GRAD STUDENTS WON'T SHUT UP
ABOUT IT AND SAY WE HAVE TO KEEP
DOING THIS.
THE PRESSURE FROM THE BOTTOM IS
PROBABLY SOME OF THE MOST EFFECT
OFFICER PRESSURE ONE CAN SEE AND
FROM THE OTHER SIDE AND PRESSURE
FROM THE PROFESSIONAL SOCIETIES
FROM ONE'S COLLEAGUES IN THE
FIELD ARE THE TWO VECTORS THAT
CAN HELP CAUSE BEHAVIOR CHANGE
IN MORE SENIOR PEOPLE.
>> WE'VE LOOKED AT LIBRARIES AS
SORT OF THE CENTRAL PLACE FOR
THIS INFORMATION AND WE
CHALLENGE INSTITUTIONS TO PUT
THEIR RESOURCES INTO THE
LIBRARIES.
WE HAVE A LOT OF MEMBER
ORGANIZATIONS AT THE CARPENTRIES
THAT ARE INSTITUTIONAL MEMBERS
THROUGH THE UNIVERSITY LIBRARY
THE NATIONAL LIBRARY OF MEDICINE
SO WE KNOW ALL RESEARCHERS HAVE
TO INTERACT WITH THE LIBRARY AT
SOME POINT AND I HAVE TO
ADVOCATE FOR THE SKILLS AND
PERSPECTIVES THROUGH THE LIBRARY
STAFF INFORMATION SCIENCE STAFF.
>> THE STUDENTS VOTED WITH THEIR
FEET AND THE OTHER THING IS THAT
THE COMMUNITY WE BUILD UP AND
PEOPLE HEARD ABOUT EFFORTS ASKED
US HOW DO YOU TEACH THIS?
HOW DO YOU IMPLEMENT THIS.
WE HAVE AN EDUCATION CREDIT FOR
PRACTITIONERS WHICH IS THE FIRST
INSTITUTE I THINK IN JANUARY AND
WE HAVE STUDENTS BUT THE MENTORS
HAVE TO BE EDUCATED AND THE
MENTORS CAME TO US AND ASKED FOR
THAT.
>> IT'S NOT ONLY THE PRESSURE,
WHICH IS PART OF THE MOTIVATION,
BUT IT COULD ALSO BE THE
REASSURANCE.
WHEN I HEAR PEOPLE AFRAID OF A
NEW PRACTICE, WHAT THEY'RE
REALLY SAYING IS I'M AFRAID I
CANNOT EQUALLY COMPETE WITH
THOSE ABLE TO EXECUTE ON THIS
PRACTICE EFFECTIVELY AND I FEEL
LIKE I CAN'T DO THAT.
HOW DO WE SUIT TO A FULL-TIME
PROFESSOR WHO WILL NOT GIVE A
FIVE-DAY WORKSHOP ON SOMETHING
BUT MAYBE SINCE THEY'RE NOT
PRACTITIONERS THEMSELVES A
TWO-HOUR INFORMATIONAL WORKSHOP
MAY BE WHAT THEY NEED TO
UNDERSTAND THE BENEFITS AND MOVE
THEIR LAB ALONG.
LET'S GO TO EVERYONE IN THE
AUDIENCE.
REMEMBER YOUR NAME AND YOUR
INSTITUTION IF YOU WANT TO SAY
MORE ABOUT YOUR BACKGROUND
BEFORE GIVING COMMENTS.
I WAS QUITE STRUCK BY COMMENTS
FROM KARI AND RUSS THOUGH THIS
IS TALKING ABOUT PRACTICAL
BEHAVIOR AND CHANGE, BOTH KARI
AND RUSS WERE TALKING ABOUT
INSTITUTIONAL BEHAVIORS AND I
WAS STRUCK BY THE GROUP THAT MET
TO COME UP WITH A TYPE OF
LANGUAGE.
I THINK YOU DESCRIBED IT AS
NIFTY AND KARI, THE DECISIONS OF
THE VARIOUS CARPENTRIES TO DO
THINGS AND THE QUESTION IS CAN
YOU ALL DESCRIBE OTHER EXAMPLES
OF THIS?
AND THE REASON WHY I'M ASKING IS
THAT I SOMETIMES HEAR A TENSION
BETWEEN BEHAVIOR'S ALWAYS
REDUCED TO A SINGULAR UNIT OF AN
INDIVIDUAL BUT NOT ACTUALLY
THINKING ABOUT THE BEHAVIORS OF
INSTITUTIONS.
WE GOT CONCRETE EXAMPLES OF
CHOICE THAT COLLECTIVES MADE
EVEN IF THEY WEREN'T TECHNICALLY
AN INSTITUTION AND HEARING MORE
WOULD BE INTERESTING TO PUT THAT
INTO CONTEXT.
>> OUR DECISION MAKING ETHOS IS
COMMUNITY DRIVEN.
A LOT OF THE CORE VALUES
MENTIONED THERE WAS A COMMITTEE
OF NOT ONLY OUR BOARD OF
DIRECTORS BUT A GROUP OF
VOLUNTEERS WHO DID EIGHT OF
CROWD SOURCIE IING FOR THE SKIL
AND BEHAVIORS ACROSS OUR
COMMUNITY.
IT'S ONE OF THOSE PREACHING TO
THE CHOIR EXAMPLES AND WE HAVE
MANY WHO COME TO OUR COMMUNITY
INITIALLY TO LEARN SOMETHING AND
THEN REALIZE I FEEL GOOD ABOUT
MYSELF HERE.
I FEEL LIKE I CAN LEARN HOW TO
CODE AND WANT TO BE AROUND THESE
PEOPLE MORE.
THEY END UP RUNNING THE
COMMUNITY IT'S LARD FOR ME TO
GIVE -- THAT IS MY EXAMPLE.
WE'RE SUCH A COMMUNITY-DRIVEN
ORGANIZATION, WE TEND TO ATTRACT
PEOPLE WHO HAVE THESE SKILLS AND
PERSPECTIVES AND THEN THEY SAY
AND LITTLE BY LITTLE THEY PULL
MORE PEOPLE INTO THE COMMUNITY
AND ADOPT OUR ETHOS AS WELL.
>> I WANT TO SAY TWO THINGS.
I THINK THEY'RE RELATED BUT MAY
NOT BE ONE IS A QUESTION OF WHO
IS COMING TO THE BASE TRAININGS.
YOUR COMMENT MADE THE THINK IT'S
RELATED BECAUSE THERE'S GROUP
THINK GOING ON HERE.
THE DIFFERENCE IN SHOWING
EXAMPLES THERE'S RESISTANCE FROM
TRADITIONAL METRICS OF SUCCESS
AND HAS ANYONE LOOKED AT HOW
THIS PRACTICE DOES MANIFEST IN
THE MORE TRADITIONAL METRICS?
IF YOU THINK OF THE
INSTITUTIONS, PAPERS AREN'T
GOING AWAY ANY TIME SOON IN
TERMS OF THEM BEING THE CURRENCY
WE'RE JUDGING SCIENTISTS AND IS
THERE A WAY TO PENETRATE INTO
THE COMMUNITIES THAT HAVEN'T
ENTERED THE CONVERSATION YET.
>> WHO XOMS TO OUR WORKSHOP
PRIMARILY GRADUATE STUDENTS OR
THOSE WORKING IN NATIONAL LABS
OR INDIVIDUALS LIKE THAT.
PEOPLE WHO HAVE RESEARCH OR DATA
AND NO IDEA WHAT TO DO WITH IT
OR WANT TO LEARN SKILLS AND
REPRODUCIBILITY.
PRIMARILY, OUR ORGANIZATION IS
FUNDED BY OUR MEMBERSHIP
STRUCTURE AND OTHER UNIVERSITIES
AND INSTITUTIONS PAY A FEE AND
WE INTERN ORGANIZE WORKSHOPS FOR
THEM.
IF YOU'RE INSTITUTION HAS A
MEMBERSHIP YOU'D RECEIVE A
CERTAIN NUMBER OF SEATS IN OUR
INSTRUCTOR TRAINING PROGRAM.
WE GET A LOT OF IF YOUR
MEMBERSHIP IS THROUGH YOU'RE
LIBRARY IT'S A DIVERSE GROUP OR
MAY BE THROUGH ANOTHER LAB AND
THE MEMBERSHIP IS STRUCTURED
AROUND THAT LAB.
WE GET A VARIETY OF INDIVIDUALS
ATTENDING OUR WORK SHOPS.
I FORGOT THE SECOND PART.
>> WHAT I WANT TO DO, IF I CAN,
ONE QUESTION OR COMMENT OR ONE
RESPONSE TO STIMULATE MANY
THOUGHTS AND WE'LL FOLLOW-UP AT
THE WORLD CAFE.
LET'S MOVE AHEAD AND TRY TO FARM
IT OUT TO AS MANY AS WE CAN.
>> GOOD AFTERNOON.
I'M CARMEN PROFESSOR OF BIOLOGY
AT A UNIVERSITY IN PUERTO RICO.
THERE'S SOMETHING BOTHERING ME A
LITTLE BIT THIS MORNING IS
ISSUES OF REPRESENTATION IN THE
DATA WE'RE TRYING TO REACH TO
MAKE IT RIGOROUS AND
REPRODUCIBILITY AND TO PUT INTO
PERSPECTIVE THE CONTEXT OF HOW
SIGN HAS GROWN IN A VERY
PRIVILEGED SITUATION WHERE A LOT
OF COMMUNITIES HAVE NOT BEEN
ADDRESSED AT ALL.
FOR EXAMPLE, THE BEAUTIFUL
INITIATIVE AT HOPKINS.
ARE YOU BRINGING IN THE
COMPONENT OF HISTORY AND ISSUES
OF INEQUITY AND STRUCTURAL
RACISM THAT HAS PREVAILED IN
SCIENCE AT ALL NOT ONLY IN THE
UNITED STATES BUT THROUGHOUT THE
WESTERN WORLD AND THE
NEUROIMAGING DATA.
HOW CAN WE MAKE ANY CONCLUSIONS
WHEN IT'S BASED ON A CERTAIN
GROUP SO THAT WAS MY QUESTION.
>> I DID NOT HAVE TIME TO TALK
ABOUT A MAJOR INITIATIVE OF OURS
AND SOME MENTIONED SOCIAL JUS
IT'S AS A MAJOR COMPETENCY.
THIS IS MY ABSOLUTE FAVORITE
PROJECT.
WHAT WE DO APART FROM OTHER
INITIATIVES THAT TARGET OUR
COMMUNITY WE HAVE A COMMUNITY
THAT NEEDS A LOT OF SUPPORT.
BUT THAT PARTICULAR PROJECT WE
INCLUDE AS A MAJOR COMPETENCY IS
SOMETHING IF I HAD TIME TO SHOW
ONE ON THE CAUSATION PROJECT YOU
WOULD HAVE SEEN THEY SAID ALL
THE DIFFERENT PEOPLE IN
TEXTBOOKS ON GETTING CREDIT FOR
MAJOR DISCOVERIES TEND TO BE IN
THE MAJORITY ARE WHITE MEN.
A MAJOR PROJECT IS TO GIVE THOSE
JUSTICE WHO DID NOT FIND THE
RIGHTFUL PLACE AND STANCE ON
HISTORY SO IT'S A MAJOR PROJECT
WE RUN IN THE STABLE LEVEL AND A
PROJECT THAT WE DO THROUGH
INTERVIEWS IN PRODUCING
PRODUCIBLE, REUSABLE EARNING
OBJECTS TO MAKE PROJECTS LIKE
CHICK PEOPLE MADE WHICH
DISCOVERIES AND CREDIT WHERE
CREDIT IS DUE.
BRING THE PEOPLE IN THE WEBSITE
AND COLLECTIONS SO EVERYBODY CAN
LOOK AT A FORWARD LOOKING
PROJECT SO THE NEXT GENERATION
GETS EDUCATED WITH TRAINING AND
THOSE WHO DID THE WORK ALSO GET
CREDIT.
THAT'S ONE OF THE PROJECTS BUT
I'M HAPPY TO TALK MORE.
IT'S A MAJOR INITIATIVE OF OURS.
>> CLEARLY SOMETHING TO THINK
ABOUT THE DEFINITION OF RIGOR
AND HOW IT INTERACTS WITH
CONCERNS.
>> THIS IS A MORE PRAGMATIC
POINT.
I'M A RESEARCH SCIENTIST AND
ALSO A FUDDY-DUDDY.
I WANTED TO MENTION THE
UNIVERSITY OF CALIFORNIA SYSTEM
LIBRARIES HAD A SYSTEM-WIDE
PROJECT THEY CALLED LOVE DATA
WEEK AND ALL KINDS OF COURSES
AND THE PLACES TO DEPOSIT DATA
AND THE PROCESS TO DO IT AND IT
WAS DONE WHERE IT WAS MORE
ACCESSIBLE TO PROFESSORS AND
RAISED AWARENESS THAT'S WHERE WE
CAN GO FOR STEWARDSHIP AND HELP
BECAUSE I THINK WE WEREN'T AWARE
UNTIL THAT.
>> THANK YOU.
>> I'M AN ASSOCIATE DEAN AND WE
HAVE A PROGRAM IN TRAINING.
GREAT SESSION.
A LOT OF WHAT WAS SAID RESONATES
IN THINKING ABOUT THE PATH AND
TALK ABOUT TRAINING WHAT YOU
WERE SAYING ABOUT HAVING
APPLICABLE EXERCISES AND THINK
OF WHAT'S NEXT FOR TRAINEES AND
YOU TALK ABOUT GRAPHS AND SHOW
DATA BETTER IN A WAY THAT'S
RIGOROUS SCIENCE AND WHAT'S ALSO
IMPORTANT IS WHEN STUDENT
TRAINEES AND FACULTY ARE SHOWING
OUR FINDINGS AT CONFERENCE.
THERE'S A ROLE FOR OUR SOCIETY
AS WELL TO SET EXPECTATIONS OF
ELEMENTS OF RIGOR AND
REPRODUCIBILITY AND POVERTIER
PRESENTATIONS IN ORAL
PRESENTATIONS AND THAT CAN COME
WITH INCENTIVES AND NOT ONLY
ONLY HAVE EXPECTATIONS IN
SHOWING DATA AND WHO TO
EMPHASIZE TO SHOW WHERE THE DATA
IS BEING PRESENTED AND AVAILABLE
FOR OTHERS TO USE OR IF THEY'RE
USING OPEN DATA SOURCES TO SHOW
IN THAT IN THEIR PRESENTATIONS.
I'VE SEEN MORE AND MORE IN THE
LAST FEW YEARS OUR TALKS ARE
MOVING TO TED TALKS.
IT'S BECOME MORE SMOOTH AND
TELLING MORE SMOOTH STORIES,
GLANCING OVER ALL THE RIGOROUS
ELEMENTS OF OUR SCIENCE.
I THINK THAT'S A MISTAKE.
WE'RE SHOWING OUR TRAINEES YOU
CAN DO THIS IN THE LAB WHEN IT
COMES TO YOUR TALK AND I THINK
YOU NEED TO BRING BACK ELEMENTS
OF TRULY SHOWING NOT ONLY SAMPLE
SIZE AND HOW YOU DERIVE IT BUT
THE REPLICATION ELEMENT AND WHAT
ARE YOU REPRODUCING AND THE
RIGOR AND INCENTIVIZE THE
ELEMENT AND WONDERING TRACEY AND
GUNDULA IF ANY OF THIS HAS COME
UP IN YOUR RESEARCH.
>> CONFERENCES AREN'T A PLACE I
LOOKED AT BUT THE FIRST RESEARCH
STUDY ON BAR GRAPHS WAS ME
SITTING AT A CONFERENCE
FRUSTRATED WITH THE BAR GRAPHS
AND I STARTED DOWNLOADING PAPERS
AND CHECKING THEM ON THE WAY
HOME TO SEE HOW MANY HAD BAR
GRAPHS.
I THINK IT'S AN IMPORTANT ISSUE.
ONE OF THE EARLY CAREER
RESEARCHERS I HAD SHARED A STORY
HOW THEY HAD GONE TO A
CONFERENCE AND ORGANIZED A
SESSION ON EXPERIMENTS THAT
DIDN'T WORK AND IT DIDN'T WORK
OUT AND WE WOULD NEVER DO IT
AGAIN.
HERE'S WHAT WE LEARN FROM THE
EXPERIENCE AND IT WAS HUGELY
POPULAR THEN THE ORGANIZERS
WOULD NEVER ALLOW EM TO DO IT
AGAIN.
-- THEM TO DO IT AGAIN.
I THINK CREATING A SPACE FOR
LESS PRETTY SCIENCE AND
UNSUCCESSFUL THINGS IN
CONFERENCES WOULD GO A LONG WAY
BECAUSE IF YOU SEE EVERYONE ELSE
PRESENTING BEAUTIFUL STORIES YOU
FEEL YOU HAVE TO HAVE ONE TOO.
THIS ALSO GOES TO HOW I THINK
ABOUT BAR GRAPHS.
PEOPLE DON'T WANT TO SWITCH TO
DOT PLOTS BECAUSE THEY FEEL
EVERYONE ELSE IS PERFECT AND
EVERYONE'S DATA IS A MESS BUT
YOU'RE JUST NOT SEEING AND IF WE
WERE SEEING EVERYONE'S MESS
YOU'D KNOW THAT AND WOULD HAVE A
DIFFERENT THRESHOLD FOR WHAT
SCIENCE LOOKS LIKE AND SO YEAH I
THINK CREATING A SPACE AT
CONFERENCES FOR THAT AND FINDING
CREATIVE WAYS TO DO IT WOULD BE
VERY VALUABLE.
>> I LOVE THAT COMMENT I CAN
LINK A LOT OF THINGS WE HEARD
HERE TO THE ANTI-TED TALK AND
WHAT THAT WOULD BE LIKE.
LET'S HEAR FROM STEVEN.
>> I WANT TO ASK A PRACTICAL
QUESTION AND BRING IT BACK TO
RESEARCH RIGOR.
MOST FOCUS ON ISSUES AROUND DATA
MANAGEMENT AND STATISTICAL
ANALYSIS WHICH IS ALMOST
AGNOSTIC TO FIELD THOUGH THE
APPLICATION IS SPECIFIC AND WHEN
WE GET TO RIGOR IT GOES TO
RANDOMIZATION AND WHERE YOU USE
INTERNAL REPLICATION AND
VALIDATE WITH MULTIPLE LABS.
IT'S DIFFICULT TO TEACH AND
SECOND BECAUSE OF THE T32
REQUIREMENTS WE GET STUDENTS
REQUIRED TO TAKE THESE COURSES
AND WE'RE GETTING NEUROSCIENCE
AND CHEMISTS AND CANCER
BIOLOGISTS AND THEY'RE STUDYING
VERY DIFFERENT PHENOMENON WITH
EXTREMELY DIVERSE AND
HETEROGENEOUS EXPERIMENTAL
TECHNIQUES WHERE THE RIGOR IS
QUITE DIFFERENT AND MAY NOT BE
RELEVANT TO SOMEONE USING
ANIMALS THERE'S A WHOLE RANGE OF
THINGS AND SOMEONE USING PLATES
OR WESTERN BLOTS OR YOU NAME IT
IT'S A DIFFERENCE SUITE OF
THINGS.
AND IF WE'RE FOCUSSING ON RIGOR
EXPERIMENTAL DESIGN ISSUES AND
NOT ANALYSIS AND DATA MANAGEMENT
THEY'RE ALIGNED BUT NOT EXACTLY
THE SAME, HOW DO WE BRIDGE OR
TAKE INTO THE CONFIDENT THE
HETEROGENEITY IN THE LAB
SCIENCES?
>> I'VE BEEN TRYING TO KEEP US
ON TIME AND WE'RE GOING GOOD.
>> THE COMPONENTS OF SCIENTIFIC
RIGOR COMPRISE DESIGN AND LOGIC
AND STATISTICS THEY CANNOT BE
SEPARATED.
FOR YOUR COMPLETE QUESTION.
IN MY VIEW AND IN OUR EXPERIENCE
WE STARTED WITH WHAT I MENTIONED
EARLIER WITH SEEING HOW THINGS
CAN GO WRONG.
YOU MAY REMEMBER FROM THE SAME
INSTITUTION I WAS WE HAVE A WIDE
VARIETY OF PEOPLE FROM DIFFERENT
DISCIPLINES AND WE HAVE PEOPLE
FROM DIFFERENT FIELD WHO HAVE
DIFFERENT RESEARCH DESIGN
PROBLEMS.
BASICALLY WE TALK ABOUT THE MAIN
BASIC ISSUES AND BRING THEM
TOGETHER THIS IS WHERE SCIENCE
CAN GO WRONG ON THE AMENDMENTAL
LEVEL AND NOW LET'S LOOK TO THE
RESPECT DISCIPLINES AND FIELD
AND HOW THE PROBLEM LOOKS IN
YOUR RESEARCH REALITY AND DO
YOUR BEST TO TELL THE OTHERS IN
UNDERSTANDABLE WORDS WHAT YOUR
PROBLEM IS WITH THAT ASPECT OF
RIGOR THEN IN PRACTICE WE TRIED
TO COLLABORATE CLOSELY WITH THE
P.I.s AND I THINK TRACEY HAD A
WONDERFUL LINE IN YOUR ACTION
ST
STEPS IN MAKING IT A REQUIRED
COMPONENT AND WHAT WE NEED TO DO
GOING FORWARD WE NEED TO
COLLABORATE.
EDUCATORS AND THE MENTOR IN THE
LAB TO MAKE SURE THESE
PARTICULAR PRINCIPLES ARE BEING
INTEGRATED AND WE CAN FOLLOW-UP
CONNECTING THE COURSE WORK TO
GET STARTED.
I THINK THAT'S A GOOD START.
>> WE HAVE A NUMBER OF
TECHNIQUES WE USED TO DEAL WITH
IN OUR COURSE.
FIRST OF ALL IT'S REALLY
IMPORTANT TO HAVE DIFFERENT
FIELD AND TO BE INTERACTING WITH
DIFFERENT FIELD AS AN INSTRUCTOR
SPECIFICALLY WHEN PEOPLE DO
APPLIED WORK BECAUSE THE
FEEDBACK FROM STUDENTS WILL TELL
YOU WHERE WE HAVE GABZ AND WHERE
THINGS AREN'T WORKING AND WILL
HAVE LEGITIMATE REASONS WHY IT
WILL NOT WORK AND SOMETIMES YOU
CAN HELP PROBLEM SOLVE AND IT
WILL MAKE YOU A BETTER TEACHER
FOR FUTURE STUDENTS.
FIRST WE TRY TO MAKE SURE OUR
ADVERTISEMENTS ARE AS CLEAR AS
POSSIBLE SO PEOPLE KNOW WHAT
SKILLS WE'RE DISCUSSING AND
GAUGE WHETHER A COURSE IS
APPROPRIATE FOR THEIR EXPERTISE.
SECOND WE TEAM TEACH TO HAVE
PEOPLE WITH DIFFERENT EXPERTISE
WITHIN OUR INSTRUCTOR PANEL.
I HAD MORE EXPERIENCE IN SMALL
SAMPLE SIZE HUMAN STUDIES.
THE PERSON I CO-TEACH WITH HAS
MORE EXPERIENCE WITH ANIMAL AND
IN VITRO WORK AND WE ARE HONEST
ABOUT OUR LIMITATIONS.
WE'LL SIMPLY TELL THEM WE CANNOT
BE EXPERTS IN ALL THE THINGS YOU
DO AND THAT'S OBVIOUS TO THEM
BECAUSE THEY REALIZE HOW BROAD
THEIR EXPERTISE AND EXPERIENCES
ARE.
THE NEXT THING IS WHEN WE HAVE
MODULES LIKE DATA VISUALIZATION
WHERE WHAT YOU NEED DEPENDS ON
WHAT YOU'RE DOING.
WE MIGHT GIVE THEM THREE HOURS
OF LECTURE CONTENT AND SAY IF
YOU NEED WESTERN BLOTS GO TO
THIS TIME STAMP.
IF YOU NEED IMAGE-BASED FIGURES
GO TO THIS TIME STAMP AND IF YOU
NEED IT AVAILABLE FOR A BROADER
AUDIENCE GO TO THIS TIME STAMP
SO PEOPLE CAN EASILY NAVIGATE
AND FIND TOOLS RELEVANT TO THEM
AND THE PEER TEACHING IS
IMPORTANT BECAUSE SOMETIMES
OTHER PEOPLE IN THE COURSE HAVE
THE ANSWERS.
THEY'RE NOT BLANK SLATES.
PEOPLE WHO ENROLL IN THESE
PROGRAMS OFTEN HAVE EXPERIENCE
FROM OTHER PLACES AND WE HAVE
HAD TIMES WHERE THE TWO OF US AS
INSTRUCTORS ARE LIKE THAT'S AN
EXCELLENT QUESTION, NO CLUE.
AND TWO OR THREE OTHER STUDENTS
HAVE THE ANSWERS AND HAVE A GOOD
DISCUSSION AMONGST THEMSELVES
AND THOSE ARE ALL THE STRATEGIES
WE USE TO ADDRESS THE SCENARIOS
AND AGAIN VALUABLE FOR STUDENTS
TO HAVE AN UNDERSTANDING HOW
THINGS APPLY BEYOND THEIR OWN
RESEARCH FIELD AND IN OTHER
AREAS THEY MIGHT ENCOUNTER IN
THE FUTURE.
IT ENHANCES THE LEARNING
EXPERIENCE FOR EVERYONE.
IT'S IMPORTANT TO EMBRACE THAT
AND NOT TO BE AFRAID OF IT AS AN
INSTRUCTOR.
>> FOR THOSE THAT DON'T KNOW THE
CARPENTRIES DOES NOT EXIST AT AN
UNIVERSITY OR SINGLE
INSTITUTION.
BY DEFINITION EVERYTHING THEY
BUILT IS BY CONSENSUS AND DON'T
HAVE A DEPARTMENT OR PROFESSOR
FROM ANY OF THE DISCIPLINES BUT
YET YOU'VE CREATED MULTIPLE
CURRICULA THAT ADDRESS THE DATA
NEEDS OF THAT PARTICULAR
DISCIPLINE AND I THINK OUR
PROBLEM IS SIMILAR WHERE THERE'S
RESEARCH RIGOR NEEDS THAT ARE
DISCIPLINE SPECIFIC AND AT THE
SAME TIME IT MAY BE A BIT OF A
FAILURE AS THEY AGREE TO THEIR
OWN SEPARATE AND CAN BE SPECIFIC
TO WHAT THEY WOULD WANT.
COULD YOU SPEAK TO HOW THAT AND
THEN WE HAVE OTHERS TO GET US TO
THE WRAP-UP OF THE SESSION.
>> THAT'S WHERE COMMUNITY
BUILDING IS EXTREMELY IMPORTANT
FOR US.
AS JASON SAID, WE'RE A STAFF OF
PEOPLE THAT ALL WORK REMOTELY
AND I'M AN ENGINEER.
I'M NOT A SCIENTIST.
I DON'T HAVE THE SUBJECT MATTER
EXPERTISE IN THE VARIOUS TOPICS
BUT WE HAVE COMMUNITY MEMBERS
WHO COME TO OUR WORK SHOPS TO
LEARN ABOUT REPRODUCIBILITY AND
LEARN ABOUT BASIC SOFTWARE
DEVELOPMENT AND ACKNOWLEDGE
OTHER PEOPLE IN THE SAME
DISCIPLINE AND BUILD THE
INFRASTRUCTURE AROUND THE
PRINCIPLES THEY LEARN FROM
CARPENTRIES WORK SHOPS.
BEING PART OF THE SUBCOMMUNITY
HAS PROVEN VALUABLE.
THEY GO ON TO LAUNCH CURRICULUM
ADVISORY PROGRAMS AROUND LESSONS
FOR GENOMICS AND GEOSPATIAL.
IT'S ALL ABOUT CREATING THE
INFRASTRUCTURE WHERE PEOPLE
WOULD LIKE TO COME TOGETHER AND
CREATE SOMETHING BIGGER BASED ON
WHAT THEY'VE LEARNED IN THOSE
FOUNDATIONAL WORK SHOPS WITH US.
>> I'M THE REPRODUCIBILITY
LIBRARIAN AT THE UNIVERSITY OF
FLORIDA AND IN MY PAST AS A
RESEARCHER ONE AREA WAS COMPLEX
SYSTEMS.
I WANT TO THANK KAREN FOR
BRINGING THAT UP.
-- THE IMPORTANCE OF SYSTEMS
THINKING.
AND I THINK EVERYONE IN THE ROOM
I THINK APPRECIATES YOUR
PRESENTATION AND HOW WE COULD DO
REPRODUCIBILITY TRAINING.
THERE'S A BIG SYSTEMIC ISSUE OF
HOW WE ARE SUPPORTING THAT
TRAINING BECAUSE I AM IN THE
FORTUNATE POSITION OF HAVING A
FULL-TIME JOB WHERE I GET TO
FOCUS ON THAT BUT IT SEEMS
PROBLEMATIC TO ME THOUGH I LOVE
THE CARPENTRY'S INVOLVEMENT AND
I'M AN INSTRUCTOR AND
INSTRUCTOR-TRAINER AND YOU TEACH
ESSENTIAL DATA AND COMPUTATIONAL
SKILLS AND YOUR WORKSHOP
VOLUNTEER LED.
WHY ARE THE INSTITUTIONS NOT
COVERING THAT ESSENTIAL NEED?
I THINK IT'S A MESSED UP BROKEN
PART OF OUR SYSTEM AND KIND OF
AMAZING IT HASN'T YET BEEN
SWALLOWED UP BY ALSOVIR, FOR
INSTANCE.
>> I AGREE.
>> TO CONTINUE ALONG THE
QUESTION STEVE BROUGHT UP AND WE
WERE DISCUSSION THIS OVER LUX.
IN DINKING ABOUT THE DIFFERENT
DISCIPLINES I THINK THERE'S ONE
THING IN COMMON WITH ALL THE
DISCIPLINES THERE'S ALWAYS A
VERTEBRATE ANIMAL IN THE ROOM
THERE'S THE RESEARCHER AND'S
EASY TO THINK OF WHAT TO FOCUS
ON FIRST BECAUSE RISK OF
UNCONSCIOUS BIAS AND THERE'S A
HUGE CONTRIBUTION TO THE ISSUES
AS MY ESTEEMED COLLEAGUES
SITTING NEXT TO ME HERE HAS
POINTED OUT MANY MANY TIMES.
A COMMON ELEMENT WOULD BE GREAT
PLACE TO START AND WE'D MAKE A
LOT OF PROGRESS IF WE ADDRESSED
RISK OF BIAS AND GET THE
COMMUNITY TO DO TO BRING ABOUT
CHANGE.
>> MAYBE I COULD CALL OUT RUSS
ON THE COMMENT TO COME BACK TO
WHAT YOU SAID.
ONE THING I HEARD THE NEED TO
UNBURDEN THE INDIVIDUAL AND IF
THE ACTIONS OF PEOPLE AND IF WE
CAN HAVE STRUCTURES NOT
EXPLOITIVE ONES AND SO IN
INSTRUCTORS CAN BE PLACED IN A
POSITION TO BE MORE RIGOROUS
BECAUSE THEY DON'T HAVE TO THINK
THROUGH EVERY SINGLE THING.
THERE'S STANDARDS AVAILABLE AND
TAGE JUR DATA COULD -- PERSONAL
BIAS AS AN ELEMENT.
ONE THING WE HAVE TO FIGURE HOW
TO ADDRESS THE COMMON ELEMENT OF
PERSONAL BIASES THAT STAND IN
THE RAY OF DEVELOPING RIGOROUS
PRACTICE.
>> INTERESTING.
WE BRING BIAS TO EVERYTHING WE
DO.
PART OF THE ISSUE IS TRYING TO
SCULPT THE BIASES OF THE FIELD,
I THINK, SO WE CAN BE BIASSED IN
A WAY THAT MEETS WITH THE THINGS
WE WANT TO INCENTIVIZE.
I THINK A LOT OF IT DOES COME
DOWN TO MAKING IT EASY FOR
PEOPLE TO DO THE RIGHT THING.
THERE'S ANOTHER -- I DON'T KNOW
IF YOU THINK OF THIS AS THE BIAS
BUT WHAT CAME TO MIND IN THE
DISCUSSION WAS WHEN YOU THINK OF
BEHAVIOR CHANGE IN MORE SENIOR
RESEARCHERS WHAT IT TAKES FOR
SOMEBODY TO SAY WE NEED TO
CHANGE WHAT WE'RE DOING IS
SOMETHING WAS WRONG WITH THE
RESEARCH I BECAME FAMOUS OR
DOING AND THERE'S A LOT OF THE
COGNITIVE DISSIDENCE THAT CAN GO
ON AND FOR SENIOR RESEARCHERS
FIGURING OUT HOW TO RANK THE
WATERS OF CHANGING WITHOUT
NECESSARILY HAVING TO EVERYTHING
YOU DID WAS PROBLEMATIC.
I DON'T KNOW WHAT THE RIGHT WAY
IS TO DO THAT BUT I THINK THAT'S
AN IMPORTANT FEATURE OF THIS.
>> I'LL LEAVE THE THOUGHT AND
MIND AS WE WRAP THIS UP.
IT'S HARD TO CONVINCE SOMEBODY
ON SOMETHING IF THEIR JOB
DEPENDS ON NOT UNDERSTANDING
WHAT YOU'RE TRYING TO
COMMUNICATE.
I SEE WALKING UP TO THE STAGE
AND WE'LL SET THE STAGE TO WE
KNOW WHAT TO DO AND WHAT THE
NUMBERS ANDERS ON THE BACK OF
OUR BADGES MEAN.
THANK YOU ONCE AGAIN TO OUR
PANELISTS AND SPEAKERS.
THANK YOU.
>> THANK YOU, JASON FOR
EXCELLENT MODERATION.
SO NOW THE FUN CONTINUES.
WE'RE GOING TRY TO DO BREAKOUT
SESSIONS
ALL RIGHT.
WE HAVE REACHED THE END OF THE
FIRST DAY OF THE WORKSHOP.
I HAVE THE -- FIND MYSELF IN THE
UNFORTUNATE POSITION TO BE
BETWEEN YOU AND ESCAPING THE
CAMPUS FOR ENJOYING A MEAL OR
BEVERAGE SO I WILL BE FAST.
AND I JUST WANT TO SHARE A FEW
SORT OF VERY HIGH LEVEL THOUGHTS
THAT I KEPT FROM TODAY.
IT'S NOT I'M NOT ATTEMPTING TO
ACTUALLY SUMMARIZE THE DAY.
THIS WOULD BE A FOOL'S ERRAND
BUT SOME OF THE THINGS THAT
WE'VE HEARD TODAY AND I THINK
MIGHT GIVE YOU FOOD FOR THOUGHT
COMING INTO TOMORROW --
INCREASING RIGOR IS ABOUT
CHANGING NORMS AND BEHAVIORS.
THAT IS WHAT WE'VE HEARD.
IT'S NOT A SIMPLE CONTAGION.
IT'S COMPLEX.
IT HAS A SOCIAL ELEMENT.
IT FACES RESISTANCE AND --
INFLUENCES.
IT'S IMPORTANT TO APPROACH THIS
WAS A COORDINATED ATTITUDE.
THE IMPORTANCE OF BUYING AND
ENGAGEMENT AND COMMUNITIES AND
COMMUNITIES OF PRACTICE.
IN TERMS OF ENGAGEMENT WE'VE
HEARD THAT INDIVIDUALS CAN HAVE
BE A VERY GOOD ENTRY POINT FOR
INDIVIDUALS TO HAVE AN IMMEDIATE
BENEFIT TO REFLECT UPON TO GET
ENGAGE WE HAD THAT BUT THEN
THERE IS THE WHOLE ASPECT OF THE
COMMUNITY AND IN CONTINUING THIS
ENGAGEMENT THAT IS IMPORTANT.
WE'VE HEARD TODAY IN VARIOUS
FORMS THE IMPORTANCE OF FOCUSING
ON APPEALING TO VALUES.
AND I THINK IT'S TRUE FOR
INSTITUTIONS BUT IT'S ALSO TRUE
FOR INDIVIDUALS.
WHO IN THIS ROOM GOT INTO
SCIENCE SO THAT THEY COULD
PUBLISH IN SOME SPECIFIC
JOURNALS.
[ LAUGHTER ]
I MEAN THE POINT IS REALLY I
THINK IT IS -- AND WE ARE
GETTING INTO SCIENCE BECAUSE WE
ARE BELIEVING I THINK IN THE
VALUES AND THE PHILOSOPHY OF
SCIENCE AND THE NORMS OF SCIENCE
AND WE DID HEAR VERY IMPORTANT
MESSAGES ABOUT THE IMPORTANCE OF
FRAMING OF THESE MESSAGES AS
WELL.
APPEALING TO THE VALUE
FRAMEWORKS BUT ALLOWING THE BEST
PRACTICES TO BE DEFINED BY THE
PRACTITIONERS AND NOT BY
EXTERNAL INFLUENCERS.
I'M GOING TO QUOTE MY FAVORITE
SCOTTISH THINKER.
HOW IS YOUR FIELD REWARDING?
BEST SCIENCE, EVOLUTION, NOT
REVOLUTION AND AUDITING FOR
IMPROVEMENT AND NOT
ACCOUNTABILITY.
WE ALSO HEARD FROM RUSS WHO
JOINED US REMOTELY TO BE AWARE
OF THE OPEN SCIENCE ZEALOTS AND
AS A SELF-DEFINED OPEN SCIENCE
ZEALOT I NEED TO RESPOND.
OPEN SCIENCE IS EXTREMELY
IMPORTANT BECAUSE IT'S A
FRAMEWORK OF VALUES.
THE OPENNESS IS ACTUALLY WHAT
DRIVES THE AVAILABILITY OF ALL
OF THE UNDER PINNINGS OF SCIENCE
TO BE CRITICIZED.
IT ALLOWS ORGANIZED SKEPTICISM
AND ALL OF THESE VALUES AND
THESE NORMS THAT ARE IMPORTANT
TO SCIENCE.
BUT STEVE STARTED TODAY AND HE
IS RIGHT IS THAT OPENNESS IN
ITSELF IS NOT SUFFICIENT AND THE
RIGOR DEFINITIONS ACTUALLY MUCH
MORE COMPLEX.
AND SO I THINK ALL OF THAT AND
IT'S DISCIPLINE SPECIFIC.
SO I THINK THAT IT'S VERY
COMPLEX SITUATION THAT WE HAVE
HAVE AHEAD OF US.
I THINK OPENNESS IS NUMBER ONE.
THE PREREQUISITE BUT IT'S NOT
THAT WE SHOULD STOP THERE.
I'M GOING TO LEAVE YOU WITH A
FEW OPEN QUESTIONS TO REFLECT
OVERNIGHT.
ONE THAT I HEARD A LOT OF THE
TENSE TODAY SORT OF WHAT IS THE
OPTIMAL BALANCE BETWEEN TOP DOWN
AND BOTTOM UP.
WE NEED BOTH AND WHERE AND HOW
CAN PEOPLE WHO ARE DRIVING THESE
BOTTOM UP APPROACHES?
AND PEOPLE AT THE TOP WHO'S
WILLING TO SUPPORT THEM.
HOW DO THEY INTERACT.
WHAT IS THE OPTIMAL BALANCE.
AND THE OTHER QUESTION THAT I
THINK IS VERY IMPORTANT IS THE
BALANCE OF THE VALUE FRAMEWORK
AND THE SPECIFIC BEHAVIORS AND
PRACTICEs THAT ARE NEEDED IN
THE FIELD AND WHAT IS AGAIN THIS
OPTIMAL VALUE?
THE OTHER THING THAT I WOULD
LIKE TO ASK YOU TO REFLECT UPON
AND COME TOMORROW WITH IS
BASICALLY WHAT WHAT IS -- WHAT
IS IT GOING TO TAKE FOR US TO
COME OUT OF THIS ROOM AND
ACTUALLY HAVE ACTIONABLE THINGS
TO THINK B. WHAT IS THE NEXT
STEP FROM THIS CONVERSATION?
WE HAVE AN INCREDIBLE RICHNESS
AND DIVERSITY OF PERSPECTIVE IN
THE ROOM.
HOW DO WE BUILD ON THAT?
HOW DO WE GET ACTIONABLE
INSIGHTS FROM THE DISCUSSION.
WE HEARD A LOT OF VERY POSITIVE
THINGS.
A LOT BEFORE TRAINING AND THINGS
THAT ARE ACTUALLY TACTICS AND
APPROACHES TO TRAINING THAT
WORK.
WE'VE HEARD THAT THERE ARE
MOTIVATIONAL ASPECT THAT'S WORK.
HOW DO WE MAKE THEM WORK?
WHAT IS IT GOING TO TAKE FOR
THIS TO BE A COMMUNITY OF
PRACTICE AND FOR THOSE CHAMPIONS
TO REALLY CONTINUE -- YOU ARE
THE CHAMPIONS.
WHAT ARE YOU GOING TO NEED TO
CONTINUE THIS WORK STARTING THE
DAY AFTER TOMORROW.
SO I WOULD ENCOURAGE TO YOU
REFLECT ON THAT OVERNIGHT AND WE
COME BACK TOMORROW REFRESHED AND
A LITTLE LESS JET LAGGED THAN
TODAY AND WE'LL CONTINUE THIS
CONVERSATION AND WE'LL WRAP UP
WHAT WE HEARD IN THE BREAK OUT
SESSIONS TOMORROW MORNING AND
WE'LL START THE DAY AGAIN.
THANK YOU VERY MUCH FOR THE
INCREDIBLE RICHNESS AND
IMPORTANCE OF ALL OF THE
PERSPECTIVES WE HEARD TODAY AND
YOUR PARTICIPATION AND HAVE AN
ENJOYABLE EVENING AND WE'LL SEE
YOU AGAIN TOMORROW.