>> GOOD AFTERNOON. WELCOME. MY NAME IS JOHN GLOWA, I'M A CHOSEN SUBSTITUTION FOR THE USUAL PERSON THAT INTRODUCES PEOPLES AT THESE OBSSR LECTURE SERIES. I'M HERE TO INTRODUCE OUR SPEAKER WHO IS DR. JONATHAN SCHOOLER. HE'S A PROFESSOR AND -- IN THE DEPARTMENT OF O PSYCHOLOGICAL AND BRAIN SCIENCES AT THE UNIVERSITY OF CALIFORNIA, SANTA BARBARA. FORMERLY HELD ACADEMIC POSITIONS IN THE DEPARTMENT OF PSYCHOLOGY AT THE UNIVERSITY OF BRITISH COLUMBIA AND UNIVERSITY OF PITTSBURG. HE TOOK HIS BA IN HAMILTON COLLEGE EARLIER AN Ph.D. AT THE UNIVERSITY OF WASHINGTON. SINCE THAT TIME HE HELD A NUMBER OF GRANTS FROM FEDERAL AND PRIVATE FOUNDATIONS TO SAY A NUMBER OF CONCEPTS AROUND COGNITION, CONSCIOUSNESS AN MENTAL PROCESSES. I NOTED WITH ELFISH INTEREST HIS -- SELFISH INTEREST HIS INTEREST INCLUDES MINDFULNESS IN AREA OF O RESEARCH THAT I OVERSEE HERE AT NIH SO HE MAYBE ABLE TO HELP ME FIND A BIOMARKER FOR SOME ASPECTS OF THAT. SO HE'S ON A NUMBER OF EDITORIAL ADVISORY BOARDS, PUBLISHED OVER A HUNDRED PAPERS AND RESPECTED JOURNALS BUT HIS CITATION OF MEDIA COVERAGE CAUGHT MY ATTENTION. ARTICLES LIKE THAT IN THE WASHINGTON POST ENTITLED TOO MUCH HAPPINESS CAN MAKE YOU UNHAPPY. OR COSMIC HABITUATION OR THE EUREKA HUNT. AND THE NEUROSCIENCE -- IS NEUROSCIENCE THE DEATH OF FREE WILL? PERHAPS INAPPROPRIATE FOR THIS BUILDING. THAT I ILLUSTRATE JOURNALISTS MADE HIM A CENTER OF ATTENTION. IN FACT ONE OF THE ARTICLES, THE TRUTH WEARS OFF, IN THE NEW YORKER, AND ANOTHER RECENT SPOT IN NATURE WHICH I BROUGHT WITH ME IF ANYONE WANTS TO SEE IT, KIND OF FORE SPOKE THE TOPIC OF THIS TALK AND I'M GOING TO DECLINE TO ATTEMPT TO STEAL HIS THUNDER ALONG THOSE LINES AN INSTEAD JUST WELCOME JONATHAN. PLEASE TELL US MORE ABOUT THE DECLINE EFFECT. [APPLAUSE] >> THANK YOU. IT'S A REAL PLEASURE TO BE HERE AND HONOR TO SPEAK TO YOU. I HAVE TOING A KNOW LEDGE -- ACKNOWLEDGE AT THE BEGINNING THAT I'M SPEAKING WITH SOME TREPIDATION ABOUT THIS BECAUSE FOR A VARIETY OF REASONS, ONE IS THAT IN SOME WAYS DECLINE EFFECT CHALLENGES SCIENCE AND CHALLENGES THE WAY WE THINK ABOUT SCIENCE IN SOME RATHER FUNDAMENTAL WAYS. AND WHEN ALWAYS FEELS A LITTLE UNCOMFORTABLE CHALLENGING THE MAINSTREAM. IN ADDITION, I'M GOING TO ALLUDE TO SOME VERY CONTROVERSIAL RESEARCH T KIND OF RESEARCH THAT MANY OF YOU MAY BE STARTLED TO SEE ME EVEN TALKING ABOUT LET ALONE TAKING SO MUCH SERIOUSLY. SO FOR THOSE REASONS I ACKNOWLEDGE THE TREPIDATION. BUT I THINK AND I EVEN CONSIDERED NOT TALKING ABOUT SOME OF THESE LINES OF RESEARCH BUT I DO THINK THAT IT'S APPROPRIATE TO AT LEAST SEE THEM IN THE CONTEXT OF THE LARGER ISSUE THAT I'M GOING TO BE TALKING ABOUT TODAY. SO WITH NO FURTHER ADIEU LET'S MOVE ON TO SEE WRITE I'M GETTING AT HERE. SO LET ME BEGIN WITH A QUOTE FROM QEULLO WHO WROTE THE POPULAR BOOK THE AL CHEMIST WHICH SAYS EVERY SEARCH BEGINS WITH BEINGERS LUCK AND ENDS WITH THE VICTOR BEING SEVERELY TESTED. THIS HAS BEEN MY EXPERIENCE IN SCIENCE IN A WHOLE VARIETY OF DIFFERENT CONTACTS WHERE I START OFF IN A NEW AREA AND GET SOME REALLY REMARKABLE RESULTS WHICH BECOME THEN INCREASINGLY DIFFICULT TO REPLICATE ENTIRELY. USUALLY I SEE THEM BUT THEY'RE NOT AS STRONG AS WHEN I FIRST STARTED OFF. TURNS OUT THIS EXPERIENCE I HAVE HAD PERSONALLY, AND I'LL FLESH IT OUT MORE SOON, HAS BEEN EXPLORED IN A VARIETY OF DIFFERENT DOMAINS, AND IT SEEMS TO BE A UBIQUITOUS ASPECT OF RESEARCH THAT INITIAL ATTEMPTS ARE ASSOCIATED WITH THE LEVEL SUCCESS THAT IS HARD TO REPLICATE THAT'S THE THESIS THAT I WANT TO EXPLORE WITH YOU TODAY. BUT I AGAIN AS I WARNED YOU, IT WAS FIRST -- THE TERM WAS COINED AND FIRST IDENTIFIED IN THE RATHER CONTROVERSIAL AREA OF PARAPSYCHOLOGY. O RYAN, WHO FAMOUSLY HAD THIS PARAPSYCHOLOGY LAB IN DUKE WAS STUDYING A CLAIRVOYANT THAT HAD ABSOLUTELY SORT OF REMARKABLE STATISTICALLY SIGNIFICANT SOME INCREDIBLE LEVEL WITH THIS INDIVIDUAL AND THEN AS HE KEPT RUNNING THE EFFECT GOT SMALLER AND HE COINED THE TERM DECLINE EFFECT AND DEAN GRADEN, PERHAPS ONE OF THE BEST KNOWN PARAPSYCHOLOGISTS WROTE IN HIS BOOK ENTANGLED MINDS, A FREQUENT OBSERVATION WHEN A NEW EXPERIMENT IS CONDUCTED THE OUTCOMES ARE SUCCESSFUL THEN OTHERS TRY TO REPLICATE THE EFFECT BEGINS TO FADE. AND THIS BASIC IDEA WAS POPULARIZED BY JOANA LEHRER IN A RECENT NEW YORKER, ARTICLE CALLED THE TRUTH WEARS OFF. IN THIS THIS ACTUALLY ARTICLE CAME FROM A DISCUSSION HE HAD WITH ME IN WHICH HE WAS INTERVIEWING MY MORE MAINSTREAM RESEARCH ON MIND WANDERING AND CREATIVITY AND I JUST HAPPENED TO MENTION TO HIM THAT I HAD THIS EXPERIENCE OF DECLINE EFFECT AND I OBSERVED IT IN A NUMBER OF AREAS AND HE BECAME INTERESTED IN THIS AND LOOKED INTO THE AREAS THAT I TOLD HIM ABOUT AND FOUND INDEED THERE WERE STRIKING EXAMPLES OF DECLINE EFFECT AND FOUNDED IN OTHER AREAS AS WELL SO CERTAINLY INTERESTING COLLABORATION BETWEEN SCIENCE AND THE MEDIA. BUT AS A CONSEQUENCE OF THIS, THE ARTICLE ACTUALLY WENT VIRAL, IT WAS THE NUMBER ONE MOST EMAILED NEW YORKER ARTICLE FOR SEVERAL MONTHS. I WAS ACTUALLY WARNED BEFORE LEHRER WROTE IT THAT IF I ALLOWED HIM TO INTERVIEW ME AND IF I ACKNOWLEDGED SOME OF THESE CONCERNS THAT IT WOULD BE DO IN DO IN MY CAREER AND STILL MAY BUT NOT SO FAR. IN FACT, AS A CONSEQUENCE OF THIS ARTICLE I WAS INVITED TO -- BY NATURE TO WRITE A PIECE DESCRIBING THE DECLINE EFFECT. SO SO FAR I HAVEN'T PAID THE PRICE BUT TIME WILL TELL. LET ME TELL YOU SOME OF MY PERSONAL EXPERIENCES WITH THE DECLINE EFFECT. IT MAYBE THAT I'M JUST UNIQUELY LUCKY OR UNLUCKY, IT'S A PECULIAR MIX OF LUCK AND LACK OF LUCK BECAUSE YOU HIT IT FIRST THEN YOU YOU HAVE A HARD TIME GETTING IT AGAIN. BUT MY ORIGINAL DISSERTATION WAS KNOWN AS A VERBAL OVERSHADOWING WHERE IT'S KIND OF A FUN FINDING, PEOPLE WITNESSED A VIDEOTAPE OF A BANK ROBBERY AND THEN ATTEMPTED TO DESCRIBE IN AS MUCH DETAIL AS POSSIBLE. THEN THEY WERE GIVEN A LIONUP INCLUDE -- LINEUP INCLUDING A DIFFERENCE PICTURE OF THE PERPETRATOR AND SEVERAL SEVERAL DISTRACTERS, VERBAL DESCRIPTION ACTUALLY IMPAIRED PERFORMANCE QUITE SUBSTANTIALLY. THIS EFFECT WE REPLICATED AND ENDED UP WITH SIX EXPERIMENTS SERIES OF REPLICATIONS AND OTHERS REPLICATED IT BUT IT BECAME INCREASINGLY DIFFICULT TO REPLICATE. I FINE IT HARD TO REPLICATE THIS BASIC EFFECT THAT WAS SO EASY TO GET AT FIRST. I EXPERIENCED THIS WITH A HOST OF OTHER PARADIGMS AS WELL. PERHAPS I'M UNIQUELY LUCKY OR UNLUCKY. WE HAVE GOTTEN IT WITH NOT ONLY FACES BUT ALSO COLOR, WE HAVE A ROBUST EFFECT, THAT WAS REPLICATED BUT THAT WAS DIFFICULT TO REPLICATE AGAIN WITH MUSIC, WITH MAPS, AND THEN RELATED PARADIGMS SUCH AS THING OUTED YOU WHILE SOLVING INSIDE PROBLEMS THAT DISCORRUPTED PERFORMANCE BUT THAT'S PROVEN DIFFICULT TO REPLICATE. IMPLICIT LEARNING PARADIGM, WHEN PEOPLE DESCRIBE AN IMPLICIT RULE THEY LEARNED THAT INTERFERES WITH PERFORMANCE, AT LEAST IT DID BUT NOW NOT SO MUCH. SO I HAVE BEEN HAUNTED BY THIS EXPERIENCE OF GETTING EFFECTS, GETTING SEVERAL TIME, NOT JUST LIKE ONCE BUT THEN FINDING IT INNINGLY DIFFICULT TO REPLICATE. I -- HERE IS AGAIN WHERE I GO INTO THE MORE CONTROVERSIAL AREAS. I WAS EXPOSED TO AND INTRIGUED BY SOME RESEARCH THAT DARRYL BEM CONDUCTED AND DECIDED TO FOLLOW-UP ON THAT. SO THIS IS HAVING TO DO WITH REALLY REMARKABLE CLAIM THAT THE HOUR OF TIME MIGHT UNDER SOME SITUATIONS GO OPPOSITE DIRECTION. SO PHYSICISTS ACKNOWLEDGE NOTHING INHERENT IN THE LAW OF PHYSICS THAT PRECLUDE TIME FROM FUTURE TO PAST. BRIAN GREEN, A RESPECTED PHYSICIST THOUGH I DOUBT VERY MUCH HE WOULD SEE HIS -- THIS AS SUPPORTING IDEA PRE-COGNITION BUT HE SAYS THE LAWS OF PHYSICS ARTICULATED FROM NEWTON THROUGH MAXWELL AND I UP UNTIL TODAY SHOW SYMMETRY BETWEEN PAST AND FUTURE. NOWHERE IN THESE LAWS DO WE FINE STIPULATION TO APPLY ONE WAY IN TIME AND NOT THE OTHER O. THOUGH EXPERIENCE REVEALS OVER AND OVER AGAIN THERE'S AN ARROW OF HOW EVENTS FOLD OVER TIME BUT NOT FOUND IN THE FUNDAMENTAL LAWS OF PHYSICS NOT ONLY DO KNOW KNOWN LAWS FAIL TO TELL US TO SEE IN ONE ORDER BUT IN THEORY CAN UNFOLD IN REVERSE ORDER. TURNS OUT THERE IS A AREA OF RESEARCH KNOWN AS PRE-COGNITION WHICH IS FOUND IN A VARIETY OF DIFFERENT PARADIGMS THAT THERE SEEMS TO BE SOME EVIDENCE THAT PEOPLE CAN BE INFLUENCED BY EVENTS NOT YET TAKEN PLACE. THIS IS A META ANALYSIS IN THE JOURNAL OF PARAPSYCHOLOGY BUT NEVERTHELESS INVOLVED A FEW STUDIES, 309 STUDIES, 62 INVESTIGATOR, 2 MILLION INDIVIDUAL TRIALS. 50,000 SUBJECTS AND SMALLER LIABLE EFFECT SIZE WHICH REQUIRE AS FILE DRAWER, 30% STUDIES WERE SIGNIFICANT IMPORTANTLY IN THIS FIELD, THEY PUBLISH NON-SIGNIFICANT RESULTS. AND THERE WAS NO RELATIONSHIP BETWEEN THE QUALITY OF STUDY AN SIZE OF EFFECT. THIS STUDY CONCLUDED THERE WOULD HAVE TO BE 46 UNREPORTED STUDIES FOR EACH REPORTED STUDY FOR A FILE DRAWER TO ACCOUNT FOR IT. IN A MORE WELL RESPECTED JOURNAL AT LEAST FORMALLY RPED JOURNAL, SOME ARGUE IT LOST RESPECT AS A RESULT OF THIS PAPER BEING PUBLISHED, DARRYL BEM A WELL RESPECTED FORMERLY WELL RESPECTED RESEARCHER PUBLISHED A SERIES OF NINE STUDIES SHOWING WHAT APPEARED TO BE EVIDENCE OF PRE-COGNITION IN A VARIETY OF DIFFERENT PARADIGMS FOR EXAMPLE, IN ONE PARADIGM PEOPLE WERE EXPOSED TO HAD TO SAY HOW AROUSING THE STIMULI WERE AND PRESENTED AGAIN IT WAS FOUND LESS AROUSING. BOTH NOT JUST ONE BUT NINE EXPERIMENTS. SO BASED ON KNOWLEDGE OF THIS WE DECIDED TO TRY A VARIATION OF THIS PARADIGM FOR OURSELVES. WE DID IMPLICIT PRIMING PARADIGM IN WHICH WE BASICALLY THERE'S A STANDARD FINDING IN COGNITIVE PSYCHOLOGY, IF YOU SEE AM IMAGE AND SEE IT AGAIN, THE SECOND TIME YOU SEE IT'S IT'S MORE FLUID. YOU SEE IT MORE READILY THE SECOND TIME F. THAT'S O THE CASE AND THERE'S SOMETHING TO PRE-COGNITION YOU MIGHT EXPECT IF YOU WERE TO PRESENT A STIMULUS AND PRESENT IT AGAIN, THAT THE -- IF IT'S GOING TO BE PRESENTED A SECOND TIME THAT THAT WOULD ACTUALLY FACILITATE ORIGINAL PERCEPTION. SO IN THIS PARADIGM, A BRIEFLY FLASHED IMAGE, A NOISE MASK AGAIN AND THEY INDICATED WHETHER OR NOT THEY KNEW WHAT WAS PRESENTED AND THE CRITICAL MANIPULATION ON SOME CASES THE IMAGE WAS PRESENTED IN OTHER CASES IT WAS NOT. SO LIKE THIS FIXATION POINT, BRIEFLY FLASHED IMAGE, IF YOU KNOW WHAT IT WAS, PRESS THE UP ARROW, IF NOT PRESS THE DOWN ARROW AND IT'S PRESENTED AGAIN. OR SAME BASIC PARADIGM. BUT THERE'S NOTHING. AND OUR ORIGINAL RESULT WAS SMALL BUT SIGNIFICANT, THE PRIME STIMULI, THESE ARE STIMULI THAT ARE PRIMED IN FUTURE WERE LIGHTLY MORE LIKELY TO BE IDENTIFIED THAN UNPRIMED STIMULI THERE ARE COMPLICATIONS SHOWN TO CLEAN UP A FEW ISSUE, THE SAME RESULT. WE REPLICATED THIS EXPERIMENT ON THE ORDER OF 16 TIMES. AS YOU CAN SEE THIS IS THE DECLINE EFFECT. YOU'LL NOTICE HERE NOT JUST LIKE WE GOT IT AND THEN IT WENT AWAY. WHAT WE GET IS THIS LINEAR EFFECT SO WE GET A LINEAR EFFECT AND IF YOU LOOK AT THE FIRST OVERALL FIRST ENTIRE EFFECT IS SIGNIFICANT OVERALL 00017 LEVEL AND IF WE LOOK AT THE SECOND HALF IT'S STILL SIGNIFICANT POINT LESS THAN 05 SO WE'RE GETTING A RATHER MARKED LINEAR DECLINE EFFECT. THINGS LIKE PUBLICATION BIAS DON'T APPLY BECAUSE THEY WERE ALL RUN IN MY LAB. SO CONCLUSIONS, THE OVERALL AFFECT REMAINS SIGNIFICANT. BUT THERE WAS A MASSIVE DECLINE EFFECT AND EARLIER STUDIES HAD A SMALLER BUT ETCH WHEN YOU TAKE THAT INTO ACCOUNT YOU STILL HAVE A SIGNIFICANT DECLINE EFFECT. SO I'M NOT THE ONLY PERSON TO OBSERVE A DECLINE EFFECT IN PARAPSYCHOLOGY, THIS IS ANALYSIS DECLINE EFFECTS OVER YEARS IN DICE THROWING TO INFLUENCE THE OUTCOME OF DICE THROWS. AS YOU CAN SEE THERE'S A DECLINE EFFECT IN THAT. HERE IS A DECLINE EFFECT IN THE GUN CELLED EFFECT. THE ON CELLED EFFECT WAS WRITTEN UP ALSO BY DARRYL BEM IN PSYCHOLOGICAL BULLETIN RESPECTED JOURNAL IN WHICH IT'S BASICALLY A FORM OF THREPTHY WHERE ONE SEES AN IMAGE AND THE OTHER TRIES TO SEE WHAT THE IMAGE S. IT'S RELATIVELY CAREFULLY CONTROLLED. BUT WHAT YOU SEE HERE IS THAT THE MAGNITUDE OF THAT EFFECT DECLINED OVER TIME BETWEEN -- UP UNTIL 2001. THERE'S A CURIOUS THING ABOUT THE DECLINE EFFECT. SO HERE THEY DID ANALYSIS AN DEMONSTRATED DECLINE EFFECT. CAN CAN I ASK THE QUESTION WHAT WOULD HAPPEN YOU HAVE NOW DEMONSTRATED THE EFFECT, WHAT HAPPENS IF YOU SEE WHAT HAPPEN IT IS THE DECLINE EFFECT, DOES DECLINE EFFECT DECLINE? THERE'S A CURIOUS THING AFTER THEY FOUND THIS THERE'S ACTUALLY A 2010 ARTICLE IN PSYCH BULLETIN WHERE YOU SEE THE DECLINE EFFECT BUT IN A PE PECULIAR WAY THE DECLINE EFFECT DECLINES AND EFFECTS RETURNS THERE BEING OVERALL SIGNIFICANT EFFECT. BUT PARTICULAR DECLINE OF DECLINE EFFECT. AND AGAIN, THE SAME THING IN ANOTHER PARADIGM OF TELEKINESIS. SO CONCLUSIONS FROM RESEARCH IS THERE'S DECLINE EFFECTS IN A NUMBER OF DOMAINS. AFTER DECLINE EFFECTS ARE OBSERVED IN SOME DOE -- DO MAIN, IN SOME CASES RETURNS, SUGGESTS REMOTELY PLAUSIBLY THIS COULD BE DECLINE IN DECLINE EFFECT. IF THE DECLINE EFFECT WERE IN MY LAB AND PARAPSYCHOLOGY, I WILL NOT EXPECT YOU TO TAKE IT SERIOUSLY. AS I HOPE TO SHOW YOU THIS IS NOT JUST LIMITED TO ME AND THAT'S TEARIC --s SEW TEARIC FIELD OF SCIENCE BUT IT SEEMS PREVALENT IN A VARIETY OF OTHER DOMAINS AS WELL. P WHEN WE CONSIDER THE LESSON FROM PARAPSYCHOLOGY, ONE POSSIBILITY IS THAT THERE'S THIS ARTIFACT IN SCIENCE AND THAT THAT ARTIFACT IS ENTIRELY WHAT DRIVES WHAT'S GOING ON IN PARAPSYCHOLOGY SO YOU SEE THE DECLINE EFFECT, THE DECLINE EFFECT IS EMBLEMATIC OF ARTIFACT BUT THAT ARTIFACT ALSO PERMEATES MAINSTREAM SCIENCE AS I'LL HOPEFULLY PERSUADE YOU. THE OTHER TERM IS PARASIGH CHOG THERE'S SOMETHING TO IT AND THEY PICKED UP ON SOMETHING PREVALENT IN OTHER AREAS AS WELL. I THINK IT'S AN OPEN QUESTION WHICH ONE OF THOSE POSSIBILITIES IS THE CASE THOUGH I CERTAINLY APPRECIATE PEOPLE HAVE STRONG OPINIONS AB WHICH ONE IS MORE LIKELY. DECLINE EFFECTS IN CONVENTIONAL SCIENCE, INTO TALK AB THREE DIFFERENT AREAS OBSERVED. DRUG TREATMENTS, HIGHLY CITED MEDICAL INTERVENTIONS AND BUYOLOGY META ANALYSES. SO IS -- SOME OF YOU MAY EVEN KNOW MY MOTHER NINA SCHOOLER WHO WORKED AT NIH FOR MANY OR NIMH MANY YEARS AND HAS BEEN IN A NUMBER OF OTHER INSTITUTIONS, UNIVERSITY OF PITTSBURG FOR EXAMPLE. AND I WAS IN HER OFFICE ONE DAY AND SAW THIS ARTICLE WHICH SAID WHY DO THE EFFECTS OF ANTIPSYCHOTIC DRUGS DECLINE OVER TIME? I HAD NO IDEA IT THIS WAS AN ISSUE BUT THIS IS A META ANALYSIS BUT A NUMBER OF RESEARCHERS INCLUDING MY MOTHER NINA SCHOOLER WHICH BASICALLY SHOWS THE EFFICACY OF ANTIPSYCHOTIC DRUGS RELATIVE TO PLACEBO HAS DECLINED OVER TIME. THIS IS NOT THE ONLY PLACE IN WHICH DECLINE EFFECTS HAVE BEEN OBSERVED. BOB KAPLAN WHO UNFORTUNATELY WASN'T ABLE TO BE HERE TODAY WHO IS ASSOCIATE DIRECTOR IN THE NIH OFFICE OF BEHAVIORAL AN SOCIAL SCIENCES RESEARCH HAS LOOKED AT A NUMBER OF DIFFERENT TYPES OF MEDICAL TREATMENTS IN WHICH HE'S OBSERVED DECLINE EFFECTSCH THIS IS A DECLINE EFFECT IN STATIN TO TREAT CHOLESTEROL. HERE IS A DECLINE EFFECT IN A BETA BLOCKER. AND HERE IS A DECLINE EFFECT IN LA -- A TREATMENT FOR GLAW CO-MA'AM THAT'S NOT ALL -- GLAUCOMA. THAT'S NOT ALL. A VERY INSIGHTFUL ARTICLE LOOKED AT THE MOST HIGHLY CITED MEDICAL JOURNAL ARTICLES AND HE FOUND OF THE 49 HIGHLY CITED ORIGINAL CLINICAL RESEARCH STUDIES, 45 CLAIM THE INTERVENTION WAS EFFECTIVE. OF THESE, SEVEN WERE CONTRA INDICATED BY SUBSEQUENT STUDIES, ACTUALLY WENT THE OPPOSITE WAY. 7 SOUND EFFECTS STRONGER THAN DOSE OF SUBSEQUENT STUDIES, 44 WERE REPLICATED AND 24 REMAIN LARGELY UNCHALLENGED SO IN EFFECT OVER 40% WHICH A REPLICATION WAS ATTEMPTED, EITHER ASSOCIATED WITH DECLINED EFFECT SIZE OR OUTRIGHT CONTRA INDICATED SO THESE ARE THE MOST INFLUENTIAL STUDIES CONSIDERED THE BEST OF THE BEST OR AT LEAST THE MOST THE MOST HIGHLY REFERRED TO AND YOU SEE THIS RATHER MARKED DECLINE EFFECT. THAT'S NOT ALL. THIS IS A META ANALYSIS OF META ANALYSES BY (INDISCERNIBLE) AND MOLAR AND THEY FOUND THEY LOOKED AT 44 PEER REVIEWED META ANALYSES AND FOUND THERE WAS A SMALL BUT SIGNIFICANT DECLINE IN EFFECT SIZE WITH THE PUBLICATION AND THEY DID FIND„i THE FIRST STUDIES HAD SMALLER ENTHAN LATER STUDIES, THAT COULD BE PART OF THE EFFECT BUT EVEN WHEN YOU HAVE THAT TAKEN INTO ACCOUNT, WHEN YOU TAKE INTO THE ACCOUNT THE EFFECT SIZE YOU STILL SEE REDUCTION OVER TIME. THIS BASICALLY SUMMARIZES THE DATA HERE. HERE WE SEE DECLINE EFFECTS NOT ONLY IN MY LAB, NOT ONLY IN PARAPSYCHOLOGY BUT ALSO IN A VARIETY OF DOMAINS INCLUDING DRUGS SUCH AS ANTI-PSYCHOTICS CHOLESTEROL LOWERING DRUG, TREATMENT OF GLAUCOMA, MEDICAL INTERVENTIONS AND BUYOLOGY. AND THERE ARE ALSO OTHER ANALYSES GOING ON CURRENTLY JOHN CROSNIC AT STANFORD IS LOOKING TO PSYCHOLOGY FINDINGS AND ALSO FINDING LOTS OF DECLINE EFFECTS IN PSYCHOLOGY AS WELL. WHAT'S GOING ON HERE, THERE'S A NUMBER OF POSSIBLE REASONS FOR THE DECLINE EFFECT. I THINK LIKELY WHEN THE DUST SETTLES THE CONCLUSION WILL BE THAT THERE'S -- IT'S MULTIPLY DETERMINED THE IF THERE'S A NUMBER OF DIFFERENT THINGS THAT ARE GOING ON HERE. I WANT TO EXPLORE SOME OF THE DIFFERENT POSSIBLE EFFECTS. AND DO THIS OPEN MINDEDLY. (INDISCERNIBLE) ARGUMENT WAS INITIAL STUDIES HAVE SMALLER END. AND HE FOUND THAT AMONG RANDOMIZED STUDIES THAT WERE CONTRAINDICATED, THAT THEY WERE INITIALLY STRONGER EFFECT SIZES WERE SMALLER THAN REPLICATEED OR UP CHALLENGED STUDIES. SORRY. SO HE FOUND TWO DIFFERENT THINGS ONE THE INITIAL STUDIES HAD A SMALLER END AND THAT'S ONE POSSIBLY REASON TO GET A LARGER EFFECT SIZE BECAUSE THE LAW OF LARGE NUMBERS AND THEN IN ADDITION RAP DOCUMENTIZED STUDIES TENDED TO BE LATER ON AND NON-RANDOMIZED INITIALLY. BOTH LEAD TO LARGER EFFECT SIZES INITIALLY. HE ALSO FOUND NON-RANDOMIZED STUDIES ARE UNRELIABLE NOT SURPRISINGLY AND FIVE OF SIX HIGHLY CITED NON-RANDOMIZED STUDIES HAD BEEN CONTRAINDICATED SO CLEARLY TWO FACTORS THAT COULD BE INVOLVED ARE SMALL IN INITIAL STUDIES AN LACK OF RANDOMIZATION IN INITIAL STUDIES. HOWEVER, THAT DOESN'T SEEM TO EXPLAIN EVERYTHING. BECAUSE DECLINE EFFECTS ARE OBSERVED WHEN N IS CONTROLLED FOR AS I MENTIONED IN THE (INDISCERNIBLE) AND MILLER META ANALYSIS, ALSO DECLINE EFFECTS OBSERVED IN MANY DOMAINS WHERE THERE HAVE BEEN RAN DOCUMENT ASSIGNMENT SO THAT DOES NOT EXPLAIN THE DECLINING EFFECT SIZE. SO WHAT ARE OTHER CONVENTIONAL ACCOUNTS OF DECLINE EFFECT? THE MOST STRAIGHT FORWARD ACCOUNT IS REGRESSION O THE MEAN. FINDING YOU'RE GOING TO RUN A BUNCH OF STUDIES, SOMETIMES THE OTHER VARYIANS IS GOING TO BE IN THE SAME DIRECTION AS THE EFFECT SIZE AND WHEN THAT HAPPENS IT'S GOING TO EXAGGERATE THE EFFECT AND YOU'RE MORE LIKELY TO PUBLISH IT. IF IT GOES IN THE OPPOSITE DIRECTION YOU WON'T GET A SIGNIFICANT RESULT AND SO YOU WON'T PUBLISH ANYTHING AT ALL. SO IT MAKES SENSE THAT THERE WOULD BE BIAS IN WHICH STUDY (INDISCERNIBLE) AS THE ACTUAL EFFECT SIZE TO BE PUBLISHED. THIS IS SURELY PART OF THE STORY BUT IT DOESN'T EXPLAIN THE FACT WE SEE LINEAR DECLINE AS OPPOSED -- IF THIS WERE THE CASE YOU SHALL GET A BIG RESULT AND THEN FLUCTUATE AROUND THE MEAN FROM THERE ON. SO THIS LINEAR CHARACTERISTIC OF THE CLIENT EFFECT IS HARD TO EXPLAIN BY REGRESSION TO THE MEAN. THIS IS AN ELEGANT EXPLANATION, IS WHAT A COLLEAGUE OF MINE TERMED DEGRADATION OF PROCEDURE. HERE THE IDEA, DAN SMELLICK COINED THAT TERM. EFFECT MAY DEPEND ON UNDERAPPRECIATED IMPORTANCE OF ARBITRARY METHODOLOGICAL ELEMENTS NOT INCLUDED IN REPLICATION. WHEN WE RUN STUDYSTHEY EAR THESE ARBITRARY DECISIONS THAT WE HAVE TO MAKE. AND WE DON'T NECESSARILY APPRECIATE THAT SOME ASH TRAISH DECISION MAYBE POTENT AND DRIVES THE EFFECT. TIME OF DAY OR THE PARTICULAR EXPERIMENT OR USING A WHOLE HOST OF DIFFERENT THINGSCH THESE FACTORS MAY NOT BE INCLUDED IN THE METHODOLOGY AND WE MAY NOT EVEN APPRECIATE THEM TO BE IMPORTANT OURSELVES SO AS THE STUDY IS REPLICATED THESE FACTORS WE DON'T RECOGNIZE BEING IMPORTANT MAYBE INCREASINGLY LIKELY TO DROP OUT OF THE EXPERIMENT AND THAT MAY CONTRIBUTE TO SUBTLE CHANGES WHICH NEVERTHELESS CAN LEAD TO A DWINDLING EFFECT SIZE. THIS IS AN ELEGANT EXPLANATION AND ALSO HAS THIS NICE PART WHICH PROVIDES ONE WAY OFtQr CONCEPTUALIZING THE LINEAR QUALITY. THE MORE TIME THAT GOES BY, THE MORE OF THESE ARBITRARY DECISIONS ARE LIKE LICK TO BE CHANGED AS -- IEWB LUKELY TO BE CHANGED AS PEOPLE THAT DON'T APPRECIATE THEM TO BE IMPORTANT. CURIOUS THING ABOUT THIS EXPLANATION IS WHY SO LUCKY AT FIRST? WHY DONE WE INTRODUCE THINGS THAT AREN'T QUITE RIGHT AN AS WE REPLICATE YOU MIGHT AS WELL EXPECT WE GET IT BETTER, THAT THERE ARE ARBITRARY, THAT WE DO IT IN THE AFTERNOON INSTEAD OF MORNING THAT SHOULD WORK FOR US AS OFTEN AS AGAINST US SO NOT OBVIOUS WHY DEGRADATION PROCEDURE WORKS FOR A DECLINE, YOU THINK OFTENTIMES THESE ARBITRARY THINGS CANCEL OUT AND GO IN THE OPPOSITE DIRECTION AN INDEED YOU THINK THAT AS PEOPLE RANDOM EYE -- RAN THE EMPERIMENT THEY GET THE HANG OF CONDITIONS THAT MAXIMIZE THE EFFECT AND YOU THINK THAT WOULD WORK IN THE OPPOSITE DIRECTION. ANOTHER POSSIBILITY IS REFINEMENT OF PROCEDURE. THAT THERE MAYBE IMPROVED METHODOLOGIES THAT REMOVER ROANEIOUS SOURCES OF POSITIVE EFFECTS SO YOU -- PART OF YOUR EFFECT IS DUE TO SOMETHING THAT REALLY ISN'T THE ACTUAL EFFECT SIZE, AND SO REFINE YOUR PROCEDURE YOU GET IT BETTER THANKER RAN SOURCES OF EFFECT DOES APPEAR. BUT SHOULD METHODOLOGICAL REFINEMENTS WORK IN THE OTHER DIRECTION? SHOULDN'T PEOPLE UNDERSTAND EFFECTS BEAR AND MAKE IT EASIER IN SOME SITUATIONS TO GET EFFECT SMS THE FACT THAT OUR UNDERSTANDING WOULD LEAD TO BE HARDER TO GET EFFECTS SO CONSISTENTLY IS HARD TO UNDERSTAND, ENTIRELY. ANOTHER POSSIBILITY, AGAIN I THINK ALL THESE HAVE MERIT IS PUBLICATION BIAS. PUBLISHED STUDIES ANDND EDITORS FAVOR POSITIVE RESULTS AND QUITE LIKELY FAVOR STRONG POSITIVE RESULTS. LET ME INTRODUCE EVIDENCE FOR THIS, SO BARTOE AND (INAUDIBLE) THAT SHOULD BE 2012, DID A RESEN META ANALYSIS OF 386 -- 3,867 STUDIES INCLUDING 5 # META ANALYSES IN ECOLOGY. THEY FOUND THE FIRST PAPER PUBLISHED IN AREA WHILE IF OBSERVED EFFECT WAS NEGATIVE THE EARLIER PAPERS REPORTED THE GREATEST NEGATIVE EFFECTS. HIGH IMPACT FACTOR JOURNALS PUBLISH THE EXTREME EFFECTS AND MORE IMPORTANT IN DATA QUALITY FOR MANY PUBLICATIONS. THE PAPERS REPORTING STRONG EFFECTS ARE LOWER DATA QUALITY THAN PAPERS REPORTING WEAKER EFFECTS. AND HIGH IMPACT FACTOR JOURNALS PUBLISH STRONG EFFECTS IN THE ABSENCE OF ANY CORRELATION WITH DATA QUALITY. BASICALLY WHAT YOU SEE HERE IS A BIAS FOR STRONG EFFECT SIZES AND THAT BIAS FOR STRONG EFFECT SIZES MAY CONTRIBUTE TO THE PUBLICATION BIAS THAT MAY LEAD TO THIS DECLINE EFFECT. SO WHAT ARE SOME OTHER EXPLANATIONS? THE LAST ONE IS ONE THAT'S REALLY PICKED UP MOMENTUM IN THE LAST YEAR OR SO WHICH IS THIS IDEA OF TOO MANY DEGREES OF FREEDOM. THE IDEA HERE IS WHEN WE CONDUCT SCIENCE, THERE ARE A WHOLE LOT OF OPPORTUNITIES FOR US TO MAKE JUDGMENT CALLS ABOUT B WHICH DEPENDENT MEASURES TO REPORT. WHICH CO-VARIANTS TO USE. SO ON. AND THAT BECAUSE WE HAVE SO MANY DEGREES OF FREEDOM THEY MAY CONTRIBUTE TO OUR ABILITY TO CHERRY PICK OUR RESULTS. INDEED THERE HAVE THERE WAS A RECENT STUDY THAT CAME OUT IN PSYCHOLOGICAL SCIENCE WITH THE CATCHY TITLE, FALSE POSITIVE PSYCHOLOGY BY SIMONS ET ET AL WHICH THEY MADE A CASE FOR DANGERS FOR EXCESSIVE DEGREES OF FREEDOM, THEY DID THIS IN TWO DIFFERENT WAYS. ONE WAY WAS TO EXAMINE FIVE DIFFERENT SOURCES OF EXPERIMENT DEGREES OF FREEDOM INCLUDING AMONG DEPENDENT VARIABLES, VARYING THE SAMPLE SIZE SO BASICALLY PEEKING AND IF THE EFFECT IS SIGNIFICANT STOPPING AND IF IT'S NOT CONTINUING, USING CO-VARIANTS, REPORTING A SUBSET OF EXPERIMENTAL CONDITIONS AND THEN COMBINING THESE, THEN THEY DID A MONTE CARLO SIPLATION OF EXPERIMENTS TO SEE WHAT MIGHT HAPPEN TO TAKE ADVANTAGE OF THESE DIFFERENCE EXPERIMENT DEGREES OF FREEDOM. AND WHAT THEY FOUND IS THAT IF THEY -- I HAVE A TABLE HERE, IF YOU TAKE A LOOK AT THE P LESS THAN .05 LEVEL HERE AND YOU COMBINE ALL THE DIFFERENT EXPERIMENTAL DEGREES OF FREEDOM TOGETHER MONTE CARLO SIMULATION SAYS A 60% CHANCE YOU GET A FALL RESULT. SO WHEN YOU TAKE ADVANTAGE OF ALL THE WIGGLE ROOM THAT WE HAVE AS SIGNTISES THERE'S A REAL OPPORTUNITY TO INTRODUCE FALSE POSITIVE RESULTS. THEY ALSO DID A VERY CLEVER STUDY IN WHICH THEY EXPOSED PARTICIPANTS TO LISTENING TO WHEN I'M 64 AND THEY TOOK ADVANTAGE OF EXPERIMENTAL DEGREES OF FREEDOM THEY HAD. IN DOING THAT THEY WERE ABLE TO SHOW THE PARTICIPANTS REPORTED A YOUNGER AGE WHEN THEY WERE EXPOSED TO WHEN I'M 64 RELATIVE TO CONTROL. I THINK MOST OF US WOULD AGREE THIS IS VERY UNLIKELY LISTENING TO WHEN I'M 64 MAKES YOU YOUNGER. HOW DID THEY ACHIEVE THIS? THEY DID TWO WRITE UPS. ONE THEY TOOK ADVANTAGE OF EMPERIMENTAL DEGREES OF FREEDOM AND OTHER WHERE THEY DIDN'T. THIS IS THE WRITE UP WHERE THEY IGNORE ACKNOWLEDGE EMPERIMENTAL DEGREES OF FREEDOM, LESS THAN OR EQUAL (INDISCERNIBLE) SENSIBLY UNRELATED TASK THEY INDICATED THE BIRTHDAY, AND REVEALED PRETICK DICKTIVE EFFECT PEOPLE WERE A YEAR AND A HALF YOUNGER LISTENING TO WHEN I'M 64 RATHER THAN CLUM BA. WHAT THEY DIDN'T TELL YOU IS THIS OTHER STUFF HERE THEY HAVE THE ORIGINAL REPORT -- IT WAS 34 UNDERGRADUATES FROM THE SAME POOL OF STUDY, LESS THAN 2, EITHER WHEN I'M 64 BY OR HOT POTATO BY THE WIG SOL THEY LEFT OUT ALL THESE OTHER DETAILS AND THEN IN THIS VERSION HERE THEY SHOW UP A THE THINGS THAT THEY LEFT OUT SO SURE, WHEN YOU ONLY REPORT SOME OF THE SAMPLES WITH SOME OF THE MEASURES AND SO FORTH YOU CAN GET SIGNIFICANT RESULTS THEY TOOK ADVANTAGE OF EXPERIMENT OR DEGREES OF FREEDOM. NOW BUT DO PEOPLE REALLY DO THIS? SURE, MAYBE THIS IS SOMETHING YOU MIGHT DO BUT PEOPLE DON'T ACTUALLY DO THIS. IN FACT, IN A -- ANOTHER PAPER THAT'S JUST CAME OUT IN PSYCHOLOGICAL SCIENCE, JOHN LOWENSTEIN AND PRELIC DID A SURVEY OF OVER 2,000 PSYCHOLOGISTS ABOUT THEIR INVOLVEMENT IN QUESTIONABLE RESEARCH PRACTICES USING ANONYMOUS ELICITATION FORMATS BY INCENTIVES FOR HONEST REPORTING. THERE WERE TWO CONDITIONS, ONE CONDITION WHERE THEY RECEIVED STANDARD INSTRUCTIONS AND THE OTHER THEY USED THIS BAYESIAN TRUTH SERUM SCORING ALGORITHM WHERE THEY DONATE TO CHARITY IF FURTHER CHOICES. AND DEPEND ON THE TRUTHFULNESS OF THEIR RESPONSES. AND THIS IS SHOWN IN THE PAST TO BE PARTICULARLY SENSITIVE TO THOSE AREAS, LIKELY TO DISTORT THE TRUTH A LITTLE BIT. HARD TO SEE THE NUMBERS BUT HERE WE HAVE A NUMBER OF POSSIBLE ACTIVITIES SUCH AS IN A PAPER, ALL THESE DEPENDENT MEASURES, 63% DID THAT. DECIDING WHETHER TO CHECK MORE DEATH AFTER SEEING WHETHER THE RESULTS WERE SIGNIFICANT, 55% SAID THEY DID THAT. STOP CHECKED DATA EARLIER BECAUSE ONE FOUND THE RESULT ONE HAD BEEN LOOKING FOR, NEARLY A THIRD HAD DONE THAT. SO VIRTUALLY ALL THE DEGREES OF FREEDOM THAT SIMONS ET ALL STUDY, IN THIS SUR VA. COULD SELECTIVE REPORTING ACCOUNT FOR THE EFFECT? IT COULD BE AN IMPORTANT FACTOR, INITIAL INVESTIGATORS MAYBE MOTIVATED TO USE DEGREES OF FREEDOM TO OBTAIN THE GREATEST POSSIBLE EFFECTS. NO QUESTION MAJOR PROBLEM FOR THE FEEL BUT IT DOES NOT PERSON EXPERIENCES, I WATCHED DOING THE SAME PARADIGMS IT'S ALSO HARD TO EXPLAIN TO SEE HOW THESE THINGS PRODUCE SYSTEMATIC LINEAR DECLINE EFFECTS OBSERVED ACROSS FIELDS. HERE I GO A LITTLE BIT AGAIN INTO THE AREA I'M HESITANT TO GO BUT I CAN'T RESIST PUTTING THIS OUT. THERE. SO NON-CONVENTIONAL ACCOUNTS TO DECLINE EFFECT. IF YOU ASK DARRYL BEM WHAT HE THINKS IS GOING ON BIAS INFLUENCES EFFECTS IN NON-CONVENTIONAL WAY. PARAPSYCHOLOGY INFLUENCING THE EXPERIMENTER HIMSELF, SO HE FINDS THAT THE -- THOSE -- THERE HAVE BEEN A NUMBER OF ATTEMPTS TO REPLICATE HIS PARADIGM, AND THOSE PEOPLE WHO BELIEVE IN HIS EFFECTS HAVE FOUND EFFECTS OR THOSE WHO TEND TO BE SKEPTICAL HAVE FOUND NON-EFFECTS AND BEM ARGUES THAT THIS MAYBE THAT THE ACTUAL BELIEFS OF THE EXPERIMENTER IS SOMEHOW IN A PARAPSYCHOLOGICAL WAY INFLUENCING THE OUTCOME OF RESULTS. YOU MIGHT THIS IS LUDICROUS. BUT I ASKED BOB ROSE THAT WILL WHO IS THE PERSON WHO DISCOVERED EXPERIMENT EXPECTANCY EFFECTS WHAT HE THOUGHT ABOUT THIS POSSIBLE INTERPRETATION. YOU MIGHT BE SURPRISED TO HEAR WHAT HE SAID. BOB ROSETHAL IS FORMER CHAIR OF THE PSYCHOLOGY DEPARTMENT AT HARVARD UNIVERSITY AND A WELL RESPECTED INDIVIDUAL. DARRYL BEM IS GOOD COMPANY, GOURDEN ALSO BELIEVED THE INTERPERSONAL EXPECTANCY EFFECTS MIGHT BE MEDIATED PARA PSYCHOLOGICALLY. I HAVE NO EVIDENCE TO SUPPORT THAT POSITION NOR THE POSITION THAT THE PARAPSYCHOLOGICAL PHENOMENON ARE NOT INVOLVED IN MEDIATION OF INTERPERSONAL HE CAN EXPECT EXPECTANCY EFFECTS. WE ARE A LISTENING WAY FROM EXPLAINING ALL THE MECHANISMS THAT SERVE TO MEDIATE THE OPERATION OF INTERPERSONAL EXPECTANCY EFFECTS. SO ROSETHAL I THINK IS THE SAME PERSON THINKS IT'S NOT INAPPROPRIATE TO KEEP OUR MIND OPEN TO THE POSSIBILITY THAT SOMEHOW IN SOME WAY WE DON'T YET UNDERSTAND THE EXPECTATIONS OF THE EXPERIMENTER MAYBE INFLUENCING THE OUTCOME OF RESULTS. IF THAT IS NOT RADICAL ENOUGH I CAN RESIST ONE STEP FURTHER WHICH IS THAT IT'S POSSIBLE AS A FUNCTION OF OBSERVATION IT'S KNOWN IN OTHER AREAS AT LEAST SUGGESTED INTERPRETATION EFFECTS IN QUANTUM MECHANICS THAT SCIENTIFIC OBSERVATION SOMEHOW AFFECT A PHENOMENA I DON'T THINK WE CAN ENTIRELY RULE OUT THOUGH I DON'T THINK WE SHALL JUMP ON THE BAND WAGON YET GENUINE EFFECTS FADE WITH OBSERVATION IN PECULIAR WAY WE CAN'T YET UNDERSTAND WHICH WOULD SUGGEST THAT THE LAWS OF NATURE ARE NOT IMMUTABLE WHICH AGAIN IS A RADICAL VIEW BUT ALSO ENTERTAINED BY PHYSICISTS. I DON'T THINK WE CAN RULE THEM OUT UNTIL WE GET A BETTER HANDLE ON THE ACTUAL SOURCE OF THE DECLINE EFFECT. SO HOW CAN WE GET HER A BETTER HANDLE ON THE SOURCE OF THE DECLINE EFFECT? HERE IS WHERE I THINK PEOPLE IF YOU TOTALLY RULED OUT MORE RADICAL IDEAS I PRESENTED HERE YOU MAY FIND SOME SYMPATHIES. SO I THINK IN ORDER TO GET A HANDLE ON THIS WE NEED MORE TRANSPARENCY IN SCIENCE. WE NEED A PROCESS TO LET SCIENTISTS LOG HYPOTHESES AND METHODOLOGIES BEFORE AN EXPERIMENT AND THE RESULT AFTERWARDS REGARDLESS OF OUTCOME, RIGHT NOW THERE'S THIS BIAS IN SCIENCE WHERE ONLY THOSE FINDINGS THAT HAVE BEEN VETTED THROUGH THIS PROCESS THAT IS QUITE BIASED, BECOME AVAILABLE. I THINK THE TIME HAS COME FOR PROVIDING WARE ALL DATA REGARDLESS OF OUTCOME IS THE ULTIMATELY MADE AVAILABLE AND SIGNTISES BEFORE THEY RUN EXPERIMENTS LOG WHAT THEY'RE GOING TO DO AN PLANNING TO DO, SO ON. SO THERE'S CHALLENGES TO SUCH APPROACH. IT WOULD REQUIRE AUTOMATED PROTOCOL TO ENABLE STUDY METHODS AND RESULTS. AND RETRIEVED. IT REQUIRES A WAY TO ASSESS QUALITY OF THE WORK AND THERE WILL BE SOME OPPORTUNITIES HERE TOO. PERHAPS THERE COULD BE OPEN ACCESS COMMENTARIES, MODERATED IN MANNER SIMILAR TO WIKIPEDIA. IT REQUIRES A WAY TO ENSURE THE QUALIFICATIONS OF RESEARCHERS WHO USE IT, AND MAINTENANCE OF BLACK OUT PERIOD TO PROTECT BLACK OUT PERIOD AND FINDINGS AND PRIOR TO PUBLICATION, INCENTIVES AND NEW RULES FROM FUNDERS TO TAKE PART. I THINK THAT THIS IS NOT BEYOND THE CAN OF IMAGINATION. I THINK THIS IS SOMETHING THAT COULD HAPPEN AND IF THE RIGHT INCENTIVES ARE PUT IN PLACE AND THE CARROT STICKS TOOK PLACE IF PEOPLE FELT THIS WAS SOMETHING APPROPRIATE TO DO, THAT IT COULD BE DONE. CERTAINLY WE HAVE THE TECHNOLOGY TO DO IT. BRIAN NOSAK AT THE UNIVERSITY OF VIRGINIA ESTABLISHED AND IS ABOUT TO HAVE ENTER BETA PILOTING OF A WEBSITE TO DO ALL THESE THINGS. AND AS MANY OF YOU KNOW THERE'S SOMETHING SIMILAR ALREADY HAPPENING FOR CLINICAL TRIALS. THAT ALTHOUGH NOT PERFECT IS GONE A LONG WAY TOWARDS INCREASING THE TRANSPARENCY OF CLINICAL TRIALS. SO WHAT WOULD THE BENEFITS OF SUCH AN OPEN ACCESS REPOSITORY BE? >> IT WOULD REVEAL HOW PUBLISHED STUDIES FIT INTO A LARGER SET OF CONDUCTED STUDIES. IN UNDERSTANDING THE DECLINE EFFECT RIGHT NOW WE DON'T KNOW HOW THE PUBLISHED STUDIES RELATE TO ALL THIS GIANT MASS OF UNPUBLISHED WORK. IT WOULD OVERCOME IN METASTASIS PROBLEMS FROM STEMMING FREEDOM REGARDLESS OF CLIENT EFFECT THERE'S ISSUE OF DEGREES OF FREEDOM THAT NEEDS TO BE ADDRESSED, IT MAKES THE SCIENTIFIC PROCESS MORE TRANSPAIRN. WE WILL HAVE A CLEARER UNDERSTAND OF EXACTLY WHAT'S GOING ON IN SCIENCE INSTEAD OF THIS PECULIAR WAY WHICH WE GET THIS CHERRY PICKED VERSION OF SCIENCE. IT LIKELY REVEAL THE SOURCE OF THE CLIENT EFFECT SO THE BOTTOM LINE, THE DECLINE EFFECT HAS HAUNTED ME MY ENTIRE CAREER AND I WONDER HOW MANY OF YOU HAVE HAD EXPERIENCES OF A SIMILAR SORT. IT IS ALSO OBSERVED IN DOMAINS OF SIGN, FACTORS ARE LIKELY TO CRINT TO THE DECLINE EFFECT. IT HIGHLIGHTS THE IMPACT AND WE CANNOT UNDERSTAND THE DECLINE EFFECT UNTIL SCIENCE DOES A BETTER JOB OF MAKING AVAILABLE ALL STUDIES, NOT JUST THOSE TAILORED FOR PUBLICATION. THIS IS THE POINT THAT I REALLY THINK IS IMPORTANT. I THINK THE TIME HAS COME TO CONSIDER WAYS TO INCREASE THE TRANSPARENCY OF SCIENCE WHERE RESEARCHERS ARE EXPECT PED TO MAKE ALL OF THEIR PROTOCOLS AND ALL THEIR FINDINGS AVAILABLE. OPEN ACCESS REPOSITORY WOULD GO A LONG WAY BOTH TOWARD CORRECTING SCIENTIFIC DEGREES OF FREEDOM AND REVEALING THE SOURCES OF THE DECLINE EFFECT. THANK YOU VERY MUCH. [APPLAUSE] (OFF MIC) >> I THINK A LOT OF THE DRUG STUDIES SHOWING DECLINE EFFECTS IN THOSE DOMAINS WOULD COPSEN INSTITUTE -- CONSTITUTE EXAMPLES OF THAT SORT. BUT I DO AGREE THAT MY EXPECTATION IS IF YOU WERE TO LOOK AT THE GAL LAY OWE'S, -- GALILEO -- THE RATE AT WHICH BALLS HAVE BEEN GOING DOWN AT A CERTAIN RATE, PROBABLY THE SAME RATE EVER SINCE IT WAS FIRST OBSERVED SO THERE IS SOME CATEGORY OF KINDS OF EXPERIMENTS THAT MAYBE PARTICULARLY VULNERABLE TO DECLINE EFFECTS AND IF I WERE TO GUESS WHAT IT WAS I WOULD SAY THE COUNTER INTUITIVE ONES SO FAILURE TO REPLICATE RESEARCH BY JOHN BARGE ON IMPLICIT GOALS. I HAVE ALWAYS SAID THAT SEEMS LIKE THE KIND OF FINDING THAT YOU HAVE TO GET EXACTLY RIGHT IN ORDER TO OBSERVE. SO YOUR QUESTION IS A GOOD ONE. THE DRUG STUDIES SUGGEST DOMAINS THAT DON'T SEEM THAT COUNTER INTUITIVE WHERE DECLINE EFFECTS HAVE BEEN OBSERVED BUT I DO THINK IT'S THE SURPRISING COUNTER INTUITIVE RESULTS THAT MAYBE THE MOST VULNERABLE. (OFF MIC) >> RIGHT. THAT'S -- I MEAN, I HAVE GOTTEN DONE THE EXACT SAME PARADIGM AND FOUND HARDER TIMES DOING EXACTLY THE SAME THING, WHICH IS WHAT MAKES ME SUSPICIOUS BUT I THINK THIS IS -- THERE ARE A LOT OF QUESTIONS THAT ARE OPEN AND WHAT WE REALLY NEED TO DO IS HAVE A CAREFUL ANALYSES AND A VARIETY OF DIFFERENT DOMAINS TO SEE THE MAGNITUDE OF THESE EFFECTS. YES. IT DEPENDS HOW YOU DEFINE SUBJECTIVE. YOU FINE IT IN ECOLOGY, YOU FINE IT IN BIOLOGY. YOU FIND IT IN MEDICINE. NOT SURE IF THOSE WERE SUBJECTIVE. ALL SEEM TO HAVE BIOLOGICAL ORGANISM INVOLVED SO THEY ALL INVOLVE -- BUT SOME OF THESE CASES THEY'RE PRETTY LOW LEVEL LIFE FORMS SO IT'S HARD TO KNOW, I DON'T KNOW ANY GOOD EXAMPLES OF IT IN PHYSICS BUT AGAIN, THERE ARE -- PHYSICS HAS A LEVEL OF CONTROL THAT WE DON'T SEE IN THESE OTHER DOMAINS AS WELL. (OFF MIC) >> YES. YES. (OFF MIC) >> WHY DOESN'T -- SO ALSO HAS TO ASSUME THE PLACEBO COMPONENT IS NOT ADDITIVE WITH THE DRUG COMPONENT BECAUSE IF THERE'S A PLACEBO COMPONENT AND A DRUG COMPONENT, AND IF PLACEBO COMPONENT IS GOING UP, WHY DOES THAT TAKE AWAY FROM THE COMPONENT THAT WOULD HAVE OTHERWISE BEEN DRIVEN BY THE DRUG PART? YOU NEED TO THROW IN THAT OTHER MECHANISM IN ORDERTOR THAT TO EXPLAIN. THERE'S NO OBVIOUS REASON WHY THAT SHOULD BE GOING ON. YES. (OFF MIC) >> YEAH. WELL, I CERTAINLY AGREE WITH THE FIRST PART OF WHAT YOU'RE SAYING. I THINK THAT A BIG PART OF THIS INVOLVES THE TRANSPARENCY OF THE PROCEDURE AND AS I MENTIONED THE DEGRADATION OF PROCEDURE, THAT WHOLE MECHANISM IS DRIVEN ON THE IDEA THAT THERE MAYBE A WHOLE SET OF THINGS THAT WE THINK ARBITRARY BUT ARE PROVING TO BE VERY IMPORTANT. I ALSO THINK THAT BECAUSE WE NOW HAVE THE INTERNET, WE HAVE THIS BASICALLY UNLIMITED AVAILABILITY FOR SPACE AND DOCUMENTATION, THERE'S AN OPPORTUNITY TO PROVIDE MUCH MORE DETAILS THAN WE COULD HAVE DONE WHEN WE WERE LIMITED BY THE SPACE OF PAPER IN JOURNAL REPORTING SO WE'RE REALLY IN A NEW AGE WITH NEW OPPORTUNITIES TO ADDRESS THE TRANSPARENCY ISSUES THAT YOU RAISE (OFF MIC) >> YEAH. >> YEAH. I THINK THAT'S A REAL ISSUE. WHAT HAPPENS IS PEOPLE SEE THEM DESCRIBED AS SUPPLEMENTAL THEY CONSIDER IT SORT OF OPTIONAL. AND I QUITE AGREE THAT THAT'S A BIG ISSUE. AND WE NEED TO THINK ABOUT THIS VERY CAREFULLY BECAUSE WE NEED TO MAKE SURE THAT IN INCREASING THE AMOUNT OF INFORMATION THAT'S MADE AVAILABLE THAT WE DON'T MAKE REVIEWING PROCESS EVEN MORE ONEROUS THAN IT ALREADY IS. SO I AGREE, THERE'S A BALANCING ACT THAT HAS TO BE SORTED OUT HERE. YES. (OFF MIC) >> YEAH. AGAIN, THIS IS ONE OF THE EXPLANATIONS THAT I SUGGESTED, IT DOES MAKE SENSE THAT AS -- IF PEOPLE MAKE ERRORS THAT ARE BIASED TOWARDS FINDING RESULTS AND THAT IS THEY REFINE THEIR PROCEDURE TO GET RID OF THES ERRANT SOURCES OF POSITIVE EFFECTS AND THAT MAYBE A SIGNIFICANT PERHAPS EVEN THE ENTIRE EXPLANATION FOR THE CLIENT EFFECT BUT YOU ALSO EXPECT THAT AS -- AND I'M SURE IS THE CASE IN MANY SITUATIONS, THESE PEOPLE REALLY UNDERSTAND THE PHENOMENA BETTER AND KNOW EXACTLY WHAT CAUSES IT AND CONDITIONS RESPONSIBLE, THEY SHOULD BE ABLE TO MAGNIFY THE EFFECT SIZE SO IT'S NOT OBVIOUS WHY IT WOULD ALWAYS WORK IN THIS ONE DIRECTION. YOU HAD A QUESTION. (OFF MIC) NO, I HAVEN'T SEEN THAT. I CAN'T SPEAK TO THAT. I'M SORRY. IT'S ONLY BEEN COUPLE OF WEEKS SO THAT'S MY EXCUSE. YEAH. THANK YOU. YES. (OFF MIC) >> •„ [LAUGHTER] >> THANK YOU. >> WELL, OBVIOUSLY THIS IS -- THERE IN LIES THE RUB. AND THERE'S NO SIMPLE ANSWER TO THAT. ONE THING IS IN MANY CASES THE EFFECT SIZE SHRINKS BUT NOT TO NOTHING. SO THOUGH I WOULD PREDICT IF YOU WERE TO DO A STUDY LOOKING AT THE GRANT PROPOSALS AND PILOT DATA AND MAGNITUDE OF THE EFFECT OF PILOT DATA COMPARED TO FINDINGS PEOPLE REPORTED IN THE ULTIMATELY PUBLISHED STUDIES THAT CAME FROM THE GRANT THAT YOU WOULD CONSISTENTLY FIND THAT THE PILOT DATA HAD LARGER EFFECT MAGNITUDE THAN THE PUBLISHED STUDIES BUT NEVERTHELESS THE PATTERN WOULD HAVE SOME SIMILARITY -- RELEVANCE TO ONE ANOTHER SO PILOT DATA WHILE LIKELY EXAGGERATED ACCORDING TO THIS ANALYSIS, IS OFTENTIMES AT LEAST NOT GOING TO BE ENTIRELY MOOT. FROM (OFF MIC) >> YES. (OFF MIC) >> SO I THINK THE ANSWER TO THAT WOULD BE THE ANSWER TO THE DECLINE EFFECT. SO THAT I THINK IS THE $64,000 QUESTION, WHAT IS THE EXPLANATION FOR THE NATURE OF THE SLOPE. IT SEEMS AS IF THE DECLINE EFFECT DECLINES TO AN ASEMTOTE TYPICALLY ABOVE ZERO THOUGH IN THE PARAPSYCHOLOGY DOMAINS IT SEEMS TO GO TO ZERO. AGAIN, THAT MAYBE THAT THERE IS SOME ARTIFACT ERROR VARIANT, SOMETHING LIKE THAT DRIVING THE EFFECT COMBINED WITH THE REAL EFFECT IN THAT AS YOU KEEP REPLICATING YOU GRADUALLY DWINDLE DOWN THE ARTIFACT AND THEN GET TO CLOSER TO THE TRUE VALUE, IN THE CASE OF PARAPSYCHOLOGY THE TRUE VALUE LEADS YOU TO ZERO SO THAT WOULD BE ONE I THINK VIABLE EXPLANATION FOR WHAT'S GOING ON THERE. I THINK THERE ARE A NUMBER OF OTHER OF POSSIBILITIES, AS I MENTION THIS PECULIAR THING IN THE PARAPSYCHOLOGY YOU GET THIS CURVE LINEAR EFFECT WHERE IT SLOPES BACK UP AFTERWARDS AS WELL. SO I THINK YOU REALLY NAILED A QUESTION THAT I DON'T HAVE A DEFINITIVE ANSWER TO. BUT WHAT I THINK NEEDS TO BE DONE IS MUCH MORE CAREFUL THOUGHTFUL QUANTITATIVE EMPERIMENTAL INVESTIGATION OF THE PHENOMENA SO THAT WE CAN GET A HANDLE ON EXACTLY THAT QUESTION. LOOKS LIKE THE QUESTIONS HAVE DECLINED. WELL, THANK YOU VERY MUCH.