SO I THINK WE'RE GOING TO GET STARTEDDED BEFORE WE GET TOO FAR BEHIND. I EXPECT THAT SOME PEOPLE DIDN'T REALIZE WE WERE STARTING QUITE EARLY THIS MORNING BUT WE HAVE A LOT TO PACK IN TODAY SO WE'LL GET STARTED NOW. SO I'M HELEN MEISSNER, AND ON BEHALF OF THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCES RESEARCH IT'S MY PLEASURE TO WELCOME -- YOU CAN'T HEAR ME? IS MIC ON? MAYBE I NEED TO BE CLOSER TO IT. SO IT'S ME PLEASURE TO WELCOME YOU TO TODAY'S WORKSHOP USING MIXED METHODS TO OPTIMIZE DISSEMINATION AND IMPLEMENTATION OF HEALTH INTERVENTIONS. DEFINED AS RESEARCH IN WHICH THE INVESTIGATOR COLLECTS, ANALYZES AND INTENTIONALLY INTEGRATES OR COMBINES QUALITATIVE AND QUANTITATIVE DATA, MIXED METHODS APPROACHES OFFER AN OPPORTUNITY TO CONTEXTUALIZE INFORMATION IN SUCH A WAY AS TO GAIN A MORE COMPREHENSIVE AND COMPLETE UNDERSTANDING OF A PROBLEM. IT IS I GUESS I'M NOT -- I NEED TO BE REALLY CLOSE TO IT, I GUESS. UP CLOSE AND PERSONAL WITH THE MIC. WE HAVE SEEN A GROWING INTEREST IN USING MIXED METHODS IN HEALTH RESEARCH BECAUSE IT CAN ADDRESS COMPLEX PROBLEMS FOR WHICH NEITHER A QUALITATIVE NOR QUANTITATIVE APPROACH BY ITSELF IS SUFFICIENT TO UNDERSTAND THE PROBLEM. ONE SUCH AREA FOR MIXED METHODS APPROACHES YET ONE IN WHICH WE REALLY HAVE MADE SURPRISINGINGLY LITTLE PROGRESS TO DATE IS THE CHALLENGE OF CLOSING THE GAP BETWEEN WHAT WE KNOW WORKS TO IMPROVE HEALTH AND WHAT'S CURRENTLY APPLIEDED IN PRACTICE. HOW BEST TO GET RESEARCH FINDINGS INTO PRACTICE IS THE FOCUS OF SIGNS OF DISSEMINATION AN IMPLEMENTATION AND TODAY WE'LL BE DISCUSSING THE POTENTIAL OF MIXED METHODS APPROACHES PROSIDING INSIGHT INTO PROCESSES AND OUTCOMES OF IMPLEMENTING HEALTH INTERVENTIONS IN DIVERSE COMMUNITY AND CLINICAL SETTINGS. THE GROWING INTEREST IN MIXED METHODS RESEARCH PROMPTED THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCES RESEARCH TO COMMISSION THIS REPORT BEST PRACTICES FOR MIXED METHODS RESEARCH IN THE HEALTH SCIENCES BECAUSE THERE WAS REALLY VERY LITTLE GUIDANCE OUT THERE FOR NIH INVESTIGATORS AS WELL AS REVIEWERS HOW TO DEVELOP AND EVALUATE MIXED METHODS RESEARCH APPLICATIONS. I CAN TELL YOU WE MUST HAVE MET A NEED BECAUSE HITS TO OUR WEBSITE WHEN WE RELEASED THIS DOCUMENT WENT UP EXPONENTIALLY AN EIGHT MONTHS LATER THIS IS THE MOST FREQUENTLY VISITED SITE ON THE OBSSR WEBSITE. I'M PLEASED ALSO TO ANNOUNCE TODAY IN ADDITION TO THE WEB VERSION AND THE PDF YOU CAN -- YOU OR YOUR STUDENTS AND YOUR MAY LIKE THIS EVEN MORE TO DOWNLOAD TO iPAD OR KINDLE. EVEN WITH THE GUIDANCE AVAILABLE WE HAVE HEARD A LOT FROM INVESTIGATORS ABOUT THE NEED FOR MORE TRAINING IN MIXED METHODS RESEARCH, SO SEEM AS LOGICAL THING TO OFFER THIS WORKSHOP TODAY. SO TODAY YOU'LL BE HEARING FROM A NUMBER OF EXPERTS BOTH IN MIXED METHODS RESEARCH AND ALSO DISSEMINATION RESEARCH. OUR GOAL REALLY IS TO SHOW POTENTIAL OF MIXED METHODS IN IMPROVING DISSEMINATION AND IMPLEMENTATION RESEARCH. SO TODAY WE WILL BASICALLY START WITH AN OVERVIEW OF MIXED METHODS AND HOW IT MIGHT BE USED IN DNI RESEARCH. THEN YOU'LL BE HEARING FROM INVESTIGATORS WHO ACTUALLY ARE DOING MIXED METHODS STUDIES AND HAVE NIH SUPPORT FOR THAT. THEN WE'LL ALSO FOCUS THE REST OF THE DAY ON SOME OF THE CHALLENGES OF DOING THESE TYPES OF APPLICATIONS INCLUDING QUALITATIVE PERSPECTIVES, CHALLENGES OF INTEGRATING METHODS AS WELL AS DESIGNING APPLICATIONS, REVIEWING THEM AND GETTING RESEARCH RESULTS PUBLISHED. BEFORE I GET STARTED TAKE A MOMENT TO ACKNOWLEDGE THE PEOPLE WHO HELPED PLAN TODAY'S MEETING, PARTICULARLY JENNIFER WISDOM AND TISHA WILEY WHO WERE INSTRUMENTAL MAKING THIS HAPPEN TODAY. I WOULD ALSO LIKE TO ACKNOWLEDGE COLUMBIA AND NEW YORK UNIVERSITIES FOR CO-SPONSORSHIP AND SUPPORT FOR THE MEETING AND I DON'T KNOW IF YOU HAVE BEEN TO GOVERNMENT SPONSORED MEETING LATELY BUT WE'RE NOT ALLOWED TO SERVE COFFEE, LET ALONE ANY REFRESHMENTS SO IT WAS REALLY FORTUITOUS AND LUCKY FOR US TO HAVE CO-SPONSORSHIP FROM COLUMBIA SO WE CAN THANK KIMBERLY HOAGWOOD FOR THE CAFFEINE TO KEEP US AWAKE TODAY. NOW I JUST LIKE TO TURN IT OVER TO KIMBERLY HOAGWOOD WHO IS OUR COLLABORATOR AT COLUMBIA AND NYU AND PRINCIPAL INVESTIGATOR ON A RECENTLY FUNDED NIMH P-30 ADVANCED CENTER FOR DISSEMINATION AND IMPLEMENTATION AND GREAT EXAMPLE OF USE OF MIXED METHODS IN DNI RESEARCH. >> WONDERFUL TO BE HERE. I CAN'T TELL YOU HOW TIMELY THIS MEETING IS. IT'S JUST REMARKABLE, HOW IT'S COMING AT A TIME WHEN MANY OF US, MY GUESS IS IT'S EVERYBODY IN THIS ROOM, IS STRUGGLING WITH SOME KIND OF LARGE SCALE STUDY INVOLVING MULTIPLE LAYERS OF CONTEXT WHERE WE'RE TRYING TO CAPTURE SOME OF THOSE PROCESSES OF CHANGE. THAT CERTAINLY WHERE WE'RE AT. AT COLUMBIA NOW MOVING TO NYU AND ADVANCED IDEA CENTER WORKING WITHIN STATES TO TRY TO TRY TO IDENTIFY WHAT ARE PROCESSES BY WHICH YOU CAN IMPLEMENT HIGHER-QUALITY PRACTICES FOR KIDS IN STATE SYSTEMS. STATE SYSTEMS YOU'RE TALKING L POLICY LEVELS, POLITICS, CHANGES AT THE LEVEL OF THE PROVIDERS AND LAYERS THERE. ALL THE WAY DOWN TO THE CLIENTS. SO AS WE'RE STRUGGLING WITH WAYS TO IMPROVE THE DELIVERY OF HIGHER QUALITY SERVICES WE HAVE GOT TO FIND WAYS TO CAPTURE PROCESSES WHICH ARE CHANGING. WE HAVE BEEN DOING IT AS I THINK MANY OF US IN THE FIELD HAVE, FAIRLY UNSYSTEMATICALLY WITH A PIECE MEAL APPROACH, WE'LL DO AN EXPERIMENTAL INTERVIEW HERE, QUALITATIVE HERE, MULTIPLE BASELINE OR SURVEYS TO PULL IT TOGETHER BUT THE PIECE MEAL APPROACH DOES NOT SERVE US WELL AS WE'RE TRYING TO IMPROVE THE PUBLIC HEALTH SERVICES IMPROVING HEALTH SERVICES FOR KIDS AND FAMILIES AN ADULTS IN THE PUBLIC HEALTH ARENA. SO IF WE'RE GOING TO DO THAT SYSTEMATICALLY AND BE ABLE TO CAPTURE THOSE PROCESSES, IN ORDER TO IMPROVE THE OUTCOMES WE HAVE TO BE ABLE TO THINK MORE STRATEGICALLY. I SEE THAT AS THE REAL GOAL OF TODAY'S MEETING, TO SAY HOW CAN WE USE DIFFERENT METHODS, TOGETHER IN AN INTEGRATIVE WAY TO LEARN MORE AN BENEFIT THE PEOPLE, THAT WE'RE SERVING WITH TAXPAYER DOLLARS. SO I'M JUST SO GRATEFUL FOR HELEN FOR ORGANIZING THIS MEETING FOR THE TIMING OF IT ALL. I ALSO WANT TO ACKNOWLEDGE COLLABORATORS FOR WHOM I HAVE LEARNED AND CONTINUING TO LEARN EVERY DAY AS WE'RE STRUGGLING WITH MIXED METHODS THAT INCLUDES (INAUDIBLE) JENNIFER WISDOM, LARRY POLINKAS T COLLABORATORS HELPING US TRYING TO SRT OUT THE SERVICES. I'M GOING TO STOP AND TURN IT TO HELEN FOR INTRODUCTIONS. WE HAVE A TERRIFIC DAY, TERRIFIC SPEAKERS. THANK YOU ALL FOR COMING. >> I WOULD ALSO LIKE THE MENTION AND SUGGEST THAT PEOPLE MOVE UP. OUR ORIGINAL IDEA FOR THE WORKSHOP WAS TO HAVE RELATIVELY SMALL GROUP TO ALLOW FOR MORE INTERACTION AND DISCUSSION. THE WORKSHOP ACTUALLY FILLED UP IN THE DAY, GOT TO BE A RECORD. SO WE WERE ABLE TO GET A LARGER SPACE AND OPEN IT UP TO SOME ADDITIONAL PEOPLE WHO WERE ON THE WAIT LIST BUT NEVERTHELESS WE WANT TO ENCOURAGE YOUR PARTICIPATION AND HOPE TO HAVE DIALOGUE TODAY NOT JUST TALKING HEADS. SO PLEASE COME FORWARD AND PARTICIPATE. NOW I'LL TURN IT TO JOHN CRESWELL AND JENNIFER WISDOM TO PROVIDE AN OVERVIEW OF MIXED METHODS RESEARCH IN DISSEMINATION AND IMPLEMENTATION. I HAVE TO SAY ONE FUNNY THING, I I HER THIS MORNING WHEN JOHN WHO MANY OF YOU KNOW AS MR. MIXED METHODS, WRITTEN QUITE A FEW BOOKS AND ARTICLES ON THE SUBJECT. WHEN HE CAME THROUGH THE NIH SECURITY THIS MORNING THE GUARD SAID, OH, ARE YOU JOHN CRESWELL? JOHN THOUGHT HE WAS MAYBE ON SOME TERRORIST LIST OR SOMETHING BUT IT ACTUALLY TURNED OUT THE GUARD WAS READING JOHN'S TEXTBOOK FOR GRADUATE CLASS THAT HE WAS TAKING. [APPLAUSE] SO >> THAT'S GREAT. ANYWAY, JOHN, I'M GOING TO TURN IT OVER TO YOU NOW. >> CAN EVERYONE HEAR ME OKAY? THANK YOU, HELEN. AND ALL THAT ORGANIZED THIS DAY, IT'S WONDERFUL TO SEE EVERYONE COME AND SHARE IDEAS AB MIXED METHODS RESEARCH AND IMPLEMENTATION RESEARCH. I DID TELL THE INDIVIDUAL AT THE DESK THAT I WASN'T A TERRORIST, THAT -- AND HE BREATHE AD SIGH OF RELIEF THERE. I'M AN AUTHOR AND WRITTEN SEVERAL BOOKS ON MIXED METHODS. I WANT TO TALK ABOUT AN INTRODUCTION TO MIXED METHODS RESEARCH. THOSE FAMILIAR WITH THIS TOPIC I APOLOGIZE FOR THE INTRODUCTORY NATURE. THE WAY I SEE ID METASTASIS AN INTRODUCTION IS HELPFUL TO SET EVERYONE ON THE SAME PAGE SO I'M GOING TO INTRODUCE AT A FAIRLY BASIC LEVEL BUT I'M GOING TO SHARE QUITE A BIT OF INFORMATION WITH YOU IN A SHORT PERIOD OF TIME AND REALIZE THROUGHOUT THE DAY THERE WILL BE PROJECTS THAT WILL ILLUSTRATE THESE POINTS IN MORE DETAIL. SO THIS IS AN OVERVIEW. I'M PRESENTING ON MIXED METHODS, I THINK I HAVE 22 POWERPOINT PRESENTATIONS BETWEEN FEBRUARY AND JULY ON MIXED METHODS, ALL OVER THE WORLD. IN A LOT OF DIFFERENT HEALTH SCIENCE SETTINGS SO MOVING AROUND THROUGH DIFFERENT HEALTH SCIENCE AREAS RAPIDLY RIGHT NOW. THERE'S QUITE A BIFT OF INTEREST. -- A BIT OF INTEREST. SO TOPICS TODAY THAT WILL BE COVERED, HELEN MENTIONED SOME OF THESE, THIS WILL BE AN INTRODUCTION AND JENNIFER WISDOM WILL TALK AB DISSEMINATION IMPLEMENTATION RESEARCH, THERE WILL BE MANY EXAMPLES OF DOING GOOD MIXED METHODS RESEARCH. WE'LL BRING QUALITATIVE INQUIRY INTO MIXED METHODS. THE TOPIC OF INTEGRATION, HOW DO YOU INTEGRATE QUALITATIVE AND QUANTITATIVE DATA, IT'S VERY IMPORTANT. AND OFTEN DISCUSSED TOPIC. THEN QUALITY PRACTICES. AND THE WAY JENNIFER AND I SEE THIS DAY UNFOLDING IS THAT WE ARE REALLY BRIDGING TWO FIELDS, THE MIXED METHODS AREA WHICH TENDS TO COME OUT OF THE SOCIAL SCIENCES, WITH THE HEALTH SCIENCE AREA DISSEMINEIGH IMPLEMENTATION. SO I PUT TOGETHER THIS LITTLE PICTURE HOW I WOULD INTRODUCE MIXED METHODS RESEARCH. THIS IS AN EVOLVING PICTURE BUT I WOULD TALK ABOUT ORIGINS OF IT, HOW IT CAME ABOUT AND WHY IT DEVELOPED, A DEFINITION SECTION GAINED TO KEY CHARACTERISTICS, SOME OF THE PURPOSES, ESPECIALLY THE RATIONALES FOR DOING MIXED METHODS RESEARCH. TECHNICAL DEVELOPMENTS THAT OCCURRED OVER THE YEARS ESPECIALLY SINCE AROUND 2003 WHEN THE HANDBOOK ON MIXED METHODS RESEARCH WAS ISSUED BY SAGE PUBLICATION. THERE'S AREA OF CONCEPTUAL THEORETICAL DEVELOPMENTS IN MIXED METHODS OFTEN DISCUSSED, ESPECIALLY AMONG THE SOCIAL SCIENTISTS. AND OF COURSE WE'RE BECOMING MORE AWARE OF THE PRACTICAL APPLICATIONS, EXAMPLES OF MIXED METHODS INQUIRY. HOW DO WE FORM RESEARCH TEAMS, SOME OF THE ADAPTATIONS THAT ARE OCCURRING ACROSS DIFFERENT DISCIPLINES AND COUNTRIES. AND OF COURSE WHAT THE BEST PRACTICES WEBSITE, THE QUESTION OF HIGH-QUALITY MIXED METHODS PROJECTS. NOW I'M GOING TO TOUCH ON ALL THESE TOPICS BUT I'LL HIT ON SEVERAL OF THEM ESPECIALLY THROUGH THE ORIGINS DEFINITIONS TECHNICAL DEVELOP MS AND PRACTICAL APPLICATIONS. ONE CONFUSING POINT OFTEN FOR INDIVIDUALS LEARNING AB MIXED METHODS FOR THE FIRST TIME IS THAT THEY SEE IT AS CHECKING BOTH QUALITATIVE AND QUANTITATIVE DATA AND OFTEN KEEPING THESE AS SEPARATE STRANDS. WE HAVE BEEN DOING THAT IN THE HEALTH SCIENCES AND SOCIAL SCIENCES FOR YEARS. THEY HAVE BEEN KEPT SEPARATE, SEPARATE STRANDS OR SEPARATE COMPONENTS IN A PROJECT. THE MIXED METHODS GROUP REALLY CAME IN AND DREW A LINE ACROSS, THEY'RE THINKING ABOUT THIS INTEGRATION ACROSS QUANTITATIVE AND QUALITATIVE. THEY CALLED IT MIXED METHODS RESEARCH. INTO THIS MIXED METHODS CIRCLE HAVE ENTERED A NUMBER OF DIFFERENT WAYS OF THINKING ABOUT MIXED METHODS. ONE CAN THINK ABOUT IT AS THE INTEGRATION OF METHODS, THE DATA COLLECTION AND ANALYSIS PRIMARY PRIMARILY. I'M IN THIS GROUP SO I'LL HAVE A METHODS ORIENTATION. I HAVE OFTEN STARTED FROM THIS POINT BECAUSE I THINK IT'S EASIEST WAY TO GET INTO MIXED METHODS AND UNDERSTAND IT. AT A CONCRETE LEVEL. THERE ARE OTHERS THINKING ABOUT MIXED METHODS SPANNING ACROSS METHODOLOGY, WHICH RUNS FROM THE PHILOSOPHY THROUGH THE QUESTIONS THROUGH THE DATA COLLECTION TO THE ANALYSIS THROUGH INTERPRETATION ALL ACROSS THE PROCESS OF RESEARCH AND HOW DO YOU MIX QUALITATIVE AND QUANTITATIVE DATA. IN SOCIAL SCIENCES WE HAVE HAD PHILOSOPHERS INVOLVED IN MIXED METHODS RESEARCH BRINGING PARADIGM PERSPECTIVES. THERE IS A GROUP OF PEOPLE NOW, THIS IS A TREND I'M WATCHING CAREFULLY, THAT IS TO USE MIXED METHODS QUANTITATIVE AND QUALITATIVE DATA WITHIN OTHER TYPES OF DESIGNS. FOR EXAMPLE, AN INTERVENTION TRIAL EMPERIMENTAL TRIAL TO BRING QUALITATIVE AND QUANTITATIVE IN. AND NARRATIVE ANALYSIS. SO THERE ARE DIFFERENT GROUPS ENTERING INTO THIS THIS CIRCLE MIXED METHODS I BELIEVE DWOFFED IN THE LATE 1980s, EARLY 1990s. PRIMARILY SOCIAL SCIENTISTS WERE INVOLVED. AROUND THE WORLD. JENNIFER GREEN IN EVALUATION WAS THE FIRST TO WRITE IN THE MID 1980s. HUNTER AND BREWER A COUPLE OF SOCIOLOGISTS STARTED WRITING ABOUT MIXED METHODS AND PUT OUT A BOOK IN 1989. HELEN BRIEMAN IN THE MANAGEMENT FIELD IN ENGLAND, 1988 HAD HIS FIRST BOOK ON MIXED METHODS RESEARCH. I WAS BEGINNING TO WRITE ABOUT IT IN TERMS OF GENERAL RESEARCH DESIGN. WHEN MY BOOK ON RESEARCH DESIGN CAME OUT IN 1994 I RECEIVED MORE EMAILS ABOUT CHAPTER ON MIXED METHODS ARE SEARCH, I CALL IT COMBINED RESEARCH THAN ANY OTHER CHAPTERS IN MY BOOK. THE FIELDINGS COUPLE OF SOCIOLOGISTS FROM THE UK WERE DEVELOPING WORK ABOUT THE MIDDLE 1980s. AND JANUARY MORRIS FROM THE FIELD OF NURSING IN CANADA AT THAT TIME WAS BEGINNING TO DEVELOP THINKING ABOUT MIXED METHODS RESEARCH. THE INTERESTING THING ABOUT THIS GROUP, NONE WERE IN TOUCH WITH EACH OTHER, WE'RE ACTING INDEPENDENTLY BUT WE FELT QUALITATIVE HAD COME OF AGE TO A CERTAIN EXTENT. IT WAS GOING TO BE A MATTER OF TIME BEFORE PEOPLE STARTED THINKING HOW TO INTEGRATE QUALITATIVE AND QUANTITATIVE VERGE. YOU CAN SEE IT WAS A WORLD PHENOMENON FROM THE BEGINNING. AND IT WAS ALSO A MULTI-DISCIPLINARY PHENOMENON FROM THE VERY BEGINNING. SO THIS ACCEPTANCE OF QUALITATIVE RESEARCH WAS A BIG FACTOR IN MY THINKING AT THE TIME. I THINK IT WAS COMMONLY ACKNOWLEDGED THAT BOTH QUANTITATIVE AND QUALITATIVE RESEARCH HAVE INDIVIDUAL STRENGTHS. THEY SPEAK TO DIFFERENT INTENTS, THE QUANTITATIVE GET AT MR. CXFC TRENDS, PROBABLY CAUSATION, THE CONTEXT, THE INDIVIDUAL VOICES, PARTICIPANTS. THAT WAS ANOTHER FACTOR. FORMATION OF INTERDISCIPLINARY TEENS CHA IS CHAMPIONED BY NIH. I OFTEN TALK HOW MIXED METHODS ISHZ– INTUITIVE. I KEEP EXPANDING MY LIST OF PRACTICAL EXAMPLES, WHERE WE SEE WITHIN THIS WORLD WE LIVE IN EXAMPLES OF QUANTITATIVE AND QUALITATIVE RESEARCH TOGETHER. SOY USE THE TITANIC WHICH IS ONE OF MY LATEST ONES. YOU MAY HAVE SEEN JAMES CAMERONS DOCUMENTARY ON THE TITANIC, BEAUTIFUL MIXED METHODS EXAMPLE. BECAUSE HE STARTS -- HE'S TRYING TO FIGURE OUT WHY THE SHIP SANCTION AND HOW THE TITANIC SANCTION. HE STARTS WITH MAPPING THE DEBRIS FIELD ON THE OCEAN FLOOR. AND GETTING THAT LARGER CONTEXT WHICH IS VERY QUALITATIVE. THEN HE STARTED WITH THE QUANTITATIVE TRYING TO FIGURE OUT THE TRAJECTORY OF THE SHIP AS IT SANEK, HOW FAST IT SANEK, ALL THIS NUMERIC QUANTITATIVE DATA SO YOU'RE SITTING THERE LOOKING AT THIS DOCUMENTARY SAYING THIS IS AN EXAMPLE IEG I COULD ALSO GET TO PROVIDER PATIENT EXAMPLES HERE. THIS LIST OF MIXED METHODS RESEARCH, I ALWAYS SAY TOOK ME ABOUT 20 YEARS TO FORMULATE. YOU CAN SEE I WAS WORKING BACK THERE IN THE 19 -- LATE 1980s ON MIXED METHODS RESEARCH. SO IT'S BEEN ALMOST 25 YEARS. BIT'S THE COLLECTION OF QUANTITATIVE AND QUALITATIVE DATA. I STARTED THAT POINT. THEN THE RIGOROUS ANALYSIS OF BOTH FORMS OF DATA SO A GOOD MIXED METHODS STUDY HAS RIGOROUS QUANTITATIVE AS WELL AS QUALITATIVE PROCEDURES. FURTHERMORE, THE QUANTITATIVE AND QUALITATIVE PROCEDURES ARE INTEGRATED. BROUGHT TOGETHER. TO RELY ON THE STRENGTH OF BOTH APPROACHES TO RESEARCH AND UNDERSTANDING OUR PROBLEMS. FURTHERMORE THERE ARE SPECIFIC DESIGNS THAT HAVE EMERGED THAT INVOLVE A CONCURRENT, THAT'S WHERE YOU'RE BRINGING BOTH FORMS OF DATA TOGETHER AND MERGING TWO TYPES OF DATA OR CONVERGING THEM OR A SEQUENTIAL TYPE OF INTEGRATION. WHERE ONE BILLION DOLLARS ON THE OTHER. MORE PEOPLE ARE COMING TO ME WITH SEQUENTIAL DESIGNS. BEGINNING MOST INDIVIDUALS THAT CAME TO ME -- HAVE COME WITH PROJECTS ARE THINKING HOW THEY CAN ACTUALLY COMBINE OR MERGE THE QUANTITATIVE AND QUALITATIVE DATA. I SHARE WITH THEM THERE IS ANOTHER MODEL OUT THERE TO THINK IN TERMS OF THE SEQUENTIAL BUILDING OF DATA. SO FOR EXAMPLE I MIGHT DO A SURVEY STU DI AND FOLLOW UP WITH QUALITATIVE FOCUS GROUP INTERVIEWS TO UNDERSTAND THE SURVEY RESULTS. A SEQUENTIAL PATTERN. IT'S ALSO AN APPROACH THAT CAN HAVE PHILOSOPHICAL ASSUMPTIONS, FOUNDATIONS TO IT. THERE ARE DIFFERENT THEORETICAL OR YEN TAILGS USED IN MIXED METHODS. THERE'S ASSUMPTIONS WE'RE WORKING WITH HERE IN TERMS OF MIXED METHODS QUANTITATIVE AND QUALITATIVE DATA COLLECTION TEND TO BE DIFFERENT. GATHERING DIFFERENT TYPES OF INFORMATION. THAT THEY'RE ANSWERING DIFFERENT TYPES OF QUESTIONS WHERE QUANTITATIVE IS ANSWERING MORE PROBABLY CAUSE AND EFFECT TYPES OF QUESTIONS AN GROUP COMPARISON QUESTIONS AND QUALITATIVE IS ANSWERING QUESTIONS RELATING TO EXPLORING AND DISCERNING THE MEAN BY SUBJECTS OR PARTICIPANTS IN A STUDY. WE KNOW QUANTITATIVE AND QUALITATIVE HAVE DIFFERENT STRENGTHS AND WEAKNESSES, HERE IS THE KEY ASSUMPTION BEHIND MIXED METHODS RESEARCH. THAT IT COMBINES THE STRENGTHS OF BOTH QUALITATIVE AND QUANTITATIVE RESEARCH AND THAT NEITHER QUANTITATIVE OR QUALITATIVE BY ITSELF IS SUFFICIENT TO UNDERSTAND THE COMPLEX PROBLEMS WE'RE FACING TODAY. CALL QAWN CALL IF YOU WERE GOING TO DO A SEARCH ON PUBMED WHAT TERMS DO YOU PUT IN TO FIND MIXED METHODS STUDIES? OFTEN CALLED MULTI-METHOD RESEARCH. AMONG SOCIAL SCIENTISTS WE DON'T USE THIS TERM. BECAUSE<  MULTI-METHOD MEANS USING MULTIPLE METHODS BUT IT DOESN'T MEAN NECESSARILY INTEGRATING THE TWO METHODS. SO WE HAVE SHIFTED OVER TO MIXED METHODS TO CONVEY THE IDEA OF INTEGRATING QUALITATIVE AND QUANTITATIVE. YOU CAN FIND ARTICLE BUSINESS SEARCHING FOR QUALITATIVE AND QUANTITATIVE RESEARCH, YOU MIGHT EVEN NARROW DOWN TO SPECIFIC METHODS OF DATA COLLECTION SUCH AS FOCUS GROUP INTERVIEWS, EXPERIMENTS. WHAT HAPPENED IN THE SOCIAL SCIENCES? I THINK WITH INCREASE FREQUENCY IN THE HEALTH SCIENCES NOW IS THE TERM IS MIXED METHODS RESEARCH. THAT PERHAPS STARTED AROUND 2003 WHEN THE HANDBOOK ON MIXED METHODS RESEARCH WAS ISSUED BY SAGE. AUTHORS OF DIFFERENT CHAPTERS IN THAT HANDBOOK WERE USING DIFFERENT TERMS. SAGE VISITED WITH US AND ASKED WHAT WOULD BE BEST TERM TO USE. WE SUGGESTED MIXED METHODS. THE JOURNAL OF MIXED METHODS WAS FIRST PUBLISHED IN 2007. SO NOW YOU CAN GO INTO THE NIH DATABASE AND FIND STUDIES USING THE TERM MIXED METHODS. SO WHAT ARE THE METHODS ASSOCIATED WITH QUANTITATIVE AND QUALITATIVE RESEARCH? THERE'S A LOT OF CONTROVERSY ABOUT THIS SEPARATION AS THE STAND THAT I HAVE TAKEN IS QUANTITATIVE DATA CONSISTS OF MORE CLOSE-ENDED TYPES OF INFORMATION WITH PRE-DETERMINED SCALES. WHEREAS QUALITATIVE IS MORE OPEN-ENDED. WITHOUT THE PRE-DETERMINEDDED RESPONSE. PEOPLE REFER TO DIFFERENT FORMS OF DATA IN A VERY SIMPLE WAY, THAT QUANTITATIVE CONSISTS OF NUMERIC DATA AND QUALITATIVE TEXT ARE IMAGE DATA. AND IF THAT IS A USEFUL DIFFERENTIATION FOR YOU, I THINK THAT WOULD BE FINE. OVERALL THE QUANTITATIVE SIDE WE KNOW WE GATHER DATA THROUGH INSTRUMENTS, OBSERVATIONAL CHECKLISTS, AND OF COURSE RECORDS LIKE PATIENT RECORDS THAT MIGHT HAVE SOME NUMERIC DATA IN THEM. ON THE QUALITATIVE SIDE WE TYPICALLY THINK ABOUT OPEN ENDED INTERVIEWS OPEN ENDED OBSERVATIONS, DOCUMENTS THAT RUN FROM FIELD NOTES TO JOURNALS TO MINUTES OF MEETINGS, TO NEWSPAPER ARTICLES, AND OF COURSE THE AUDIO-VISUAL AREA IS ANOTHER AREA IN QUALITATIVE WHICH CONSISTS OF PHOTOGRAPHS, VIDEOTAPES, ARTIFACTS, PICTURES. WHAT WE'RE EXPERIENCING IN THE QUALITATIVE AREA IS AN ENLARGEMENT OF THE FORMS THAT ARE CONSIDERED QUALITATIVE DATA. ESPECIALLY THROUGH THE TECHNOLOGY AREA SO NOW EMAIL MESSAGES, TEXT MESSAGES, WE'RE THINKING ABOUT LEAVING -- SEE STUDIES WHERE SOUNDS ARE COLLECTED. SO QUANTITATIVE HAS A LITTLE NARROWER LIST IN MY MIND OF THE FORM OF DATA COLLECTION WHEREAS QUALITATIVE IS A LITTLE BIT LARGER LIST AND KEEPS EXPANDING TO NEW AREAS. A LITTLE BIT LARGE E PICTURE THAN METHODS WE HAVE DESIGNS. AND I REALLY FEEL A GOOD MIXED METHODS PROJECT WOULD HAVE A RIGOROUS QUANTITATIVE DESIGN TO IT AS WELL AS QUALITATIVE DESIGN. SO THAT WOULD CONSIST OF THE WAY YOU FORM A QUESTION, YOUR DATA COLLECTION, YOUR DATA ANALYSIS, YOUR INTERPRETATION I LISTED TYPICAL QUANTITATIVE METHOD WE SEE IN HEALTH SCIENCES OVER HERE. QUALITATIVE METHODS THAT WE SEE OVER HERE. THERE MIGHT BE AN RCT DEVELOPED THAT'S FOLLOWED BY IN DEPTH CASE STU DI TYPE OF DESIGNS. WE HAVE TWO TYPES OF DISIENS, THE MOST RIGOROUS WAY TO LOOK AT QUALITATIVE AND QUANTITATIVE METHODS. FROM THESE DESIGNS I THINK THERE ARE CERTAIN RESULTS WE'RE AFTER. ON THE QUANTITATIVE SIDE WE WANT TO IDENTIFY TRENDS, WE WANT TO BE ABLE TO GRAPH THEM, COME UP WITH STATISTICS. WE WANT TO MAKE THEM INTERPRETATIONS BASED ON PROBABLE CAUSE AND EFFECT. WE WANT TO COMPARE GROUPS, WE HAVE TO START WITH AN A PRIORI THEORY. THERE ARE CERTAIN BENCHMARKS OF GOOD QUANTITATIVE RESEARCH, CERTAINLY VALIDITY, RELIABILITY, GENERALIZABILITY. CONTROL. OVER ON THE QUALITATIVE SIDE WHAT WE'RE HOPING TO COME UP WITH WOULD BE INDIVIDUAL STORIES. PATIENTS, NARRATIVES OF THEIR ILLNESSES. WE WANT TO PRESENT A PICTURE OF DIFFERENT PERSPECTIVES RECOGNIZING THAT WHEN WE TALK WITH PEOPLE WE'RE NOT JUST GOING TO COME UP WITH ONE PERSPECTIVE, WE'LL SEE AN ARRAY OF PERSPECTIVES, ARRAY OF POSSIBILITIES. WE WANT TO PAINT THE COMPLEXITY, RATHER THAN NARROWING TO A FEW VARIABLES, A REALLY GREAT QUALITATIVE STUDY PRESENTS THE COMPLEXITY OF THE SITUATION. WE WROTE AN ARTICLE ON WAITING FOR A LIVER TRANSPLANT THAT YOU MAY HAVE SEEN IN QUALITATIVE INQUIRY AND OUR INTENT THERE WAS TO JUST SIMPLY TALK ABOUT HOW COMPLEX THAT SITUATION IS AVOIDING. GETTING THE PROCESS UNDERSTANDINGS, BRINGING IN THE CONTEXT, THE SETTING, THE SITUATION, JUST LIKE JAMES CAMERON BROUGHT IN THE DEBRIS FEEL FIRST, TO SET THE LARGER PICTURE FOR WHY THE TITANIC WENT DOWN. OFTEN WE WORK MORE INDUCKTIVELY, QUALITATIVELY, COMING UP WITH AN INTERESTING PERSUASIVE ACCOUNT, REALISTIC AND BASED IN A SETTING. SO WHEN I LOOK AT QUALITATIVE AND QUANTITATIVE I SEE DIFFERENT RESULTS OCCURRING FROM OUR PROJECTS. WHEN WE BEGIN TO INTEGRATE THESE TWO, WE NEED A GOOD RATIONALE WHY WE WANT TO INTEGRATE THEM, WHY IT'S IMPORTANT. OVER THE YEARS THERE'S BEEN LONG LISTS DEVELOPED FOR RATIONALES FOR DOING MIXED METHODS RESEARCH. ON THIS SLIDE I HAVE HIGHLIGHTED IN MY MIND SOME OF THE MAJOR ONES THAT I HAVE WORKED ON AND THAT I HEAR ABOUT. FIRST OF ALL MIXED METHODS IS USED TO HELP EXPLAIN WHY A MECHANISM WORKS OR DOESN'T WORK. SO WE MIGHT GATHER QUALITATIVE DATA TO FOLLOW-UP ON THE QUANTITATIVE CORRELATIONAL STUDY. THE WAY I SEE IT IN QUANTITATIVE RESEARCH WE GAIN VALUABLE DATA BUT IT'S A SURFACE PICTURE, IT'S A PICTURE CREATED FROM P VALUES AND SIGNIFICANT EFFECTS AND CONFIDENCE INTERVALS. SO A GENERAL TREND WHICH IS VERY IMPORTANT, BUT TO GET TO THAT LEVEL OF DETAIL WHI THOSE TRENDS MIGHT HAVE OCCURRED, NOW WITH FOLLOW-UP ON QUALITATIVE DATA. OTHER ASPECTS THERE'S VALIDITY PURPOSE. HOW DOES THE QUALITATIVE INSTRUMENT DATA COMPARE TO INTERVIEW DATA THAT WE'RE GATHERING. ONE WHERE PEOPLE FILL OUT AN INSTRUMENT ON THEIR OWN TIME AND ANOTHER WHERE WE'RE TALKING TO THEM. WE HOPE THESE TWO PICTURES WOULD CONVERGE AND THERE MIGHT BE SOME VALIDITY TO THESE TWO DATABASES. ANOTHER POPULAR USE FOR MIXED METHODS RESEARCH IS TO HELP DESIGN A GOOD PSYCHOMETRICALLY SOUND STRUM FOR MEASUREMENT. SKILL DEVELOPMENT HAS BEEN AROUND FOR MANY YEARS. SYSTEMATIC HOW WE TALK ABOUT IN THE MIXED METHODS FIELD. WHERE YOU FIRST GATHER QUALITATIVELY SOME DATA PHASE 1, PHASE 2 WOULD BE TO TAKE THAT -- THE RESULTS FROM THAT QUALITATIVE DATA AND DEVELOP AN STRUM THAT FITS POPULATION YOU'RE STUDYING. PHASE 3 TO TEST OUT THAT INSTRUMENT WITH A LARGER SAMPLE FROM A POPULATION INSTRUMENT DESIGN. IN INTERVENTION WORK WE'RE INTERESTED IN WHAT ARE THE EXPERIENCE OF PARTICIPANTS IN THE ACTUAL EXPEERMT, THE PROCESS AND INTERESTED IN OUTCOMES, HOW DID THEY SCORE AT THE END OR DIFFERENT INTERVALS DURING THE INTERVENTION SO TO HELP EXPLAIN THE PROCESS AND THE OUTCOMES OF AN INTERVENTION TRIAL WOULD BE ANOTHER REASON FOR USING MIXED METHODS. ANOTHER REASON IS TO USE MIXED METHODS AND DIFFERENT PHASES IN A MULTI-YEAR MULTI-PHASE PROJECT. WE MIGHT BEGIN QUALITATIVELY TO UNDERSTAND PARTICIPANTS PERSPECTIVES, WE MIGHT DEVELOP AN INSTRUMENT, WE MIGHT THEN PUT THAT INSTRUMENT WITHIN A LARGER PROGRAM WE WANT TO EVALUATE OUT OF THE COMMUNITY, WE MIGHT WANT TO ASSESS HOW THEY RESPONDED TO THAT PROGRAM. QUANTITILY OR QUALITATIVELY. CAN YOU SEE MIXED METHODS FLOWING INTO THE A PROJECT TO SEE QUALITATIVE, QUANTITATIVE, QUALITATIVE QUANTITATIVE MUL PHASE PROJECTS STREAMED TOGETHER IN A MULTI-PHASE STUDY. WE SEE THIS IN PROGRAM EVALUATION, WE SEE THIS IN PROGRAM IMPLEMENTATION RESEARCH. THERE ARE PROBABLY SOME OTHER REASONS AND PERHAPS MY P COLLEAGUES WOULD EXTEB THIS LIST BUT THESE ARE PRIMARY ONE IS SEE PRIMARY REASONS FOR USED MIXED METHODS RESEARCH. SO WHAT HAPPENED OVER THE LAST DECADE OR SO IS THERE'S BEEN A LOT OF FOCUS BY THE MIXED METHODS RESEARCH COMMUNITY ON DIFFERENT DESIGNS, RESEARCH DESIGNS FOR DOING MIXED METHODS RESEARCH. WE STARTED DRAWING PICTURES OF THESE DESIGNS. SO MY COLLEAGUE AND I CALL THESE THE BASIC DESIGNS. WHERE WE'RE GOING TO COLLECT QUALITATIVE AND QUANTITATIVE AND COMPARE THE RESULTS AND MAKE AN INTERPRETATION. FOR THIS ONE AN EXPLANATORY DESIGN WHERE WE START QUANTITATIVE RESEARCH, COME UP WITH THE RESULTS AND FOLLOW-UP QUALITATIVELY AND EXPLORATORY, WHICH IS THE REVERSE OF EXPLANATORY, WE'RE GOING TO START WITH QUALITATIVE DATA FIRST, MAYBE BUILD AN INSTRUMENT AND TEST OUT THE INSTRUMENT QUANTITATIVELY. SO WE HAVE PICTURES EMERGE TO WHAT THE DESIGNS LOOK LIE WHICH ADVANCES MIXED METHODS RESEARCH, SECONDLY WE HAVE COME UP WITH NAMES AND YOU CAN SEE THAT THESE ARE SIMPLE DIAGRAMS. SO WHEN WE HAVE PEOPLE DESIGN THEIR FIRST DRAW THEY'RE DESIGN WE LIKE TO SUGGEST A FAIRLY SIMPLE DESIGN BUT MORE COMPLEX HERE IN A MINUTE. WE ALSO HAVE IMBEDDED DESIGNS. ACTUALLY WE CAME UP WITH THAT TERM BY LOOKING AT A NUMBER OF BMJ ARTICLES MIXED METHODS STUDIES, THEY USE THE TERM IMBEDDED OR NESTED BUT IT'S PUTTING A QUALITATIVE COMPONENT INTO AN INTERVENTION TRIAL. IT DOESN'T NECESSARILY NEED TO GO THAT WAY. YOU COULD PUT A QUANTITY TAI ACTIVE SURVEY WITHIN A LARGER QUALITATIVE ETHNOGRAPHIC PROJECT BUT MORE FREQUENTLY WE SEE QUALITY FLOWING INTO A QUANTITATIVE INTERVENTION TRIAL OR QUALITATIVE COULD FLOW INTO A CORRELATIONAL STUDY THERE IS A NUMBER OF OPTIONS. WE'RE CALLING THIS SET MORE COMPLEX DESIGNS HERE. TRANSFORMATIVE, WHICH WE DONE SEE A LOT OF IN THE HEALTH SCIENCES RIGHT NOW BUT FOR EXAMPLE THIS IS USING A LENS, A THEORETICAL LENS LIKE A FEMINIST PERSPECTIVE AND HAVING THAT ENCASE THE MANY MIXED METHODS PROJECT. SO FEMINIST PROJECT IN CASE MIXED METHODS MIGHT DIRECT WHAT THE PROBLEM IS, HOW YOU FRAME THE QUESTION, HOW YOU CHECK DATA FROM PEOPLE, FROM PARTICIPANTS, WHAT TYPE OF CHANGES YOU MIGHT RECOMMEND IN THE END. SO A RACIAL LENS, ETHNIC LENS, A DISABILITY LENS, I'M WORKING WITH THE SOCIAL SECURITY ADMINISTRATION RIGHT NOW IN THE DISABILITY AREA. WHEN I THINK ABOUT THEIR MIXED METHODS PROJECTS WE'RE BEGINNING TO SKETCH UP DISABILITY FRAMEWORK SURROUNDS DIFFERENT PARTS OF THE STUDY. THEN ANOTHER COMPLEX DESIGN IS THE MULTI-PHASE DESIGN RIGHT DOWN HERE, WHERE THERE'S OVERALL PROGRAM OBJECTIVE AND THEN YOU MIGHT COLLECT QUALITATIVE AND QUANTITATIVE AND WITH ONE BILLING ON THE OTHER THROUGHOUT A PROJECT THAT MIGHT SPAN 3 TO 5 YEARS. PROGRAM EVALUATION, EVALUATING INTERVENTION APPLIED IN THE COMMUNITY WITH THE EXAMPLES OF THIS. ANOTHER THING THE MIXED METHODS RESEARCH THEY HAVE COME UP WITH A SHORTHAND WAY OF DRAWING DIAGRAMS, CONVENE WHAT MIXED METHODS RESEARCH IS. BASE ON THE WHOLE IDEA WHEN YOU'RE GATHERING MULTIPLE FORMS OF DATA YOUR SPENDING CAN BE COMPLEX. IT WAS ABOUT 2005 OR SO I WAS TALKING TO A PROGRAM OFFICER AT NIH AND THIS PERSON WAS SAYING JOHN, WE LIKE THESE MIXED METHODS STUDIES BUT THEY ARE SO COMPLEX, WE CAN'T UNDERSTAND, YOU GOT ALL THESE DIFFERENT FORMS OF DEATS, ALL THESE DIFFERENT ANALYSES, IT WAS AT THAT POINT WE BEGAN DRAWING DIAGRAMS. THESE ARE EXTREMELY HELPFUL CONVEYING THE OVERALL PROCEDURE. IF YOU THINK A MINUTE AT LEAST IN SOCIAL SCIENCE AN HEALTH SCIENCE RESEARCH WE HAVE DIAGRAMS FOR THEORIES. WHY CAN WE HAVE DIAGRAMS FOR OUR METHODS AS WELL SO WE DRAW THE PICTURES AND WHICH NEED ADINOATION SYSTEM. SO YOU CAN SEE HERE THAT IF YOU HAVE LOWER CASE LETTERS, MEAN LESS PRIORITY. IT'S POSSIBLE TV A LARGER EXPERIMENT QUANTITATIVE IN CAPITAL LETTERS AND SMALLER AS MUCH CAPITAL LETTERS SO THE PRIORITY GIVEN TO THE QUALITATIVE AND QUALM CAN BE DISPLAYED BY LOWER CASE LETTERS OR CAPITAL LETTERS SO IF IT'S A CAPITAL THAT MEANS IT'S A HIGHER PRIORITY. CONVERGENT WE USE A PLUS SIGN, SEQUENTIAL WE USE AN ARROW. THESE WERE ALL STARTED BY JANUARY MORRIS IN THE NURSING FIELD IN THE EARLY 1990s. SO WHEN WE STARTED THINKING ABOUT IMBEDDED DESIGNS WE THINK ABOUT THE IMBEDDED PART IN PARENTHESES, IT'S POSSIBLE TO GO BACK AND FORTH SO WE HAVE A RECURSIVE SYMBOL ON THAT MULTI-PHASE STUDY, YOU WILL SEE THEM DIAGRAMMED WITH BOXES AROUND THE DIFFERENT STUDIES THAT ARE LINKED TOGETHER. THEN JANUARY MORRIS FROM THE NURSING FIELD CAME ONE THE IDEA OF EQUAL SIGN TO INDICATE THE REASON FOR MIXING. SO IF YOU WERE TO VISIT WITH ME ABOUT YOUR MIXED METHODS PROJECT I WOULD DRAW A PICTURE AND THAT WOULD BE SOMETHING LIKE THIS. YOU'RE GOING TO FIRST START QUANTITATIVELY WITH A MAJOR EMPHASIS ON QUANTITATIVE RESEARCH, FOLLOWED THOSE RESULTS UP WITH QUALITY DATA COLLECTION, AND THE REASON FOR DOING THIS TO HELP EXPLAIN THE RESULTED. WE HAVE THIS SHORTHAND LABELS FOR MAPPING OUT THESE DESIGNS THAT BECOME POPULAR THE LAST FEW YEARS. WE HAVE DRAWN OUR VISUALS IN A MORE COMPLEX WAY, YOU SAW VERY SIMPLE MODELS OF BASIC DESIGN. THIS IS A CONVERGENT DESIGN GATHERING QUANTITATIVE AND QUALITATIVE DATA BUT OFF TO THE LEFT OF THESE LARGER BOXES THE DETAILED PROCEDURES ARE LISTED. TO THE RIGHT OF THE BOXES YOU SEE PRODUCTS. OFTEN EACH PHASE IN A PROJECT THERE'S DIFFERENT PRODUCTS ESPECIALLY FOR STATE PROJECTS FOR FEDERAL PROJECTS, THE INVESTIGATORS NEED TO COME FORWARD WITH. SO HERE IS YOUR PROCEDURES. ALONGSIDE HERE, HERE ARE THE PRODUCTS SO EACH STAGE WE CAN SEE EXACTLY WHAT'S GOING ON. SO ALL THE NIH PROJECTS I WORK ON THESE DAYS WE PUT A DIAGRAM LIKE THIS FOR THE DESIGN, WE PUT THIS LEVEL OF DETAIL IN SO WE CAN SEE IT. I WANT TO SHOW OTHERS HERE, WHEN WE PUT QUALITATIVE INTO AN INTERVENTION TRIAL ANOTHER NURSING RESEARCHER (INDISCERNIBLE) DOWN AT NORTH CAROLINA CAME UP WITH THIS SCHEME TO ADD QUALITATIVE BEFORE THE EXPERIMENT BEGINS, LIKE THE HEALTH RECRUIT PARTICIPANTS. SO I CAN POINT TO A NICE DMJ ARTICLE THAT USES THAT APPROACH. COME IN DURING THE EXPERIMENT TO UNDERSTAND BARRIERS AND FACILITATORS. I'M WORKING ON A PROJECT THAT USES THAT MODEL EXPLORING WOMEN WITH CANCER AND THEIR TREATMENTS QUALITATIVE COME IN AFTER THE EXPERIMENT TO HELP EXPLAIN SOME OF THE OUTCOMES. THERE'S GREAT MIXED METHODS PROJECTS THAT BRING QUALITATIVE IN AT THAT POINT. SO I'M FORESHADOWING TOPICS, ONE IS BRINGING QUALITATIVE DATA INTO INTERVENTION TRIALS. HERE IS AN ARTICLE BY ROGERS AND OTHERS ON THE MANAGEMENT OF ANTIPSYCHOTIC MEDICINE. THAT IS ACTUALLY BRINGING QUALITATIVE IN BEFORE THE INTERVENTION BEGAN, CONDUCTING THE EXPERIMENT, AND BRINGING QUALITATIVE IN AFTER THE INTERVENTION TO EXPLAIN RESULTS. THEY DID NOT HAVE THIS PICTURE IN THEIR JOURNAL ARTICLE BUT WE DREW THIS. I THINK IT'S AN ACCURATE REPRESENTATION OF THE FLOW OF THEIR IDEAS. YOU CAN SEE MAKING THE PRESENTATION ON THIS PROJECT HAVING A VISUAL PICTURE LIKE THIS, CAN BE AN IMMENSE AID.ca x SO THIS IS PAIN MANAGEMENT IN ONCOLOGY AN NIH PROJECT. ANOTHER EXAMPLE OF QUALITATIVE IN TO AN INTERVENTION RCT, BRINGING QUALITATIVE RCT, IT WAS BROUGHT IN THIEWL CHIEWLLY DURING WHILE THE EXPERIMENT WAS GOING ON. THAT HE GATHER QUALITATIVE DATA WITH HIGH DOSE CONDITION AN GATHER QUALITATIVE DATA IN THE LOW DOSE CONDITION. ONE OF THE CHALLENGES BEFORE US, IN THE MIXED METHODS COMMUNITY IS HOW TO DRAW AN IMBED DESIGN PUTTING QUALITATIVE DATA INTO AN INTERVENTION TRIAL. SUDDENLY MORE DIAGRAMS TO SEE WHAT THE MODELS LOOK LIKE. I LIKE THIS ONE QUITE A BIT. HERE IS SEQUENTIAL EXPLANATORY DESIGN WHERE WE START WITH QUANTITATIVE STRAND AND COME UP WITH RESULTS AND THEN WE DESIGN QUALITATIVE IN ORDER TO EXPLAIN THOSE. SO IT COULD BE ORGANIZED IN A VERTICAL WAY, JUST LIKE THIS. THIS IS A MULTI-PHASE NIH PROJECT LOOKING AT INTERVENTIONS FOR YOUTH IN SRI LANKA, PUBLISHED IN THE JOURNAL OF MIXED METHODS RESEARCH. IF YOU LOOK AT THE STUDY, THEY HAVE QUALITATIVE AND QUANTITATIVE FLOWING THROUGHOUT MANY DIFFERENT PHASES HERE. THIS IS WORKING WITH EXISTING THEORY, DEVELOPING A CONCEPTUAL MODEL, DEVELOPING INSTRUMENTS, THEN EVALUATING THE INSTRUMENTS, PUTTING THE PROGRAM IN PLACE, EVALUATING THE PROGRAM. SO THIS IS A CASE WHERE THEY HAD QUALITATIVE AND QUANTITATIVE STUDIES AT DIFFERENT PHASES THROUGHOUT THEIR MULTI-PHASE PROJECT. THERE'S SOME GOOD EXAMPLES OF THIS. WHAT RECOMMENDATIONS WOULD I MAKE FOR DESIGNS AND IMPLEMENTATION DISSEMINATION? CERTAINLY THE IMBEDDED DESIGN IS A VERY POPULAR ONE IN THE HEALTH SCIENCES. YOU'RE IMBEDDING QUALITATIVE DATA WP AN RCT, A VERY RIGOROUS DESIGN. BUT EXPLANATORY, WHEN I DID WORK ON TRAUMA RESEARCH, MOST STUDIES WERE EXPLANATORY, MIXED METHODS WERE EXPLANATORY WHERE THEY GATHERED QUANTITATIVE DATA AND GATHERED QUALITATIVE DATA TO HELP EXPLAIN THE QUANTITATIVE RESULTS. OF COURSE WITH PROGRAM IMPLEMENTATION, THE MULTI-PHASE DESIGN, ANOTHER POPULAR ONE. YOU MIGHT BE CURIOUS AT THIS TIME WHAT A SOCIAL SCIENTIST HAS TO SAY ABOUT HEALTH SCIENCES. OUR CONCEPTUALIZATION IN THE SOCIAL SCIENCE OF MIXED METHODS WAS WE'RE TRYING TO BALANCE TWO EQUAL ENTITIES HERE, QUANTITATIVE AND QUALITATIVE INTEGRATE TO TWO DATABASES. WHEREAS THE MODEL THAT I SEE EMERGING IN THE HEALTH SCIENCES IS TO PUT QUALITATIVE WITHIN QUANTITATIVE. QUALITATIVE WITHIN INTERVENTION TRIAL FOR EXAMPLE. KIND OF SHIFT WITH MY THINKING WORKING WITH HEALTH SCIENCES. AS MIXED METHODS RESEARCH GOES OUT ACROSS DIFFERENT FIELDS, ALL TYPES OF DISCIPLINES, THERE WILL BE A DISCIPLINE ADAPTATION, HEALTH SCIENCE ADAPTATION OF WHAT MIXED METHODS IS. I SEE IT MOVING THAT DIRECTION. ANOTHER TREND IN THE HEALTH SCIENCES IS EVERYONE WANS TO KNOW THE DETAIL PROCEDURES ON DOING THINGS. SO IF YOU LOOK IN OUR BOOK THAT I WROTE WITH PLANO CLARK FOR EACH OF THESE DESIGNS, WE TAKE THE READER THROUGH DETAILED STEPS ANALYSTED EACH PHASE, HERE IS THE QUANTITATIVE PHASE, THEN THE STRATEGIES TO FOLLOW UP ON THE RESULTS, THEN QUALITATIVE PHASE AN INTERPRETING IT. SO WE'RE BEGINNING TO GET QUITE A BIT OF DETAIL HOW TO DO THESE DIFFERENT TYPES OF DESIGNS. WE ARE ALSO GETTING GOOD STUDIES. TODAY WE'LL HEAR SOME GREAT MIXED METHODS PROJECTS AND WE CAN HEAR THE DETAIL PROCEDURES THEY'RE USING. THE TOPIC OF INTEGRATION WE'LL BE VISITINGm!np‡ER IN THE DAY TOO. AND INTEGRATION CAN MEAN SEVERAL THINGS JUST FOR BASIC IDEA WHEN WE'RE CONVERGING OUR DATA WE'RE BRINGING THE TWO DATABASES TOGETHER, CONVERGING INFORMATION OR INTEGRATING. WHEN WE COLLECT OUR QUALITATIVE DATA, COME UP WITH THE RESULTS, FOR EXAMPLE, DEVELOPMENT INSTRUMENT, THAT INTERSECTION BETWEEN COMING WITH RESULTS AN FORMING THAT INSTRUMENT WE'RE INTEGRATING, THINKING HOW ONE FORM OF DATA BUILDS ON THE OTHER. WHEN WE CONDUCT A RANDOMIZED CRIMINAL TRIAL AND PUT QUALITATIVE DATA INTO IT, WE'RE THINKING HOW WE'RE INTEGRATING THE TWO DATABASES TO HELP US UNDERSTAND THE PROBLEM SO I SEE INTEGRATION FLOWING IN THIS WAY. ANOTHER THING WE HAVE DONE IN THE MIXED METHODS FIELD IS COME ONE A WHOLE NEW TYPE OF RESEARCH QUESTION. CALLED A MIXED METHODS QUESTION. BECAUSE IF YOU HAVE ONE OF THESE DESIGNS YOU'RE TRYING TO ANSWER A LARGER QUESTION. BASICALLY I'M ENCOURAGING INDIVIDUALS TO PUT INTO THEIR STUDY AIMS A MIXED METHODS AIM OR QUESTION THAT WOULD COMBINE BOTH QUANTITATIVE AN QUALITATIVE. WHAT WAYS DO THE QUALITATIVE THEMES PROVIDE AN EXPLANATION FOR THE QUANTITATIVE STATISTICAL RESULTS? THAT'S KIND OF A METHODS ORIENTATION. BUT WE CAN SHIFT IT INTO MORE OF A CONTENT N WHAT WAYS DO THE PROVIDER PATIENT COMMUNICATION PROVIDE AN EXPLANATION FOR THE PREDICTORS OF QUALITY OF HEALTH. YOU CAN SEE WHERE I HAVE A QUALITATIVE PART AND I HAVE A QUANTITATIVE PART. SO WE'RE BEGINNING TO THINK ABOUT WHAT IS THE QUESTION WE'RE TRYING TO ANSWER WITH OUR MIXED METHODS PROJECT AND HOW DO WE GO ABOUT WRITING IT. WE KNOW THAT REGARDLESS OF OUR METHODS WE WAN THEM RIGOROUS, SYSTEMATIC, STATE-OF-THE-ART, EASY TO UNDERSTAND, USEFUL INFORMATION. SO WHAT WE'RE SEEING NOW THESE AREN'T EASY TO SEE, WE'RE SEEING EXPANSION OF MIXED METHODS THIS IS AN AREA OF IMAGING, THIS AN ARTICLE ON THE USE OF MIXED METHODS RESEARCH AND STATISTICAL THERAPY, THIS IS AN ARTICLE THAT I WORKED ON THAT TALKS ABOUT THE APPLICATION OF MIXED METHODS RESEARCH AND TRAUMA. THIS IS ONE I WORKED ON WITH DR. FEATHERS HERE IN PRIMARY CARE. THIS IS ONE IN MENTAL HEALTH NURSING, THESE ARE ALL DISCUSSIONS USING MIXED METHODS WITHIN A PARTICULAR HEALTH SCIENCES FIELD. THIS IS ONE ON INFORMATION TECHNOLOGY, THIS IS ONE ON MEDICAL EDUCATION THIS IS PALLIATIVE CARE, THIS IS ONE ON LONGITUDINAL JERN TO LOGIC DATA. THIS IS ONE HEALTH PSYCHOLOGY. THIS IS ONE ON FORMING AN ACADEMIC TEAM THAT LESLIE AUTHORED HERE. WHAT'S HAPPENED IS THAT THE WAY MIXED METHODS SEEMED TO BE SPREADING ACROSS THE HEALTH SCIENCES, SOMEONE MR. PLACE AN ARTICLE IN ONE OF THE LEADING JOURNALS TALKING ABOUT THE USE OF MIXED METHODS RESEARCH BASICALLY WHAT IT IS, WHAT THE DESIGNS MIGHT BE AND WE'LL GIVE SPECIFIC ILLUSTRATIONS WITHIN THE FIELD. SO FOR EXAMPLE, IF YOU LOOK AT MY ARTICLE ON TRAUMA RESEARCH IN THE MENTAL HEALTH FIELD, IT BEGINS BY AN OVERVIEW OF MIXED METHODS RESEARCH AND THEN WE'LL BRING IN THREE OR FOUR TRAUMA RESEARCH PROJECTS THAT ARE MIXED METHODS TO ILLUSTRATE HOW INDIVIDUALS ARE USING MIXED METHODS. THIS HAS PROFOUND IMPACT ACROSS THE SCIENCE FIELD WHEN SOMETHING LIKE THIS OCCURS. THERE ARE BOOKS OUT THERE, I WOULD LOVE TO COUNT, WE MUST BE SOMEWHERE AROUND 16, 17, 18 BOOKS ON MIXED METHODS RESEARCH, EACH YEAR NEW ONES COME OUT. THERE ARE COUPLE OF BOOKS THAT ARE TARGETED INTO THE HEALTH SCIENCES, A BOOK FROM AUSTRALIA ON MIXED METHODS RESEARCH HEALTH SCIENCES AN NURSING. JANUARY MORRIS'S BOOK ON MIXED METHODS DESIGN BRINGS IN MANY EXAMPLES FROM THE HEALTH SCIENCES. MY ADVICE IS TO BEGIN READING SOME OF THIS LITERATURE THERE IS NO REASON TO REINVENT THE WHEEL AS YOU DO YOUR HEALTH SCIENCE RESEARCH. WE BRING INTO OUR BOOKS MANY HEALTH SCIENCE EXAMPLES TO SHARE. LET'S LINK TO DISSEMINATION IMPLEMENTATION RESEARCH AND JENNIFER WILL BEGIN BUILDING THAT BRIDGE FOR US NOW. IML >> THANK YOU JOHN, I'M HAPPY TO BE HERE TO TALK TO YOU ABOUT HOW THIS APPLIES TO DISSEMINATION IMPLEMENTATION RESEARCH. IP ALSO WANT TO THANK HELEN AND KIMBERLY AND THE GROUP THAT HELPED PUT THIS ON, IT'S A PLEASURE TO WORK WITH KIMBERLY SPENCER THE APPLY MIXED METHODS RESEARCH TO DISSEMINATION IMPLEMENTATION. SO I HAVE THE PLEASURE OF TALKING WITH YOU ABOUT WHAT TO ME HAS BEEN ONE OF THE MORE COMPLEX COMPLICATED AREAS OF RESEARCH THAT I HAVE BEEN INVOLVED WITH WHICH IS DISSEMINATION IMPLEMENTATION. SO FOR -- ACCORDING TO THE PROGRAM ANNOUNCEMENT FOR THAT NIH HAS OUT FOR DISSEMINATION IMPLEMENTATION RESEARCH IT INCLUDES ALL THESE THINGS SO LET'S WALK THROUGH THEM, DISSEMINATION IMPLEMENTATION RESEARCH INCLUDES IDENTIFYING UNDERSTANDING AN OVERCOMING BARRIERS SO YOU'RE LOOKING AT BARRIERS AND FACILITATORS TO THE ADAPTION ADAPTATION ADOPTION AND INTEGRATION OF EVIDENCE BASED INTERVENTIONS TO -- THAT PREVIOUS RESEARCH HAS SHOWN TO BE EFFICACIOUS AN EFFECTIVE BUT UPTAKE IS LIMITED OR SIGNIFICANTLY DELAYED SO EACH SECTION HAS SOMETHING IMPORTANT TO THINK ABOUT AND STARTING WITH -- THIS IS EVIDENCE BASED INTERVENTION SO DISSEMINATION IMPLEMENTATION RESEARCH IS NOT ABOUT DEVELOPING INTERVENTION, IT'S NOT ABOUT STARTING FROM SCRATCH AND TRYING TO FIND SOMETHING THAT WILL WORK, IT'S MORE ABOUT TAKING SOMETHING THAT IS -- HAS ALREADY BEEN SHOWN TO WORK, SOMETHING THAT IS EFFECTIVE OR EFFICACIOUS BUT IS NOT BEING TAKEN UP IN THE ENVIRONMENT IN WHICH IT WOULD BE MOST HELPFUL SO YOU COULD PROBABLY THINK OF EXAMPLES ONE CLEAR EXAMPLE IS HAND WASHING, NOT TAKEN UP AS MUCH AS IT COULD BE IN HOSPITAL SETTINGS. SO AN EXAMPLE COULD BE TO LOOK AT WHAT ARE BARRIERS TO HAVING HAND WASHING AT FREQUENCY RECOMMENDED IN THAT KIND OF ENVIRONMENT. THERE'S ALSO ADDITIONAL AREAS AROUND DISSEMINATION IMPLEMENTATION RESEARCH THAT INCLUDES LOOKING AT METHODS FOR DISSEMINATION IMPLEMENTATION. WHAT TRAINING FOR EXAMPLE WOULD BE BEST AMONG LOTS OF OPTIONS FOR TRAINING FOR DISSEMINATING AND EVIDENCE BASED TREATMENT. IS THIS A ONE TIME TRAINING, IS IT A COURSE, DOES IT INVOLVE ONGOING CONSULTATION, IS IT WEB-BASED, IS IT SOME OTHER MODERN WAY, ARE YOU CHECKING BACK IN WITH PEOPLE, WHAT FREQUENCY. ALL THOSE ARE QUESTIONS THAT FALL FULLY WITHIN THE DISSEMINATION IMPLEMENTATION RESEARCH FIELD. ALSO LOOKING AT TOPICS OF HEALTH BEHAVIOR CHANGE INTERVENTION, PREVENTION DETECTION, DIAGNOSTIC TREATMENT AND QUALITY OF LIFE IMPROVEMENT INTERVENTION. SO THE TYPES OF INTERVENTIONS INCLUDED UNDER THIS ARE QUITE BROAD. BUT AGAIN, THEY DO NEED TO BE THINGS THAT HAVE BEEN SHOWN EFFICACIOUS AN EFFECTIVE. YOU NEED TO BE LOOKING AT WHAT ARE THE BEST WAYS TO GET THEM ACTUALLY USED IN THE SETTING WHICH THEY'RE DESIGNED TO BE USED. AGAIN, LOOKING AT THIS A DIFFERENT WAY DISSEMINATION IS THE TARGETED DISTRIBUTION OF INFORMATION AND MATERIALS TO A PARTICULAR AUDIENCE AN INTENT IS TO SPREAD KNOWLEDGE AND INTERVENTION AND IMPLEMENTATION IS STRATEGIES TO INTEGRATE INTERVENTIONS WITHIN THE SPECIFIC SETTINGS. SO IN ADDITION TO THOSE TWO, THERE'S ALSO TWO OTHER STAGES, OF COURSE THE STAGE BEFORE AND STAGE AFTER WHERE WE HAVE EXPLORATION AN ADOPTION WHICH IS THE INITIAL PROCESS OF ASSESSING THE MATCH BETWEEN WHAT THE ORGANIZATION OR COMMUNITY NEEDS, WHAT THE PRACTICE IS, WHAT THE PROGRAM NEEDS AN RESOURCES TO MAKE A DECISION TO PROCEED OR NOT PROCEED I'M WORKING ON A REVIEW RIGHT NOW RELATED TO ADOPTION WHERE THERE'S COUPLE OF DIFFERENT PARTS OF ADOPTION, PRE-ADOPTION STAGE FIGURING OUT WHAT'S GOING ON AND ADOPTION STAGE WHERE YOU'RE MAKING A DECISION. THERE ARE A LOT OF COMPLICATED FACTORS AROUND THIS. SO FOR EXAMPLE, IN A STATE SYSTEM WHICH IS WHERE I WORK, SOMETIMES ADOPTION IS VIEWEDDED AS AN AGENCY IS THINKING WE HAVE A LOT OF CLIENTS WHO HAVE THIS KIND OF ISSUE PERHAPS WE SHOULD ADOPT A TRAUMA BASED INTERVENTION BECAUSE WE HAVE CLIENTS WITH TRAUMA. THEY GO OUT TO THE LITERATURE, AND LOOK AROUND AT WHAT THE OPTIONS ARE FOR TRAUMA-BASED INTERVENTIONS AN THINK ABOUT WHAT THE TRAINING IS AND THEY DECIDE THEY DO WANT TO ADOPT SOMETHING AND THEN THEY BEGIN THE PROCESS OF TRAINING THEIR STAFF AND ACTUALLY IMPLEMENTING IT. IN THE REAL WORLD IT DOESN'T ALWAYS WORK LIKE THAT, SOMETIMES IT'S LIKE THIS WHERE THE STATE SAYS WE UNDERSTAND WE HAVE AN INTERVENTION THAT YOU MIGHT BE ABLE TO BENEFIT FROM. WOULD YOU LIKE TO DO IT OR NOT. EAN HEN THE STATE CAN PROVIDE TRAINING, SO THE PROCESS FOR MAKING A DECISION FOR ADOPTION CAN BE VERY DIFFERENT BUT THAT'S SOMETHING THAT'S REALLY KEY IF YOU CAN IMAGINE DIFFERENT PROCESSES TO DECIDING TO ADOPT MAY HAVE SERIOUS IMPLICATIONS FOR WHETHER THE PROJECT IS ULTIMATELY IMPLEMENTED. AND I WOULD SAY SUSTAINED WHICH IS THE FINAL STAGE WHICH IS THE SURVIVAL AND CONTINUED EFFECTIVENESS OF THE INTERVENTION, AND DIFFERENT HEALTH AREAS OF THE AWD YNS WHERE THERE'S A BIG INTERVENTION TO IMPLEMENT THAT NEW STRATEGY, MAYBE A YEAR LATER IT'S NOT WORKING ANY MORE, PEOPLE AREN'T ACTUALLY DOING IT, THERE'S TURN OVER, PEOPLE STOP DOING IT, THE SYSTEMS WEREN'T IN PLACE, MAYBE THE STAFF WERE TRAINED DOING THEIR PART BUT THE HEALTH IT SYSTEMS OR THE PROCESSES ARE NOT ADJUSTED TO BE ABLE TO ENCOURAGE SUSTAINABILITY SO THAT'S ANOTHER PART OF THE THIS PROCESS SO DISSEMINATION AND IMPLEMENTATION RESEARCH REALLY INCLUDES ALL FOUR OF THESE STEPS. SO SINCE WE'RE AT NIH I WANT TO LOOK WHERE DISSEMINATION IMPLEMENTATION FALLS WITHIN THE REALM OF TRANSLATIONAL RESEARCH SO WE ARE RIGHT THERE AT T-3. SO AS YOU PROBABLY KNOW STARTING AT 0 LOOKING AT BASIC RESEARCH QUESTIONS INTHE PHASE 1 AND 2 CLINICAL TRIALS, PHASE 3 CLINICAL TRIALS, WE ARE RIGHT THERE IN THE FUSION RESEARCH AND PHASE 4 CLINICAL TRIALS AND THERE'S SOME EXAMPLES THERE WHAT COULD HAPPEN. NOTE THIS IS DIFFERENT FROM THE ACTUAL OUTCOMES RESEARCH AT THE POPULATION LEVEL. SO THERE ARE LOTS OF MODEL MODELS, PRACTICAL ROBUST IMPLEMENTATION SUSTAINABILITY MODEL. THERE ARE I BELIEVE AT LAST COUNT TALKING WITH (INDISCERNIBLE) A FEW WEEKS AGO AROUND 16 MODELS RELATED TO DISSEMINATION AND IMPLEMENTATION. THIS IS A BETTER ONE, YOU CAN -- LET'S LACK AT THE PARTS OF IT. YOU HAVE CHARACTERISTICS OF THE INTERVENTION, AND ON -- IN THIS MODEL YOU HAVE THE ORGANIZATIONAL PERSPECTIVE FROM THE PERSPECTIVE OF STAFF, MANAGERS AND LEADERS, YOU ALSO HAVE PATIENT PERSPECTIVES THAT THE CHARACTERISTICS OF INTERVENTION, INTERACT WITH THE RECIPIENT AND WHO THE PEOPLE ARE WHO ARE GOING TO BE USING THE INTERVENTION. THERE'S THE IMPLEMENTATION SUSTAINABILITY INFRASTRUCTURE, THERE'S ALSO THE EXTERNAL ENVIRONMENT THAT RELATES TO BEING ABLE TO IMPLEMENT AND SUSTAIN INTERVENTION THEN THERE'S STAGES OF ADOPPLER ADOPTION AND MAINTENANCE AND HOW EFFECTIVE THE INTERVENTION ACTUALLY IS. IT'S A VERY QUICK AND BRIEF DESCRIPTION OF THE MODEL BUT IT HIGHLIGHTS THE MULTITUDE OF COP SEPTEMBERS THAT WE HAVE THAT WE'RE LOOKING AT. THESE ARE NATURE MULTI-LEVEL BECAUSE THEY INCLUDE A PRACTICER AND SOME SORT OF PATIENT WORKING ON IMPLEMENTING SOMETHING. THERE MAY ALSO BE -- THERE ARE LIKELY OTHER LAYERS INCLUDING MANAGERS, THE LEADERS IN THE ORGANIZATION, EXTERNAL ENVIRONMENT AND THEY MAY OR MAY NOT BE MULTIPLE PATIENTS SUCH AS HAVING CHILDREN AND FAMILIES OR PATIENTS AND THEIR FAMILIES. SO I I WENT THROUGH A NUMBER OF DESIGNS THAT CAN BE USED IN COMBINATION WITH MIXED METHODS RESEARCH, THESE ARE A NUMBER OF DIFFERENT DESIGNS USED IN DISSEMINATION AN IMPLEMENTATION RESEARCH. THIS IS DEFINITELY AN INCOMPLETE LIST AND YOU CAN SEE THE SAME POINT JOHN WAS MAKING HOW MANY OF THESE ARE DESIGNED AS QUANTITATIVE RESEARCH DESIGNS TO IMPLEMENT TO INCORPORATE A QUALITATIVE COMPONENT TO HELP UNDERSTAND THE PHENOMENON BEAR. LET'S TALK ABOUT THE OUTCOMES DISSEMINATION IMPLEMENTATION STUDIES. IN MOST HEALTH STUDIES YOU MAYBE FAMILIAR WITH THE OUTCOME VELTED TO HOW THE PATIENT IS DOING, REDUCED BLOOD PRESSURE, REDUCED INCIDENCE OF CANCER, BETTER QUALITY OF LIFE, IN DISSEMINATION IMPLEMENTATION THERE'S DIFFERENT OUTCOMES. IDEALLY WHEN YOU DO DISSEMINATION IMPLEMENTATION STUDY YOU WANT TO LOOK AT THESE OUTCOMES AND THE PATIENT OUTCOMES SO LET'S GO THROUGH THESE. THESE ARE FROM (INAUDIBLE) FANTASTIC REVIEW THIS YEAR LOOKING -- TRYING TO IDENTIFY OUTCOMES WE SHOULD BE LOOKING AT. SO WHAT IS ACCESSIBILITY? STAKEHOLDERS AGREEING THE INTERVENTION WORKS FOR THEM, THIS IS AT THE INDIVIDUAL STAFF LEVEL, IT COULD ALSO BE AT THE LEADER LEVEL. IF YOU'RE LOOKING AT MULTIPLE SYSTEMS. AND THE MEASUREMENT OF THAT CAN BE IN QUALITATIVE OR QUANTITATIVE WAYS, SUR FAYS INTERVIEWS OR ADMINISTRATIVE DATA. YOU CAN LOOK AT THE REACH OF THE INTERVENTION SO THE BREADTH WITH WHICH THE HEALTH INFORMATION SPREADS, FOR EXAMPLE, MAYBE YOU'RE DOING LEARNING COLLABORATIVES IN A STATE WHERE YOU WANT TO TRAIN PEOPLE IN HOSPITALS DOING A CERTAIN TYPE OF INTERVENTION AND YOU WANT TO LOOK AND SEE HOW FAR DOES THAT GO. IS IT ONLY THE PEOPLE WHO ARE ATTENDING THE TRAINING WHO ARE DOING THE INTERVENTION OR HAS THAT SPREAD WITHIN THE HOSPITAL? HAD IT SPREAD TO OTHER HOSPITALS? HOW WIDELY IS THAT INFORMATION GOING OUT. THERE ARE QUALITATIVE WAYS. ADOPTION, WE CAN TALK WHETHER THE INTERVENTION IS ADOPTED THE EXAMPLE THAT I USE BEFORE WHERE A STATE SYSTEM OR MAYBE SOMEONE WHO HAS NIH FUNDING APPROACHES A NUMBER OF DIFFERENT ORGANIZATIONS WHERE THEY ADOPT A CERTAIN INTERVENTION. SOME WILL SAY YES AND NOT END UP ADOPTING IT. SOME WILL SAY YES AND WILL END UP ADOPTING IT. SOMETIME IT IS MERE ISSUE HAS IT BEEN ADOPTED IS A USEFUL OUTCOME, IF YOU'RE IN A BROAD SYSTEM. THIS IS MEASURED IN A NUMBER OF WAYS, OF COURSE PEOPLE CAN SAY YES AND THEN NOT ACTUALLY DO IT. SO USING SURVEYS, USING OBSERVATIONS TO SEE IF FOR EXAMPLE A PARTICULAR SCREENING TOOL HAS BEEN IMPLEMENTED, CAN BE USEFUL DOING INFORMATIVE INTERVIEWS OR FOCUS GROUPS OR TYPE OF INTERVENTION PICKED UP ON ADMINISTRATIVE DATA, FOR EXAMPLE, DRUG TESTING OR URINALYSIS THAT'S A NEW INTERVENTION WITH EVERY PATIENT WHO HAS THIS PARTICULAR ISSUE. YOU CAN TAKE THAT UP IN THE AD MINUTE STRAIFTIVE -- ADMINISTRATIVE DATA AS WELL ADOPTED. LOOKING AT APPROPRIATENESS, HOW MUCH IS THIS INTERVENTION RELATED TO THE PARTICULAR CONTEXT AND HOW IS IT VIEWED AS BEING COMPATIBLE WITH WHAT'S DONE ALREADY SO A GREAT EXAMPLE IS (INDISCERNIBLE) THOSE FAMILIAR WITH SUBSTANCE ABUSE TREATMENT, IT'S A PHARMACOLOGICAL INTERVENTION FOR TREATING SUBSTANCE ABUSE, WHICH HAS HAD A LOT OF CHALLENGES BEING IMPLEMENTED WIDELY IN THE TREATMENT AGENCIES, AND SOME CHALLENGES AROUND THERE ARE IN HOW COMPATIBLE IT'S VIEWED WITH THE TRADITIONAL MODEL, SOME TRADITIONAL MODELS OF SUBSTANCE ABUSE TREATMENT. THERE ARE PLENTY OF OTHER EXAMPLES OF INTERVENTIONS Y RESEARCHERS HAVE THE BEST IN MIND WHAT THEY WAN DONE BUT IT DOESN'T QUITE FIT WELL WITH THE ACTUALTURE OF ORGANIZATION OR AGENCY OR HOSPITAL BEING IMPLEMENTED SO THERE ARE WAYS TO EXPLORE THAT. IT DOESN'T MEAN A DEAD END BUT IT DOES MEAN YOU WANT TO LOOK WHAT THE'S GOING ON. FEASIBILITY IS ALSO SOMETHING IMPORTANT THE EXTENT TO WHICH A PRACTICE IS CARRIED OUT, THERE ARE SOME FABULOUS INTERVENTIONS THAT REQUIRE THINKING OF MENTAL HEALTH INTERVENTIONS IN MY FEEL THAT REQUIRE 16 VISITS WITH THE FAMILY, THAT MAY NOT BE FEASIBLE. THAT MAY NOT BE SOMETHING THAT WORKS WITH THE POPULATION THAT IS BEING SEEN AT THAT CLINIC. THAT CAN BE MEASURED, THAT FEASIBILITY CAN BE MEASURED AS AT THE INDIVIDUAL LEVEL WHETHER THAT CAN HAPPEN WITH THE PARTICULAR PATIENTS AT THAT AGENCY, AT THE ORGANIZATION LEVEL, IS IT FEASIBLE FOR THE ORGANIZATION TO SEND ALL THEIR PEOPLE TO ONE WEEK TRAINING? HOW IS THAT GOING TO WORK? OR IS THE TRAINING MORE IN DEPTH, HOW DOES THAT WORK? POLICY LEVEL, IT'S SOMETHING GOING BACK TO BUPRONORFIN, IS IT A FEASIBLE INTERVENTION TO REQUIRE ALL SUBSTANCE ABUSE TREATMENT AGENCIES ADMINISTER BUPRONORFAN. I WOULD SUGGEST NOT BUT THAT'S A QUESTION TO LOOK AT RELATED TO DISSEMINATION IMPLEMENTATION. FIDELITY. THIS IS AN AREA THAT IS OFTEN CHALLENGING, AS I MENTION, A COMPLEX AREA SO YOU -- WHEN TRYING TO PUT A STUDY TOGETHER AND YOU HAVE THE PEOPLE WHO ARE TAKING PART IN THE INTERVENTION IN YOUR TRAINING YOU DO WANT TO FIND OUT WHETHER THEY'RE ACTUALLY USING THE INTERVENTION WITH FIDELITY. THE DEGREE TO WHICH THE PRACTICE WAS ACTUALLY IMPLEMENTED AS DESCRIBED OR INTENDED SOMETIMES IT HAPPENS SOMETIMES IT DOESN'T HAPPEN SO YOU CAN ASSESS THE DEGREE TO WHICH IT HAPPENS USING OBSERVATION WHICH IS RESOURCE INTENSIVE WAYS TO ASSESS IT USING CHECKLISTS OR CONTENT ANALYSIS OR SOME WAYS SELF-REPORT. THE COST OF INTERVENTION, THIS IS A WHOLE DIFFERENT AREA THAT DESERVES ITS OWN AREA OF STUDY LOOKING AT COST EFFECTIVENESS AND THE DEGREE TO WHICH COST IS RELATED TO SUCCESS OF IMPLEMENTING A PRACTICE AND SUSTAINABILITY OF IT PENETRATION, INTEGRATION OF PRACTICE INTO A STUDY, THIS IS DIFFERENT THAN THE REACH WE TALKED ABOUT BEFORE, THE REACH TALKED ABOUT THE BREADTH AND THE PENETRATION ESSENTIALLY TALKED ABOUT THE DEPTH. SO WHAT DEGREE IS THIS INTERVENTION INTEGRATED WITHIN A SETTING TO THE POINT WHERE IT'S ACCEPTED AS REGULAR PRACTICE. IF IT IS A POLICY, SOMETHING PEOPLE DO EVERY TIME THEY'RE SUPPOSED TO DO IT. WHATEVER THE INTERVENTION IS. THEN INTERVENTION SUSTAINABILITY WHICH IS MAINTENANCE OF THE PRACTICE IN ONGOING OPERATIONS, THIS CAN BE ASSESSED IN A NUMBER OF WAYS AS WELL. SO I HAVE A COUPLE OF SIMPLE DESIGNS, THE GRAPHICS -- I WANTED TO GIVE YOU AN OVERVIEW. WE'LL BE SEEING LATER TODAY QUITE A FEW EXAMPLES OF BRILLIANT MIXED METHODS PROJECTS RELATED TO DISSEMINAILG IMPLEMENTATION. SO -- DISSEMINATION AND IMPLEMENTATION SO THIS IS A SNEAK PREVIEW A STUDY OF EVIDENCE BASED PRACTICES AN COMMUNITY HEALTH, A MULTI-PHASE QUALITATIVE AND QUANTITATIVE STUDY WHERE FIRST PHASE INFORMS THE NEXT. SO FOR THE QUALITATIVE PORTION, THE CONDUCTED INTERVIEWS OF PRACTITIONERS TO UNDERSTAND CHARACTERISTICS, COMPETENCIES, CONDITIONS, CONTEXT, ASSOCIATED WITH ADOPTION OF EVIDENCE-BASED PRACTICES AS WELL AS FACILITATORS AN BARRIERS SO THOSE ARE A LOT OF PARTS FROM PROCTORS LIST OF DISSEMINATION IMPLEMENTATION OUTCOMES. SO THEY'RE ASSESSING BY INTERVIEW AND QUANTITATIVELY USING EVIDENCE BASED PRACTICE ASSESSMENT SCALE, GREG SCALE HE MAYBE TALKING ABOUT LATER TO DETERMINE THE DEGREE OF ADOPTION AND TO ANALYZE THAT CHANGE OVER TIME. SO AT MULTIPLE TIME POINTS THEY'RE COLLECTING THE SCALE THAT SHOWS QUANTITATIVELY WHAT'S BEING DONE AND ASSESSING QUALITATIVELY WITH ENTERRUE WHAT'S HAPPENING. SO HERE IS AN EXAMPLE OF IMPLEMENTATION, THIS IS A WEIGH MANAGEMENT PROGRAM EVALUATION, THIS IS AN IMBEDDED DESIGN WHERE THE QUANTITATIVE PORTION OF 55 INDIVIDUALS TRACKS THEIR PROGRAM COMPLETION RATES AND WEIGHT LOSS AND THEN THE QUALITATIVE SUB SAMPLE IS4 PORTION OF THOSE INDIVIDUALS SELECTED FOR THEIR SUCCESS OR FAILURE IN THE PROGRAM AND IDENTIFYING THE MECHANISMS BY WHICH THEY WERE ABLE TO BE SUCCESSFUL OR BY WHICH THEY WERE NOT ABLE TO BE SUCK SESTLES. -- SUCCESSFUL. HERE IS A THIRD DESIGN WHICH IS AN EXPLANATORY DESIGN WHERE YOU HAVE -- THEY'RE IDENTIFYING BEST PRACTICES FOR LIFESTYLE INTERVENTION PROGRAM CALLED NATIVE AMERICAN WOMEN CALL WAS WM. THEY TAKE THE HIGH PERFORMING SITES AND LOW PERFORMING SITES AND CONDUCT QUALITATIVE INTERVIEWS AND THOSE ARE BOTH QUALITATIVE. SO QUALITATIVE INTERVIEWS AND QUALITATIVE FOCUS GROUPS AND OBSERVATIONS USING QUANTITATIVE DATA TO DIVIDE THE TWO GROUPS FOR REVIEW AND THEN THEY ASSESS THE HIGH PERFORMANCE DATA FOR EACH OF THEM AND USED A LOT OF QUALITATIVE DATA TO DETERMINE THE CONTEXT THE PRACTICES THAT LED TO SUCCESS TO HIGH PERFORMANCE OR LOW PERFORMANCE ON BEING ABLE TO IMPLEMENT THE INTERVENTION. A FANTASTIC INTRODUCTION TO MIXED METHODS AND PROVIDE A NUMBER OF DESIGNS THAT CAN HELP YOU REALLY THINK THROUGH WHAT IT IS YOU'RE TRYING TO STUDY AND HOW TO DO THAT IN THE MOST EFFECTIVE WAY. YOU CAN CONSIDER WHICH DESIGNS ARE USEFUL TO YOU AND CONSIDER THE SKILLS AND KNOWLEDGE NEEDED. THERE'S A BIG ISSUE AROUND HAVING A TEAM THAT INCLUDES MIXED METHODS RESEARCHERS, QUALITATIVE RESEARCHERS AND QUANTITATIVE RESEARCHERS AND IN AN IDEAL WORLD ALL OF THOSE PEOPLE ARE TALKING TO EACH OTHER AND IN A LESS IDEAL WORLD SOMETIMES THEY DON'T TALK TO EACH OTHER AND THAT CREATES PROBLEMS ON YOUR STUDY AND THAT CAN BE DIFFICULT. BUT IDEALLY YOU WANT QUALITATIVE AND QUANTITATIVE RESEARCHERS LIKE WE HAVE HERE TODAY WHO ARE OPEN TO MIXED METHODS AND WHO ARE INTERESTED IN LEARNING HOW TO BRIDGE THESE THINGS TOGETHER INSTEAD OF HAVING A BATTLE WITHIN YOUR STUDY, NOT FUN FOR ANYONE. FOR MIXED METHODS RESEARCHERS LOOKING TO DISSEMINATION IMPLEMENTATION RESEARCH THINK IN TERMS OF WHAT ARE THE DETAIL PROCEDURES, THE ACTUAL PARTS OF DISSEMINATION IMPLEMENTATION YOU'RE LOOKING TO STUDY. LOOK ON HOW TO CONSIDER USING A STANDARD INTERVENTION TRIAL AND ONE OF THE STRAIGHT FORWARD WAYS RATHER THAN STARTING WITH A NEW DESIGN TO ANSWER A DIFFERENT KIND OF QUESTION. HOW WELL DELINEATED ARGUMENTS, ONE OF THE GREAT BENEFITS THAT I FOUND FOR STUDYING THE MIXED METHODS LITERATURE IS THAT IT REALLY STRONGLY SUGGESTS THAT YOU WALK TRUE EVER PART OF YOUR RESEARCH DESIGN AND DON'T TAKE ANYTHING FOR GRANTED FIGURING OUT THE BEST WAY TO ANSWER WHY YOU'RE DOING WHAT YOU'RE DOING, BECAUSE IT ANSWER IT IS QUESTIONS P YOU WANT THE ASK. FOR -- FINALLY FOR MIXED METHODS RESEARCHERS UNDERSTANDING THE PROCESSES OF RESEARCH ADOPTION, DISSEMINATION IMPLEMENTATION AND SUSTAINABILITY AND HOW THEY MIGHT BE USED IN DIFFERENT PHASES OF THOSE. SO WE HAVE AGREED, WE HAVE A GREAT SHOW FOR YOU TONIGHT, WE HAVE A GREAT WORKSHOP FOR YOU TODAY WITH SOME REALLY FANTASTIC RESEARCHERS, I THINK WE MIGHT HAVE ONE MINUTE FOR QUESTIONS. THANK YOU SMOWCH FOR BEING HERE, HOPE YOU HAVE A FANTASTIC DAY. [APPLAUSE] COUPLE OF QUESTIONS. JOHN IS COMING BACK UP. >> THERESA BETCOURT, HARVARD. I WOULD LIKE TO HEAR MORE ABOUT IMMUNITY BASED PARLTS PATORY RESEARCH BECAUSE OF THE POWER SHARING DYNAMIC, YOU CAN'T NECESSARILY FOLLOW THE SAME LINEAR PROCEDURES HOW YOU INCORPORATE THAT SORT OF ABILITY TO CHANGE COURSE AND EVEN HOW YOU WRITE PROPOSALS OR THINK AB THOSE DESIGNS. >> GREAT QUESTION AROUND COMMUNITY BASED PARTICIPATORY RESEARCH AND HOW TO INTEGRATE WITH MIXED METHODS. THIS IS NOT A WORKSHOP ON COMMUNITY BASED PARTICIPATORY RESEARCH, THERE'S WAYS WHICH VERY COMPATIBLE ONE IS THAT ONE OF THE PRIMARY ONES IS THAT THEY'RE FLEXIBLE DESIGNS. SO WHEN YOU'RE WORKING WITH COMMUNITY MEMBERSŤi ABOUT TRYING TO IDENTIFY WHAT ARE THE OUTCOMES YOU'RE LOOKING FOR AND PROCESSES YOU WANT TO ASSESS, MIXED METHODS IS VERY FLEXIBLE IN BEING ABLE TO LOOK AT EACH OF THOSE IN TURN SEQUENTIALLY OR CONCURRENTLY. AS FAR AS HOW TO WRITE IT UP, I THINK THERE'S A WORKSHOP, THERE'S PART OF THE WORKSHOP LATER TODAY THAT TALKS ABOUT WRITING OF MIXED METHODS RESEARCH. AND THERE'S ALSO THE NIH GUIDELINES THAT CAME OUT LAST FALL THAT ALSO COULD BE HELPFUL ABOUT THAT BUT I THINK PARTICULARLY GIVEN THAT CDPR HAS A LITTLE BIT LESS STRICT DESIGN AND THAT MIXED METHODSES CAN POTENTIALLY AS WELL TO THE EXTENT THAN YOU CAN DESCRIBE WHAT YOU PLAN TO DO, USE A MODEL, USE A VISUAL TO DESCRIBE EXACTLY WHAT YOU PLAN TO DO AT EACH STAGE AND WHAT YOU PLAN TO GET OUT OF IT AND EXPLAIN WHAT COULD HAPPEN IF THINGS GET DERAILED OR MOVED TO A DIFFERENT DIRECTION BECAUSE OF COMMUNITY INVOLVEMENT. OTHER THOUGHTS, CHUCK? >> I HAVE WRITTEN A CHAPTER ON THE TOPIC FOR AMERICAN EPA, AMERICAN PSYCHOLOGICAL ASSOCIATION BOOK THAT RECENTLY CAME OUT. I WOULD BE HAPPY TO SHARE THAT BE YOU. BASICALLY WE TAKE THE STEPS AND IMPLEMENTATION OF COMMUNITY BASED RESEARCH PROJECT, WE TALK ABOUT HOW AT EACH POINT YOU CAN GATHER QUALITATIVE AND QUANTITATIVE DATA AND HOW YOU MIGHT CONVERGE THE DATABASES TO UNDERSTAND EACH STEP IN THE PROCESS. WE CAME ONE A THEORETICAL MODEL, BASED UPON STUDIES OF COMMUNITY BASED PROJECTS. THIS, THE WHOLE NEW AREA FOR MIXED METHODS RESEARCHERS AND NOT MUCH HAS BEEN WRITTEN IN THE FIELD ABOUT IT. AND BUT I THINK IT'S AN EXCELLENT ILLUSTRATION OF AN OPPORTUNITY TO GATHER QAWP AND QUALITY DATA PHASES OF PROJECT. WE NEED GOOD MODELS THAT ARE MORE THAN JUST SAYING THIS IS A COMMUNITY BASED STUDY WITH QUALITATIVE AND QUANTITATIVE BUT HAS HAS A SPECIFIC TYPE OF MIXED METHODS DESIGN A RIGOROUS APPROACH. SO BE HAPPY TO SHOW THAT TO YOU. >> PETER NEWMAN, UNIVERSITY TORONTO. THIS MAY REFLECT IGNORANCE BUT I WONNER MIGHT THERE BE A TIME IN 10, 20 YEARS WHEN WE DON'T -- WE MAY STILL TALK ABOUT IMPLEMENTATION RESEARCH BUT THAT WE WOULD LOOK CRITICALLY SOMEONE WANTING TO RUN A LARGE SCALE RANDOMIZED CRIMINAL TRIAL WITHOUT DOING APPROPRIATE FORMATIVE RESEARCH UP FRONT WITHOUT IMBEDDING QUALITATIVE RESEARCH TO ANSWER KEY QUESTIONS WE COULD ANTICIPATE. SUCH THAT WE WOULD STILL HAVE IMPLEMENTATION SCIENCE AND THAT NEED BUT THAT THAT MIGHT BECOME SMALLER ACTUALLY, WON'T BE MUCH LARGER THAN BECOME A NORM THAT YOU HAVE TO DO MIXED METHODS RESEARCH TO UNDERSTAND MANY LARGE SCALE PROBLEMS. >> I HOPE THAT IS OUR FUTURE WELL. DOING QUALITATIVE AND QUANTITATIVE AND EXTENSIVE STUDIES I THINK IT IS HARD TO THINK OF A SITUATION THAT WOULD NOT BENEFIT FROM HAVING MULL PLT METHODS WHERE YOU CAN'T CHANGE THE QUESTION TO REALLY GET MORE OUT OF THE STUDY BY ADDING A METHOD. THAT'S A WAY TO GO. >> I SAY NEXT DECADE OR SO EVERYONE WILL BE DOING MIXED METHODS RESEARCH. BUT THERE ARE AREAS WHERE IT'S NOT GOING TO BE APPROPRIATE. I HAVE RUN INTO A COUPLE RECENTLY. ONE WAS WOORKING WITH THE SOCIAL SECURITY ADMINISTRATION AND HEARD A BRIEFING BY THE CENSUS BUREAU, IT'S BECOMING IMPRACTICAL TO GAIN QUALITATIVE DATA LARGE SCALE TRYING TO DISCERN THE BROADER TRENDS. SO UNQUESTIONABLY MORE QUANTITATIVE APPROACHES WILL BE USEFUL. ON THE QUALITATIVE SIDE, I WAS VISITING WITH INDIVIDUALS, PROJECT OFFICER FROM THE DRUG ABUSE INSTITUTE OF NIH WHO IS TALKING ABOUT STUDIES DONE IN ALASKA AND THESE PROJECTS, THIS PROJECT WAS LOOKING AT A PARTICULAR COMMUNITY WORRIED ABOUT HAVING A SUFFICIENT NUMBER OF INDIVIDUALS FOR STATISTICAL ANALYSIS. AND THEY WERE CALLING ON ALL THESE QUANTITATIVE STATISTICIANS TO COME UP WITH TECHNIQUES FOR SMALL SAMPLE SIZES WHICH IS OKAY, BUT THE REALITY IS A MORE QUALITATIVE STUDY APPROACH IS MORE USEFUL. SO I DON'T THINK MIXED METHODS CAN BE APPLIED TO EVERY PROBLEM BUT IT CAN BE APPLIED TO MANY PROBLEMS. [APPLAUSE] >> >> THANK YOU FOR THAT OVERVIEW. FOR THOSE HANGING OUT IN THE BACK, THERE'S LOTS OF ROOM UP HERE SO PLEASE FEEL FREE TO COME UP AN JOIN US. MEAN THIS TO BE INTERACTIVE AND THERE'S PLI PLENTY OF SPACE IN THE FIRST ROW SO YOU DON'T HAVE TO HANG OUT WHERE YOU CAN'T P SEE. WHEN WE WERE PLANNING THIS WE THOUGHT IT HELPFUL TO HAVE CONCRETE EXAMPLES OF EXCELLENT WORK THAT BRIDGES DISSEMINATION IMPLEMENTATION SCIENCE WITH MIXED METHODS RESEARCH. SO WE HAVE OUR NEXT PANEL IS THREE SPEAKER WHOSE ARE ALL DOING EXCELLENT EXAMPLES OF THAT KIND OF WORK. AND I KNOW YOU'RE IN THE HERE TO LISTEN TO ME SO I'M GOING TO BRIEFLY INTRODUCE EACH OF THEM AND THEN I THINK THEY'RE GOING TO TALK ABOUT THEIR BACKGROUND, AND I'LL REFER YOU TO THE BIOS IN YOUR REGISTRATION PACKET FOR MORE DETAILS. OUR FIRST SPEAKER IS DR. GREGORY AARONS, A CLINICAL ORGANIZATIONAL PSYCHOLOGIST, AND PROFESSOR OF PSYCHIATRY AT THE UNIVERSITY OF CALIFORNIA SAN DIEGO SCHOOL OF MEDICINE. OUR SECOND SPEAKER WILL BE DR. DR. ADRIENNE FIELDSTEEN A PHYSICIAN AT KAISER PERMANENTE NORTHWEST AND CLINICAL DIRECTOR FOR POPULATION CARE AT THE CENTER FOR HEALTH RESEARCH AND IS ALSO AN INVESTIGATOR THERE. OUR THIRD SPEAKER IS DR. CARLA GREEN, WHO IS A SOCIOLOGIST AND SENIOR INVESTIGATOR ALSO AT THE CENTER FOR HEALTH RESEARCH AT KAISER PERMANENTE NORTHWEST. DR. GREEN IS ASSOCIATE PROFESSOR OF PUBLIC HEALTH AND PREVENTIVE MEDICINE AT THE OREGON HEALTH AND SCIENCE UNIVERSITY. AGAIN, WHERE EACH OF THESE SPEAKERS ARE GOING TO GIVE AN EXAMPLE OF RESEARCH THEN WE'LL HAVE A FEW MINUTES AFTER THEIR PRESENTATION FOR QUESTION AND WE SHOULD HAVE A FEW MINUTES AT THE END FOR QUESTIONS THAT CUT ACROSS ALL THREE. SO I'M GOING TO TURN OVER TO DR. AARONS TO TALK ABOUT HIS WORK. >> SINCE I WAS STANDING THERE AND WANTING TO MAKE A COMMENT I'LL TAKE A COUPLE OF SECONDS. WANTED TO RESPOND TO THE THE QUESTION ABOUT COMMUNITY BASED PARTICIPATORY RESEARCH, ONE WAY OF FRAMING CDPR IN TERMS OF MIXED METHODS AND IMPLEMENTATION RESEARCH IS DEGREE WHICH YOU BRING YOUR STAKEHOLDERS INTO THE RESEARCH PROCESS. IN FORMULATING WHAT ARE THE IMPORTANT IMPLEMENTATION QUESTIONS, WHAT THEIR ROLE WILL BE IN TERMS OF COLLECTING DATA AND ANALYZING DATA, INTEGRATING DATA. CAN USE THE CBBR E THOSE IF NOT THE REAL METHODS TO TALK ABOUT HOW YOU'RE INTEGRATING YOUR STAKEHOLDERS INTO THE RESEARCH PROCESS. SO TODAY I'LL TAWNG ABOUT ONE STUDY, THE TITLE IS FUNDED BY THE NIMH TO LOOK AT IMPLEMENTATION ACROSS THE WHOLE STATE OF AN EVIDENCE BASED PRACTICE TO REDUCE CHILD NEGLECT. FIRST I WANT TO ACKNOWLEDGE MY COLLEAGUES MARK KHAFIN WHO HAD AN EFFECTIVENESS TRIAL, WE BUILT THE IMPLEMENTATION STUDY ON TOP OF EFFECTIVENESS TRIAL AND THEN JOHN LANCEBERG AN ADVANCED CENTER TO PEDIATRIC CARE WHICH PROVIDED SOME SUPPORT. BUT IMPORTANTLY IN THE SPIRIT OF CBPR OUR AGENCY PROGRAM PARTNERS IN THE STATE OF OKLAHOMA WHO WERE INTEGRAL IN HELPING US REALLY UNDERSTAND THE CONTEXT CHALLENGES AND SUCCESSES AND HOW YOU PROVIDE EFFECTIVE SERVICES.NNT A NUMBER OF PARTNERS, LARRY POLANKIS AN ANTHROPOLL GIST I WORKED WITH ON A NUMBER OF STUDIES AND HE AND I WORKED CLOSELY ON THE STUDY DESIGN. AND DATA COLLECTION BOTH QUALITATIVE AND QUANTITATIVE DATA COLLECTION IN FIELD. SO THIS IS ONE OF MY FAVORITE LITTLE COMMENTS ABOUT MIXED METHODS. WE'RE DRIVING ACROSS THE BRIDGE AND HE SAYS THIS BRIDGE IS WONDERFUL, IT SHORTENS MY COMPUTE COMMUTE BY 4.7-MILES. SHE SAID WHO WOULD HAVE THOUGHT A STRUCTURE OF STEEL WOULD BE BEAUTIFUL STRONG AN DELICATE AT THAT SAME TIME, THIS GETS AT THAT DICHOTOMY OF THE HARD COLD NUMBERS VERSUS EXPERIENCE OF IMPLEMENTATION AND CHANGE AND DIFFERENT PHENOMENON AND DIFFERENT PHILOSOPHICAL AND RESEARCH PERSPECTIVES THAT QUALITATIVE AND QUANTITATIVE METHODS BRINGS. I'M GOING TO SKIP THIS BECAUSE WE -- I THINK JIM CRESWELL COVERED THIS BUT IN MIXED METHODS YOU'RE COMBINING QUALITATIVE AND QUANTITATIVE APPROACH BUS THERE'S COLLABORATION AMONG RESEARCHERS AND WORKING OUT DIFFERENT PERSPECTIVES AND DIFFERENT PHILOSOPHICAL APPROACHES OF THE DIFFERENT METHODS AND IN MY WORK ACROSS A NUMBER OF STUDIES WITH DIFFERENT ANTHROPOLL GISTS, I HAVE NEVER -- I GUESS IF YOU HAVE SEEN ONE ANTHROPOLL GIST YOU HAVE SEEN ONE ANTHROPOLL GIST, VERY DIFFERENCE APPROACH IT IS TO QUALITATIVE METHODS AND NEGOTIATING YOUR APPROACH YOUR DESIGNS IS OFTEN A LEARNING EXPERIENCE. SO THIS STUDY I'M GOING TO TALK AB WAS IMPLEMENTATION OF SAFE CARE, CHILD NEGLECT INTERVENTION ACROSS OKLAHOMA STATEWIDE CHILDREN SERVICES SYSTEM. WE INTEGRATED ORGANIZATIONAL AND CLINICAL THEORIES, HYPOTHESES, WE COMBINED EXPLORATORY AN CONFIRMATORY APPROACHES IN THE STUDY AND LOOKED AT RECIPROCAL EFFECTINGS OF IMPLEMENTATION OF THE PRACTICE ON THE SERVICE SYSTEM AN PROVIDER ORGANIZATIONS AND CLIENTS BUT ALSO IMPACT OF THE NATURE OF THE SERVICE SYSTEM, THE NATURE OF THE CONTEXT ON IMPLEMENTATION. AND WE HAD EQUAL COMPONENTS OF QUANTITATIVE AND QUALITATIVE METHODS AND THIS WAS LONGITUDINAL OVER SIX YEARS AT THE ORGANIZATION AND TEAM LEVEL. I DO WANT TO SAY THAT IT DOES REQUIRE COLLABORATION ONGOING RELATIONSHIP BILLING AND MAINTENANCE NOT JUST AMONG INVESTIGATOR BUS YOUR STAKE HOLERS. SO WE DID A NUMBER OF THINGS IN THIS STUDY TO GIVE BACK TO INTERACT AND SPEND TIME WITH STAKEHOLDERS IN THE COMMUNITY. I'LL TALK A LITTLE BIT MORE ABOUT THAT AS WE GO ALONG. IN TERMS OF CONCEPTUAL MODELS THIS IS WHEN THAT WE PUBLISHED LAST YEAR, A PHASE MODEL THAT TAKES INTO ACCOUNT BOTH OUTER CONTEXT, SOCIOWHAT'S GOING ON SOCIOPOLITICAL LEVEL AT ORGANIZATIONAL LEVEL IN TERMS OF NETWORKS AN INNER CONTEXT OF WHAT'S GOING ON IN THE ORGANIZATION. WE WERE FOCUSING SPECIFICALLY ON THE IMPLEMENTATION PHASE, THIS FOUR PHASE MODEL DURING THE STUDY. THE INTERVENTION IMPLEMENTED IS CALLED SAFE CARE. AND IT'S TARGETED FOR FAMILIES WHO HAVE COME TO THE ATTENTION OF CHILD WELFARE SYSTEMS FOR CHILD NE GLEK. THE CORE COMPONENTS THAT GO THROUGH ALL THE SUBSTANTIVE COMPONENTS ARE COMMUNICATION AND PROBLEM SOLVING SKILLS FOR THE PARENTS AND THEN THERE'S THREE PARENT TRAINING MODULES THAT DEAL WITH PARENT AND CHILD INTERACTIONS, HOME SAFETY AND CHILD HEALTHCARE. THESE SERVICES ARE PROVIDED IN THE HOME AND THERE ARE A NUMBER OF STUDIES SINGLE CASE AND RANDOMIZED LARGE SCALE DESIGNS THAT HAVE SHOWN EFFECTIVENESS. IN THIS STUDY THERE WERE FOUR CONDITIONS THAT WRP EXPERIMENTALLY ASSIGNED TO THE SAFE CARE INTERVENTION WITH AN WITHOUT COACHING AND USUAL CARE WITH AND WITHOUT COACHING. THIS IS OUR CONCEPTUAL FRAMEWORK FOR WHAT WE WANTED TO LOOK AT AND WE WANTED TO LOOK AT THREE MAIN ISSUES. ONE WAS ISSUES AROUND WORK FORCE, THIS TOP -- HOLD ON. THERE WE GO. WORK FORCE ISSUES UP HERE LOOKING AT WORK ATTITUDES AND TURN OVER, I'LL TALK ABOUT THOSE RESULTS TODAY. LOOKING AT THE IMPACT OF IMPLEMENTATION ON CLINICAL PRACTICE, ON WORKING ALLIANCE BETWEEN CASE MANAGERS AND THEIR CLIENTS. AND THEN ORGANIZATIONAL PROCESSES. SO WHAT'S GOING ON IN THE ORGANIZATION THAT CAN IMPACT IMPLEMENTATION. SO MIXED METHODS WERE VERY WELL SUITED TO THIS STUDY.U/0 IN OUR DESIGN WE USE CONCURRENT AND MIXED METHODS WE HAVE A SURVEY OF PROVIDERS PERCEPTIONS OF EVIDENCE BASED PRACTICE AND WE DID SAMPLING TO IDENTIFY THOSE MOST POSITIVE ABOUT SAFE CARE AND THOSE MOST NEGATIVE. SO WE SAMPLED THOSE FOLKS. IN SUBSEQUENT WAYS WE USE ALL RELEVANT UNDERSTAND DENSE. QUALITATIVE AND QUANTITATIVE BUT AS WE MOVE THROUGH THIS AS WE ADDRESS DIFFERENT RESEARCH QUESTIONS IN OUR STUDY DESIGN AND THEORETICAL MODEL, THAT RELATIONSHIP MAY VARY DEPENDING ON THE PARTICULAR QUESTION. SEW WE TALK ABOUT INITIAL DESIGN FOR PROJECT AND YOU HAVE TO LAY THAT OUT IN GRANT PROPOSAL OR PAPER BUT IN PRACTICE, I THINK IT'S MORE FLUID. WE CONNECTED OUR DATA SETS, OUR QUALITATIVE AND QUANTITATIVE DATA AND CONNECTED THOSE. BUT WE ALSOzd WHICH WE BROUGHT STAKEHOLDERS TOGETHER FROM THE COMMUNITY INCLUDING PROVIDERS AN SUPERVISORS AN AGENCY DIRECTORS AND TOGETHER WITH THE GROUP RESEARCHERS WE WHEN THROUGH DATA AND SAID WHAT ARE THE DATA TELLING US. WHAT ARE WE LEARNING THROUGH THE DATA THAT LEADS US TO CHANGING HYPOTHESES AND CHANGING THE QUESTIONS WE WANT TO ASK IN OUR NEXT WAVE AND A NICE WAY TO INTEGRATE STAKEHOLDERS IN THE RESEARCH PROCESS AS WE GO FORWARD. SO MIXED METHODS FUTURE OF DATA RESULTS, COMPLIMENTARITY OF RESULTS AN EXPANSION OF INITIAL FINDINGS. TALK A LITTLE BIT ABOUT WHAT WE ACTUALLY DID, SUPERVISORS AGENCY DIRECTORS AN EXECUTIVE DIRECTORS. LONGITUDINAL WEB-BASED SURVEYS COMPLETED BY SERVICE PROVIDERS AND THEIR SUPERVISORS, AND ANNUAL QUALITATIVE DATA COLLECTION, FOCUS GROUPS, WITH WITH SERVICE PROVIDERS, SUPERVISORS, AGENCY PROVIDERS AB OTHER STAKEHOLDERS WITHIN THE SYSTEM. PART OF THE REASON WE WENT DOWN THE MIXED METHODS PATH WE HAD AN ENTIRE STATEWIDE SERVICE SYSTEM, 21 TEAMS BUT WHEN DOING POWER ASEIZE, IF YOU HAVE A NESTED DATA DESIGN IT GOES FROM 150 PROVIDERS TO 21 TEAMS. SO WE WANT TRIANGULATION OF METHODS TO ADDRESS THAT TO GIVE MORE CONFIDENCE TO BETTER UNDERSTAND THE FINDINGS. WE HAD EXTREMELY HIGH RESPONSE RATES GREATER THAN 95% AVERAGE OVER THE 12 WEAS OF DATA COLLECTION. THIS GIVES AN IDEA HERE AT THE TOP, THE QUANTITATIVE DATA COLLECTION WAS EVERY SIX MONTHS OVER THE COURSE OF THE STUDY AND AN ANNUAL HOW DATA ARE COLLECTED AND AN RAISED. IN OUR QUANTITATIVE COLLECTION, WE -- IT WAS BASED ON A THEORETICAL MODEL WE HAD PRESENTED UP THERE. WE DID THREE WAVES OF QUALITATIVE DATA COLLECTION IN THE FIRST IT WAS THE PURPOSE OF SAMPLING THAT I MENTIONED WITH 15 CASE MANAGERS WHO BEGAN USING SAFE CARE. IN THE WAVE WE DID INTERVIEW WITH UPPER AND MID LEVEL MANAGEMENT TO GET THEIR PERSPECTIVE WHAT IS IT MEANT TO IMPLEMENT HOW IT IMPACTED THEIR ORGANIZATIONS AND STAFF RETENTION TRAINING AND TURN OVER. THIRD WAVE LIRRY AND I DID TOUR DEOKLAHOMA WHICH WE COLLECTED DATA FROM EVERY CASE MANAGEMENT TEAM AND SUPERVISOR AND WE SPENT TWO WEEKS AND DROVE OVER 5,000-MILES IN AUGUST. AND GOING IN AUGUST IN OKLAHOMA IS NOT TRIVIAL, TYPICALLY 103 TO 110-DEGREE BUS WE GOT STREET CREDIT FOR STAKE HOLERS FOR COMING OUT THEN. SO A LOT OF OF NUANCE TO BUILDING THESE RELATIONSHIPS AND HOW YOU FRAME WHAT YOU'RE DOING AND HOW YOU APPROACH PEOPLE IN THE FIELD. AND BE ACCESSIBLE TO THEM. WE GAVE PINS WITH THE NAME OF THE STUDY AND WE WERE DOING A SET OF FOCUS GROUPS IN STILL WATER OKLAHOMA AND ONE CASE MANAGER KEPT GRABBING ORANGE ONES AND THAT WAS THE OSU COLORS SO LITTLE THINGS LIKE THAT MAKE A DIFFERENCE. THESE ARE TEAMS IN RED AND IN GREEN WERE CONTROL CONDITION TEAMS. ALL THOSE COWS ARE IN THE SHADE OF THE TREE TRYING TO STAY COOL, IT WAS 110-DEGREES THAT DAY. THIS IS ONE TEAM THAT WE MET WITH AND GOT TO KNOW FOLKS OVER THE COURSE OF THIS STUDY. UNEXPECTED THINGS HAPPEN OVER THE COURSE OF -- THIS IS 8 WAVES OF THE STUDY, WE STARTED WITH FOUR TEAMS, TEAM SPLITS, SOME NEW TEAMS WERE DEVELOPED SO THINGS CHANGE IN THE STRUCTURE OF YOUR QUANTITATIVE AN QUALITATIVE DATA COLLECTION YOU NEED TO ACCOUNT FOR. WE FOCUSED ON STAFF TURN OVER. I'M GOf‡ QUICKLY OVER THE RESULTS BUT THE HYPOTHESES WERE THE IMPLEMENTING AND EVIDENCE BASED PRACTICE, WOULD DECREASE JOB AWE TON MY BECAUSE YOU HAVE A MORE STRUCTURED INTERVENTION AND THERE'S GREATER LIKELIHOOD OF FOLKS LEAVING AGENCY, RECEIVING ONGOING COACHING OR MONITORING ALSO REDUCE JOB AWE TON MY AND LEAD TO LIKELIHOOD OF LEAVING THE AGENCY AND IMPLEMENTING THE EVIDENCE BASED PRACTICE AND RECEIVING ONGOING MONITORING IN THOSE CONDITIONS WOULD BE THE HIGHEST TURN OF OVER RATES. OUR QUALITATIVE DATA ANALYSES USED A GROUNDED THEORY APPROACH, CO-OCCURRENCE AND COMPARISON. WE RANDOMLY SELECTED ONCE WE HAD CONCURRENCE WE MOVED TO HAVING OUR TRAINED CODERS CODE USING THAT CODING STRUCTURE. WE HAVE ONGOING MEETINGS THROUGH LARRY AND MYSELF AND OUR TEAM AND CONSIDERED HYPOTHESES, CONCEPTUAL FRAMEWORK AND INTEGRATING RESULTS. SO IN TERMS OF THE STAFF TURN OVER IN OUR QUANTITATIVE ANALYSIS, THE SURVIVAL ANALYSIS SHOWED DESPITE LITERATURE STAFF RETENTION WAS BEAR THAN EVIDENCE BASED PRACTICE PLUS MONITORING CONDITION. SO WE HAD A 14.9% TURN OVER RATE IN THAT CONDITION AN MORE THAN DOUBLE THOSE TURN-OVER RATES IN THE OTHER THREE CONDITIONS. SO THE QUESTION WITH MIXED METHODS IS CAN WE EXPLAIN WHY AND CAN WE EXPLAIN THE VARIABILITY. THIS IS ONE OF THE THINGS I WANTED TO HIGHLIGHT IS THAT IN YOUR QUANTITATIVE ANALYSES YOU HAVE VARIABILITY. THAT'S A CONFIDENCE LEVEL IN RESULTS, DOESN'T MEAN OTHERS AREN'T GOING ON IN THERE. THE QUALITATIVE METHODS HELP HELP UNDERSTAND THAT VARIABILITY. WE FOUND HIGHER JOB OR LOWER JOB AUTONOMY WAS RELATED TO RETENTION AND RELATED TO TURN OVER BUT WHEN WE LOOK AT CONVERGENCE OF FINDINGS WE ANSWERED THE QUESTIONS, THIS IS HOW WE DECIDED TO SHOW INTEGRATION OF METHODS TO ASK SIMILAR QUESTIONS OF CONVERGENCE OF FININGS TO SEE IF E WE'RE GETTING THE SAME ANSWERS. CON VER JEBS WE FOUND CONVERGENCE. DOES IT INCREASE RISK OF TURN OVER QUANTITATIVE? THE ANSWER WAS NO. WE HAD GREATER LIKELIHOOD OF RETENTION IN SAFE CARE CONDITION, MANY PROVIDERS REPORTED SATISFACTION WITH THE STRUCTURE. WE HEARD PRIOR TO TO SAVE CARE, THEY TRY TO UNDERSTAND WHAT'S GOING ON WITH THE FAMILY AND COME ONE A CASE PLAN. THE STRUCTURE GAVE THEM TOOLS TO USE WITH THEIR FAMILIES PERCEIVE AS HELPFUL SO WE'RE LEARNING SOMETHING ABOUT WHY THE INTERVENTION HAS HIGHER UTILITY AND HOW THAT MIGHT LEAD TO BETTER DOES IT INCREASE THE RISK OF TURN OVER, ANSWER IS YES, QUANTITATIVELY, SOME PROVIDERS REPORTED INTENTIONS TO LEAVE DUE TO SUPERVISOR MICROMANAGEMENT BUT UNRELATED TO THE EVIDENCE BASED PRACTICE. WE'RE DOING QUANTITATIVE AND UNDERSTANDING WHAT IT IS ABOUT JOB AUTONOMY NOT RELATED TO THE EVIDENCE BASED PRACTICE BUT MORE TO ORGANIZATIONAL MANAGEMENT PRACTICE. IN TERMS OF LOOKING AT COMPLIMENTARITY, SAFE CARE IMPLEMENTATION LEAD TO TURN OVER, THE ANSWER WAS NO BUT SOME PROVIDERS LOVE THE STRUCTURE PROVIDED BY SAFE CARE, SOME DISLIKE HAVING TO IMPLEMENT MODULES THEY FELT SOME FINALLY IN TERMS OF EXPANSION WHEN WE ASK ABOUT TURN OVER, WE ASKED ALSO WHY OUR PROVIDERS MORE LIKELY TO STAY, STRUCTURE, SAFE CARE, FELT THE SUPPORT THEY RECEIVED FROM THE QUOTE MONITORS WERE SEEN AS FREE SUPERVISION. AND OPPORTUNITY CASE CONFERENCE ABOUT THEIR CLIENTS. SO I WANTED TO GIVE QUICKLY THE THREE EXAMPLES OF CONVERGENCE, COMPLIMENTARITY AND EXPANSION. THESE TABLES ARE TABLES WE USED IN PUBLICATION ABOUT THIS STUDY. FOR US A WAY TO PUT QUANTITATIVE AND QUALITATIVE SIDE BY SIDE IN OUR RESULTS. IN CONCLUSION, WE USED MIXED METHODS IN MULTIPLE WAYS TO EXPAND THE QUALITATIVE TO EXPAND ON THE 4GTITATIVE TO ANSWER QUESTIONS WE HADN'T NECESSARILY ANTICIPATED AND PROVIDE INCITE WE COULDN'T HAVE WITH QUANTITATIVE METHODS ALONE. THANK YOU VERY MUCH. I DON'T KNOW IF WE HAVE TIME FOR A QUESTION OR TWO. HERE IS ONE. THIS WAS NIH'S PLAN TO HELP PARTICIPANTS GET MORE EXERCISE WHEN THEY CAME UP TO ASK QUESTIONS. >> VERY NICE PRESENTATION. I WOULD LIKE TO MAKE A SMALL COMMENT ON THE SMALL END PROBLEM YOU MENTION, I THINK JOHN ALSO MENTIONED ABOUT THE QUESTION I THINK ESPECIALLY IMPLEMENTATION RESEARCH VERY OFTEN THE ENTIRE POPULATION MIGHT BE 50 SITES. WE DO THE USUAL 5% SIGNIFICANCE, 80% TRIAL. BUT NOT ENOUGH SAMPLE SIZE. THE GOOD OPPORTUNITY TO THINK ABOUT FRAMEWORK AND ALSO TO OVERLAY FROM THIS REGIONAL 5%, SIGNIFICANCE 80% POWER. SO THIS IS A VERY INTERESTING FRAMEWORK WHERE IT'S NOT VERY OFTEN EMPHASIZED IN I GUESS RANDOMIZED TRIAL LIT TATURE, USUALLY REFERRED TO A QUESTION ABOUT PATIENT HORIZON. AND THAT REALLY MEANS. LATION SIZE. IN MOUSE TRADITIONAL RANDOMIZED TRIALS THE PATIENT HORIZON IS INFINITE SO WE DO THE TRIAL AND THEN A LARGE NUMBER OF PATIENTS WHO BENEFIT FROM THE NEW DRUG APPROVED. SO USUALLY WE DESIGN THE STUDY TO EMPHASIZE HIGH ACCURACY. 80% DESIGN COMES FROM. BUT FOR IMPLEMENTATION RESEARCH VERY OFTEN I THINK THE PRIMARY GOAL IS TO SOLVE THE LOCAL PROBLEM. SO THE 55 IS THE TOTAL POPULATION SO THE DESIGN STUDY REALLY SHOULD TAKE THE LIMITED POPULATION INTO CONSIDERATION AND WE DON'T HAVE THE UNLIMITED RESULT OF FUTURE PATIENTS TO CONSIDER SO IT IS SOME VERY NICE WORK LIKE (INDISCERNIBLE) AND PART OF THAT IN THE CONTEXT ADAPTIVE CLINICAL TRIAL DESIGN. I THINK (INAUDIBLE) WILL BE TALKING ABOUT. SOME OF THE FINDINGS SHOULD USE NO MORE THAN ONE-THIRD OF THE TOTAL POPULATION IN THE TRIAL AND THEN RESERVE TWO-THIRDS TO BENEFIT FROM KNOWLEDGE GAINED. WHAT THAT MEANS IS THAT IN SOME OF THE RESEARCH RECENT RESEARCH, I COLLABORATED WITH COLLEAGUES AT THE VA AND THE TRIAL -- THE TEST MIGHT BE CONDUCTED WITH SIGNIFICANCE LEVEL OF 25% WITH A POWER OF 50%. SO THAT'S VERY DIFFERENT FROM THE TRADITIONAL STATISTICAL DESIGN. BUT THAT IS THE UNIQUE NATURE OF THE FRAMEWORK FOR THE IMPLEMENTATION RESEARCH WHERE WE HAVE A LIMITED POPULATION AND THE FOCUS IS SOLVING THE LOCAL PROBLEM. SO I THINK THE SMALL END PROBLEM IS A GOOD OPPORTUNITY FOR US TO RETHINK HOW THE STATISTICAL METHODS COULD BE OR SHOULD BE APPLIED IN THE FRAMEWORK. SO I APPRECIATE THE IMPORTANCE OF LOOKING IN THAT -- IN DEPTH INTO THE QUALITY METHODS TO DEAL WITH THE SMALL SITUATION BUT AT THE SAME TIME WE SHOULD NOT OVERLOOK THE OPPORTUNITY THAT MIGHT BE STILL SOME GOOD STATISTICAL METHOD THAT CAN BE USED. >> I'LL JUST SAY WE NEED YOU BACK ON STUDY SECTION BECAUSE STATISTICIANS MIGHT NOT HAVE THAT SAME PHILOSOPHY. SO I THINK WE MIGHT NEED EDUCATING OF OUR STATISTICIANS AND METHOD DOLL GISTS MORE BROADLY ABOUT ISSUES IN THE CONTEXT OF IMPLEMENTATION. >> DEFINITELY VERY IMPORTANT."eu THIS LITERATURE STARTED IN THE '60s BUT IT HAS BEEN LARGELY IGNORED BECAUSE MOST OF THE CLINICAL STUDIES ARE DEVELOPED IN THE FRAMEWORK OF ESSENTIALLY INFINITE PATIENT HORIZON. >> THANKS, EVERYBODY. [APPLAUSE] >> SO I'M GOING TO TALK TO YOU ABOUT ANOTHER STUDY THAT ALSO INTEGRATES QUANTITATIVE AND QUALITATIVE METHODS, THIS IS CALLED FOCUSING IMPLEMENTATION TO BRING EFFECTIVE REMINDERS. AND THIS FOCUSES ON COLON KAREN SCREENING QUALITY IMPROVEMENT. SO THIS WAS A MIXED METHODS STUDY STILL GOING ON, A FOUR YEAR STUDY CONDUCTED AT KEISER PERM NEANTY NORTHWEST, ALSO HAD SOME ADDITIONAL HEALTH PLANS INVOLVED IN THE QUALITATIVE RESEARCH ASPECTS. IT LOOKED AT IMPLEMENTATION OF A CANCER SCREENING PATIENT REMINDER PROGRAM, BASICALLY IT WAS AN AUTOMATED CALL IN CAMPAIGN. WE WERE INTERESTED IN THE EFFECTIVENESS OF THE CAMPAIGN AND WHAT MADE IT MORE OR LESS EFFECTIVE BUT ALSO SURROUNDED QUALITY IMPROVEMENT SO JUST GENERAL ORGANIZATIONAL METHODS TO IMPROVE COLON CANCER SCREENING. SO THIS IS A MULTIPLE METHOD STUDY, A MULTI-LEVEL, IT INCLUDED A PRACTICAL RANDOMIZED CONTROL TRIAL OF THIS AUTOMATED CALLING SYSTEM. AND THEN LOOKED AT BOTH THE SYSTEM LEVEL EFFECTIVENESS IN CRC SCREENING IMPROVEMENT AND FACTORS THAT LED TO THAT PCP LEVEL ANALYSIS, AND PATIENT COHORT STUDY. USES ESSENTIALLY ELECTRONIC AND SURVEY DATA BOTH AT THE INITIAL PART OF THE TRIAL AS WELL AS ALL PATIENTS CONTINUE TO RECEIVE THE PROGRAM OVER A COUPLE OF YEARS PERIOD. IT INCLUDE AD QUALITATIVE EVALUATION AT ALL LEVELS. AS I MENTION INCLUDING COMMUNITY SITES AROUND WHETHER OR NOT THIS WOULD BE TRANSFERABLE TO OTHER SITES. ESSENTIALLY WHAT WE WERE INTERESTED IN WAS LOOKING AT THE FACTORS THAT MIGHT BE ASSOCIATED WITH IMPLEMENTATION SUCCESS. AGAIN, ACROSS MULTIPLE LEVELS. THE PATIENT LEVEL FOR IMPROVING SCREENING, THE PRIMARY CARE PROVIDER, AND THEIR TEAM, SPECIALISTS INVOLVED IN COLON CANCER SCREENING AND OTHER HEALTH PLAN STAFF. SO THE BACKGROUND I DON'T HAVE TO TELL YOU MUCH ABOUT THAT SCREENING, THE LOWEST AMONG EVIDENCE BASED CANCER SCREENING PROGRAMS. AND WE THOUGHT THAT AN AUTOMATED PHONE REMINDER SYSTEM TO PATIENTS MIGHT BE A COST EFFECTIVE WAY TO INCREASE SCREENING RATES ACROSS LARGE POPULATIONS. BUT ALSO IT WAS OF INTEREST WHAT WAS HAPPENING AT THE TEAM AND PCP LEVEL TO PROMOTE SCREENING AND HOW THIS MIGHT INTERACT WITH PATIENT REMINDERS OR OTHER PATIENT PROMOTION TYPES OF PROGRAMS. LITTLE SEARCH KEYPAD MINUTED THE USE OF AUTOMATED REMINDERS WITHIN -- TO PROMOTE CRC SCREENING. SO AS WAS MENTIONEDDED EARLIER, WE USED WHAT'S CALLED THE PRISM MODEL, THE PRACTICAL ROBUST IMPLEMENTATION AN SUSTAINABILITY MODEL, YOU HEARD A LITTLE BIT ABOUT THIS EARLIER. BASICALLY CONSIDERS FACTORS AROUND THE INTERVENTION, THE RECIPIENTS, IMPLEMENTATION AN SUSTAINABILITY INFRASTRUCTURE, WHAT'S HAPPENING IN THE EXTERNAL ENVIRONMENT AND HOW THOSE WILL INTERRELATE TO NUMBER ONE, PROMOTE ADOPTION AND THEN FULL IMPLEMENTATION MAINTENANCE OF A PROGRAM AND ULTIMATELY THEIR REACH AND EFFECTIVENESS TO PATIENTS. AND THE QUALITATIVE WORK WAS GUIDED BY A SERIES OF SUCCESS FACTORS THAT WE HAD ALREADY DEVELOPED WITHIN VARIOUS AREAS OF MODEL SO WE FOCOCUSED ON THAT. CERTAINLY THIS WORK AS WELL AS OTHER STUDIES WE HAVE CONDUCTED IN IMPLEMENTATION CONTINUALLY HELP TO REFINE THE MODEL AND WHAT SUCCESS FACTORS MIGHT BE. SO WE HAD AGAIN IN OTHER STUDIES PRESENT AD MULTI-DISCIPLINARY STAFF INVOLVED WITH GENERAL PREVENTIVE AND INTERNAL MEDICINE, A TEAM THAT WAS VERY EXPERIENCED IN IMPLEMENTATION RESEARCH TOGETHER STATISTICIAN, QUALITATIVE RESEARCH EXPERTS AND ECONOMISTS. SO WE STARTED WITH THIS PRACTICAL RCT WHICH AS I MENTIONED ESSENTIALLY LOOKED AT WHAT WAS THE EFFICACY OF THIS AUTOMATED CALLING SYSTEM. SO RANDOMIZED PATIENTS 3,000 PATIENTS INTERVENTION AND 3,000 TO USUAL CARE AND THESE WERE ESSENTIALLY PATIENTS WHO WERE ALREADY DUE OR OVERDUE FOR COLON CANCER SCREENING IN THE -- IN 51 TO 80 YEARS OF AGE. SO THIS WAS EFFECTIVE INTERVENTION INCREASED LIKELIHOOD OF SCREENING BY ABOUT 30% AND SO OUR NEXT QUESTION THOUGH, THIS IS THE AREA THAT I'M GOING TO FOCUS ON TO DEMONSTRATE TO KIND OF THIS MIXING OF METHODS FOR QUANTITATIVE AND QUALITATIVE, THE NEXT THING WE WERE INTERESTED IN WAS WHY WERE WE SEEING THIS VARY YEAR AGO IN PRIMARY CARE PROVIDERS SCREENING WITHIN THEIR PANELS OF PATIENTS. EVEN THOUGH THESE REMINDER CALL WERE EFFECTIVE WE STILL SAW VARIATION THAT WAS HIGHWAY HAPPENING. SO WHAT WAS KNOWN IS PROBABLY THE STRONGEST PREDICTOR OF COLON CANCER SCREENING IS PRIMARY CARE PROVIDER RECOMMENDATION TO COMPLETE IT BUT DOES IT MATTER WHAT SPECIFIC SCREENING STRATEGIES PRIMARY CARE PROVIDERS TOOK WITHIN THEIR PRACTICE PROMOTING ONE SCREENING OVER ANOTHER OR DIFFERENT TYPES OF COMMUNICATION ABOUT SCREENING AND SO FORTH. SO WHAT WE SET OUT WAS TO DESCRIBE BASICALLY PCP SCREENING TEST RECOMMENDATION PATTERNS, WHAT WERE THE ASSOCIATED FACTORS FOR CERTAIN KINDS OF PATTERNS OF SCREENING WITHIN A PATIENT PANEL P AND WHAT WAS THE RELATIONSHIP OF THAT PATTERN TO PATIENT TEST COMPLETION IN THEIR PRACTICE. THE DESIGN WAS THIS WAS ACROSS -- A CROSS SECTIONAL STUDY, USED A PCP SURVEY, ELECTRONIC HEALTH REPORT DATA FROM THEIR PATIENTS. AND IN DEPTH INTERVIEWS WITH PCPs. THIS WAS A SEQUENTIAL DESIGN THAT INCLUDED PROBABLY EXPLORATORY AND EXPLANATORY ELEMENTS TO DEVELOP A SURVEY TO BEST DESIGN THE SURVEY. THE SURVEY WAS SENT TO 195 PCPs, 74% RETURN THEM AND AMONG THOSE 92% COMPLETED THE QUESTIONS SCREENING TESTS. AS I MENTIONED WE DID SEMISTRUCTURED INTERVIEWS WITH 20 PCPs TO EVALUATE THE SURVEY AN UNDERSTAND THE RESULTS. WE USED AN INTERVIEW GUIDE AND TRAINED RESEARCH SPECIALIST. FOR THE QUANTITATIVE DATA ANAL SITS WE DID CLUSTER ANALYSIS BASED UPON THE PCP REPORTED FREQUENCY OF GUIDELINE RECOMMENDED CRC TESTS, NAMELY SCHOOL TESTING, FLECKABLE SIGMOIDOSCOPY STOOL TESTING OR COLONOSCOPY, THIS IS SURVEY REPORTED FREQUENCY. OF USE OR RECOMMENDATION OF THESE TESTS. BASICALLY WE SAID OFTEN ON SCALE OF 1 TO 5 DID YOU RECOMMEND EACH OF THESE FOLLOWING SCREENING TESTS OR THEIR COMBINATIONS. THEN WE VALIDATED CLUSTERS BY LOOKING ACTUALLY AT THE ORDERING PATTERNS IN THE ELECTRONIC MEDICAL RECORDS OF THE CLINICIANS. THEN WE COMPARED THESE CLUSTERS OF CLINICIANS WHO REPORTED VARYING PATTERNS OF ORDERING WITH BOTH THEIR CHARACTERISTICS, NAMELY GRAPHICS OR PANEL CHARACTERISTICS AND WHAT THINGS TEND TO INFLUENCE THEIR SCREENING, CHOICE OF TEST, WHAT WERE THEIR CONCERNS AB TESTS, AND TO WHAT DEGREE DID THE THEY JOINTLY DECIDE WHAT THE PATIENT, WHAT TEST TO ORDER VERSUS STRONGLY RECOMMEND A PARTICULAR TEST. AND THEN FINALLY WE LOOKED AT THE PCP OVERALL SCREENING RATES. SO THESE WERE THE FOUR PREVALENT CLUSTERS. WHAT WE CALL THE BALANCE CLUSTER WAS THE LARGEST CLUSTER, 54 PRIMARY CARE PROVIDERS. AND THEY PRETTY MUCH RECOMMENDED EQUALLY THE FECAL TEST OR FECAL TEST PLUS FLEXIBLE SIGMOIDOSCOPY OR COLONOSCOPY. EQUAL IN SIZE BUT SMALLER, THE FECAL TEST ONLY GROUP PREDOMINANTLY, THE NEXT GROUPING WHICH WAS THE FECAL TEST PLUS SIGMOIDOSCOPY FINALLY THE COLONOSCOPY PLUS FOBT GROUP. THE CHARACTERISTICS OF THESE INDIVIDUALS BASICALLY THE CLUSTERS THAT TENDED TO USE MORE ENDOSCOPY NAMELY COLONOSCOPY OR FLEXIBLE SIGMOIDOSCOPY HAD MORE INTERNAL MEDICINE SPECIALISTS VERSUS FAMILY MEDICINE. THEY TENDED TO HAVE MORE COMMUNITY PRACTICE EXPERIENCE VERSUS HMO EXPERIENCE. THE CLUSTERS WERE OTHERWISE SIMILAR IN THEIR AGE OR GENDER, ET CETERA. SO WHAT TENDED TO DRIVE THESE DIFFERENT PATTERNS OF ORDERING OF TESTS? WHAT WE FOUND WAS THAT THE CLUSTERS THAT HAD MORE FECAL TESTING TENDED TO BE MORE INFLUENCED BY CLINICAL GUIDELINES IN GENERAL. THEY TENDED TO BE DRIVEN BY HIGHER LEVEL OF EVIDENCE AROUND FECAL TESTING OVER TIME. FOR EXAMPLE COMPARED TO COLONOSCOPY. GROUPS RECOMMENDING MORE ENDOSCOPY, HOWEVER THEY WERE MORE CONCERNED ABOUT TEST PERFORMANCE. SO THEY TENDED TO BE MORE ACCURACY TYPES OF PEOPLE. AND THEY WERE CONCERNED FOR EXAMPLE THAT FECAL TESTS GENERATED HIGH NUMBERS OF FALSE POSITIVES AND THAT FECAL TEST CAN MISS CANCERS. THEY WERE ALSO LESS CLEAR ABOUT WHEN FECAL TESTS SHOULD BE APPROPRIATELY USED. WHEREAS THE GROUPS THAT TENDED TO USE MORE FECAL TESTS ALONE OR COMBINATION WITH FLEXIBLE SIGMOIDOSCOPY, THEY WERE MORE LIKELY TO BE FAIRLY FORCEFUL ABOUT RECOMMENDING A PARTICULAR STRATEGY COMPARED TO THE COLONOSCOPY PLUS FOBT OR BALANCE GROUPS THAT TEND TO HAVE A BALANCE OF TESTS IN THEIR PRACTICE. THOSE GROUPS BALANCED OR COMBINED COLONOSCOPY IN THEIR PRACTICE MORE LIKELY TO LET THE PATIENT DECIDE VERSUS PUTTING FORTH A STRONG RECOMMENDATION. THE FOBT GROUP WAS MORE CONFUSED ABOUT FLEXIBLE SIGMOID OTIONSCOPY AN FELT GROUP HAD COMPLICATIONS OR DIDN'T TOLERATE THE TEST. THERE WERE NO GROUP DIFFERENCES IN THE NUMBER OF ELEMENTS THAT THEY USED IN TALKING ABOUT COLON CANCER SCREENING SUCH AS COMPLICATIONS, BENEFITS, THAT SORT OF THING. WHAT DID WE FIND, WAS THERE ANY DIFFERENCE IN OUTCOMES SCREENING OUTCOMES AMONG THESE PATTERN? NO. IF YOU LOOK TT TOP ROW, AMONG PATIENTS DUE OR OVERDUE WHO WERE FOLLOWED FOR SIX MONTHS PERIOD ALL OF THE PRACTICE PATTERN GROUPS HAD SIMILAR, 20% OR SO SCREENING OVER THE NEXT SIX MONTHS AMONG PATIENTS WHO NEEDED IT. AS A PREVALENCE SCREENING RATE IN THEIR PRACTICE, PRETTY SIMILAR IN THE LOW 60% AT THIS TIME. THIS WAS IN 2009. SO BASICALLY WE THEN WENT AND TRIED TO LOOK AT HOW DOES OUR INTERVIEW INFORMATION SUPPORT OR EMBELLISH UPON THESE RESULTS. WE FOUND THE CONTENT ANALYSIS FROM INTERVIEWS SUPPORTED THE QUANTITATIVE FINDINGS SO THE KEY FINDINGS WERE CONFIRMED SO WE FELT CONFIDENT ABOUT THEM. THAT WAS FOR EXAMPLE IN THE CLUSTERS OF CLINICS THAT USED MORE FECAL TESTING TENDED TO SUPPORT GUIDELINES, ONE OF THE THINGS THAT CAME OUT IN THE INTERVIEWS THOUGH THAT IT WASN'T ONLY NATIONAL GUIDELINES, THEY WERE PERHAPS AS LITTLE BIT MORE FOR LACK OF BETTER TERM, ORGANIZATIONAL ANIMALS SO IF THEY FELT THE ORGANIZATION WAS PROMOTING FECAL TESTING WHICH IT LIKELY WAS, THEY TENDED TO GO MORE IN THAT DIRECTION. SO THAT CAME OUT AS A NUANCE IN THE INTERVIEWS. THEY WERE ALSO -- IT WAS CONFIRMED IN INTERVIEWS MORE LIKELY TO RECOMMEND A SPECIFIC STRATEGY NAMELY FECAL TESTING. AND THE BALANCED GROUP OF PCPs WERE LIKELY TO USE A MUTUAL OR JOINT APPROACH TO DECISION MAKING. AND WE GOT MORE INFORMATION ABOUT EXACTLY HOW THEY ACCOMPLISH THAT OR TENDED TO CARRY IT OFF IN THE DIFFERENT WAYS THEY DID THAT IN THE INTERVIEWS AS COMPARED TO SURVEY RESULTS. ANOTHER THING THAT CAME OUT IN THE QUALITATIVE DATA WAS PARTICULARLY THE INDIVIDUALS WHO WERE IN THE FECAL TESTING CLUSTERS THEY TENDED TO RELY MORE ON LOCAL ORGANIZATIONAL EXPERTS IN ADDITION TO THE ORGANIZATIONAL GUIDELINES THEMSELVES. SO THE QUALITATIVE DATA PROVIDE AD MORE NUMBER ONE CONFIRMED VALIDATED QUANTITATIVE FINDINGS IN THE SURVEY BUT ALSO PROVIDE AD MORE NUANCED INFORMATION ON INFLUENCES AND ALSO COUNSELING METHODS. SO WHAT DID WE CONCLUDE FROM THIS? THAT PROVIDER RECOMMENDATIONS TO SCREEN AND NOT THE SPECIFIC NATURE OF THE RECOMMENDED TEST IS THE MAJOR DRIVER OF CRC SCREENING COMPLETION AMONG PATIENTS. SO WE RECOMMEND THAT HEALTHCARE PROVIDERS ENTHUSIASTICALLY RECOMMEND CRC SCREENING IN A WAY THAT FITS CURRENT PRACTICE STANDARDS AND PRACTICES WITHIN THEIR INDIVIDUAL CLINIC SYSTEMS. AND SETTINGS. AND THAT SYSTEMS REALLY SHOULD ROBUSTLY SUPPORT MULTIPLE CRC SCREENING METHODS BOTH TO SUPPORT THIS VARIATION IN PRACTICE AN VARYINGS -- VARIATIONS IN WHAT PATIENTS WANT. WHAT DID WE LEARN FROM THIS? A QUANTITATIVE AND QUALITATIVE DATA CAN BE SUCCESSFULLY COMBINED TO ENHANCE MANY ASPECTS OF IMPLEMENTATION RESEARCH AND WE DIDN'T NECESSARILY KNOW HOW ALL THE DATA WAS GOING TO BE USED AT THE START BUT THAT THAT WAS OKAY. WE WERE OPEN TO NEW FINDINGS AND REFINEMENT ALONG THE WAY AND STAKEHOLDERS DID GUIDE THE MOST IMPORTANT QUESTIONS SO WE DID ADJUST OUR METHODS TO THAT. THAT WAS QUITE USEFUL. SO QUESTIONS, THOUGHT? [APPLAUSE] >> IT'S NICE TO BE HERE TODAY. I THINK IT'S A GREAT OPPORTUNITY FOR A LOT OF INTERESTED PEOPLE TO GET TOGETHER AND TALK ABOUT HOW TO DO THESE THINGS. I'M GOING TO TELL YOU TODAY ABOUT A PROJECT THAT IS NOT A DNI PROJECT BUT REPRESENTS A REALLY GOOD EXAMPLE OF WAYS IN WHICH QUALITATIVE DATA AND QUANTITATIVE DATA CAN BE MIXED IN A REALLY BALANCED APPROACH. I HOPE YOU'LL AGREE IT WAS WORTH LEAVING DISSEMINATION AN IMPLEMENTATION RESEARCH AREA TO LOOK AT THIS ONE. THIS STUDY IS STUDY OF STRATEGIES STUDIES OF TRANSITIONS AN RECOVERY STRATEGIES, OTHERWISE KNOWN AS AS FAR AS NIMH PROJECT. IT'S A MIXED METHODS EXPLORATORY STUDY SEEKING FACTORS TO IDENTIFY RECOVERY AMONG PEOPLE WILL THE SERIOUS MENTAL ILLNESS P. THE SPECIFIC AIMS WERE PRETTY LARGE, I'M ONLY GOING TO SHOW YOU A SMALL SET OF ANALYSES WE CONDUCTED IN THE TALK TODAY. OVERALL WE WERE SEEKING TO IDENTIFY THE PROCESSES FACTORS ACTIONS AND EXPERIENCES THAT FACILITATE RECOVERY, DESCRIBE ROLES PLAYED BY MENTAL HEALTH PROVIDERS AN HEALTHCARE SYSTEM AND FACILITATE IMPROVEMENTS IN FUNCTIONING, QUALITY OF LIFE AND HAPPINESS, AND IDENTIFY PATTERN AN HEALTH PLAN SERVICE USE TO IDENTIFY VOLUME AND TYPE REPORTS OF SYMPTOM LEVELS, FUNCTIONING LIFE DIFFICULTIES AN LIFE SATISFACTION. SO THE RATIONALE FOR USING MR. WAS THAT WE KNEW VERY LITTLE ABOUT RECOVERY AT THE TIME. WE WANTED TO LEARN ABOUT CONSUMER PERSPECTIVES AND EXPERIENCES RATHER THAN DECIDING A PRIORI WHAT FACTORS ARE IMPORTANT. AND BE ABLE TO THIS IS A LONG TUDNAL STUDY WITH THE KAISER PERMANENTE HEALTHCARE PLAN, WE HAD A ANALYSIS AN DATA WERE CHECKED CONCURRENTLY. WE HAD A MULTI-DISCIPLINARY TEAM WITH REPRESENTATION FROM SOCIOLOGY, PSYCHOLOGY, PSYCHIATRY SYSTEM SCIENCE PUBLIC HEALTH ANTHROPOLOGY AN BIOSTATISTICS. AND WE HAD BOTH CONSUMER CONSULTANTS THAT HELPED TO DESIGN THE STUDY AND PROVIDED CON STULTING THROUGHOUT THE STUDY AS WELL AS CONSUMER ADVISORY PANEL THAT WAS COMPOSED OF CONSUMERS FROM WITHIN HEALTH PLAN AND ALSO OUTSIDE THELT PLAN. ALL -- THE HEALTH PLAN AND ALL THESE INDIVIDUALS INCLUDING THE CONSUMER ADVISORY PANEL HELPED US INTERPRET RESULTS AFTER EACH WAVE OF DATA COLLECTION AND THEN THEY HELPED US MODIFY THE STUDY AS WE MOVE FORWARD SO THAT THE NEXT WAVE OF DATA COLLECTION WAS RESPONSIVE TO THE FINDINGS FROM THE FIRST WAVE. SO THIS IS A RECURSIVE STUDY AND IT HAD I THINK IT HAD REALLY IMPORTANT IMPLICATIONS FOR OUR FINDINGS. BECAUSE WE WERE ABLE TO ADAPT AS WE WENT. AND AGAIN THAT'S ONE OF THE NICE THINGS AB THESE MIXED METHODS DESIGNS. SO WE CHECKED A GREAT DEAL OF QUALITATIVE DATA. WE HAD TWO BASELINE INTERVIEWS THAT RAN ON AVERAGE AN HOUR EACH AND TWO FOLLOW-UP INTERVIEWS, ONE AT ONE YEAR AND TWO YEARS ABOUT AN HOUR EACH. AT THE BASELINE WE USED PHOTOGRAPHY, WE GAVE PARTICIPANTS A DISPOSABLE CAMERA AND ASKED HEM TO TAKE PHOTOGRAPHS OF THINGS IN THEIR DAILY LIVES THAT THEY FELT AFFECTED THEIR RECOVERY FOR GOOD OR BAD. AT THE SECOND BASELINE INTERVIEW THEY SEND THE CAMERAS BACK THE US, WE DEVELOP THE FILM AND BROUGHT THE PHOTOS TO THE SECOND INTERVIEW AND INTERVIEWED THEM ABOUT WHAT THE PHOTOS MEN TO THEM. THE INTERVIEWS INCLUDED QUESTIONS ABOUT THEIR MEMBER TALL HEALTH HISTORY, SYMPTOMS, TREATMENT, THE RELATIONSHIPS WITH CLINICIANS, DAILY LIVE, RECOVERY PROCESSES QUALITY OF LIFE, AND AS THE STUDY WENT ON MUCH, MUCH MORE. IN TERMS QUANTITATIVE DATA, WE COLLECTED PAPER AND PENCIL QUESTIONNAIRES AND THE QUANTITATIVE INSTRUMENTS THAT WERE INCLUDED IN THE QUESTIONNAIRES WERE CHOSEN BASED ON A MODEL OF RECOVERY, THEORETICAL MODEL OF RECOVERY THAT GUIDED THE PROJECT. THEY INCLUDED QUALITY OF LIFE, MEASURE OF RECOVERY, MENTAL HEALTH SYMPTOMS, SATISFACTION WITH CLINICIAN WITH MENTAL HEALTH CLINICIAN, SATISFACTION WITH MEDICATION, AGAIN MENTAL HEALTH MEDICATION, RECOVERY ORIENTED CARE AND OVER TIME PATIENT ACTIVATION AND TRAUMA HISTORY AND THEN ALSO A GREAT DEAL MORE THAT I DON'T HAVE TIME THE TALK ABOUT TODAY. WE ALSO COLLECTED DATA FROM HEALTH PLAN AND THOSE DATA INCLUDED DIAGNOSES, BASIC DEMOGRAPHIC DATA, SERVICE USE, HOSPITALIZATIONS, CONTINUITY OF CARE WITH USUAL MENTAL HEALTH CLINICIAN, THEIR USE OF MEDICATIONS AND GAPS IN THEIR MEDICATIONS, CO-MORBID CONDITIONS INCLUDING MEDICAL CONDITIONS, MENTAL HEALTH AND ADDICTION. AND WE ALSO COLLECTED BMI. SO TODAY I WANT TO TALK ABOUT ONE SET OF ANALYSES WE CONDUCTED USING THESE DATA. THE KEY QUESTIONS WERE HOW DOES RELATIONAL CONTINUITY OF CARE, THAT IS, CONTINUITY OF CARE WITH THE SAME CLINICIAN OVER TIME, EFFECT CLINICIAN PATIENT RELATIONSHIPS. WHAT ARE ASSOCIATIONS BETWEEN RELATIONAL CONTINUITY, CLINICIAN PATIENT RELATIONSHIPS, QUALITY OF LIFE AND RECOVERY, AND THEN WHAT MEDIATES OR MODERATES THOSE RELATIONSHIPS. I'M GOING TO SKIP THAT, COME BACK. REALIZE THIS MORNING THAT I WANTED THAT SLIDE IN A DIFFERENT PLACE. SO FIRST I WANT TO GO THROUGH EXAMPLES OF FINDINGS WE HAD. THESE WERE BASED ON THEMEATIC ANALYSIS, USING A CONSTANT COMPARATIVE PROCESS AS WE COMPLETED THESE INTERVIEWS. WHAT WE FOUND WAS THAT CONSUMERS NEED AND DESIRE A GOOD FIT WITH THEIR CLINICIAN AND A COMFORTABLE RELATIONSHIP. A CARING COMPASSIONATE APPROACH WITH WITH THE CLINICIAN WHO LISTENS, UNDERSTANDS, BELIEVES AND KNOWS THE PATIENT AS A PERSON AND WHO PROVIDES HOPE FOR RECOVERY. THEY ALSO WANTED FRIENDSHIP AND MUTUALITY IN THE RELATIONSHIP, THEY WANTED MUTUAL TRUST AS PART OF THAT. THEY WANTED CLINICIANS TO BE COMPETENT, FLEXIBLE AND CREATIVE. AND WANTED COLLABORATIVE RELATIONSHIPS AND IN PARTICULAR THIS WAS CRITICAL FOR MEDICATION RELATED DECISIONS. WE FOUND CONTINUITY WITH THE SAME CLINICIAN PROVIDED NEEDED KNOWLEDGE ABOUT PAST TREATMENT EFFORTS AND AS TIME PASSED IT FACILITATED COLLABORATION AND TRUST THAT PROVIDED THE BASIS FOR THIS GOOD RELATIONSHIP. THIS IS AN EXAMPLE, I GUESS WHEN YOU THINK ABOUT A MEASURE OF WORKING ALLIANCE, YOU MIGHT HAVE A SCORE ON A SCALE. YOU HAVE A NUMBER. BUT WHAT DOES THAT REALLY MEAN? I THINK THIS GETS TO THE KEY -- ONE OF THE KEY ASPECTS OF WHAT MIXED METHODS OFFERS. BECAUSE THIS IS WHAT A GOOD WORKING ALLIANCE MEANT TO ONE OF OUR PARTICIPANTS. YOU MENTIONED YOU WORRY, YOU HOPE YOUR DOCTOR WILL OUTLIVE YOU. DEFINITELY THAT. BECAUSE I KNOW THE NEXT DOCTOR I WOULD GO TO WOULD PROBABLY SAY YOU DON'T NEED TO COME HERE. WHO ARE YOU TRYING TO KID? OR HOW COME YOU HAVE BEEN COMING TO KEISER SO LONG? OR HOW COME THAT DOCTOR LET YOU KEEP COMING? WHAT'S GOING ON WITH YOU? WHAT'S REALLY WRONG WITH YOU? THERE'S NOTHING WRONG WITH ME NOW. THERE'S NOTHING WRONG, PSYCHIATRIST, HE DON'T LOOK AT IT THAT WAY. HE KNOWS SOMETHING COULD GO WRONG AND HE'S SEEN ME COME FROM THE DEPTHS ALL THE WAY UP AND HE'S GOT THE MAINTAIN ME. BUT HE'S YOUNGER THAN ME. I WANT TO KEEP IT THAT WAY. I NEED THAT DOCTOR. I NEED THAT PSYCHIATRIST. I NEED THAT GUY JUST ABOUT MORE THAN ANYTHING IN MY WHOLE LIFE I NEED THAT GUY. HE'S THERE. HE'S LIKE MY BROTHER WATCHING OVER ME. HE'S LIKE AN OLDER BROTHER SAYING YOU'RE A GREAT GUY. WE'RE HAVING GREAT FUN IN HERE TALKING ABOUT ALL THOSE CRAZY STUFF. I'M HERE. I'M GOING TO HELP YOU. DON'T WORRY. YOU'RE MY FRIEND. I THINK THAT PROVIDES A REALLY, REALLY GOOD EXAMPLE ABOUT THE DEPTH OF UNDERSTANDING THAT YOU CAN GET FROM THESE KIND OF MEASURES. OR THESE KIND OF DATA, COMPARED TO QUANTITATIVE MEASURES. WE HAVE ALL HAD THE EXPERIENCE OF TRYING TO FILL OUT A SURVEY WHERE NONE OF THE RESPONSE CATEGORIES QUITE FIT. CERTAINLY WHERE WE CAN'T EXPLAIN THE DEPTH THAT WE SEE IN THIS KIND OF ANSWER. BASED ON A THEMEATIC ANALYSES WE PRODUCED AN INDUCKTIVELY DERIVED THEORETICAL MODEL THE WAYS RELATIONSHIPS WITH CLINICIAN AN CLINICIAN TREATMENT APPROACHES AFFECTED CONSUMERS QUALITY OF LIFE AND RECOVERY. SO WHAT YOU SEE HERE IS WE POSITED THAT RECOVERY ORIENTED PATIENT DRIVEN CARE WOULD LEAD TO IMPROVED SATISFACTION WITH MEDICATION, REDUCE SYMPTOMS, IMPROVE RECOVERY. THAT THAT SAME RECOVERY ORIENTED PATIENT DRIVEN CARE WOULD LEAD DIRECTLY TO IMPROVED QUALITATIVE OF LIFE AND TO SATISFACTION WITH CLINICIANS. WE PROPOSE -- WE POSITED THAT RECOVERY ORIENTED PATIENT-DRIVEN CARE WOULD LEAD TO A BETTER FIT WITH CLINICIANS TO INCREASE CONTINUITY OF CARE AND TO SATISFACTION WITH CLINICIANS AND THAT THIS TOO WOULD AFFECT SYMPTOMS RECOVERING QUALITY OF LIFE. SO WE WENT BACK TO OUR QUANTITATIVE DATA AND SAID WHAT DO WE HAVE TO TEST THIS MODEL? IN FACT, WE HAVE GOODEN PRETTY CLOSE WITH OUR DATA COLLECTION. BUT WE IDENTIFIED ONE CONSTRUCT FOR WHICH WE DID NOT HAVE A QUANTITATIVE MEASURE, THAT WAS THE FIT WITH CLINICIAN. SO WE WENT BACK AND REREAD ALL THE INTERVIEWS AND DETERMINED WE COULD QUANTITYTIZE OUR DATA TO FIT WITH CLINICIAN VARIABLECH IT WAS A THREE LEVEL VARIABLE. POOR, MOTD RAT AND GOOD FIT. THEN WE TESTED THE MODEL USING THE QUESTIONNAIRE HEALTH PLAN INTERVIEW BASED DATA IN A COIR VARIANT STRUCTURE MODEL -- CO-VARIANT STRUCTURE MODEL AND YOU SEE HERE THE RESULTS OF THE MODEL. SO THE SQUIGGLY LINES ARE ONES THAT WERE NOT SIGNIFICANTLY STATISTICALLY SIGNIFICANT BUT ALL THE OTHER RELATIONSHIPS WERE STATISTICALLY SIGNIFICANT AND VARIANTS EXPLAINED WAS EXTRAORDINARILY HIGH. 56% RECOVERY AND 56% RECOVERY OF QUALITY OF LIFE. IF I REPEAT THIS IN MY RESEARCH CAREER, I WILL BE A VERY HAPPY CAMPER. SO WHAT IS THIS TELL YOU? FOR ME, THIS SHOWS THE VALUE OF TRAY YANGLATION TRY TRIANGULATION OF DATA, IT PROVIDED THE QUALITATIVE ANALYSES AND RESULTING THEORETICAL MODEL ACCURATELY REPRESENTED THE DATA. THE FIT INDICES WERE GOOD EXCELLENT, VARIANTS EXPLAINED WAS HIGH. SO MUCH BETTER THAN TYPICAL IN SOCIAL SCIENCE RESEARCH. AND INDICATES THE VALUE OF USING QUALITATIVE DATA AS PART OF THE MODEL BUILDING PROCESS. SO FOR ME TOGETHER THESE ANALYSES SIGNIFICANTLY INCREASED CONFIDENCE IN FINDINGS AND IN OUR CONCLUSION. THE NEXT STEP OF COURSE IS TO TEST THIS MODEL IN A DIFFERENT POPULATION, WHAT WE SAID WAS OKAY, THIS IF WE LOOK AT THIS BOTH QAWLT QUALITATIVELY AND QUANTITATIVELY WE COME UP WITH THE SAME PICTURE IN THIS GROUP OF PEOPLE. NOW, HOW DOES IT APPLY ELSE ELSEWHERE? THAT'S WHAT I HAVE TO SAY. ANYONE HAVE QUESTIONS? I WANTED TO POINT OUT, I PUT THE REFERENCE FOR THE DATA I PRESENTED TODAY THE SECOND REFERENCE FOR AN ARTICLE THAT SEQUENTIAL RESEARCH THAT MIGHT BE A DIFFERENCE DESIGN TYPE. >> I WILL TAKE THE PREROGATIVE AS MODERATOR TO ASK THE QUESTION. I WAS WONDERING YOU WERE DESCRIBINGING QUALITATIVE LONGITUDINAL DATA COLLECTION. HOW ARE YOU CONNECTING THE DATA ACROSS7@$ APPROACH TO DOING THAT AND GETTING INSIGHT FROM THAT DATA. >> THAT'S A VERY GOOD QUESTION. I WOULD SAY WELL IN SOME CASES NOT SO WELL IN OTHER CASES AND DEPENNED ON THE RESEARCH QUESTIONS AND WE'RE WORKING ON ANALYZING THESE DATA. PROBABLY WILL BE ANALYZING THESE DATA FOR THE REST OF MY CAREER. WE HAVE SO MANY. BUT THEY'RE WONDERFUL DATA TO WORK WITH. BECAUSE OF THE -- I THINK WHAT WE TRY TO DO IS LOOK FOR THEMES OVER TIME IN THE DATA AND LOOK FOR CHANGES. WE ALSO TRY TO DO SOME QUANTITATIVE ANALYSES LOOKING AT CHANGES OVER TIME. WE ARE WORKING ON A PAPER LOOKING AT TRAJECTORIES OF RECOVERY. THAT WILL I HOPE WE WILL DO A GOOD JOB COMBINING AGAIN THE QUALITATIVE AN QUALITATIVE DATA THERE. IT'S A DIFFICULT PROBLEM. I DID WANT TO MENTION TOO ACTUALLY WE HAD A FOLLOW-UP RATE AMONG LIVING PARTICIPANTS OF 98% AT 24 MONTHS. ONE OF THE REAL VALUES OF THIS DATA COLLECTION IS PEOPLE REALLY -- IF YOU ARE TRULY INTERESTED IN THEM, THEY WANT TO TELL THEIR STORYND THEY WILL BE INVOLVED IN THE RESEARCH PROJECT. THAT'S ANOTHER ADVANTAGE OF THIS APPROACH. ANYONE ELSE HAVE ANY SPECIFIC QUESTIONS SPECIFICALLY FOR DR. GREEN? WE DO HAVE A QUESTION. CAN YOU GO TO THE MICROPHONE, PLEASE? I WOULD LIKE TO INVITE THE REST OF THE PANEL TO COME UP, WE HAVE TEN MINUTES FOR QUESTIONS FOR THE PANEL AS A WHOLE. PLEASE IF YOU HAVE ANY QUESTIONS YOU MIGHT HAVE FOR THEM AS A GROUP. >> HI, I JUST HAD A SIMPLE QUESTION, I WANTED TO KNOW WHAT -- >> I CAN'T HEAR YOU. >> I HAD A SIMPLE QUESTION I WANTED TO KNOW WHERE THE STUDY TOOK PLACE, THE SETTING. >> WITHIN CIEZ PERMANENTE NORTHWEST. -- KAISER PERMANENTE NORTHWEST IN OREGON, NORTHWEST OREGON, SOUTHWEST WASHINGTON STATE. >> ANY OTHER QUESTIONS? >> DOES ANYONE HAVE ANY QUESTIONS FOR THE PANEL AS A GROUP? WHILE YOU'RE THINKING ABOUT IT AND COMING UP WITH YOUR QUESTIONS, I ACTUALLY HAD A QUESTION FOR DR. AARONS I WANTED TO ASK. I WAS CURIOUS ABOUT THIS MAY APPLY TO DR. GREEN, AND DR. FELLSTEEN'S WORK AS WELL IN DR. AARONS' WORK YOU'RE TALKING ABOUT A VERY POLITICALLY CHALLENGING CLIMATE, HOW DO YOU HANDLE ISSUES OF CONFIDENTIALITY TRYING TO CONNECT THE QUALITATIVE DATA WITH SOME OF THE QUANTITATIVE DATA IF YOU FACE CHALLENGES WITH THAT AND HOW YOU DEAL WITH THOSE CHALLENGES. >> IN TERMS OF CONFIDENCIALITY, WE USE STANDARD PROCEDURES OF >> OKAY. THANK YOU FOR COACHING ME. WE HAVE AN INTERACTION. SO YEAH. WE DID A NUMBER OF THINGS ALSO IN DEVELOPING OUR RELATIONSHIPS BECAUSE IT'S NOT JUST THAT WE WERE COLLECTING DATA, IF IT'S JUST RESEARCH DATA AND WE BRING IT IN, AS RESEARCHERS AND WE AN AND GET RESULTS WE CAN PRETTY MUCH ASSURE CONFIDENTIALITY BECAUSE WE DON'T SHARE THE DATA OUT AT THE INDIVIDUAL TEAM LEVEL OR INDIVIDUAL RESPONDENT LEVEL BUT ONE OF THE QUICK WITH THESE AGENCIES IS WE GO BACK AND PROVIDE DATA BACK TO THE TEAMS AND TO THE ORGANIZATIONS ABOUT THINGS LIKE THE CLIMATE OF THEIR ORGANIZATIONS OR CULTURE OF THEIR ORGANIZATIONS WHICH THEY REALLY VALUED HIGHLY AND REALLY HELPED US CREATE THAT ONGOING RELATIONSHIP THAT I THINK ALSO WAS HELPFUL WITH OUR HIGH RESPONSE RATES AS WELL. BUT FOR EXAMPLE WHEN WE WERE GOING TO EACH OF THE TEAMS OUT IN AL VA, OKLAHOMA OR HUGO THAT MET IN THE BAPTIST CHURCH DOWN THERE WHO WOULD SAY THESE ARE YOUR DATA AS A TEAM. WE'RE NOT GIVING THIS TO THE ORGANIZATION. IF YOU WANT TO SHARE THIS WITH YOUR ORGANIZATION YOU CAN DO THAT BUT WE CAN PUT IT IN THEIR HANDS IN TERMS OF HAVING CONTROL OVER THE DATA AND FEEDBACK OF THE DATA. >> ALSO REITERATE PHYSICIANS ARE NOTORIOUSLY DIFFICULT TO RECRUIT BOTH FOR INTERVIEW AND FOR OUR SURVEY SO THIS MIXED METHOD APPROACH WE ALSO FOUND WAS VERYRh– HELPFUL IN NUMBER ONE, ENGAGING THEM, THEY WERE INTERESTED IN TALKING ABOUT THEIR PRACTICE AND THEIR THINGS THAT DROVE IT AND WE WARN THEM DURING THE QUALITATIVE ENTERRUES THAT WE WOULD BE DEVELOPING A SURVEY. SO I THINK THAT DEVELOPMENT OF THE RELATIONSHIP IMPROVES PROBABLY SURVEY RESPONSE RATES TOO. >> I THINK ONE OF THE THINGS THAT ANYONE WHO DOES QUALITATIVE RESEARCH HAS TO BE CAREFUL OF IS CONFIDENTIALITY OF PARTICIPANTS. YOU CAN USE ALL SORTS OF STANDARDS PRACTICES WHICH WE DO. PEOPLE CAN STILL BE IDENTIFIABLE IF YOU'RE NOT CAREFUL. AND SO WE TRY VERY, VERY HARD TO MAKE SURE THAT THERE WERE NO THERE WAS NO INFORMATION THAT LEAKED OUT AS PART OF QUOTE THAT WE USE THAT COULD BE USED TO IDENTIFY PEOPLE. SO FOR EXAMPLE, WE HAD -- WE DID QUALITATIVE ONLY PAPER THAT JENNIFER LED ACTUALLY ON PET OWNERSHIP AND ITS RELATIONSHIP TO RECOVERY BECAUSE THAT WAS ONE OF THE INTERESTING UNEXPECTED FINDINGS WE FOUND. WE HAD SOME PEOPLE WITH UNUSUAL PETS. I CAN ASSURE YOU WE CHANGED THOSE PETS. BECAUSE HOW MANY PEOPLE OWN A -- I WON'T SAY IT. BUT I THINK THESE KIND OF DATA ARE PARTICULARLY AT RISK AND PARTICULARLY IF YOU'RE LOOKING LONGITUDINALLY WITH TRAJECTORIES OVER TIME TRYING TO PRESENT THOSE DATA AND THEN NOT IDENTIFY THE PERSON, YOU HAVE TO BE VERY, VERY CAREFUL S. >> THANK YOU. START WITH HELEN. >> FOR SOME GREAT EXAMPLES OF MIXED METHODS STUDIES, ALBEIT VERY DIFFERENT STUDIES AN WHILE WE HAVEN'T BEEN READING AS THE INTRODUCTIONS YOUR BIOGRAPHIES, THEY ARE IN THE FOLDERS AND I NOTICED THAT EACH OF YOU ARE FROM VERY DIFFERENT DISCIPLINES AN PRESUMABLY ONES WHERE YOUR PRIMARY TRAINING WAS IN QUANTITATIVE RESEARCH SO I WONDER IF YOU COULD JUST BRIEFLY TELL FOLKS IN THE AUDIENCE WHO ARE INTERESTED IN GETTING INTO MIXED METHODS HOW YOU TRANSITIONED TO THIS AND HOW YOU WOULD RECOMMEND PUTTING TOGETHER TEAMS FOR DOING MIXED METHODS STUDIES. >> I DID COME FROM A QUANTITATIVE BACKGROUND. I DID COME FROM A QUANTITATIVE BACKGROUND AND REALLY GOT INTO QUALITATIVE IN A SENSE BECAUSE WE HAD TO, TO UNDERSTAND THE NATURE OF THE PROBLEM AND TO DO THE WORK THAT WE WANTED TO DO. AND I WOULD SAY ACTUALLY OTHER THAN THE QUALITATIVE RESEARCHERS, MOST OF OUR TEAM WAS THAT WAY. WHAT WAS REALLY INTERESTING THOUGH IS THAT EVEN THESE SORT OF DEDICATED QUANTITATIVE RESEARCHERS LOVED THE QUALITATIVE PROCESS AND BEING ABLE TO GET BEYOND -- GET INTO CONTEXT AND UNDERSTANDING. SO I THINK HAVING CERTAINLY SOMEONE WHO IS QUALIFIED WITH EXPERIENCE ON YOUR TEAM AND METHOD LOGICAL THINGS AND GETTING STARTED, GETTING EVERYONE ON TEAM TRAINED BECAUSE I THINK HAVING THE PERSPECTIVE OF QUANTITATIVE FOLKS INVOLVED REALLY ENHANCED THE QUALITATIVE WORK IN THE END TO. >> MY STORY IS SIMILAR, I HAD TO AND I WAS FORTUNATE ENOUGH, REALLY IT WAS THE SMALL END ISSUE THAT I TALK ABOUT AND TEAMED UP WITH LARRY THINKING ABOUT HOW WE WOULD INTEGRATE THE MIXED METHODS, HOW WE WOULD DESIGN THE STUDY TO ADDRESS THOSE ISSUES AND HOW WE WOULD INTEGRATE THE PROCESS IN TERMS OF THEORIZING IN TERMS OF DATA COLLECTION IN TERMS OF OUR ANALYSES. I HAVE WORKED WITH A NUMBER OF ANTHROPOLL GISTS ANDETH NOTHING FERS IN THAT STATEMENT OF NO TWO ARE THE SAME JUST VERY DIFFERENT APPROACHES. BUT ALSO GOT GOOD TRAINING, I WAS DOOG STUDY IN THE VA ON IMPLEMENTATION OF HIV COMPUTERIZED CLINICAL REMINDER AND I REMEMBER THE ANTHROPOLL LISTENING TO MY INTERVIEWS SAID I WOULD LIKE TO HEAR LESS OF YOU IN THOSE INSTANCES SO I GOT -- I GOT SOME GOOD COACHING AND FEEDBACK AND HOPEFULLY NOW COLLABORATING ON A COUPLE OF RO-1 STUDIES AND HOPE TO GET GOOD COACHING TO MAKE ME A BETTER MIXED METHOD RESEARCHER. >> I -- MY TRAINING ACTUALLY INCLUDED BOTH QUANTITATIVE AND QUALITATIVE RESEARCH. ALTHOUGH MORE HEAVILY FOCUSED ON QUANTITATIVE, I SAW EARLY ON THE VALUE IN TERMS OF THE RICHNESS OF THE QUALITATIVE DATA. SO WHEN I DID MY DISSERTATION I DID THE QUALITATIVE AN QUANTITATIVE DATA. I DIDN'T KNOW -- I DEN KNOW AT THAT TIME AND I DIDN'T DO A GOOD JOB OF INTEGRATING THEM BUT IT FELT IT DID HELP ME THE FIRST NIH SPONSORED PROJECT I WORKED ON ADS RESEARCH ASSOCIATE WAS A PROJECT LOOKING AT PEOPLE IN MOROCCO WHO HAD SCHIZOPHRENIA BUT NEVER BEEN TREATED. I WAS KIND OF A PROJECT MANAGER BUT I WAS ABLE TO GET A QUALITY OF LIFE MEASURE INCLUDED AND I WAS REALLY INTERESTED IN CHANGE AND QUALITY OF LIFE FOLLOWK MEDICATION TREATMENT. BUT THERE WERE NO QUALITATIVE INTERVIEWS AND I WAS FACED WITH RESULTS THAT I COULD SPECULATE ABOUT BUT NOT ANSWER WHEN IT CAME TO THE CHANGES THAT WE FOUND IN QHAWLT OF LIFE, QUALITY OF LIFE IMPROVED IN RESPONSE TO TREATMENT IN SOME DIMENSIONS BUT WAS REDUCED IN OTHERS AND I HAD SOME IDEAS ABOUT THAT. BUT I VOWED AFTER THIS TIME NEVER TO BE FACED WITH HAVING TO SPECULATE THAT I REALLY WANTED TO ASK PEOPLE WHAT HAPPENED TO THEM. SO THAT KIND OF SEALED ITTOR ME AND ALMOST EVERY STUDY SINCE THEN HAS BEEN A MIXED METHOD STUDY. >> NEXT QUESTION. >> I WAS -- IS THIS ON? I CAN'T TELL, WAS WONDERING HOW YOUR MIXED METHOD DESIGN IN SELLING THIS TO YOUR STAKEHOLDERS HELPED OR HINDERED THEIR BUY-IN. FOR EXAMPLE WE HEARD PHYSICIANS WERE HARD TO GET FOR QUESTIONNAIRES BUT THEY WERE MUCH BETTER ABOUT DESCRIBING THEIR PRACTICES. ARE THERE OTHER EXAMPLES THAT YOU HAD WITH OTHER STAKEHOLDERS WHERE THE MIXED METHODS DESIGN REALLY APPEALED TO THEM OR THEY'RE LIKE WE DON'T REALLY BELIEVE IN THIS SO MUCH WHY NOT DO IT SOME OTHER WAY. >> WHEN YOU SAY STAKEHOLDERS YOU MEAN RESEARCH PARTICIPANTS? >> PARTICIPANTS OR IN YOUR CASE THE STATE HEALTH PROGRAM IN OKLAHOMA. >> I THINK IT WAS MORE A CASE OF THEM BUILDING RELATIONSHIPS THROUGH COLLABORATORS AT THE UNIVERSITY OF OKLAHOMA HEALTH SCIENCES SENOR, IT WASN'T UNTIL WE GOT THE FUNDING THEY REALLY FIGURED WHIEWT THE STUDY WAS ABOUT. WE DID PRESENT THE DESIGN AND TALK ABOUT THERE'S SOME SURVEYS AN INTERVIEWS BUT IT WASN'T UNTIL WE ACTUALLY GOT GOING THAT WE ENGAGED THEM. I THINK THAT ONCE WE DID GET GOING THEY REALLY APPRECIATED THE CHANCE TO TELL THEIR STORIES ALONG WITH THE SURVEY METHODOLOGY WE WERE USING AND THE FACT WE INTEGRATED THEM IN THINKING ABOUT WHAT WE WERE LEARNING AS WE WENT ALONG TO OUR ANNUAL MEETINGS WHERE WE BROUGHT OUR STAKE STAKEHOLDERS AT THE ORGANIZATION AND SYSTEM LEVEL TOGETHER TO THINK AB AND TALK ABOUT WHAT HAVE WE BEEN LEARNING AND WHAT ARE OPPORTUNITIES FOR ADDITIONAL LEARNING, HOW CAN WE CHANGE WHAT WE'RE DOING IN THE RESEARCH DESIGNS AS WE GO FORWARD. >> I THINK FOR ME ONE OF THE THINGS WE DID AS WELL WAS WE ASKED CLINICIANS TO HELP US RECRUIT PARTICIPANTS. SO WE WENT BACK WHEN WE HAD RESULTS AND PRESENTED THEM TO THE CLINICIANS. SO AT THE START WE DIDN'T HAVE ANYTHING TO OFFER WE JUST ASKED WILL YOU PLEASE HELP US. BUT THEY KNEW ABOUT I US BECAUSE PATIENTS TOLD US HOW MUCH THEY ENJOYED IT. AND THEN WE SAID HERE IS WHAT WE'RE FINING SO OVER TIME WHAT THAT MEANT IS THEY HAVE BEEN MORE WILLING TO HELP RECRUIT FUTURE STUDIES. IF YOU'RE WORKING IN THE SAME GROUP OVER TIME, YOU HAVE AN OPPORTUNITY TO DEVELOP THESE RELATIONSHIPS AND IF YOU REALLY ARE COLLABORATIVE AND YOU FEEDBACK AND GIVE A ROLE IN TRYING TO PROVIDE INFORMATION THEY'RE INTERESTED IN, IT CAN BE VERY HELPFUL. >> I THINK WE HAVE TIME FOR ONE MORE QUESTION THEN WE HAVE TO TAKE OUR BREAK. >> I'M PAULA LIPMAN AT WESTTAT THE CONTRACT RESEARCH ORGANIZATION UP THE ROAD IN ROCKVILLE, MARYLAND. BASED ON WHAT I HAVE LEARNED I THINK I'M WORKING ON EXPLANATORY DESIGN LOOKING AT THE IMPLEMENTATION OF EVIDENCE BASED GUIDELINES IN PRIMARY CARE PRACTICES. FOLLOW-UP TO RESPONSES ON THE QUANTITATIVE SURVEY AND MY QUESTION HAD TO DO WITH THE SAMPLING APPROACH. HOW DO YOU START THERE SO YOU CAN APPROPRIATELY ADDRESS THE RESEARCH QUESTIONS YOU HAVE AT THE BEGINNING OF STUDY DESIGN OR RECOGNIZE HAVE A MERGE. BECAUSE IT'S EXPENSIVE AND SO FORTH AND YOU WANT TO BE STRATEGIC. >> RIGHT. WE RAN INTO THAT PROBLEM IN THE STUDY THAT I PRESENTED IN THAT WE DIDN'T EVEN KNOW WE WERE GOING TO DO CLUSTER ANALYSIS WHEN WE STARTED OUT. WE KNEW WE WERE DOING PCP LEVEL ANALYSIS. SO WE HAD DONE UP FRONT INTERVIEWS WITH CLINICIANS TO DEVELOP THE SURVEY AND SAID OH, WE SHOULD DO CLUSTER ANALYSIS, THIS IS WHAT'S GOING TO WORK, TO UNDERSTAND THE QUALITATIVE SURVEY FINDINGS. SO THEN WE WENT BACK AND LOOKED AT INTERVIEW RESULTS WITHIN THE CLUSTERS. SO WE DIDN'T RECRUIT THEM AS CLUSTERS INITIALLY. WE COULD HAVE IF WE REALLY KNEW MORE -- KNEW WHAT THE FINDINGS WERE GOING TO BE. SO THAT'S THE WAY WE DID IT. IT WAS A LITTLE BIT OF A RETROSPECTIVE GROUP SO WE DID TRY TO RECRUIT. WE DID RECRUIT IN A BALANCED WAY AMONG PERFORMANCE. WE KNEW WE WERE INTERESTED IN VARY YOITION PERFORMANCE. THAT HELPED. >> THANK YOU. >> UNFORTUNATELY THAT'S ALL WE HAVE TIME FOR. NOW WE HAVE A BREAK SO WE'LL BE BACK IN HERE AT 11:00. THANK YOU AGAIN TO THE PANEL. [APPLAUSE] >> HI, EVERYONE, WELCOME BACK. HOPE YOU HAD A GOOD BREAK. I AM THE MODERATOR OF THIS NEXT SESSION ON CHALLENGES AND INTEGRATING QUALITATIVE AND QUANTITATIVE DATA WITH AN OVERVIEW OF THE QUALITATIVE PERSPECTIVE. SO AS I INVITE THE SPEAKERS TO COME UP, RENA, KATIE AN JOHN, PLEASE COME ON UP AND HAVE A SEAT. I'LL GIVE A VERY BRIEF OVERVIEW OF QUALITATIVE METHODS AND EVEN THOUGH THIS IS QUALITATIVE METHODS I'M GOING THE START WITH A CLOSE ENDED QUESTION. HERE IS A QUESTION FOR YOU. DO YOU PRIMARILY IDENTIFY YOURSELF AS A QUANTITATIVE PERSON, A QUALITATIVE PERSON OR A MIXTURE, OTHER TYPE OF PERSON. HOW MANY OF YOU IDENTIFY AS A PRIMARILY QUANTITATIVE PERSON BY TRAINING? HOW MANY QUALITATIVE? OKAY. HOW MANY MIXED OR OTHER, SOMETHING ELSE? WOW, THERE'S ABOUT A THIRD EACH. FABULOUS. SO THIS -- THESE NEXT COUPLE OF MINUTES WILL BE FOR THE QUANTITATIVE PEOPLE TO GIVE A BRIEF OVERVIEW OF QUALITATIVE METHODS AND REST IT MAYBE A BIT OF A REVIEW. IN GENERAL QUALITATIVE METHODOLOGY USES A NATURAL SETTING FOR DATA, IT FOCUSES ON PARTICIPANTS PERSPECTIVE AND MEANING, THERE IS AMBIGUITY IN DECISION MAKING WHICH IS FRUSTRATING FOR QUANTITY FOLKS BUT THAT'S OKAY. THERE'S A DIFFERENCE ROLE OF RESEARCHER FROM QUANTITATIVE WHERE RESEARCHER IS AN ACTIVE LEARNER AND NOT AN EXPERT SEEKING TO FIND THE ONE TRUE ANSWER BUT RATHER SOMEBODY SEEKING TO UNDERSTAND WHAT'S GOING ON AND QUALITATIVE METHODS GOALS ARE OFTEN TO DEVELOP THEORY AND GAIN INFORMATION THAT MAYBE DIFFICULT TO CONVEY QUANTITATIVELY. AND„Mz CARLA GAVE AN EXAMPLE OF THAT EARLIER AND TO BETTER UNDERSTAND THE PHENOMENA. THAT'S OFTEN A SMALLER SAMPLE SIZE BUT MORE DEPTH AND VERY DIFFERENT TYPES OF QUESTIONS. IT'S OFTEN A GOOD METH FOR SITUATIONS WHERE YOU DON'T HAVE MUCH INFORMATION OR YOU NEED MORE INFORMATION TO BE ABLE TO DO SOMETHING ELSE. SO VERY, VERY BASIC OVERVIEW WITH DIFFERENCES BETWEEN THE TYPES OF GOALS, THE TYPES OF QUESTIONS ASKED, THE TYPE OF THINKING THAT'S INCLUDED AS DEDUCTIVE VERSUS INDUCKTIVE, DIFFERENT TYPES OF METHODS, AND JOHNS PRESENTATION COVERED THIS IN MUCH MORE DEPTH AND DIFFERENT TYPES OF ANALYSIS. THESE ARE SOME OF THE QUALITATIVE TRADITIONS, I THINK QUALITATIVE IS SOMETIMES VIEWED AS A HO HOMOGENOUS GROUP WHEN IT'S NOT HOMOGENOUS METHODOLOGIES AND SOME ARE FROM JOHN'S BOOK NISH LITTLE PUBLISHED IN 1997, THERE'S A NEWER VERSION NOW. BIOGRAPHY IS ONE OF THE TRADITIONS FROM THE FIELD OF HISTORY WHERE IN DEPTH EXPLORATION OF THE LIFE OF AN INDIVIDUAL PROBABLY FAMILIA WITH THAT PHENOM LOGICAL TRADITION FROM THE FIELD OF PSYCHOLOGY SEEKS TO UNDERSTAND THEs SEN OF ENS PEERNSES REGARDING PHENOMENON. GROUNDED THEORY WHICH SOME MAYBE MORE FAMILIAR WITH IS DEVELOPMENT OF THEORY THAT'S GROUNDED IN DATA IN YOUR FIELD DATA. GROUNDED FOR HEALTH SERVICES IN PARTICIPANTS EXPERIENCES OF A SITUATION AND YOU DEVELOP THEORY FROM THAT. SO THAT'S AN EXAMPLE OF HOW CARLA PUT A MODEL TOGETHER THAT WAS GROUNDED IN THE DATA THAT WE'RE COLLECTED FROM PARTICIPANTS THAT'S FROM SOCIOLOGY. INTERPRETING A PARTICULAR CULTURAL GROUP FROM A FIELD OF ANTHROPOLOGY, CASE STUDIES FROM POLITICAL AND SOCIAL SCIENCE LOOKING AT EXPLAINING A CASE OR SERIES OF CASES IN A SYSTEM OVER TIME AND THEN ONE THAT I ADDED IS CONTENT ANALYSIS WHICH I ADDED BECAUSE IT'S QUANTITATIVE, FRIENDLY, IT'S AN NAILSIS OF WORDS AND PHRASES FROM THE TEXT YOU CAN ESSENTIALLY QUANTIFY QUALITATIVE DATA. THERE'S SPECIFIC WAYS TO ASK THE QUESTIONS IN ORDER TO MAKE THAT WORK THAT ESSENTIALLY YOU CAN QUANTIFY INFORMATION AND PUT ANYTIME A QUANTITATIVE ANALYSIS SHOULD YOU CHOOSE TO DO THAT. BASIC OVERVIEW OF WHAT'S GOING ON AND NEXT SLIDE SHOW COMES UP FOR DR. PASICK. DR. PASICK WILL COME UP FIRST FROM UNIVERSITY OF CALIFORNIA SAN FRANCISCO. FOLLOWED BY KATIE WILLING AND FINAL IS JOHN RICH FROM DREXEL UNIVERSITY. THANK YOU. >> THANK YOU VERY MUCH. GREAT MEETING. I'M EXCITED TO BE PART OF IT. TITLE OF MY TALK IS CULTURE AND HEALTH BEHAVIOR THROUGH THE LENS OF MIXED METHODS. THE BEST -- TRY THAT. OKAY. MY WORK FOCUSES ON CANCER DISPARITY, THE DISPROPORTIONATE BURDEN OF CANCER BORNE BY COMMUNITIES OF COLOR AND THOSE OF LOW INCOME. OVER THE PAST 40 YEARS NOT ONLY HAVE WE MADE ALMOST NO PROGRESS IN REDUCING AND ELIMINATING DISPARITIES BUT BECAUSE OF PROJECTIONS DEMOGRAPHIC PROJECTIONS IN TERMS OF THE AGING AND INCREASING DIVERSIFICATION OF THE POPULATION, DISPARITIES ARE PROJECTED TO INCREASE. THE REASONS FOR THIS ARE COMPLEX AS YOU MIGHT IMAGINE BUT AMONG THEM, DON'T NANS OF BIOMEDICAL APPROACH TO COMMUNICATION AND RESEARCH ON HEALTH BEHAVIOR. IT'S BEEN OUR PATTERN TO STUDY BEHAVIOR IN ISOLATION FROM THE CULTURAL CONTEXT AN RELEGATING CULTURE TO THE BACKGROUND. SO THE PREMISE OF MY TALK IS THAT CANCER DISPARITIES RESEARCH HAS BEEN LIMITED BY THIS BIOMEDICAL APPROACH, CULTURE IS RELEGATED TO THE BACKGROUND OR OPERATIONALIZED AS A SET OF DECONTEXTUALIZED CON COGNITIVE CONSTRUCTS SUCH AS FATALISTIC BELIEFS. IN THE REAL WORLD, CULTURE IS BACKGROUND, FOREGROUND, CULTURE IS CONTEXT. THERE'S A LOT OF DEFINITION OF CULTURE, MY COLLEAGUES AND I HAVE CHOSEN THE FOLLOWING. CULTURE AS PATTERN PROCESSES OF PEOPLE MAKING SENSE OF THEIR WORLD AND THE CONSCIOUS AND UNCONSCIOUS ASSUMPTIONS, EXPECTATIONS, KNOWLEDGE AND PRACTICES THEY CALL ON TO DO SO. WE EMPHASIZE THE UNCONSCIOUS ASSUMPTIONS AND EXPECTATIONS AND SO FORTH BECAUSE THESE ARE INNATE DYNAMICS THAT GO ON FOR EACH OF US IN THE WORDS WE LIVE IN. I ARGUE THESE CAN'T BE TAPPED BY SURVEY QUESTIONS OR RESPONSES IN FOCUS GROUPS. WE NEED AL ATTORNEYTIVE APPROACHES WHERE MIXED METHODS COMES IN. SO I WANT TO SHARE MY PATH TO THE REAL WORLD WITH BOTH FEET DEDUCTIVE QUANTITATIVE RESEARCH PARADIGM LOOKING AT STUDIES OF BREAST AND SERVE CALL CANCER SCREENING. THIS WAS A DECADE, CALL IT MY DECADE UNDER THE DOMINANT PARADIGM, WE WERE DOING A LARGE MULTI-ETHNIC RANDOMIZED CONTROL TRIAL AND SURVEY STUDIES ALL DUTIFULLY APPLYING THE DOMINANT BEHAVIORAL THEORIES BECAUSE WE KNEW THAT REVIEWERS WANTED TO SEE CONCEPTUAL FRAMEWORK. I GOT GOOD HAVING THE FRAMEWORK LINKING IT TO THE MEASURES AN INTERVENTIONS THE REALITY IS WORKING FOR YEARS IN THESE DIVERSE COMMUNITIES I CAME TO FEEL THAT THEORY NOT INFORMING WHAT THE QUESTIONS WE WERE TRYING TO ANSWER. NOT ONLY WORKING WITH DIVERSE COMMUNITIES IN CLINICS AND COMMUNITIES AND ALL OUR WORK IS MULTI-LINGUAL. WE WERE GETTING IN THERE AND EXPERIENCING HOW THINGS WORK BUD NOT ABLE TO MEASURE AN DESCRIBE IT. HERE ARE SOME OF THE QUESTIONS WE WERE LEFT WITH AFTER THESE STUDIES, HOW AND WHY DID INTERVENTIONS WORK, HOW COMPARABLE ARE SURVEY QUESTIONS ACROSS CULTURES. WHY DON'T THEORIES ACCOUNT FOR MORE VARIABILITY AND BEHAVIOR AND ARE THEORETICAL CONSTRUCTS MEANINGFUL AND APPROPRIATE ACROSS CULTURES. MY COLLEAGUES WOULD COMPLAIN ABOUT THE THEORIES BUT WE WENT AHEAD USED THEM BECAUSE THAT'S WHAT WE HAD TO DO TO GET FUNDED. SO I STARTED SEARCHING BEYOND THE PUBLIC HEALTH LITERATURE READING AND SOCIAL SCIENCES AND ANTHROPOLOGY AND SOCIAL PSYCHOLOGY. IT WAS IN THAT WAY THAT I DISCOVERED THE KEY TO ENLIGHTENMENT. IT LED TO A WHOLE SET OF STUDIES UNDER AN INDUCKTIVE QUALITATIVE AND MIXED METHODS PARADIGM. I'M GOING TO TALK TO YOU ABOUT ONE OF THESE TODAY, THE STUDY IS CALLED THE BEHAVIORAL CONSTRUCTS AND CULTURE AND CANCER SCREENING. THIS IS BASIC BEHAVIORAL RESEARCH TO ASSESS CULTURAL APPROPRIATENESS OF FIVE BEHAVIORAL THEORY CONSTRUCTS. WE USED MIXED METHODS INCLUDING SURVEYS FROM SOME OF OUR EXISTING INTERVENTION TRIALS, IN DEPTH INTERVIEWS WITH KEY INFORMANT SCHOLARS, COMMUNITY GATE KEEPERS, WOMEN IN THE COMMUNITY, THESE WERE ALL LATINA AND FILL PIANO WOMEN. FILL PIANO THESE ARE THE DOMINANT CONSTRUCTS USED IN STF DIS OF CANCER -- STUDIES OF CANCER PROCEEDINGS PERCEIVED BENEFIT AND SUSCEPTIBILITY AND SUBJECTIVE NORMS. I'M GOING TO HIGHLIGHT FIRST WE GOT IT FUNDED BACK IN 2001 AND IT WAS BY THE GRACE OF THE LATE SABRA WOOLEY AN ANTHROPOLL GIST AND PROGRAM OFFICER IN THE NATIONAL CANCER INSTITUTE SHE RECOGNIZED THIS AS SOMETHING TO SEE GOING FORWARD AND MADE IT HAPPEN. ALWAYS GRATE. TO HER. OUR TEAM WAS VERY MULTI-DISCIPLINARY AND TURNED INTO A TRULY TRANSDISCIPLINARY TEAM. WE HAD THREE ANTHROPOLL GIST, BEHAVIORAL SCIENCE, BIOSTATISTICS, HEALTH PSYCHOLOGY, PUBLIC HEALTH AND SOCIOLOGY. THIS WAS THE BLESSING -- CURSE AND BULLETING OF THE STUDY. AS YOU CAN IMAGINE IT WAS DIFFICULT. WE STRUGGLED TO COME TOGETHER TO ESSENTIALLY CHECK SOME OF OUR DOG MA AT THE DOOR AND COME TOGETHER WITH A NEW APPROACH TO THE QUESTIONS WE ALL WANTED TO ANSWER. IT WAS VERY EXCITING. I LEARNED A LOT. I'M STRICTLY PUBLIC HEALTH. TALK ABOUT ACTIVE LEARNER, I WAS PI AND ACTIVE LEARNER IN THIS STUDY THROUGHOUT. SO THIS IS A QUOTE FROM (INAUDIBLE) ONE OF THE FATHERS OF THEORY OF PLANNED BEHAVIOR, VERY DOMINANT THEORY IN HEALTH RESEARCH. IT IS A UNIVERSAL TRUTH IN BEHAVIORAL SCIENCE IN THE FINAL ANALYSIS A PERSON'S BEHAVIOR IS EXPLAINED BY CONSIDERING HIS OR HER BELIEFS. YOU DON'T NEED ANYTHING MORE THAN THAT, YOU NEED THE PERSON YOU ASK THEM QUESTIONS, YOU GET ANSWERS, YOU LEARN WHAT YOU NEED TO KNOW. SO LET'S TAKE THE CONCEPT OF INTENTION. THIS IS A CURRENT PAGE ON THE NCI WEBSITE. AND YOU'LL SEE WHAT IT SAYS ABOUT INTENTION. MOST THEORIES OF HEALTH BEHAVIOR SHARE A BELIEF THE SINGLE BEST PREDICTOR OF AN INDIVIDUAL BEHAVIOR IS HIS OR HER INTENTION TO ENGAGE IN THAT BEHAVIOR. INTENTION IS REGARDED AS SO POWERFUL THAT MANY STUDIES EVEN USE IT AS THE MAIN OUTCOME WHEN NOT ABLE TO STUDY CANCER SCREENING ITSELF. TO BRING ABOUT A CHANGE IN PERSON'S INTENTION MEANING THEIR STATED INTENT TO GET A MAMMOGRAM IN THE NEXT YEAR, THAT IS AS GOOD AS MEASURING MAMMOGRAPHY. INTENTION IS DRIVEN BY A SET OF POSITIVE AND NEGATIVE BELIEFS ABOUT BEHAVIOR. SO IF YOU CAN CHANGE THOSE BELIEFS YOU SHOULD CHANGE INTENTION, BINGO, YOU HAVE CHANGED BEHAVIOR. BUT WE WANTED TO KNOW, IS INTENTION MEANING AN MEASUREMENT ACROSS CULTURES. SO THIS IS THE FIRST PART OF OUR STUDY DESIGN. THIS IS A SEQUENTIAL QUANTITATIVE TO QUALITATIVE STUDY. AND THE QAN PART CONSISTED OF LONGITUDINAL SURVEYS, THE PRE- AN POST SURVEYS IN A MULTI-ITSELF IN THIS CASE STUDY OF CANCER SCREENING. WE WERE MEASURING -- THIS IS THE USUAL MEASURE OF INTENTION. OR MOST COMMON WAY OF GOING IT. I INTEND TO GET A MAMMOGRAM IN THE NEXT 12 MONTHS, EXTREMELY UNLIKELY TO LIKELY ON A 7 POINT SCALE. WE COULDN'T DO THAT OVER THE PHONE IN FOUR LANGUAGES WITH WOMEN OF HIGH AND LOW LITERACY SO WE HAD TO SIMPLIFY THE QUESTION TO DO YOU PLAN TO HAVE A MAMMOGRAM IN THE NEXT 12 MONTHS, YES AND NO. VERY BRIEFLY, THIS IS OUR FINDING. THIS IS FINDINGS FROM THE BASELINE AND THE FOLLOW-UP STUDY WHERE WOMEN AT BASELINE WERE ASKED THEIR INTENTION TO GET A MAMMOGRAM IN THE NEXT 12 MONTHS AND THEN ABOUT 18 MONTHS LATER THEY WERE SURVEYED TO SEE WHETHER THEY HAD HAD A MAMMOGRAM. I'M NOT GOING THROUGH ALL THE DETAILS OF THE TABLE IN THE INTEREST OF TIME BUT WHAT THIS SHOWS IS THAT INTENTION PREDICTED MAMMOGRAPHY ONLY FOR WHITE WOMEN, NOT FOR THE OTHER RACE ETHNIC GROUPS. THIS MAKES HEALTH PSYCHOLOGIST AS LITTLE NUTS. I'M NOT AWARE OF ANYONE HAVING REPLICATED THIS OR ATTEMPTED TO DO A LONGITUDINAL STUDY OF -- LIKE THIS IN DIVERSE POPULATIONS. O THE NEXT PART OF OUR STUDY WAS -- AN INTENSIVE INDUCKTIVE QUALITATIVE ASSESSMENT TO INVESTIGATE AND ELUCIDATE CONNECTIONS BETWEEN SOCIOCULTURAL CONTEXT AND MAMMOGRAPHY AMONG PHILIPPINO AND LATINO WOMEN. OUR INFORMANTS WERE KEY INFORMANTS, THESE WERE ACADEMICS WHO WERE THEMSELVES LATS KNOW OR -- LATINO OR PHILIPPINO. SCHOLARS IN VARIOUS FIELDS WHO WERE VERY KNOWLEDGEABLE IN THEIR CULTURE AND ALSO REALLY ABLE TO STEP BACK AND TALK ABOUT THEIR CULTURE. WE HAD -- THERE WERE 11 KEY INFORMANTS, 13 GATE KEEPERS, GATE KEEPERS WERE PEOPLE WHO HAD A LOT OF EXPOSURE TO THE POPULATION. PUBLIC HEALTH NURSE, SOCIAL SERVICE AGENCY MANAGERS AND SO FORTH. THEY WERE ALSO THEMSELVES LATINA OR PHILIPPINO. WE INTERVIEWED 29 WOMEN, LAY WOMEN. THIS WAS A VERY IMPORTANT PART OF THE STUDY BECAUSE THIS REALLY GAVE US INSIGHT FROM MULTIPLE PERSPECTIVES. I WAS SO BLOWN AWAY AS I CAME TO SEE THE DATA UNFOLD AND UNDERSTAND THE IMPORTANCE OF THESE MULTIPLE PERSPECTIVES IN ADDING DIMENSION TO WHAT WE WERE UNDERSTANDING ABOUT THE WAY WOMEN LIVED. AND I COULD REALLY SEE THE VAST DIFFERENCE BETWEEN JUST TALKING TO LAY WOMEN VERSUS GATHERING THESE MULTIPLE PERSPECTIVES. I'LL GIVE YOU SOME EXAMPLES OF THAT. THE INDUCKTIVE APPROACH WAS ALSO PARAMOUNT. IT WAS STRESSFUL FOR ME BECAUSE OUR FIRST SETS OF INTERVIEWS WITH KEY INFORMANTS WERE WIDE RANGING, ASKING ABOUT THE DAILY LIFE OF WOMEN IN THIS COUNTRY VERSUS THEIR HOMELAND, ABOUT FAMILY RELATIONSHIPS AND I'M THINKING WHEN ARE WE GOING TO TALK ABOUT THE CONSTRUCTS BECAUSE I WAS IN THIS SPACE OF PUBLIC HEALTH BEHAVIORAL SCIENCE AND WHILE I WANTED TO DO THIS IN DEPTH HE CAN PLOR RATION, I KIND OF THOUGHT -- EXPLORATION I THOUGHT WE WOULD TALK MORE DIRECTLY ABOUT THE SUBJECT MATTER OF OUR STUDY. LITTLE BY LITTLE AS WE WENT THROUGH THESE INTERVIEWS THOUGH, I BEGAN TO SEE PATTERNS OF LIFE THAT REALLY WERE DISCONNECTS WITH THE FUNDAMENTAL ASSUMPTIONS OF OUR THEORETIC CONSTRUCTS. NOT ONLY LIGHT BULBS GOING OFF BUT I HAD BEEN LOOKING THROUGH A PINHOLE AT THE BEHAVIORS AND SUDDENLY I COULD SEE LIFE IN FULL COLOR AND THREE DIMENSIONS, I WILL GIVE YOU SOME EXAMPLES. BUT THIS WAS REALLY TRANSFORMATIVE FOR ME TO BE ABLE TO WORK ALONGSIDE THESE ANTHROPOLL GISTS ON THIS COMMON QUESTION OF INTEREST. SO FROM THESE DATA WE WERE ABLE TO SEE REALLY FLOWING THROUGHOUT ALL LEVELS OF OUR INTERVIEWS, THREE OVERARCHING DOMAINS THAT WE CALLED SOCIAL CAPITAL, TRANSCULTUREATIN AN TRANSMIGRATION AN RELATIONAL CULTURE. IN THE INTEREST OF TIME I CAN'T GO INTO ALL THESE, I AM GOING TO ZERO IN ON RELATIONAL CULTURE. AFTER WE IDENTIFIED AND DEFINED AND HAD A NUMBER OF THEMES THAT RELATED UNDER THESE DOMAINS, WE THEN STARTED TO LOOK AT THE CONSTRUCTS THROUGH THE LENS OF THESE SOCIAL CONTEXT, SOCIOCULTURAL CONTEXT DOMAINS. WE LOOKED AT SELF-EFFICACY AND WE COULD HAVE LOOKED AT CONSTRUCTS UNDER ALL THE DOMAIN BUS WE DIDN'T HAVE THE PERSON POWER TO DO THAT IN THE END. THE PUBLICATIONS THAT RESULTED ARE THESE. WE LOOKED AT SELF-EFFICACY IN THE CONTEXT OF SOCIAL CAPITAL, PERCEIVED BENEFITS AND SUSCEPTIBILITY IN THE CONTEXT OF THE TRANSCULTUREATION AN INTENTION AND SUBJECTIVE NORMS IN THE CONTEXT OF RELATIONAL CULTURE, IT'S THAT LAST ONE I'M GOING TO TELL YOU ABOUT. SO RELATIONAL CULTURE, WE DEFINED FROM OUR DATA AS THE PROCESSES OF ENTERDEPENDENCE AND INTERCONNECTEDNESS AMONG INDIVIDUALS WITHIN FAMILIES AND GROUPS AND THE PRIORITIZATION OF THESE CONNECTIONS ABOVE VIRTUALLY ALL ELSE. CONNECTEDNESS WAS ALL THE DIFFERENT WAYS THAT PEOPLE EXPRESS RELATIONSHIPS TO OTHERS WHETHER FAMILY MEMBERS OR DOCTORS IN A CLINIC, CONCEPT OF SELF WAS REALLY IMPORTANT. PEOPLE CLEARLY DID NOT SEE THEMSELVES AS AUTONOMOUS AND INDEPENDENT, THEY ONLY THOUGHT OF THEMSELVES OFTEN IN RELATION TO OTHERS. AND THEY PRIORITIZE THOSE OTHERS. THEY PUT THEIR OWN NEEDS SECOND AND OTHERS CAME FIRST. YOU'LL PROBABLY RECOGNIZE THIS AS THE DIFFERENCE BETWEEN COLLECTIVISM AND INDIVIDUALISM. BUT WE CHOSE NOT TO GO WITH THAT CONCEPT BECAUSE IT'S MORE A BOWNTED -- BOUNDED WAY OF LOOKING AT THIS AND WE SAW THIS AS MUCH MORE DYNAMIC AND COMPLEX. WE SAW DECISION MAKING AS CONSULTATION WHERE PEOPLE REALLY DID NOT MAKE DECISIONS ON THEIR OWN, IT JUST DIDN'T EVEN OCCUR TO THEM TO SIT THERE IN FRONT OF A DOCTOR AND MAKE A DECISION THAT SOMETHING THAT YOU DO IN THE CONTEXT OF THE PEOPLE YOU MAKE DECISIONS WITH. WHICH IS USUALLY YOUR FAMILY. TRUST, ACCOMPANIMENT, PEOPLE GO PLACES WITH OTHER PEOPLE THEY'RE CLOSE TO, THEY DON'T REALLY WANT TO GO ALONE. THIS JUST CAME OUT LOUD AND CLEAR IN SO MANY DIFFERENT WAYS IN ALL OUR INTERVIEWS. HERE ARE SOME QUOTE, RELATIONAL CULTURE. RELATIONAL YOU'RE NOT AN INDIVIDUAL THAT LIVES IN YOUR HEAD BUT AN INDIVIDUAL THAT LIVES IN PROCESSES WITH OTHER HUMAN BEINGS SO IT'S VERY DISTRESSING TO ME WHEN PEOPLE WRITE AS YOU'RE JUST THIS UNDERDEVELOPED PERSON WHO NEEDS OTHER PEOPLE OR ELSE YOU DON'T KNOW WHO YOU ARE. SPEAKING ABOUT THE EXPRESSION YOU AND I ARE ONE WHICH IS A COMMON EXPRESSION AMONG PHILIPPINOS, IT'S VERY, VERY DEEP, IT MEANS OTHER LITERALLY AND INNER SELF-SO IT EMPHASIZES THE RELATIONSHIP BETWEEN MYSELF AND OTHERS. THERE ARE WORDS FOR THIS IN OTHER LANGUAGES, THAT SHOWS HOW BASIC AND FUNDAMENTAL IT IS. SO AGAIN, WE SCRUTINIZE THE CONSTRUCTS THROUGH THE LENS OF THESE DOMAINS. AND WHEN WE LOOKED AT INTENTION THROUGH THE LENS OF RELATIONAL COULD CHU THESE ARE THE THINGS WE FOUND, THE MEANING OF STATED INTENTION FOR MANY REASONS INCLUDING THE SUBTLETIES OF INDIRECT COMMUNICATION, PEOPLE WILL SAY YES BUT MEAN NO. IT'S LIKE THEY SAY, WELL, YAY YEAH, I MADE THE APPOINTMENT OR I'LL BE THERE BUT THEY ARE NOT REALLY PLANNING TO COME. IT'S LATINA NURSE KNEW THAT. SHE UNDERSTOOD IT INHERENTLY. YOU HAVE TO TELL THEM IT'S IMPORTANT THAT YOU COME, IF NOT YOU JUST TELL ME IT'S OKAY FOR YOU TO TELL ME THAT YOU CAN'T COME. YOU HAVE TO TELL THEM THAT. THAT'S HOW I DEAL WITH IT. IT'S OKAY FOR YOU TO TELL ME THAT YOU CAN'T COME, YOU KNOW I WON'T GET MAD. SO THIS IS AN INSIDER TALKING TO OTHER INSIDERS WHO UNDERSTAND THAT SHARE THIS UNDERSTANDING OF WHAT IT MEANS TO SAY YES I WILL. ABSOLUTELY WHAT INTENTION IS ABOUT. LACK OF INTENTION BUT WILLINGNESS TO BE SCREENED. INTENTION DUE TO RELATIONSHIPS NOT THE POSITIVE AND NEGATIVE BELIEFS ABOUT BEHAVIOR. IN RELATIONSHIP CULTURES IT'S LIKELY TO FIND WOMEN GETTING SCREENED BECAUSE THEY PERCEIVE THE PERSON ASKING TRULY CARESCH THIS IS NOT BASED ON PERCEIVED BENEFIT OF THE TEST. INDEED THEY MAY NOT GO IN THE ABSENCE OF THE RELATIONSHIP. IF YOU DON'T HAVE THAT RELATIONSHIP WITH THEM THEY'RE LESS LIKELY TO DO IT. I THIS THEY DO IT FOR US BECAUSE WE'RE RECOMMENDING THEY GET THIS MAMMOGRAM. THEY THEMSELVES DON'T UNDERSTAND WHY THEY NEED IT. YEARS BEFORE THIS STUDY, I SAW THIS IN OUR OUTREACH WHERE LAY HEALTH WORKERS WOULD BRING IN LITTLE OLD GRANDMAS AND THEY WERE NOT THERE BECAUSE THEY WANTED TO FIND CANCER EARLY ENOUGH TO BE TREATED. THEY WERE THERE BECAUSE OF LAY HEALTH WORGER. A LOT OF PEOPLE FEEL THEIR WORD IS HONORABLE. MAYBE DON'T FEEL LIKE GOING BUT THE THOUGHT OF THE WORD THAT RELATIONSHIP BECAUSE THEY SID THEY WERE GOING BECAUSE THEY HAD THAT CONNECTION AND THEY SAID THEY WERE GOING TO GO SO THEY WOULDN'T WANT TO DISAPPOINT HERE, YES, IT'S AN HONOR THING. I'M SORRY I'M CUTTING THIS BACK AN OVERSIMPLIFYING IT BUT I'M HOPING YOU GET THE IDEA HERE. THESE ARE RELATED THEMES RELATED TO INTENTION WE FOUND. THE MEANING OF STATED INTENTION FOR REASON INS COLLUDING SUBTLETIES OF INDIRECT COMMUNICATION PEOPLE WILL SAY YES BUT MEAN NO. DESIRABILITY AND POLITENESS, PART OF RESPECT IS BEING AGREEABLE AND NOT SAYING NO. INTENTION BASED ON RELATIONSHIP, I ALREADY DESCRIBED THAT. INTENTION IN THE ABSENCE OF BELIEF, WOMEN WILL INFORM TO GET MAMMOGRAM WE SAW MANY EXAMPLES OF THAT. ACCEPTANCE OF PUTTING THINGS OFF. WOMEN MAYBE ACCUSTOMED TO BEING UNABLE TO DO ALL THEY INTEND. THEY COPE BY ACCEPTING THAT SOME IMPORTANT THINGS MUST BE PUT OFF. SO OUR CONCLUSIONS AB INTENTION, THE MEANING AND RELEVANCE OF INTENTION ACROSS CULTURES IS THE CONSTRUCT SHOULDN'T BE USED ADS UNIVERSAL DUE TO DIFFERENCES IN MEANING OF STATED INTENTION, DIFFERENCES IN RELATIONSHIP OF INTENTION TO BEHAVIOR, MISSING PREDICTORS AND DIFFERENT ISSUES AN RELATIONSHIPS THAT INFLUENCE THE BEHAVIOR. OUR CONCLUSIONS OF THE -- IN THE STUDY OVERALL, THE SCIENCE OF CANCER HEALTH DISPARITIES WILL EVOLVE WHEN COLLABORATING WITH SOCIAL SCIENTISTS TO FORM TRANSDISCIPLINARY TEAMS WHEN WE START AT THE BEGINNING INDUCKTIVELY, WHEN WE EMBRACE THE COMPLEXITY OF THE REAL WORLD AND EXPLORE MULTIPLE COMPLIMENTARY PERSPECTIVES AND INTEGRATE INDUCKTIVE AND DEDUCTIVE PARADIGMS IN THE FORMATIVE AND EVALUATION STAGES OF RESEARCH. IN OTHER WORDS, USING MIXED METHODS. THIS IS A MONOGRAPH THAT WAS PUBLISHED THANKS TO BOB CROYLE AT NCI AND ELAINE OLD IN THE SOCIETY OF PUBLIC HEALTH EDUCATION, IT HAS SEVEN PAPERS AN FIVE COMMENTARIES ON THIS STUDY AND IT WAS PUBLISHED IN A SUPPLEMENT TO HEALTH EDUCATION AND BEHAVIOR IN 2009. SO YOU'LL FIND EVERYTHING AND I COULDN'T TELL -- THAT I COULDN'T TELL YOU HERE THIS MORNING. THANK YOU. [APPLAUSE] DO YOU WANT QUESTIONS NOW? AT THE END? HI MY NAME IS KATIE WILLING WILLGING. I'LL TALK ABOUT A LARGE STALE SCAIL IMPLEMENTATION STUDY TO DO WITH HEALTH REFORM IN NEW MEXICO. I'M GOING TO START WITH INFORMATION ABOUT ME SINCE I'M SO FASCINATING. I'LL MOVE INTO THE REFORM, I'LL TALK ABOUT MIX STUDY, MIXED METHODS STUDY OF REFORM. SHARE WITH YOU TWO SAMPLE ANALYSES. IF WE HAVE TIME I MIGHT TOUCH ON IN PROGRESS ANALYSES, TALKS ABOUT THE STRENGTH AND CHALLENGES OF DOING THIS TYPE OF WORK AND SOME DATA PRESENTATION CONSIDERATIONS. SO I WAS TRAINED IN YOUR TRADITIONAL ANTHROPOLOGY DEPARTMENT. YOU GO OUT AS SOLO AND YOU DON'T REALLY COLLABORATE WITH QUANTITATIVE RESEARCHERS. IN FACT IN MANY DEPARTMENTS QUANTITATIVE RESEARCH IS FROWNED UPON, IN THE DEPARTMENT THAT I RECRUIT MOST OF MY STUDENTS FROM, I BEG THEM TO DO DISSERTATION RESEARCH PROJECTS ON SOME OF THE PROJECTS THAT I HAVE AND THEY REFER TO IT AS TURNING TO THE DARK SIDE. SO I HAVE SOME DISCIPLINARY BIASES WITHIN MY PROFESSION. MY AREAS OF INTEREST INCLUDE GENDER AND SEXUALITY, HEALTH POLICY, MANAGED CARE, MENTAL HEALTH SERVICES RESEARCH, I'M AN APPLIED ANTHROPOLL GIST AS WELL AND HERE PEOPLE DO APPRECIATE QUANTITATIVE APPROACHES. I SPECIALIZE HERE IN COMMUNITY-BASED PARTICIPATORY RESEARCH, ADAPTING DEVELOPING AND EVALUATING INTERVENTIONS, INSTITUTIONAL AND QUALITATIVE RESEARCH METHODS. I DIDN'T HAVE ANY BACKGROUND IN MIXED METHODS RESEARCH UNTIL I TOOK ON A POST-DOC POSITION AT THE UNIVERSITY OF NEW MEXICO WITH HOWARD WADE SKIN WHERE I WAS INTRODUCED TO A LARGE SCALE STUDY OF MEDICAID MANAGED CARE REFORM AND I WAS PART OF A LARGE ETHNOGRAPHIC TEAM, I NEVER WORKED IN THAT COLLABORATIVE ENVIRONMENT BEFORE AN THIS WAS MY FIRST INTERACTION WITH QUANTITATIVE FOLKSCH MORE RECENTLY I HAVE BEEN INVOLVED IN QUANTITATIVE STUDIES OF WOMEN, COMMUNITY AMONG WOMEN LEAVING PRISON AND INTERVENTIONS TAWRGTING RURAL AND GENDER MINORITIES. IT WAS ORIGINALLY ANNOUNCED IN OCTOBER 2003, CAME ONLINE IN JULY 2005, BASICALLY THE STATE IMPLEMENTED A GIANT MANAGED CARE CARVE OUT FOR ALL PUBLICLY FUNDED MENTAL HEALTH AND SUBSTANCE USE TREATMENT SERVICES SO THIS INCLUDED A CARVE OUT FOR MEDICAID DOLLARS AND EVERYTHING ELSE. BASICALLY ALL THE MONEY THAT 15 STATE AGENCIES AN DEPARTMENTS HAD TO FUND BEHAVIORAL HEALTH SERVICES, WERE PULLED TOGETHER. THESE AGENCIES CREATED A PURCHASE COLLABORATIVE WHICH CONTRACTED WITH A WITH A SINGLE FOR PROFIT CORPORATION TO OVERSEE THE DELIVERY OF SERVICES LOCALLY. THERE WERE SEVERAL GOALS ASSOCIATED WITH THIS REFORM, THE REFORM WAS VERY AMBITIOUS IN SCOPE. I'M ONLY FOCUSING ON SELECT GOALS HERE THAT PERTAIN TO THE ANALYSES I'M GOING TO SHARE WITH YOU. GOALS INCLUDED MAXIMIZING ACCESS TO CULTURALLY COMPETENT EVIDENCE BASED CARE ENHANCING SERVICE QUALITY AN INCORPORATING SERVICE USER AND FAMILY VOICE INTO SYSTEM OPERATIONS, AND REDUCING BUREAUCRACY AND COST FOR SERVICE PROVIDERS. OUR STUDY FUNDED BY THE NATIONAL INSTITUTE OF MENTAL HEALTH WAS TO DESIGN TO INVESTIGATE THE PROCESS OF INTRODUCING THIS REALLY LARGE SCALE MASSIVE REFORM LOCAL LEVEL FOCUSING ON IMPACT ON BEHAVIORAL HEALTH SAFETY NET INSTITUTIONS IN RURAL VERSUS URBAN REGIONS AND BY SAFETY NET INSTITUTION I'M REFERRING TO PROVIDER ORGANIZATIONS THAT HAVE TRADITIONALLY CARED FOR PEOPLE ON MEDICAID AND OTHER LOW INCOME POPULATIONS. AND WE WERE ALSO INTERESTED IN THE DEGREE TO WHICH THE REFORM RESULTED IN IMPROVED ACCESS AND QUALITY OF SERVICE FOR PLEA INCOME ADULTS WITH SERIOUS MENTAL ILLNESS. WE WERE FOCUSING ON PEOPLE OF DIFFERENT ETHNICITIES INCLUDING LATINO OR HISPANIC, NATIVE AMERICAN, WHITE AND ANGLO. FOR THOSE OF YOU UNFAMILIAR WITH OUR STATE, IT DOES REPRESENT A CHALLENGING CONTEXT WHICH TO PLAN FOR AND DELIVER BEHAVIORAL HEALTHCARE. WE'RE SPARSELY POPULATED WITH A LITTLE OVER 2 MILLION PEOPLE SPREAD ACROSS 121,000-MILES. WE ARE A MAJORITY MY NRTY STATE MEANING ETHNIC MINORITIES MAKE UP A BULB OF. LATION, POOR STATE WE RANKED 43RD IN PERSONAL INCOME PER CAPITA, FIFTH IN POVERTY LEVEL AND FIFTH HEALTH INSURANCE, 32 COUNTIES OUT OF 33 ARE DESIGNATED MENTAL HEALTH PROFESSIONAL SHORTAGE AREAS AND WE TOP THE CHARTS IN TERMS OF ALCOHOL AND DRUG INDUCED DEATH RATES AND UP THERE WITH SUICIDE AND HOMICIDE. ALL TOP TEN. OUR STUDY OF BEHAVIORAL HEALTH REFORM IS INTERDISCIPLINARY. IT IS LED BY AN ANTHROPOLL GIST UNUSUAL FOR THIS TYPE OF WORK. ECONOMICS EPIDEMIOLOGY, ORGANIZATIONAL PSYCHOLOGY AND SOCIOLOGY AND ONE OF MY COLLABORATOR, GREG, YOU HEARD FROM EARLIER. THIS WORK IS INFLUENCED BY THE SOCIAL CONSTRUCTIVIST TRADITION IN EVALUATION RESEARCH WHERE THERE'S RECOGNITION OF MULTIPLE TRUTHS, THIS TRADITION ALLOWS CRITICAL INQUIRY TO UNFOLD OVER TIME, ITERATIVE PROCESSES OF DATA COLLECTION AND ANALYSIS, DIVERSE PROSPECTIVES, PROVIDES A RICH DESCRIPTION AND VALUES CONSENSUS AS WELL AS CONTESTATION ARRIVING AT CONCLUSIONS. OUR DESIGN FOR USING A MIXED METHODS APPROACH WAS TO PROVIDE A MORE ROBUST MULTI-FACETTED UNDERSTANDING OF THE REFORMS EFFECTS ON SAFETY NET INSTITUTIONS PROVIDERS AN SERVICE USERS. YOU COULDN'T JUST DO QUANTITATIVE RESEARCH ALONE, THIS TYPE OF RESEARCH PRESENTS SPECIAL CHALLENGES TO RURAL STATES WHERE AGENCY, TARGET POPULATIONS ARE DISPERSED, HARD TO E RECRUIT, OFTEN DIFFICULT TO GENERATE SAMPLES LARGE ENOUGH TO DEMONSTRATE TRENDS AND COMMONLY USED MEASURES ARE NOT APPROPRIATE FOR LOCAL POPULATION S. THESE METHODS DID NOT LEAD TO PRECISE ESTIMATES OF A POPULATION EXPERIENCE BUT AS MENTIONED EARLIER, THEY CAN HELP DEFINE, DEVELOP AND REFINE QUANTITATIVE TOOLS, CONTEXTUALIZE AND INTERPRET QUANTITATIVE FINDINGS AND SHED LINE ON IMPLEMENTATION BARRIERS AN INTENDED AND UNINTENDED CONSEQUENCES OF REFORM. THE STUDY IS MULTI-LEVEL. WE TARGETED SIX COUNTIES, THREE COUNTIES ARE URBAN AREAS, THREE PREDOMINANTLY RURAL. AND THEY WERE PAIRED UP ONE SET OF COUNTIES HAD A LARGE NATIVE HISPANIC POPULATION, ANOTHER HAD A LARGE MEXICAN AMERICAN POPULATION AND ANOTHER PAIRING HAD A LARGE NATIVE AMERICAN POPULATION. WE STARTEDDED OUR WORK IN SAFETY NET INSTITUTION WHERE IS WE ENGAGED IN ETHNOGRAPHIC OBSERVATIONS OF ADMINISTRATIVE AND CLINICAL PRACTICE AND WE ALSO INTERVIEWED ALL THE STAFF IN THESE INSTITUTIONS WHO WORKED WITH PEOPLE WITH SERIOUS MENTAL ILLNESS. WE STARTED OUR STUDY NINE MONTHS AFTER IMPLEMENTATION OF THE REFORM AND WE FOLLOWED UP WITH THESE FOLKS 18 AND 36 MONTHS LATER. WE ALSO CONDUCTED DOCUMENT REVIEW, COMPILING AN INVENTORYING ALL TEXT WE COULD GET OUR HANDS ON RELATED TO THE REFORM. ADULTS WITH SERIOUS MENTAL ILLNESS AS WELL AS FAMILY AND FRIENDS. WE WERE INTERESTED HOW THEY NAVIGATE IN THE HEALTHCARE SYSTEM AND WHETHER OTHER RESOURCES DID THEY USE THE GET MENTAL HEALTHCARE NEEDS MET. WE CONDUCTED TWO STATEWIDE SURVEYS OF CLINICAL AND CHIEF MEDICAL OFFICERS OF MENTAL HEALTH AGENCIES. WE CONDUCTED SEMISTRUCTURED INTERVIEWS WHICH STATE POLICY MAKERS, MANAGED CARE PERSONNEL AND COMMUNITY LEADERS, AT TWO TIME POINTS AS WELL. WE DIDN'T PLAN TO DO THIS, WE UNDERTOOK A SUPPLEMENTAL ETHNOGRAPHIC STUDY OF COMMUNITY INPUT PROCESSES AN PUBLIC FORUMS PERTINENT TO REFORM. WE DID THIS IN ALL FIVE YEARS OF OUR STUDY AND WE CONTINUED TO DO IT TO THIS DAY. THIS IS JUST ONE OF THOSE REALLY INTERESTING AREAS. FOR QUALITATIVE DATA ANALYSIS MOST INTERVIEW DATA FOR MOST INTERVIEW DATA WE ENGAGED IN QUESTION LEVEL CODING, FOR OBSERVATION AN DOCUMENT REVIEW, WE ENGAGED IN OPEN END FOCUS CODING, OUR CODING WAS FACILITATED BY IN VIVO SOFTWARE AND WE WEPT THROUGH SEVERAL ADDITIONS OF IN VIVO SOFTWARE CONLEY EVOLVING. THEY RANGE FROM T TEST TO MULTI-LEVEL REGRESSION NAIL SEIZE, EQUATION MODELING DEPENDING ON RESEARCH QUESTIONS AND DATA SETS WE WERE DRAWING FROM. MIXED METHODS ANALYTIC CONVENTIONS WERE USED FOR COMPLIMENTARITY AND EXPANSION. THE TWO‡gx g PROJECTS ISLE I'M GOING TO TALK ABOUT TODAY INVOLVE A QUAL PLUS QUAN DESIGN. IN THE QUALITATIVE DESIGN THE QUALITATIVE WORK DROVE THE FRAME FOR RESEARCH, WE IMPLEMENTED OUR -- WE COLLECTED QUALITATIVE AND QUANTITATIVE DATA SIMULTANEOUSLY. THE POINT OF THIS APPROACH WAS PRIMARILY FOR EXPLORATION AND HYPOTHESIS GENERATION WITH QUAN PLUS QUAL WE DREW UPON OUR SURVEY DATA AND OUR QUALITATIVE DATA GIVING EQUAL WEIGHT TO BOTH TYPES OF DATA. OUR FIRST ANALYSIS ENTITLED EFFECTS OF BEHAVIORAL HEALTH REFORM ON SAFETY NET INSTITUTIONS, A MIXED METHOD ASSESSMENT IN RURAL STATE. WE HAD TWO RESEARCH QUESTIONS, HOW HAS REFORM IMPLEMENTATION IMPACTED ORGANIZATIONAL DYNAMICS AND SAFETY NET PERSONNEL. AND HAVE RURAL SAFETY NET PERSONNEL PERFORMED DIFFERENTLY THAN URBAN PERSONNEL. THIS ANALYSIS IS BASED ON ALL THREE WAVES OF FIELD WORK IN THE SAFETY NET. WE HAD 177 RURAL PARTICIPANTS, 148 URBAN PARTICIPANTS. WE CONDUCTED OVER 1600 HOURS OF OBSERVATION WITHIN THESE SETTINGS. WE LOOKED SPECIFICALLY HERE AT OUR QUANTITATIVE ANALYSES OF JOB SATISFACTION AND ORGANIZATIONAL COMMITMENT MEASURES. WE ALSO INCORPORATED IN TERMS OF EXPANSION POST FIELD WORK QUALITATIVE INTERVIEWS WE CARRIED WITH SAFETY NET ADMINISTRATORS -- ADMINISTRATORS AN CLINICAL DIRECTORS AFTER THE FIRST MANAGED CARE COMPANY CONTRACTED TO OVERSEAS SERVICES REPLACEDDED BY A NATIONAL EXETERTOR. WE WERE INTERESTED IN TRANSITION ISSUES THAT WERE EMERGING. IN TERMS OF FIRST QUESTION HOW IMPLEMENTATION IS IMPACTED ORGANIZATIONAL DYNAMICS AN SAFETY NECESSARY PERSONNEL. QUALITATIVE DATA INDICATED SAFETY NET PERSONNEL FACED INCREASED STRESS RELATED TO FAULTY UNTRIED INFORMATION TECHNOLOGY SYSTEMS OR IT SYSTEMS. AND BURDENSOME PROCESSES TO ENROLL CLIENTS, PROCURE AUTHORIZATION AND SUBMIT CLAIMS AND THESE PROBLEMS WERE REALLY BIG AT THE GET GO AND THAT FIRST YEAR OF IMPLEMENTATION BUT THEY HAVE CONTINUED TO LINGER AND THEY ACTUALLY REEMERGED WHEN THE NEW MANAGED CARE COMPANY CAME ON SCENE. DELAYED PAYMENTS AND UNREIMBURSED CLAIMS LED TO FINANCIAL PROBLEMS FOR SAFETY NET INSTITUTIONS WHICH THEN HEIGHTENED WORKER CONCERNS OVER AGENCY STABILITY AND JOB SECURITY. AND I HAVE TONS AN TONS OF QUOTE TO LIKE ILLUSTRATE THESE POINTS BUT WE'D BE HERE A VERY LONG TIME. TRANSITION TO A NEW STATEWIDE MANAGED CARE CONTRACTOR INTENSIFIED THE FINANCIAL PROBLEMS EXPERIENCED BY SAFETY NETS WHEN NEW MANAGED CARE COMPANY CAME ON THE SCENE, NO ONE WAS PAID THE FIRST FOUR MONTHS FOR THE PRESENCE IN THIS STATE. WE FOUND EVIDENCE OF INITIAL DECLINES IN JOB SATISFACTION AND ORGANIZATIONAL COMMITMENT AMONG SAFETY NET PERSONNEL AS REFORM WAS IMPLEMENTED. AFTER A WHILE THESE DECREASES IN SATISFACTION AND EQUIPMENT DID DIMINISH BY THIRD WAVE. DID RURAL SAFETY NET PERSONNEL EXPERIENCE THE REFORM DIFFERENTLY FROM URBAN SAFETY NET PERSONNEL? YES. RURAL SAFETY NET PERSONNEL EXPERIENCED GREATER PROBLEMS UNDER THE REFORM. OHING TO LIMITED WORK FORCE CAPACITY. THEIR OWN IT INFRASTRUCTURE AS WELL AS LACK OF PRIOR EXPERIENCE WITH MANAGED CARE. AGAIN, NOT EVERYBODY HAD HAD PRIOR EXPERIENCE WITH MANAGED CARE IF THEY WESTERN PARTICIPATING IN THE MEDICAID SYSTEM, FOR EXAMPLE, PRIOR TO THIS REFORM. IN TERMS OF QUANTITATIVE FINDINGS RURAL SAFETY NET PERSONNEL EXPERIENCED SIGNIFICANT DECLINES IN JOB SATISFACTION AND ORGANIZATIONAL COMMITMENT WHEREAS URBAN PERSONNEL DEMONSTRATED LITTLE CHANGE. WHERE IMPLICATIONS OF THIS. OUR MIXED METHODS APPROACH DEMONSTRATED CONVERGENCE BETWEEN TWO MAJOR SETS FOR OUR STUDY. THE RURAL PERSONNEL INTENDED TO REPORT WORSES IN CONDITIONS BETWEEN THE FIRST AN SECOND WAVES BUT THEN SOME IMPROVEMENTS IN WORK ATTITUDE. THE RESULT OF RESILIENCE AMONG SAFETY NET PROVIDERS FOR THE NEW CONDITIONS. HOWEVER THE TRANSITION TO A NEW MANAGED CARE CONTRACTOR WAS NOT CAPTURED IN THE QUANTITATIVE WORK. WE BELIEVE THE SECOND SHIFT MAY HAVE UNDERMINED STAFF RESILIENCY AND ADAPTION -- ADAPTATION AS INDICATED IN OUR INTERVIEWS WITH FOLKS AFTER WE HAD COLLECTED THE QUANTITATIVE DATA AND IS EVIDENCED BY CLOSURE OF TWO SAFETY NETS AFTER WE FINISHED THE DATA COLLECTION SOON AFTER. FOR OUR SECOND ANALYSIS WE LOOKED AT„i?; CULTURALLY COMPETENT SERVICES. WE HAD TWO QUESTIONS, DID THE REFORM ACHIEVE THE DESIRED GOAL OF INCREASING AVAILABILITY OF LANGUAGE ACCESS SERVICES AND ORGANIZATIONAL SUPPORTS FOR CULTURALLY COMPETENT SERVICES AND WHAT FACTORS INFLUENCED THE AVAILABILITY OF CULTURALLY COMPETENT SERVICES DURING THE REFORM. HERE WE ANALYZE FIRST AND SECOND WAVE DATA. THIS LARGELY PERTAINED TO THE FIRST 36 MONTHS OF THE REFORM. WE STARTED WITH OUR STATEWIDE SURVEY OF MENTAL HEALTH AGENCIES LOOKING AT PREVALENCE OF LANGUAGE ACCESS SERVICES AND ORGANIZATIONAL SUPPORTS RELATED TO CULTURALLY COMPETENCE SERVICES AND SUPPORTS INCLUDED TRAINING IN CULTURAL COMPETENCY, SELF-ASSESSMENT, INTERNAL INVENTORIES OF AGENCY WIDE POLICIES AN PRACTICES RELATED TO CULTURALLY COMPETENT CARE AND WHETHER OR NOT AGENCIES WERE COLLECTING CLIENT LEVEL DATA TO RAISE ETHNICITY AND LANGUAGE. THESE WERE BASED ON NATIONAL STAN DOORDZ FOR CULTURALLY AN LINGUISTICALLY SERVICES. WE UNDERTOOK REVIEW OF STATE DOCUMENTS ABOUT THE REFORM, WE LOOKED SPECIFICALLY AT CONTRACT BETWEEN THE STATE AND THE MANAGED CARE COMPANY. WHAT BOTH PARTIES AGREE TO DO IN RELATION TO CULTURAL COMPETENCY. WE FOCUSED ON INTERNAL SERVICE DELIVERY REPORTS AND SERVICE USER DEMOGRAPHICS. WE ALSO DREW UPON OBSERVATIONS AN SAFETY NET SETTINGS AND IN DEPTH INTERVIEWS WITH PROVIDERS AND STAFF WHERE WE DID ASK VERY TARGETED QUESTIONS ABOUT KNOWLEDGE PERTAINING TO CULTURALLY COMPETENT CARE, THE PROVISION OF SUCH SERVICES AND TRAINING. DID THE REFORM INCREASE AVAILABILITY OF LANGUAGE ACCESS SERVICES AND ORGANIZATIONAL SUPPORT? IN TERMS QUANTITATIVE DATA, THE ANSWER IS MINIMALLY. THE SURVEY AS WELL AS THE DOCUMENT REVIEW INDICATED THERE WAS LITTLE TO NO INCREASE IN LANGUAGE ACCESS SERVICES OR ORGANIZATIONAL SUPPORTS. WE DID COME ACROSS A GREATER BUT NON-STATISTICALLY PERCENT OF AGENCIES EMPLOYING BILINGUAL STAFF IN OFFERING INTERPRETATION SERVICES THREA YEARS INTO THE REFORM. MOST CULTURAL COMPETENCY EFFORTS UNDER THE REFORM EMPHASIZE NATIVE AMERICAN PEOPLE VERSUS LATINOS OR ANY OTHER ETHNIC POPULATION. THERE WAS A SIGNIFICANT AVERAGE INCREASE IN AGENCY ASSESSMENT HOW WELL THE MANAGED CARE CONTRACTOR WORKED TO IMPROVE CULTURALLY COMPETENCE CARE BUT THE BEFORE WAS BARELY RAISED FROM PAR TO FAIR BY THE THIRD YEAR OF THE REFORM. IN TERMS OF QUALITATIVE DATA ANSWERING THIS QUESTION, WE -- THERE WAS A SLIGHT INCREASE IN PROVIDER UNDERSTANDINGS RELATED TO CULTURALLY COMPETENT CARE. WE OBSERVED AN INCREASE OF TRAINING THREE YEARS INTO THE REFORM BUT SOURCES OTHER THAN THE STATE GOVERNMENT OR THE MANAGED CARE CONTRACTOR PAID FOR THIS TRAINING. >> FACTORS INFLUENCED THE AVAILABILITY OF THESE SERVICES, CULTURALLY COMPETENT SERVICES. OUR ANALYSIS OF OBSERVATION AN INTERVIEW DATA SUGGEST SIX BROAD EXPLANATIONS. FIRST OFF, THERE WEREN'T ENOUGH RESOURCES PROVIDED TECHNICAL ASSISTANCE PROVIDED AND TRAINING FOR CULTURALLY COMPETENCE CARE. THERE WAS A HUGE INCREASE IN ADMINISTRATIVE WORK UNDER THE REFORM, WHEREAS YOU WOULD EXPECT WITH CONSOLIDATION OF FUNDING ACROSS THESE DIFFERENT STREAMS THERE MIGHT BE REDUCED BIEWR RO CRA SI, IT WAS MAGNIFIED AN PEOPLE WERE SPENDING MORE TIME FILLING OUT THEIR PAPERWORK THAN WITH PATIENTS AND REALLY CULTURALLY COMPETENCE CARE WAS NOT A HUGE PRIORITY. WE FOUND SOME SPECIFIC PRACTICES WERE DISADVANTAGED IN A MANAGED CARE ENVIRONMENT SPECIFICALLY THOSE RELATED TO NATIVE AMERICAN HEALING. WITHIN THE AGENCIES THERE WERE LIMITED VIEWS OF IB CAPACITY FOR LINK QISICLY APPROPRIATE SERVICES. OFTENTIMES WHEN YOU TALK TO PROVIDERS ABOUT CULTURALLY COMPETENT SERVICES THEY WOULD AUTOMATICALLY CONFLATE WITH LANGUAGE, APPROPRIATE SERVICES. THEY OFTEN OVERESTIMATED THEIR ABILITY TO PROVIDE LANGUAGE APPROPRIATE SERVICES AS WE DOCUMENTED NUMEROUS INSTANCES IN WHICH RECEPTIONISTS AND FAMILY MEMBERS WERE BEING CALLED UPON TO DO INTERPRETATION. LACK OF APPRECIATION AMONG -- THEY KNOW EVERYTHING OR THEY'RE FROM THE COMMUNITIES SO THEY GET IT, AND THAT WAS KIND OF THE LEVEL OF DISCUSSION, WE TRY TO DIG DEEPER. ULTIMATELY THE STATE GOVERNMENT, MANAGED CARE CONTRACTOR AND THE SAFETY NET INSTITUTIONS THEMSELVES FAILED TO CHAMPION CULTURALLY COMPETENT CARE THOUGH THIS IS ONE OF THE WIDELY TOUTED GOALS OF THE REFORM. IMPLICATIONS HERE ARE MIXED METHOD APPROACH LARGELY DEMONSTRATED COMPLIMENTARITY BETWEEN THE TWO SETS OF DATA COLLECTED FOR STUDY. THE SURVEYS WERE USEFUL FOR DOCUMENTING STATEWIDE DEFICIENCIES IN THE PROVISION OF CULTURALLY COMPETENT CARE ESPECIALLY IN TERMS OF CLASS STANDARDS. THE ETHNOGRAPHIC WORK WAS USEFUL FOR ILLUMINATING POSSIBLE REASONS AND BOTH DATA SETS STRENGTHENED STUDY RECOMMENDATIONS FOCUSED ON ENHANCING PROVIDER KNOWLEDGE AND SKILLS AN CULTIVATING EXTERNAL OR POLICY ENVIRONMENT AND INTERNAL OR AGENCY ENVIRONMENTS TO FACILITATE CULTURALLY COMPETENT CARE. I'M GOING TO SKIP OVER IN PROGRESS ANALYSES BUT WE'RE DOING MORE AND THESE ARE JUST TWO AND WE HAVE MORE ANALYSES UNDERWAY JUST FINDING TIME. WHY DO A MIXEDED METHODS APPROACH? THIS HAS BEEN COVERED BUT TO ENHANCE THE QUALITY AND CREDIBILITY OF RESULTS. AREAS OF KORVER YENS INCREASE CONFIDENCE IN FINDINGS, DI VER DIVERGENCE ARE IMPORTANT. SO WHEN THE DATA DON'T DISAGREE, DON'T AGREE, YOU'RE NOT GETTING THE SAME CONCLUSIONS, THAT ACTUALLY PROMPTS MORE COMPLEX UNDERSTANDING OF WHAT'S GOING ON. ULTIMATELY USING BOTH TYPES OF APPROACHES PROVIDES A FIRMER FOUNDATION FOR CONCLUSIONS AN RECOMMENDATIONS. CHALLENGES TO MIXED METHODS RESEARCH, AT LEAST FROM A QUALITATIVE PERSPECTIVE, THE ITERATIVE NATURE OF THE FIELD WORK WHICH I LOVE BUT AT THE SAME TIME THE EXPLORATION NEVER ENDS. SO IN NEW MEXICO WE'RE ON TO A BRAND NEW REFORM. YOU GET THE BUG AND YOU WANT TO STUDY THESE REFORMS. THERE'S ALWAYS NEW RESEARCH QUESTIONS TO PURSUE BASED ON EVENTS ON THE GROUND. MONEY MONEY MONEY IS A HUGE CONSIDERATION, TEAM BASEDETH NOTHINGNOGRAPHY IS NOT CHEEP. WHEN COLLECTING OBSERVE RATION DATA AND PREPARING AN ANALYZING IT, IT'S EXPENSIVE PROSPECT. PEOPLE POWER ISSUES. INITIALLY IT WAS FINDINGETHNOGRAPHERS WHO DIDN'T BALLK, THEY LAUGHED WHEN I FIRST PRESENTED THE MEASURES AND SAID NO WAY IS A PROVIDER GOING TO FILL OUT THIS TOOL HERE. ULTIMATELY, THEY CAME TO REALLY APPRECIATE THOSE MEASURES WHEN THEY SAW THAT THERE WAS A LOT OF THE RESULTING DATA REALLY COMPLIMENTED WHAT THEY WERE FINDING IN THE FIELD TALKING TO FOLK. FOR ME IT'S DIFFICULT TO FIND STATISTICIANS THAT I CAN COMMUNICATE EFFECTIVELY WITH. I HAVE SOME NOW. BUT IT'S BEEN A LEARNING PROCESS. I'M JUST NOT LINGUISTICALLY PROFICIENT WITH THAT PARTICULAR POPULATION. ON THE PLUS SIDE THERE IS AMPLE OPPORTUNITY TO OVERCOME DISCIPLINARY BASED ETHNOSEN TRISM AND LEARN FROM ONE ANOTHER. LIMB TAIXES OF HEALTH SERVICES AND PUBLIC HEALTH JOURNALS ARE ALSO A CHALLENGE WHICH I'M SURPRISED NOBODY TALKED ABOUT YET BUT IT'S EXCEEDINGLY DIFFICULT TO TELL A NUANCE STORY IN UNDER 4500 WORDS ESPECIALLY WHEN YOUR BIBLIOGRAPHY IS COUNTED IN THAT. WHAT IS COOL ABOUT MIXED METHODS RESEARCH WOULD BE THE PRODUCTS. SO FOR EXAMPLE, WE HAVE PRODUCTS THAT FOCUS ON IMPACT OF THE REFORM ON NATIVE AMERICANS, ON TURN OVER, WHAT PROVIDERS THINK ABOUT RECOVERY ORIENTED SERVICES, WE HAVE STUFF ON SOCIAL SUPPORT, WE HAVE SO MUCH IN SO MANY DIFFERENT VANTAGE POINTS AND IF WE DIDN'T DO THE MIXED METHODS APPROACH, I THINK OUR STUDY WOULD HAVE LACKED. WE TRY TO TAILOR OUR PRESENTATION APPROACH TO THE AUDIENCE WITH HEALTH AND MENTAL HEALTH SERVICES RESEARCHERS, I HATE TO SAY IT BUT WE DO ENGAGE IN FORMULAIC DESCRIPTIONS OF THE THEME, HERE IS OUR THEME, HERE IS WHAT IT MEANS, HERE IS THE QUOTE ILLUSTRATING THE THEME. IT'S A WAY TO WRITE BUT IT'S NOT THE PREFERRED KIND OF ANTHROPOLOGICAL WAY OF WRITING. FOR ANTHROPOLL GISTS AND OTHER SOCIAL SCIENTISTS WE LIKE TO NARRATE STORIES. WE INCLUDE OUR OWN VOICE IN THE STORIES WHICH ARE OFTEN WRITTEN OUT WHEN PRESENTING BEFORE HEALTH AND MENTAL HEALTH SERVICES FOLKS. AND WE PAID GREATER ATTENTION TO ISSUES OF POWER AND CRITICAL THEORY SO WHAT I'M PRESENTING FOR ANTHROPOLL GISTS OR WRITING FOR ANTHROPOLL GISTS I CAN TALK ABOUT NEOLIBERALISM, FOR OTHER AUDIENCES IF I BRING IT UP I'M ACCUSED OF BEING BIASED BUT YET THAT'S AN ACCEPTED THEORETICAL PRAIM WORK IN ANTHROPOLOGY. FOR POLICY MAKERS AND LEGISLATORS WE OFTEN FOCUS ON LESSONS LEARNED AND RECOMMENDATIONS. WE ACTUALLY HAD A HUGE COMMUNITY ENGAGEMENT PORTION TO THE STUDY WHICH I DIDN'T GOIN TO BUT WE DO A MIX OF ALL THE ABOVE WHEN TALKING TO MEMBERS OF OUR COMMUNITY ADVISORY BOAR, LOCAL HEALTH COUNCILS AN NATIVE AMERICAN INSTITUTIONAL REVIEW BOARDS THAT HAD TO APPROVE THIS RESEARCH. HERE WE TEND TO USE POWERPOINT, HOWEVER, WHICH IS NOT ALWAYS THE BEST THING. AND THAT IN A NUTSHELL IS OUR GIANT STUDY OF BEHAVIORAL HEALTH REFORM IN NEW MEXICO. [APPLAUSE] >> SO GOOD AFTERNOON. IT'S WONDERFUL TO BE HERE AND MY COLLEAGUES HAVE LAID OUT MANY OF THE CONCEPTS THAT I WILL TALK ABOUT. SO WHAT I WOULD LIKE TO DO TODAY IS TO SHARE WITH YOU A LITTLE BIT ABOUT MY EXPERIENCE LOOKING AT VIOLENCE, DUE TO STORIES OF YOUNG AFRICAN AMERICAN MEN. I CAME TO THIS WORK AS A PHYSICIAN AND I HAVE BEEN TRAINED IN CLINICAL EPIDEMIOLOGY. YET DESPITE A REASONABLY GOOD UNIVERSITY MEDICAL SCHOOL AND PUBLIC HEALTH SCHOOL, I WAS NEVER EXPOSED TO THE PHILOSOPHY OF SCIENCE THAT WOULD HAVE TOLD ME THAT THERE WAS ANOTHER WAY OF KNOWING OTHER THAN STATISTICAL INFERENCE. POSSIBLE I DIDN'T TAKE THE RIGHT CLASSES BUT IT WASN'T PART OF THE DISCUSSION CERTAINLY IN MEDICAL SCHOOL. SO IN 1990 IN BOSTON AS IN MUCH OF THE COUNTRY, THERE WAS AN EPIDEMIC OF VIOLENCE. AND I AS I WALK THROUGH THE HOSPITAL AND DEALT WITH PATIENTS, ACTIVELY SAW THAT THE PERSPECTIVE OF THIS INJURED INDIVIDUAL WAS MISSING FROM THE DISCUSSION. IN FACT, THERE WAS A TACIT ASUDGES THAT ANY PATIENT WHO WAS A YOUNG AFRICAN AMERICAN MAN WHO CAME IN WHO WAS A VICTIM OF VIOLENCE HAD SOMEHOW BROUGHT THIS ON HIMSELF. IT WAS I THINK A MATTER OF OF THESE VOICES BEING ACTIVELY SUPPRESSED. SO RACISM STIGMA ALL MADE IT UNACCEPTABLE OR MAYBE VOICES UNAVAILABLE TO US AS PROVIDERS. IN MANY WAYS THE STORIES AS I SAID WERE SUPPRESSED. PROVIDERS WHEN YOU TALK WOULD ACTIVELY SAY I TELL PATIENTS DO NOT TELL ME THE STORY STORY OF WHAT HAPPENED TO YOU. THE REASON PROVIDERS HAD CONCERNS ABOUT THEIR OWN PSYCHOLOGICAL SAFETY BUT ALSO WHETHER THEY'RE INVOLVED IN COURTS. AND THERE WAS THIS LOOMING QUESTION ABOUT WHAT REALLY HAPPENED. SO PROVIDERS WOULD OFTEN SAY I DON'T ASK QUESTIONS BECAUSE THEY ALL LIE. SO HOW DO WE DEAL WITH THAT PROBLEM WHEN STORIES FOR ALL OF US HELP US MAKE SENSE OF THE WORLD. AND THAT SENSE MAKING IS CRITICAL FOR OUR HEALING AN RECOVERY PARTICULARLY FROM TRAUMA. NOW THE OTHER THING THAT DROVE THE STUDY WAS THE FACT THAT WE KNOW FROM PAST STUDYIES FOR AN INDIVIDUAL WHO HAD A PENETRATING INJURY, A GUNSHOT WOUND OR STAB WOUND, AT FIVE YEARS, 45% OF THOSE PEOPLE HAVE BEEN SHOT OR STABBED AGAIN AND 20% ARE DEAD. SO WE HAD THESE DATA. WE DIDN'T REALLY AT BOSTON CITY HOSPITAL NEED THAT STUDY TO TELL US THAT WE SAW PATIENTS AGAIN AND AGAIN. WE KNEW THAT THERE WAS A WAY WHICH FOLKS CAME BACK. BUT AGAIN, IT WAS ASSUMED FROM OUR PERSPECTIVE THAT THEY BROUGHT THAT ON THEMSELVES. SO THE PURPOSE OF THE STUDY THAT I'LL TALK TO YOU ABOUT FUNDING BY THE NATIONAL INSTITUTES OF MENTAL HEALTH WAS TO EXAMINE THE EXPERIENCE OF VIOLENT INJURY FROM THE PERSPECTIVE OF THE VICTIM, IN THIS CASE YOUNG AFRICAN AMERICAN MALE VICTIMS OF VIOLENCE, USING LARGELY OPEN-ENDED QUALITATIVE INTERVIEWS PAIRED WITH SOME STANDARDIZED ASSESSMENTS OF DEPRESSION, EXPOSURE TO VIOLENCE, PTSD. I WON'T GO INTO MUCH DETAIL HERE BUT WE RECRUITED 18 TO 30-YEAR-OLD YOUNG AFRICAN AMERICAN MEN WHO HAD BEEN SHOT STABBED OR ASSAULTED IN THE HOSPITAL, WE RECRUITED THEM IN THE HOSPITAL, WORKED TO INTERVIEW THEM WITHIN TWO WEEKS OF INITIAL INJURY AND FOLLOWED THEM ONE TO THREE MONTHS. THEY ARE DIFFICULT TO FOLLOW-UP. THEY'RE DIFFICULT TO FIND OFTEN. AT THAT TIME IT WAS BEEPERS BUT CELL PHONES OFTEN CHANGED NUMBERS, PEOPLE MANY PEOPLE RL SEMIHOMELESS L OR NEAR HOMELESS AND DIFFICULT TO FIND. WE USE SEMISTRUCTURED INTERVIEWEDDED BUT MOSTLY OPEN ENDED QUESTIONS AND MOSTLY BEGINNING WITH THE QUERY CAN YOU TELL ME WHAT HAPPENED. THE INTERVIEWS WERE TAPED AND TRANSCRIBED AND OF COURSE LOADED IN TO IN VIVO THAT WONDERFUL TOOL ALL OF US HAVE THIS AMAZING LOVE HATE RELATIONSHIP WITH. IT'S AMAZING FOR MANAGING EVEN THE AMOUNTS OF DATA THAT I'LL TELL YOU ABOUT. SO IN THIS STUDY THERE ARE 51 PARTICIPANTS, HALF OF THEM INTERVIEWED AT A SECOND INTERVIEW, ABOUT 23 YEARS OLD. THEY WERE A MIX OF SELF-IDENTIFIED RACE ETHNICITIES, 50% HAD BEEN SHOT. WE RECORDED AS DIGITAL INTERVIEWS, THE QUANTITATIVE DATA WERE CHECKED AT EACH ENCOUNTER. THE DATA ANALYSIS TOOK TWO APPROACHES ONE WAS A CONTENT BASED APPROACH USING GROUNDED THEORY METHODS, THE OTHER WAS USING NARRATIVE ANALYSIS, A STRUCTURAL ANALYSIS WHICH I'LL TALK TO YOU ABOUT A LITTLE BIT MORE. WE GATHERED THE QUALITATIVE DATA AND CONSISTENT WITH BEST PRACTICES WE ANALYZED THE INTERVIEWS AS WE COLLECTED THEM. AND THE INTERVIEW GUIDE CHANGED AS WE GOT INSIGHTS FROM THE EARLY INTERVIEWS. THE QUANTITATIVE DATA COLLECTED IN INTEGRATED AFTER WE HAD COMPLETED THE QUALITATIVE DATA ANALYSIS AND USED THOSE QUALITATIVE RESULTS TO STRATIFY OR LOOK AT GROUPS WITHIN, FOR EXAMPLE, THE ABILITY TO COMPARE THE NARRATIVES OF INDIVIDUALS WITH POST TRAUMATIC STRESS DISORDER TO THOSE WITHOUT. SO I WANT TO COME BACK FOR A MOMENT AND TALK THE PROCESS OF WHY WE THINK ABOUT STRUCTURAL ANALYSIS. WE USED A METHOD INTRODUCED TO ME BY A WONDERFULLY GENEROUS MENTOR NAMED ELLIOT MISHLER. A METHOD DEVELOPED BY LINGUIST JIM G., JIM G.'S WORK DISCLOSED REVEALED TO HIM THAT ONE CAN BY LISTENING TO TRANSCRIPTS AN CONVERSATION IDENTIFY PITCH GLIDES OR CHANGES IN -- THAT INDICATE WHAT HE CALLS IDEA UNITS, IDEA UNITS CAN BE PUT TOGETHER TO MAKE A LINE, LINES ARE GROUPED TOGETHER IN STANZAS. AND STANZAS ARE THE WAY REALLY THAT ALL OF US TALK IN HIS FRAMEWORK. HE WOULD SAY POETRY IS REALLY A FOSSILIZED OR RITUALIZED VERSION OF WHAT WE DO ALL THE TIME ANYWAY. SO APPROACH TAKES TEXT AN PARSES INTO THESE UNITS AND COMPOSES STANZAS AND BECOMES UNITS OF ANALYSIS. THIS IS HELPFUL FOR COUPLE OF REASONS, ONE WHEN ONE DOES A GROUNDED THEORY ANALYSIS, IN SOME WAYS IT MAYBE DIFFICULT TO SHOW YOUR WORK. WHEREAS IN A STRUCTURAL ANALYSIS IT'S EASIER, I'LL SHOW YOU THIS IN A MOMENT, TO SHOW YOUR WORK AND SHOW THE UNITS OF ANALYSIS THAT YOU'RE USING. HERE IS A TYPICAL TRANSCRIPT YOU MIGHT SEE, THIS IS A YOUNG MAN DESCRIBING -- I WON'T READ IT BUT DESCRIBING WALKING OUT OF A STORE SEEING SOME YOUNG GUYS NOT THINKING HE WAS IN DANGER, THEN FINDING HIMSELF IN THE MIDDLE OF A SHOOT OUT DESCRIBING WHAT HAPPENED AFTERWARDS. HERE I HAVE TAKEN THE SAME TEXT AND PARSED IT OUT INTO THESE STANZAS AND WHAT YOU SEE ARE -- IS CONTENT THAT IS HANGING TOGETHER HERE. SO IF WE ASSIGN THIS SET OF STANZAS IF WE TAKE IT TOGETHER, THIS IS REALLY A TYPICAL ORIENTATION OR SETTING DESCRIPTION. SO HE SAYS WELL I WAS WALKING THROUGH THE STORE AND I HAVE SEEN THESE TWO YOUNG BOYS STANDING OUT IN FRONT OF STORE, THEY DIDN'T SEEM TO BE VIOLENT. SO I WALKED INTO THE STORE, BOUGHT A JUICE, CAME OUT OF THE STORE, WALKED BY THE SAME TWO KIDS. HERE HE BEGINS TO TALK ABOUT HIS FIRST AWARENESS OF DANGER, I GET MAYBE THREE OR FOUR FEET AWAY FROM THE TWO KIDS, THAT'S WHEN I START HEARING GUN SHOTS. I SAID TO MYSELF, THE GUN SHOTS ARE REAL LOUD. I TURNED AND THERE WAS A GUNPOINTED AT ME. HE THEN TALKS ABOUT DEFENDING HIMSELF, I TRIED TO IDENTIFY MYSELF, WHOEVER THAT HE WAS TRYING TO SHOOT, I KNEW IT WASN'T ME SO I'M TRYING TO TELL HIM I'M NOT THE ONE. MOMENT OF INJURY. THAT'S WHEN I GOT SHOT. I GOT HIT ONCE IN THE STOMACH AND I WAS GRAZED IN THE ARM. TRYING TO ESCAPE. THE NEXT SEGMENT. THOUGHTS OF DEATH. HE SAID IT'S FEWMY WHILE I WAS LYINGN'T GROUND I WAS JUST LOOKING IN THE AIR LIKE WHAT AM I GOING TO DO? AM I GOING TO DIE? THEN I SAID TO MYSELF, IT'S NOT UP TO ME, IF IT'S MY TIME TO GO IT'S MY TIME TO GO. IT'S UP TO GOD. SO THAT CALMED ME DO YOU KNOW ACTUALLY. IT TURNS OUT IF YOU LOOK AT THESE STORIES, THERE ARE PARTS OF INJURY STORIES THAT OCCUR IN EACH STORY. IN EACH STORY BY DEFINITION THERE IS A MOMENT YOU REALIZE YOU WERE INJURED. THERE'S A MOMENT WHERE YOU MAY HAVE REALIZED YOU WERE IN DANGER. BUT IT ALSO TURNS OUT THAT THERE ARE DIFFERENT TYPES OF STORIES. THERE'S INNOCENT BYSTANDER STORY, CRIME VICTIM, ESCALATING ARGUMENT, MISTAKEN VICTIM, THAT IS I KNOW WHO THEY WERE TRYING TO GET, IT WASN'T ME BUT I WAS MISTAKEN FOR THAT PERSON TURNS OW THESE STORIES HAVE DIFFERENT PARTS OFTEN, THERE MAYBE PARTS OF STORIES THAT ARE MISSING FROM SOME OF THESE TYPES. SO AN INNOCENT BYSTANDER IF YOU'RE CAUGHT IN THE CROSS FIRE YOU MAY NOT EVER HAVE AWARENESS OF DANGER. WHEREAS IF ENGAGED IN CONFLICT YOU KNOW THE DANGER IS COMING. THAT MAYBE IMPORTANT BECAUSE THE NATURE OF THE STRESS OF INJURY HAS SOMETHING TO DO WITH WHETHER YOU'LL DEVELOP POST TRAUMATIC STRESS WE KNOW. SO THAT IS PART OF THE BOTH HOW STRUCTURAL ANALYSIS HELPS ONE MAKE ONE'S WAY THROUGH THE TEXT AND TO IDENTIFY THESE STORIES. BUT WE HAVE ALSO FOUND IT USEFUL TO APPLY TO NON-NARRATIVE TEXT AS A WAY OF HELPING THE CONTENT HOLD TOGETHER. I'LL PRESENT BRIEFLY THREE THEMES THAT CAME OUT OF THIS WORK. THE FIRST WAS THE NOTION OF RESPECT OR IN THE WORDS OF THE PARTICIPANTS THE MEANING OF BEING A SUCKER. NOW, I BEGAN THIS WORK AND I STARTED TO DO THE INTERVIEWS RELATIVELY NOVICE. IN THE FIFTH INTERVIEW A YOUNG MAN BEGAN TO USE THE WORD SUCKER. AND I KNEW THIS WORD FROM MY CHILDHOOD BUT HE WAS USING IT IN A WAY THAT I HADN'T HEARD BEFORE. SO AFTER I DRILLED DOWN A LITTLE BIT WITH HIM IN THE INTERVIEW, I THEN BEGAN TO ASK IT IN SUBSEQUENT INTERVIEWS. DOES THIS NOTION, YOU EVER HEARD OF BEING A SUCKER WHAT DOES THAT MEAN TO YOU? THIS IS THE DEFINITION THAT YOUNG PEOPLE OFFERED. A SUCKER IS A PERSON IF SOMEONE SAYS SOMETHING TO THEM OR DOES SOMETHING TO THEM THEY JUST SIT THERE AND TAKE IT AND DON'T RETALIATE. SO RELATIVELY WELL KNOWN AND SHARED IDEA, PROBABLY FAMILIAR TO MANY OF US HERE DON'T LET SOMEONE DO SOMETHING TO YOU OR EVERYBODY WILL DO THAT TO YOU. IN MANY WAYS THE STRUCTURING OF TEXT HELPS NOT ONLY FOR THE ANALYSIS BUT ALSO FOR THE PRESENTATION, USING THIS AS AN EXEMPLAR OF THE THEME THAT WAS IDENTIFIED. A SECOND THEME WAS THIS WIDESPREAD MISTRU OF THE POLICE. THIS YOUNG MAN SAYS IF I NEEDED HIM PERMLY I WOULDN'T CALL THEM, I WOULD SAY THEY WOULD BE THE LAST PERSON I CALL IF I DIDN'T HAVE NO OTHER CHOICE THEN I WOULD CALL THEM. IF I HAD AN OPTION NOT THE CALL THEM I USE THAT BEFORE. A SHARED IDEA EXEMPLIFIED IN THIS CODE IS YOUNG PEOPLE DIDN'T BELIEVE THAT THEY DID NOT HAVE ALTERNATIVES. FINALLY TO EXEMPLIFY THE POETRY EXISTS TRAUMATIC STRESS WAS A COMMON DISCUSSION OF SYMPTOMS OF TRAUMATIC STRESS AMONG YOUNG PEOPLE, I WAS INTERVIEWING A YOUNG MAN AND HE TOLD ME HE HAD BEGUN TO WRITE AS A WAY TO DEAL WITH SYMPTOMS OF STRESS AND I ASKED HIM IF HE WOULD SHARE ONE OF HIS POEMS WITH ME, HE PULLED IT OUT AND READ THIS. AS THE DAY DARKENS, I'M FEELING SHOOK. I GUESS THAT'S WHY I STAY HIGH CONSTANTLY FIGHTING TO KEEP THE NIGHTMARES FROM TURNING TO REALITY. NIGHTMARES OF BLOODY DAYS AN COURT DATES CIRCULATENING MY MIND, NIGHTMARES SEEM TO ALWAYS FADE INTO REALITY AND REALITY SEEMS TO FADE INTO NIGHTMARES. AGAIN, THIS IS REALLY HIS OWN SYNTHESIS OF BRINGING TOGETHER MULTIPLE EXPERIENCES BUT IDENTIFYING THIS TENDENCY OF YOUNG PEOPLE TO SELF-TREAT -- SELF-MEDICATE WITH MARIJUANA AND ALCOHOL FOR THEIR SYMPTOMS OF TRAUMATIC STRESS. LOOKING AT THE QUANTITATIVE DATA, LAST NUMBERS I SHOW YOU, 65% IN THE FULL CRITERIA FOR PTSD OF OF THOSE SURVEYED ALL 23 SURVEYED MET CRITERIA FOR HYPERAROUSAL. SO WE BEGAN TO USE THESE DATA TO CONSTRUCT A POTENTIAL ALTERNATIVE PATHWAY BY WHICH A YOUNG PERSON WHO WAS INJURED MIGHT BE REINJURED, TO SET AS AN -- IN OPPOSITION TO THE ASSUMPTION OF PROVIDERS THESE WERE YOUNG CRIMINALS WHO SIMPLY WENT BACK TO CRIMINAL ACTIVITY. IF YOU WALK THROUGH, THERE'S A BASIC ENVIRONMENT OF POLICE MISTRUST WHICH YOUNG PEOPLE LIVE THEIR LIVES AND ALSO A COMMUNITY WIDE SENSE YOU NEED TO ESTABLISH RESPECT. IN THE SETTING OF INJURY, THREE THINGS AT LEAST HAPPEN. THIS NOTION YOU NOW HAVE TO NOT BE A SUCKER IS ACTIVATED BECAUSE YOU HAVE TO RESPOND TO SOMETHING THAT HAS HAPPENED TO YOU. AND OFTEN YOUR MISTRUST OF THE POLICE IS INTENSIFIED BECAUSE POLICE ARE INVOLVED IN THE MOMENT OF YOUR INJURY. FOR ALL THESE YOUNG PEOPLE IT TRANSLATES INTO A DISRUPTED SENSE OF SAFETY. EVEN IF THEY AT BASELINE DIDN'T FEEL COMPLETELY SAFE. FOR THOSE WITH SYMPTOMS OF TRAUMATIC STRESS SELF-TREATMENT WITH ALCOHOL OR MARIJUANA OFTEN LEADS TO POSITIVE DRUG TESTS WHICH WILL EXCLUDE YOU FROM EMPLOYMENT AND MAY HAVE YOU BACK IN THE CRIMINAL JUSTICE SYSTEM IF YOU'RE ON PROBATION. BUT FACED WITH DIMINISHING OPPORTUNITIES FOR LEGITIMATE EMPLOYMENT, MANY YOUNG PEOPLE ARE FORCED INTO ELICIT ECONOMIES, ELICIT ECONOMIES AND A COMBINED LACK OF SAFETY, HAVE POTENTIAL TO PUSH YOUNG PEOPLE IN THE DIRECTION OF OBTAINING WEAPONS AN REENTRY. WE HEARD THIS STORY FROM YOUNG PEOPLE THIS CASCADE OF EVENTS LEADING TO REINJURY SO IT REPRESENTS AN ALTERNATIVE PATHWAY OUT OF THE WORDS OF THE YOUNG PEOPLE. SO TO TALK FOR A MOMENT ABOUT IMPLICATIONS OF THIS WORK, AND QUALITATIVE RESEARCH PARTICULARLY, AS I SAID, I THINK QUALITATIVE RESEARCH IS A CRITICAL TOOL IN UNDERSTANDING THE EXPERIENCE OF PEOPLE WHO ARE AT THE MARGIN WHOSE STORIES ARE SUPPRESSED, VIEWED AS OTHER,7s– AND WHOM NO QUANTITATIVE TOOL COULD EXIST TO CAPTURE THEIR EXPERIENCE. IN FACT, THE INHE CAN FIES -- INEQUITIES IN OUR SOCIETY WOULD MEAN SOMEONE WITH THEIR EXPERIENCE IS UNLIKELY TO BE THE RESEARCHER AND BE ABLE TO INCORPORATE EXPERIENCES INTO RESEARCH DESIGN, HENCE THE QUESTION ABOUT WHAT MIGHT COMMUNITY BASED PARTICIPATORY RESEARCH BRING TO THIS PROCESS. PRESENTATION AND DISSEMINATION I THINK REQUIRES THAT WE SEARCH THOSE THEMES THAT ACHIEVE REAL SATURATION AND THAT WE FIND REALLY POWERFUL EXEMPLARS OF THOSE. WE ARE AS PARSIMONIOUS AS WE CAN BE IN PRESENTING THOSE THEMES IN AS CLEAN A FORM AS WE CAN. ONE ASPECT OR CHALLENGE OF THIS WORK IS THE DEMAND EARLIER THAT WE BRING REFLEXIVITY TO THIS. THAT IS IS, I AS AN AFRICAN AMERICAN PHYSICIAN WHEN I SIT WITH THE YOUNG AFRICAN AMERICAN MEN THEIR CHARACTERISTICS ABOUT ME THAT WILL INFLUENCE THE NARRATIVE HE TELLS, IN FACT, AS ELLIOT MISHLER SAY, NARRATIVES ARE CO-CONSTRUCTED. THEY ARE NOT LIKE A TUMOR THAT YOU CAN GO AND EXTRACT FROM SOMEONE, IT IS CO-CREATED IN THAT MOMENT. AND CREATED FOR THE RESEARCHER, BY THE RESEARCHER AND BY THE PARTICIPANT TOGETHER. THAT MEANS THAT I NEED TO DISCLOSE TO THE CONSUMER OF THE RESEARCH WHO I AM, AND HOW MY PRESENTATION IN THAT PROCESS AFFECTED WHAT CAME OUT OF IT. THAT MAKES BLINDING ON THE MANUSCRIPT DIFFICULT FOR THE JOURNAL. IF I SEND AN ARTICLE THAT SAYS I'M AN AFRICAN MERP MAN, THEY MAY HAVE PERCEIVE MED THIS WAY, I RUBS AGAINST TRADITIONAL NOTIONS OF OBJECTIVITY AND WHAT OUR PLACE IN THE RESEARCH IS. I WOULD SAY AN ONGOING CHALLENGE IS THIS IS RESOURCE INTENSIVE WORK THAT REQUIRES LOTS OF ENGAGEMENT IN ORDER TO DO THE KINDS OF FOLLOW-UP FROM A MARGINALIZED POPULATION, I THINK WE'RE MAKING PROGRESS WITH REVIEWER COMMITTEES WHO ARE LOOKING AT QUALITATIVE RESEARCH. I THINK THERE'S POTENTIALLY A TENDENCY TO UNDERRESOURCE THOSE STUDIES GIVEN WHAT YOU NEED TO DO. AND I THINK THAT THAT MAY ULTIMATELY BE A LIMITATION TO THE KINDS OF WORK WE CAN DO OR PRESENCE AN OPPORTUNITY TO EDUCATE AND DEMONSTRATE. PART OF WHAT HELPED ME WAS TO PRESENT THE NARRATIVES OF THESE YOUNG PEOPLE BACK IN A FORM THAT WAS ACCESSIBLE TO A BROADER AUDIENCE IN THE FORM OF THIS BOOK WRONG PLACE WRONG TIME. SEVERAL PARTICIPANTS IN THE RESEARCH CO-EDITED THEIR CHAPTERS, EVEN IF THEY CHOSE TO BE ANONYMOUS IN THAT WORK IN ORDER TO MAKE SURE THAT I WAS REPRESENTING OUR SHARED EXPERIENCE IN A WAY THAT WAS REASONABLE TO THEM. THE REAL WORLD TRANSLATION OF THIS I'LL TALK ABOUT, IF YOU LIKE, BUT WE'RE TRYING TO THINK HOW LESSONS INFORM OUR WORK IN HOSPITAL BASED VIOLENCE INTERVENTION BUT ALSO EQUIPPING PROVIDERS TO BETTER UNDERSTAND THE WORLD OF THE PATIENTS WE'RE INTERACTING WITH. THANK YOU. [APPLAUSE] >> THANK YOU TO THE PRESENTERS, THOSE ARE BRILLIANT PRESENTATIONS. WE HAVE FIVE MINUTES FOR QUESTIONS, SO ANYONE WHO WOULD LIKE TO ASK A QUESTION PLEASE MAKE YOUR WAY TO THE MOARK PHONE, PLEASE. MICROPHONE PLEASE. >> ALL THE DISCUSSIONS WERE FANTASTIC. I WAS THINKING THE RECURRENT THEME WHICH IS THE SELF-DISCLOSURE OF WHO YOU ARE AND HOW YOUR LENS CAN AFFECT THE DATA, THESE IDEAS OF REFLEXIVITY IN TERMS OF WHO YOU ARE AND HOW THAT DATA WHICH IS A TRADITION TYPE OF PIECE. IN TERMS OF WORKK WITH MIXED METHOD TEAMS BECAUSE HAVING BEEN EXPOSED TO THAT TRADITION MYSELF, I FEEL AS THOUGH SOMEONE WITH STRONG QUANTITATIVE BACKGROUND SAYS THIS ISTOTALLY OBJECTIVE. AND I'M WONDERING HOW IN YOUR EXPERIENCE YOU HAVE BEEN ABLE TO ADDRESS THOSE ISSUES OR HOW THOSE ISSUES HAVE PLAYED OUT. THEY'RE VERY FUNDAMENTAL TO INDIVIDUALS IN QUALITY TAITIVE RESEARCH AND'S A STRONG SENSE IF YOU'RE DOING QUANTITATIVE STUDIES YOU HAVE TO BE EXTREMELY OBJECTIVE ABOUT THAT. I SEE THAT AS ONGOING INTENTION IN MIXED METHODS PROJECTS, WONDERING IF MAYBE YOU WILL BE WILLING TO COMMENT ON YOUR EXPERIENCES OR IF YOU HAVEN'T HAD EXPERIENCES WITH THAT, INSER YOUR OWN THINKING ABOUT THAT. >> I THINK THIS RAISES SOMETHING WE WERE TALKING ABOUT EARLIER WE CAN THINK AB QUALITATIVE AN QUANTITATIVE APPROACHES AS TOOLS OR THINK AB THEM AS PARADIGMS OR WORLD VIEWS. I THINK YOU HAVE TO CONTINUE TO NURTURE THEu PARADIGMS NATURE OF TRUTH. I THINK IT IS FUNDAMENTAL MANY OF US WHO ADHERE TO THIS TRADITION THAT OBJECTIVITY IS A FANTASY IN A SENSE, INTERVIEWER EFFECTS, THERE'S ALWAYS AN EFFECT. HUMAN SCIENCES FRAUGHT WITH THESE KINDS OF BIASES ARE REALITIES AND AS ELLIOT MISHNER SAID, WOULDN'T IT BE BEAR THIS PERSON DO AN INTERVIEW THAN THIS PERSON? HE SAYS REALLY JUST DIFFERENT. IT WOULD BE DIFFERENT. THAT DOES NOT REDUCE ITS RIGOR. SO I THINK IT WOULD BE GOOD IF WE IN ALL STUDIES DISCLOSED MORE ABOUT WHO WAS DOING WHAT. SO THAT WE CAN UNDERSTAND POWER DYNAMICS AND OTHER THINGS HAPPENING. TO THE EXTENT THAT WON'T HAPPEN ACROSS THE BOARD IT'S IMPORTANT TO MAINTAIN THAT IT HAS TO HAPPEN IN QUALITATIVE ININJURIES OF THE TYPE WE'RE TALKING ABOUT, THERE'S JUST ONE PERSPECTIVE. >> AS I ALLUDED TO EARLIER, I WAS IN A STATE OF TERROR THROUGHOUT A LOT OF THIS STUDY UNTIL I HAD THE AHA MOMENT OF REALLY BEING ABLE TO SEE THE VALUE OF THE INDUCKTIVE APPROACH. I DON'T KNOW THAT I WOULD HAVE MADE TO IT THAT POINT HAD THIS TEAM NOT BEEN COMPOSED OF COLLEGIAL PEOPLE THAT I RESPECTED ENORMOUSLY. AND THAT WE ALL WERE COMMITTED TO TRYING TO UNDERSTAND OTHER WAYS OF LOOKING AT BEHAVIOR AND REALLY TRYING TO SORT OUT THE BEHAVIORAL CONSTRUCTS. BUT THIS WAS A FOUR YEAR GRANT THAT TOOK SEVEN YEARS TO COMPLETE. PART OF IT WAS ACTUALLY IN THE WRITING. ANTHROPOLL GISTS AND BEHAVIORAL SCIENCE PEOPLE WRITE VERY DIFFERENTLY. I FURTHER COMPLICATED THAT BY INSISTING THAT EDITORS REPRESENT THE TWO DIFFERENT PERSPECTIVES, THAT WASN'T THE HARD PART BUT I ALSO WANTED TO HAVE REVIEWERS FOR EACH PAPER WHO REPRESENTED EACH OF THE DIFFERENT DISCIPLINES. I CAN'T SAY THAT I WOULD RECOMMEND THAT YOU TRY THAT, YOU HAVE GOT TO HAVE NERVES OF STEEL. AND A LOT OF PATIENTS IN LONG TIME FRAME BECAUSE THEY WANT TO SEE DIFFERENT THINGS BUT I WANTED THIS TO BE ACCESSIBLE TO DIFFERENT AUDIENCES. I DON'T KNOW HOW WELL WE SUCCEEDED WITH THAT BUT WE GAVE IT THE BEST TRY, SO IT'S TOUGH, THIS IS NOT FOR EVERYBODY. >> I WOULD SAY, I HAVE BEEN VERY FORTUNATE IN MY -- THE FOLKS I HAD THE OPPORTUNITY TO COLLABORATE WITH STARTING WITH MY POST DOC WHEN I WAS INTRODUCED TO THE TREMENDOUS RESEARCH, MY PRIMARY MENTOR WAS VERY OPEN TO REFLEXIVITY. HE ALSO ADEPT AT TRAINING THE DIFFERENT AUDIENCE BUS I NEVER HAD THE HORROR STORIES OTHERS HAD AB GOING TO A QUANTITATIVE TEAM AND HAVING THESE NEGATIVE EXPERIENCES WHERE YOU'RE BEING ACCUSED OFFER INSERTING TOO MUCH OF YOURSELF INTO THE WORK. IT'S REALLY THEN BECAUSE THE CHOICE OF PARTNERS AND THE ON GOING ENGAGEMENT AND CONSTANT DISCUSSION. WE DO FIND AVENUES WHERE WE PRESENT OUR FINDINGS MORE REFLEXIVE POINT PARTICULARLY IN ANTHROPOLOGY WE USUALLY EVERY SO MANY YEARS WE'LL HAVE A PANEL WHERE THE TEAM GETS TOGETHER AND NOT NECESSARILY TALKING THE FINDINGS BUT THEY'RE TALKING THE PROCESS DOING THEIR LITTLE BIT TO THE PROJECT. THIS ADDS TO THE LEARNING EXPERIENCE FOR THE TEAM AS A WHOLE. >> I THINK WE HAVE TIME FOR ONE MORE QUESTION, PLEASE. >> THANK YOU, EVERYONE, THESE WERE GREAT PRESENTATIONS. I WAS THINKING ABOUT THE BARRIERS THAT -- AND LESSONS LEARNED THAT Y'ALL HAVE PRESENTED IN THE CONTEXT OF DOING MIXED METHS AND PARTICULARLY QUALITATIVE RESEARCH, REMEMBERING NOT TWO YEARS AGO I SAW THE SLIDE PUTTING UP THE HIERARCHY WHAT WE CONSIDER EVIDENCE, WHAT WE CONSIDER RO ROBUST SCIENCE, QUALITATIVE AT THE LOW END AND I THINK THAT THERE'S STILL A LOT OF CONFUSION ABOUT WHAT THE DIFFERENCE BETWEEN ANECDOTE AND RIGOROUS QUALITATIVE RESEARCH IS. AND I'M WONDERING HOW MUCH RECENT EFFORTS TO INCLUDE MORE SOCIAL BEHAVIORAL SCIENCES AN MEDICAL EDUCATION, A AND B SURVEY FROM A REPORT CAME OUT. WONDERING HOW MUCH OF THAT, THOSE KINDS OF EFFORTS MIGHT EVENTUALLY FEED INTO MORE WIDESPREAD IMPORTANCE OF CULTURE RESEARCH AS YOU MENTIONED, DR. PASICK. I WAS WONDERING IF YOU HAD HIGHER EDUCATIONAL BARRIERS AND WHAT WE NEED TO DO TO OVERCOME THE INCREASE ETHICS OF METHODS OF ETHICS RESEARCH. >> THERE IS DEFINITELY A WAVE IN PROGRESS AND IT'S PICKING UP SPEED. YEARS AGO I STRUGGLED TO 1„ GET MY STUDY FUNDED. I THINK THESE DAYS THERE'S MORE ACCEPTANCE. PEOPLE ARE BECOMING AWARE OF THE GUIDELINES FOR MIXED METHODS RESEARCH THAT HELEN MEISSNER, DR. CRESWELL AND OTHERS PUT TOGETHER THROUGH OBSSR OPT WEB A LOT OF PEOPLE ARE AWARE OF IT. AND INCREASINGLY THEY FIGURED IT OUT, PERFECT SENSE TO THEM. PROBLEM IS FINING THE RIGHT COURSES FOR THEM AND THE RIGHT MENTORS WHO CAN REALLY GUIDE THEM. I THINK THIS IS REALLY -- WE ARE GOING TO NOT TOO FAR DOWN THE ROAD SEE THIS, PEOPLE WILL LOOK BACK AND SAY WHAT WAS THE FUSS ABOUT. OF COURSE WE'RE DOING THIS. SO I THINK THAT WILL BE REALITY. >> THANK YOU SO MUCH TO OUR PRESENTERS WE APPRECIATE YOUR GREAT WORK HERE. THIS CURRENT SYMPOSIUM. WE NOW HAVE SOME INFORMATION FOR YOU ABOUT WHAT -- LUNCH. WE HAVE ABOUT AN HOUR FOR LUNCH. WE'LL BE BACK HERE AT 1:30 AND THERE IS A CAFETERIA IF YOU GO UP TO THE MAIN LEVEL -- THAT'S ABOUT THE ONLY GAME IN TOWN. [LAUGHTER] >> HOPE YOU ENJOY IT. [LAUGHTER] >> OKAY, WELCOME BACK. I UNDERSTAND THAT WE NEED TO BEGIN, MAY BE FEW PEOPLE THAT WILL STRAGGLE IN HERE. SO THE OVERALL QUESTION TODAY IS HOW DO WE BEGIN THINKING ABOUT MIXED METHODS RESEARCH AND DISSEMINATION AND IMPLEMENTATION AND BRINGING THE TWO TOGETHER. MAYBE INTEGRATING THE TWO. SO I THOUGHT I MIGHT GIVE A MID DAY ASSESSMENT OF WHAT I'M HEARING. AND OF COURSE THESE IDEAS COME FROM A MIXED METHODS WAY OF THINKING ABOUT ALL OF THIS. I'M VERY INTERESTED IN THE LANGUAGE THAT WE USE AS WE BUILD THIS BRIDGE BETWEEN THESE TWO AREAS. AND HOW THAT LANGUAGE IS BEGINNING TO EMERGE, FOR EXAMPLE, ONE OF THE AUTHORS THIS MORNING MENTIONED THE WORD EXPANSION. IF YOU RECALL I USED THE WORD TO EXPLAIN THE FINDINGS WITH THE SECOND PHASE. SO LANGUAGE IS EMERGING HERE. I DIDN'T SEE MANY SPECIFIC MIXED METHODS DESIGNS MENTIONED, NOR ANY FIGURE, PROCEDURES. BUT YET THOSE MODELS ARE AVAILABLE. I WAS VERY CURIOUS ABOUT KATHLEEN'S PRESENTATION WHERE SHE HAD A LOT OF COMPLICATED FACETS OF THAT PROJECT GOING ON. AND I BEGAN DRAWING IN MY MIND A PICTURE OF A PROCEDURE THAT WOULD HELP CLARIFY THE COMPLEXITY. ONE OF THE THINGS I HAVEN'T HEARD MUCH ABOUT IS -- THE REASON WHY I THINK DESIGNS ARE IMPORTANT IS BECAUSE IF WE HAVE A CLEAR PICTURE AND WE HAVE -- UNDERSTAND THE PROCEDURES THEN WE CAN MOVE TO A REALLY IMPORTANT PHASE IN RESEARCH AND THAT IS TO THINK ABOUT WHAT WOULD BE THE THREATS, WHETHER YOU WANT TO CALL THEM VALIDITY THREATS, THEY'RE CHALLENGES TO THESE DESIGNS. I DIDN'T SPEAK MUCH ABOUT THAT. BUT THE MIXED METHODS COMMUNITY IS VERY ACTIVE RIGHT NOW IN THINKING ABOUT WHAT ARE THE CHALLENGES TO THESE DESIGNS, ONE OF THE PERSONS NO GOT UP AND ASKED A QUESTION, IF YOU RECALL IT WAS ON SAMPLINGU THAT'S AN IMPORTANT CHALLENGE THAT YOU NEED TO THINK ABOUT. ANOTHER PERSON TALKED ABOUT WHAT WERE THE BARRIERS TO USING THIS DESIGN. SO SOME OF YOU WERE CURIOUS ABOUT SOME OF THESE THREATS. I THINK WE'VE SEEN SOME GREAT MODELS OF WHAT ARE BEING CALLED IN THE MIXED METHODS FIELD, JOINT DISPLAYS. SO CARLA STUDY AND GREG'S BOTH HAD TABLES. NOW IN GREG'S CASE HE HAD A TABLE OF QUESTIONS AND ANSWERS AND QUALITATIVE AND QUANTITATIVE. REMEMBER THAT TABLE? THAT'S CALLED A JOINT DISPLAY. AND CARLA HAD A PATH MODEL WHERE SHE HAD THE QUALITATIVE INTEGRATED WITH THE QUANTITATIVE. SO WHAT WE'RE DOING ACROSS THE COUNTRY RIGHT NOW IS WE'RE GATHERING EXAMPLES OF THESE JOINT DISPLAYS SO THAT PEOPLE CAN BEST PICTURE THE COMBINATION OF QUALITATIVE AND QUANTITATIVE DATA TOGETHER. THESE CONVERSATIONS ABOUT QUALITATIVE RESEARCH, ONE THING THAT I'M QUITE AWARE OF IS HOW DIVERSE THE QUALITATIVE RESEARCH COMMUNITY IS ACROSS THE UNITED STATES. AND IN FACT MIXED METHODS WILL GET THAT WAY AS IT CONTINUES TO EVOLVE AND DEVELOP. WHAT'S NEEDED IN TERMS OF QUALITATIVE RESEARCH IN THE HEALTH SCIENCES. I WOULD SAY A SYSTEMATIC, RICK RUSS APPROACH -- RIGOROUS APPROACH. EASE OF COMMUNITY PROGRAMS, WE HEARD ABOUT THAT. THE COMPLEX DATA ANALYSIS. THE EXTENSIVE QUALITATIVE DATA COLLECTION, I WAS IMPRESSED WITH JOHN RICH, IS HE BACK FROM LUNCH? HE USED A VERY SYSTEMATIC LINGUISTIC ANALYSIS QUALITATIVE COMPONENT IN HIS PROJECT. SO HE'S USING SYSTEMATIC. I THINK THAT'S THE WAY IT CAN BE, IT NEEDS TO BE NOT JUST FOCUS GROUPS, NOT JUST GENERAL OBSERVATIONS BUT NEEDS TO BE RIGOROUS AND SYSTEMATIC AS IT COMES IN TO HEALTH SCIENCES. ALSO THERE'S QUITE A BIT OF DISCUSSION ABOUT TEAM WORK AND THE INDIVIDUAL PERSPECTIVES THAT INDIVIDUALS BRING TO TEAMS IN DOING MIXED METHODS RESEARCH IN THIS AREA. I THINK THAT IS A TOPIC OF THE FUTURE THAT NEEDS MORE WORK AND MORE THOUGHT AS TO HOW TO BUILD A GOOD MIXED METHOD TEAM. MY CLASSIC EXAMPLE AS NSF PROJECT UP IN BOSTON, ON THE LEFT-HAND SIDE OF THE TABLE SIT THE QUANTITATIVE RESEARCHERS. ON THE RIGHT HAND SIDE SIT THE QUALITATIVE RESEARCHERS AND YOU KNOW WHERE I SIT, IN THE MIDDLE. AND I'M GOING BACK AND FORTH, I'M THE TONS LATER REALLY ON MANY TOPICS TRYING TO BRIDGE ACROSS. SO THE IMPORTANCE OF TEAM WORK. ON THIS TOPIC OF INTEGRATION WE'RE GOING TO TALK JUST VERY BRIEFLY ABOUT HOW IT'S BEING DEFINED, WHY IT'S IMPORTANT, WHERE DOES IT OCCUR, WHAT ARE SOME PROCEDURES FOR INTEGRATING, HOW IS IT WRITTEN IN TO A PUBLISHED MIXED METHOD STUDY AND THEN WE HAVE DR. CURRY WHO WILL BE PRESENTING A A SEQUENTIAL DESIGN AND DR. FETTERS AND DR. DUAN AND NEW MAN IS THIS MIXED SEQUENTIAL CONCURRENT TYPE OF DESIGN. WE'LL SEE A LOT OF NICE OPTIONS HERE IN TERMS OF INTEGRATION. FIRST OF ALL INTEGRATION IS A CONFUSING CONCEPT. I HAVE A VERY SIMPLE ILLUSTRATION V THIS. I WAS WORKING -- THIS PERSON IS NOW AN NIH PSYCHOLOGIST. AND HE SAID, JOHN, I WANT TO COLLECT QUALITATIVE FOCUS GROUP DATA AND THEN I'VE ALSO GOT SOME INSTRUMENT DATA THAT I HAVE NO CLUE AS TO HOW TO BRING THOSE TWO TOGETHER. THERE'S THE DILEMMA. IF YOU ARE REALLY GOING TO HAVE A MIXED METHOD STUDY IT'S NOT SEPARATE STRANDS OF RESEARCH BUT THAT INTEGRATION. IT'S A TOPIC THAT'S OCCUPIED THE ATTENTION OF MIXED METHOD RESEARCHERS FOR WHITE A BIT OF TIME HERE. THERE ARE SOME PEOPLE WRITING ABOUT ARTICLES ABOUT INTEGRATION. AND IT CAN BE IMPLEMENTED IN DIFFERENT WAYS. EARLIER YOU SAW THIS DIAGRAM I PRESENTED, INTEGRATION IS STRETCHING ACROSS THESE TWO STRANDS. ONE WAY TO INTEGRATE IS TO COMBINE A DESIGN WITH DIFFERENT TYPES OF METHODS. SO AN EXPERIMENTAL STUDY WHERE YOU'RE PUTTING QUALITATIVE DATA IN TO THE EXPERIMENT. ANOTHER WAY INTEGRATION COULD OCCUR IS BY BRINGING IN DIFFERENT PHILOSOPHIES AND SOCIAL SCIENTISTS TALK A LOT ABOUT INTEGRATING DIFFERENT PARADIGMS OR WORLD VIEWS IN TO DIFFERENT PHASES OF THE RESEARCH PROJECT. SOME PEOPLE LOOK AT INTEGRATION AS INTEGRATING A DIFFERENT PHASES IN THE PROCESS OF RESEARCH. FOR EXAMPLE, WHEN A PROPOSAL FOR FUNDING YOU MIGHT HAVE QUANTITATIVE HYPOTHESES, QUALITATIVE RESEARCH QUESTION AND MIXED METHODS QUESTION. SO YOU ARE REALLY INTEGRATING DIFFERENT TYPES OF QUESTIONS WITHIN THE RESEARCH QUESTION HYPOTHESES PHASE OF RESEARCH. ONE EXAMPLE OF THOSE JOINT DISPLAY, IS THAT HAVE ALREADY BEEN PRESENTED TO DISPLAY BOTH QUANTITATIVE AND QUALITATIVE DATA IN THE DATA ANALYSIS PHASE. SO, WHEN I ADVANCED EARLIER THIS MORNING THAT DEFINITION OF THE CHARACTERISTICS OF MIXED METHODS RESEARCH I SAID IT'S THE INTEGRATION OF TWO OR MORE DATA SOURCES AND THAT -- IT MIGHT INVOLVE THE CONCURRENT OR SEQUENTIAL OR POSSIBLE EVEN BOTH MODELS OF ORGANIZING THOSE DATABASES. WE INTEGRATE IN ORDER TO DRAW ON THE STRENGTHS OF EACH AND SOME STUFF IN THE RESEARCH PROCESS, INVESTIGATOR TYPICALLY INTEGRATES THE DATA AND THIS INVOLVES KNOWING HOW TO INTEGRATE OFF A NUMERIC INFORMATION WITH TEXT INFORMATION. SO IN ONE TYPE OF MIXED METHOD STUDY WE MAY INTEGRATE BY LOOKING AT THE RESULTS, COMPARING THE RESULTS FROM OUR QUANTITATIVE AND QUALITATIVE DATABASE. IN ANOTHER STUDY WE MIGHT INTEGRATE MY CONNECTING FROM ONE PHASE TO THE OTHER. IN OTHER STUDY WE MIGHT INTEGRATE BY ASKING DIFFERENT TYPES OF QUESTIONS, A PROCESS QUESTION, OUTCOME QUESTION INTEGRATING INTERVENTION. SO THERE ARE SOME PROCEDURES THAT HAVE BEEN IDENTIFIED FOR HOW YOU INTEGRATE NUMERIC DATA AND TEXT DATA. YOU SEE THESE IN MIXED METHODS PROJECTS, PUBLISHED MIXED METHODS PROJECTS. ONE IS WHAT WE CALL A SIDE BY SIDE COMPARISON. SO IF YOU GO TO THE CONCLUSION SECTION OF A MIXED METHODS PROJECT YOU MIGHT SEE SOMETHING LIKE THIS. WHERE THE AUTHOR FIRST MENTIONS QUANTITATIVE FINDINGS THEN MENTIONS THE QUALITATIVE THEMES AND HOW THEY TEND TO EITHER SUPPORT OR REFUTE SOME OF THESE QUANTITATIVE FINDINGS. SIDE BY SIDE. IT'S A POPULAR APPROACH YOU SEE IT IN A LOT OF MIXED METHODS PROJECTS. IN GREG'S PROJECT WE SAW AN EXAMPLE OF THIS WHERE YOU GOT YOUR QUALITATIVE DATA RIGHT HERE, IN THIS COOL I'M. AND YOUR QUANTITATIVE DATA RIGHT HERE AND OVER HERE ARE THE KEY QUESTIONS. THIS IS A VERY POPULAR MODEL. WHAT YOU ARE DOING IS INTEGRATING THE TWO DATABASES IN A SINGLE TABLE. CARLA'S PROJECT WAS A STRUCTURAL EQUATION MODEL, A PATH MODEL WHERE SHE HAD THOSE TWO INTEGRATED WITHIN A DIAGRAM. OR WE MIGHT DO, I'LL USE QUANTITATIVE TERM. A CROSS TAB OF A QUANTITATIVE CATEGORICAL VARIABLE HERE AND THEMES THAT APPEAR HERE, SO THIS IS THE QUALITATIVE THEMES, WITHIN THE TEXT -- WITHIN THE CELLS YOU CAN PUT EITHER TEXT DATA OR YOU COULD PUT NUMERIC DATA, HOW MANY PEOPLE, OR BOTH. WHAT WE'RE DOING IN THE MIXED METHODS FIELD WE'RE ACTUALLY CATEGORIZING THESE DIFFERENT TYPES OF JOINT DISPLAYS TO ENABLE PEOPLE TO THINK BETTER ABOUT WAYS TO INTEGRATE QUANTITATIVE AND QUALITATIVE DATA IN A SINGLE TABLE. WE CAN ALSO INTEGRATE BY TRANSFORMING DATA, I HAVEN'T HEARD OF ANY EXAMPLES TODAY WHICH IS A LITTLE SURPRISING TO ME. BUT THAT'S WHERE YOU COLLECT QUALITATIVE DATA THEN YOU CODE IT THEN YOU START COUNTING YOUR CODES. NOW YOU -- IN OTHER WORDS YOU TRANSFORM IT IN TO QUANTITATIVE DATABASE TO COMPARE WITH ANOTHER QUANTITATIVE DATABASE. OR YOU CAN INTEGRATE BY HAVING ONE DATABASE BUILD ON THE OTHER. SO IF YOU START QUANTITATIVELY AND ANALYZE YOUR RESULTS, WHAT RESULTS DO YOU LOOK AT. IT MIGHT BE THE SIGNIFICANT!)n RESULTS OUTLIERS, EXTREME CASE, NOT SIGNIFICANT RESULTS, IMPORTANT DEMOGRAPHICS. THEN USE THAT IN ONE TWO OF WAYS. TO IDENTIFY THE TYPES OF YES, SIR YOU USE IN QUALITATIVE PHASE AND/OR TO HELP SELECT PARTICIPANTS THAT WOULD PARTICIPATE IN YOUR QUALITATIVE PHASE. SO NOW INTEGRATION IS SOMEWHAT OF A COMPLEX MATTER. IT CAN FLOW IN TO A PROJECT IN A LOT OF PLACES. WE CAN INTEGRATE IN THE DATA COLLECTION PHASE BY ADDING TO A QUANTITATIVE SURVEY OPEN-ENDED QUESTIONS. I CALL THIS MIXED METHODS LIGHT. BECAUSE WE KNOW THAT QUALITATIVE DATABASE, THOSE ANSWERS ARE GOING TO BE VERY SHORT AND NOT VERY CONTEXTUALLIZED. BUT IT DEPENDS. DEPENDS ON YOUR PARTICIPANTS TO A CERTAIN EXTENT. IT COULD BE INTEGRATED IN THE DATA ANALYSIS PHASE WHERE WE HAVE A MATRIX, THE QUANTITATIVE DATA AGAINST THE QUALITATIVE THEMES. IT COULDU INTERPRETATION OF THE DISCUSSION PHASE, THIS SIDE BY SIDE COMPARISON OF THE QUANTITATIVE AND QUALITATIVE DATA. IT COULD BE INTEGRATED AT MULTIPLE STEPS IN THE RESEARCH PROCESS. FOR EXAMPLE, IN AN INTERVENTION TRIAL BEFORE THE TRIAL BEGINS TO HELP RECRUIT INDIVIDUALS YOU MIGHT CONDUCT SOME FOCUS GROUPS, THEN AFTER THE TRIAL IS OVER YOU MIGHT GATHER SOME MORE FOCUS GROUP DATA IN ORDER TO HELP EXPLAIN SOME OF THE SIGNIFICANT OR NONSIGNIFICANT OUTCOMES. COULD EVEN BE INTEGRATED IN TO MULTIPLE PHASES IN OUR PROJECT. THERE ARE LOTS OF DIFFERENT WAYS THAT WE CAN THINK ABOUT INTEGRATION AND MIXED METHODS STUDY. WE HAVE SOME ILLUSTRATIONS NOW. FOUR MIXED METHODS SPECIALISTS WILL SPEAK TO TALK ABOUT DIFFERENT PERSPECTIVES ON INTEGRATIONS IN THEIR PROJECT. WHERE IT OCCURS IN THEIR STUDY, THE TYPES OF INTEGRATIONS THAT THEY'RE USING, WHERE THEY WOULD WRITE IT IN TO A MIXED METHOD PROJECT AND LINK IT OVER TO THE DISSEMINATION AND IMPLEMENTATION PHASE OF RESEARCH. WE'LL START WITH DR. CURRY. >> I HAVE HAD SO MUCH FUN LISTENING AND LEARNING ALL A DAY I WISH I DIDN'T HAVE TO TALK. BUT I WILL AND I'M GOING TO KEEP MY COMMENTS TO 15 MINUTES NOT A NANOSECOND LONGER. JOHN WILL HELP ME WITH THAT BECAUSE I WANT TO HAVE TIME TO HEAR HOW THE OTHER PANELISTS SHARE SOME OF THEIR STRATEGIES FOR INTEGRATION IN THE WORK THAT THEY DO. I'VE JUST BEEN REALLY STRUCK BY THE DIVERSITY OF THE PROJECTS. I'M GOING TO DESCRIBE A FIVE-YEAR STUDY IN THESE 15 MINUTES THAT I HAVE. AND I WANT TO HELP CONVEY HOW WE THINK ABOUT INTEGRATION FROM THE CONCEPTUALIZATION OF THE PROJECT THROUGH TO THE DISSEMINATION PHASE. AND I'LL BE USING AN ILLUSTRATION FROM POSITIVE DEVIANT STUDY OF HEALTH CARE QUALITY. BEFORE I GET GOING I WANT TO ACKNOWLEDGE OUR TEAM AT YALE. A LARGE MULTI DISCIPLINARY VERY DIVERSE GREAT GROUP OF PEOPLE. THIS WAS FUNDED BY THE UNITED HEALTH FOUNDATION, DONOHUE FOUNDATION AND COMMONWEALTH FUND. JUST ONE MINUTE ON MY BACKGROUND AND INTEREST IN MIXED METHODS. MY GRADUATE WORK ADDRESSED ORGANIZATION AND FINANCING OF LONG-TERM CARE. MY MASTER'S THESIS USED QUANTITATIVE METHODS I APPLIED THEM AND FELT GOOD ABUT WHAT I LEARNED. BUT AS I RETINED MY LITTLE PIN HEAD FOR HI DISSERTATION I COULD NOT ANSWER THAT QUESTION WITH A QUANTITATIVE APPROACH. I LEARNED, APPLIED QUALITATIVE METHODS IN MY DISSERTATION. SINCE THAT TIME I'VE BEEN INTERESTED IN THE TRACTS AS PARALLEL TRACKS BUT REALLY THESE OPPORTUNITIES FOR INTERCEPTION WHICH I REALLY BELIEVE CAN GENERATE UNIQUE INSIGHTS WE'VE SEEN SO MANY EXAMPLES OF THAT TODAY. I THINKING BE A VERY POWERFUL TOOL IN OUR INCREASINGLY COMPLEX HEALTH CARE SYSTEM. VERY EXCITED TO SEE THE INTEREST AND GROWTH IN THE MIXED METHODS. THIS STUDY WE WERE INTERESTED BROADLY IN QUALITY OF HOSPITAL COMPARE FOR ACUTE MYOCARDIAL INFARCTION. LOTS OF DIFFERENT OF QUALITY OF CARE, VERY LITTLE CAN BE EXPLAINED BY THE QUANTITATIVE WORK THAT'S BEEN PUBLISHED TO DATE. OUR PARTICULAR AIM WAS TO IDENTIFY HOSPITAL LEVEL FACTORS THAT MAY BE ASSOCIATE WITH BETTER PERFORMANCE ON -- FOR CARE AS MEASURED BY RISKED STANDARDIZED MORTALITY RATES. WE ALL COME WITH OUR LANGUAGE, I TRY NOT TO USE MUCH JARGON HERE. AMI CARE QUALITY. WE WERE INTERESTED BROUGHTLY IN PROCESSES, STRUCTURES AND WHAT WE CALLED ORGANIZATIONAL ENVIRONMENT. HERE WE USE POSITIVE DEVIANCE APPROACH WHICH THE ROOTS OF POSITIVE DEVIANCE WORK ARE IN GLOBAL HEALTH ADDRESSING COMPLEX PROBLEMS OF GLOBAL HEALTH. FIRST STUDIES OF POSITIVE DEVIANCE WERE TO ADDRESS CHILDHOOD MALNUTRITION, DONE IN THE 1970s. WE ADAPTED THIS APPROACH TO BE USED AT ORGANIZATIONAL PERFORMANCE AND HEALTH CARE QUALITY MORE BROADLY. BASIC IDEA THAT WE TRY TO IDENTIFY KEY STRATEGIESED USED BY THE TOP PERFORMERS IN A GIVEN INDUSTRY. THE SOLUTIONS TO PROBLEMS OFTEN RESIDE WITHIN THAT COMMUNITY. THAT CERTAIN MEMBERS POSSESS WISDOM THAT CAN BE GENERALIZED TO OTHER MEMBERS THAT HAVE COMMUNITY IN ORDER TO IMPROVE PERFORMANCE. THE STRATEGIES, BECAUSE THEY'RE DEVELOPED IN A PARTICULAR CONTEXT WITH RESOURCES MAY BE MORE LIKELY THAN TO BE ADOPTED AND SUSTAINED OVER TIME. THAT'S THE BASIC APPROACH. WE'VE WRITTEN ABOUT THIS IN IMPLEMENTATION SCIENCE, GREG, I WANT TO THANK HIM AS EDITOR EXTRAORDINARE WE FIGURED OUT THAT HE REVIEWED THAT PAPER IT WAS MUCH IMPROVED BY THE SERIES OF REVIEWS. AND IN THIS PAPER WE DESCRIBE THE METHODOLOGY IN MORE DETAIL. THE CURVE FOR RSMRs, THE AMI CARE MEASURED BY RSMR LOOKS LIKE A NORMAL CURVE. WE'RE INTERESTED IN THAT VERY TOP TAIL OF THE CURVE. TOP 5% OF PERFORMANCE IN THIS CASE. THERE ARE FOUR STEPS TO A POSITIVE DEVIANCE APPROACH. THERE ARE ALSO CONDITIONS WHICH NEED TO BE IN PLACE FOR THIS APPROACH FOB ABLE TO APPLIED RIGOROUSLY. THOSE AGAIN ARE DESCRIBED IN THIS PAPER. BUT THE FOUR COMPONENTS ARE TO BEGIN WITH IDENTIFICATION OF THE POSITIVE DEVIANTS. THESE ARE THOSE EXCEPTIONAL PERFORMERS. RSMR DATA ARE PUBLICLY REPORTED BY CMS, WE IN THE FIRST STAGE LOOKED AT RSMR AND IDENTIFIED HOSPITALS BOTH IN THE VERY HIGH END PERFORM AN, THE TOP 5% OF PERFORMANCE FOR TWO CONSECUTIVE YEARS THEN LOWEST END OF PERFORMANCE. WE THEN IDENTIFIED THOSE HOSPITALS, RECRUITED THEM IN, WE HAD 11 HOSPITALS TOTAL, SEVEN IN THE HIGH PERFORMANCE SPAN FOUR IN THE LOW PERFORMANCE GROUP. SECOND STEP TO STUDY THEM IN TENTATIVELY USING QUALITATIVE METHODS. WE BRING OUR MULTI-DISCIPLINARY TEAM TO THE HOSPITALS FOR SITE VISITS AND CONDUCT IN DEPTH INTERVIEWS WITH KEY STAFF. IN THIS STUDY WE HAD 158 KEY STAFF. THE THIRD STEP TO GENERATE AND TEST HYPOTHESES FROM THESE QUALITATIVE DATA USING QUANTITATIVE APPROACHES. THIS STUDY WE DID NATIONALLY -- SURVEY OF NATIONALLY REPRESENTATIVE GROUP OF U.S. HOSPITALS, AND 537 AND 91% RESPONSE RATE. THE FOURTH STEP IN POSITIVE DEVIANT STUDY TOE PARTNER WITH STAKEHOLDERS FOR DISSEMINATION. WE HAVE A NATIONAL QUALITY ALLIANCE TO DRAW ON THE QUALITATIVE AND QUANTITATIVE DATA IN THE CAMPAIGN. I'VE BEEN USING THIS SLIDE ALMOST FOUR YEARS I KEEP TELLING MYSELF I HAVE TO FIX IT BECAUSE IN FACT PARTNERSHIP HAPPENS AT THE VERY BEGINNING. REALLY AT THE GRANT WRITING STAGE WE'RE WORKING VERY CLOSELY WITH THE PARTNERS WHO WE PLAN TO USE AS THE LEAD ON DISSEMINATION PIECE OF A PROJECT. I HAVE TO FIX THIS GRAPH EVERY TIME I SEE IT IT MAKES ME -- GOT TO FIX IT. GOING TO TALK BRIEFLY THROUGH THE KEY FINDINGS AND QUALITATIVE COMPONENT TO TAKE YOU THROUGH HOW WE OPERATIONALIZE SOME OF THESE IN THE QUANTITATIVE STEP OF THE PROJECT. THERE WERE SIX KEY FIND THANKS EMERGED FROM THE ANALYSIS OF THESE 1500 PAGES OF TRANSCRIPTS, I CAN EMPATHIZE WITH THE QUALITATIVE RESEARCH THAT'S BEEN DESCRIBED AND VOLUMES OF DATA AND VALUE OF SOFTWARE. THE SIX KEY THEMES THAT WE IDENTIFIED, FIRST FIVE OF THEM DIFFERENTIATED BETWEEN THE HIGH AND LOW PERFORMING HOSPITALS. LAST TEAM SOMETHING OCCURRED. I'LL DESCRIBE THESE JUST BRIEFLY. THE FIRST WAS THIS IDEA OF ORGANIZATION FALL VALUES AND GOALS ALLS SHARED COMMITMENT TO ACHIEVE EXCEPTIONAL QUALITY AMI CARE SHARED THROUGHOUT THE ORGANIZATION. THE SECOND WAS SENIOR MANAGEMENT INVOLVEMENT. NOT ONLY SOMETIMES NOT EVEN IN THE GIVING OF RESOURCES BUT IN THE USE OF DATA FOR DECISION MAKING. IN ENSURING ACCOUNTABILITY FOR AMI CARE QUALITY. NUMBER OF MANIFESTATIONS OF SENIOR MANAGEMENT INVOLVEMENT. SELF ENGAGEMENT AND EXPERTISE, THIS, TOO, MANIFEST IN MULTIPLE WAYS. EMPOWERED NURSES, SOME FANTASTIC DATA AROUND THE EMPOWERMENT OF NURSES IN THE TOP PERFORMING HOSPITALS. INVOLVEMENT OF PHARMACISTS, DEEP INVOLVEMENT OF PHARMACISTS IN AMI CARE WAS ANOTHER MANIFESTATION. COORDINATION ACROSS GROUPS, THIS, TOO, WE SAW IN A NUMBER OF WAYS VALUING THE DIVERSE ROLES AND SKILLS ON A TEAM, LOOKING AT THE INTER-DEPENDENCIES WITHIN A TEAM. PAYING CAREFUL ATTENTION TO SMOOTH INFORMATION FLOW ACROSS GROUPS. PROBLEM SOLVING AND LEARNING, THIS NOTION THAT ADVERSE EVENTS ARE REALLY OPPORTUNITIES FOR LEARNING. IN A NONPUNITIVE ENVIRONMENT. I HEARD THE TERM PSYCHOLOGICAL SAFETY EARLIER TODAY. BEING ABLE TO WORK IN AN ENVIRONMENT TO LEARN FROM MISTAKES TO INNOVATE AND DO CREATIVE PROBLEM SOLVING. ALL QUALITIES OF TOP PERFORMING A HOSPITALS REALLY DIFFERENTIATED BETWEEN HIGHS AND LOWS. BUT LAST TEAM WAS RECURRENT THESE WERE NUMBER OF DESCRIPTIONS OF HOSPITAL PRACTICES AND PROTOCOLS FOR AMI CARE. ALL OF THE HOSPITALS THAT WE TALKED WITH DESCRIBES EVIDENCE-BASED PRACTICES THAT THEY WERE USING. AND THEY WEREN'T DIFFERENTIATED, RAPID RESPONSE TEAMS WE SAW IN THE HIGHS AND LOWS. ONE INTERPRETATION MAY BE THAT WE HAVE OF THIS IS THAT THESE PRACTICES AND PROTOCOLS MAY BE NECESSARY BUT NOT SUFFICIENT. THAT USING THESE PRACTICES IN THE ABSENCE OF A BROADER SUPPORTIVE ORGANIZATIONAL ENVIRONMENT OR CULTURE MAY NOT HELP TO A ACHIEVE TOP PERFORMANCE. TAKE I THROUGH THE DATA SOURCE AND COLLECTION PHASE. THIS IS A TABLE THAT IS ONE OF THE SUPPLEMENTAL TABLES FOR PAPER THAT JUST CAME OUT TUESDAY IN THE ANNALS OF INTERNAL MEDICINE WHICH IS REPORT OUT OF THE QUANTITATIVE DATA. THIS SHOWS EXACTLY INFORMED THE SURVEY ITEMS IN THE LAST HANDLE COLUMN. RIGHT HAND COLUMN JUST ONE ILLUSTRATIVE SURVEY ITEM. THERE WERE MULTIPLE ITEMS THAT MATCHED. JUST TO ILLUSTRATE TO LOOK AT THE TOP ROW FOR BROAD -- DOMAIN OF BROAD SELF EXPERTISE THIS IS A QUOTE, AT A HIGH PERFORMING MANAGER, I STARTED WRITING MY CONSULT KNOTS. OVER THE YEARS JUST BECOME STANDARD. THAT WAS WAY OF MY BREAKING IN TO THE -- I WANT TO YOU READ IT IT'S NOT IN THE NURSES SECTION. I HAVE SOME IDEAS AND I'M OPEN TO TALKING ABOUT THEM. ONE ITEM THAT WE DEVELOPED TO THIS IDEA WAS NURSES ARE COMFORTABLE IF THEY HAVE CONCERNS ABOUT PATIENT CARE. SO WE THINK ABOUT THE VERY TIGHT WEAVING FROM THE QUALITATIVE FIND TONGS QUANTITATIVE MEASUREMENT. NEXT WE THINK ABOUT INTEGRATION ANALYSIS PHASE. ABOUT FACTORS THAT -- MORTALITY RATES. THIS IS QUALITATIVE DATA WE DO TWO THINGS WITH THIS. THE FIRST REALLY -- THE FIRST IS A STAND ALONE CHARACTERIZATION OF THE ENVIRONMENTAL CONTEXT. THIS COULD ONLY BE DONE QUALITATIVELY. AND THAT PAPER ALSO REPORTED IN THE DESCRIBES THOSE HOSPITAL FEATURES. WE ALSO THOUGH MOVE ON TO DEVELOP AND TEST THESE HYPOTHESES. THAT WAS THE SURVEY. BUT THEN LASTLY, THIS WASN'T THE MOA EXCITING PART IS THIS INTEGRATION. WHERE WE CAN NOW DESCRIBE FACTORS THAT INFLUENCE RSMRs THE IMPACT ON RSMRs. BOTH QUALITATIVELY. SO, FOR INSTANCE, THIS DOMAIN OF CREATIVE PROBLEM SOLVING AND LEARNING WHICH EMERGED AS STATISTIC ASSOCIATED VARIABLE WITH RSMR. CAN DESCRIBE WHAT THAT IS WITH REAL EXAMPLES FROM THE HOSPITALS. A CATH LAB TECHNICIAN IN A HOSPITAL WHO INNOVATED AND ALL OF THE INTERVENTIONAL CARDIOLOGISTS TOOK UP THIS SMALL INNOVATION AND TRANSFORMED THE QUALITY OF THEIR CARE. THIS KIND OF PROCESS OF INTEGRATION IS DESCRIBED IN A A PAPER IN AMERICAN HEART JOURNAL. I ALSO SHARE I THINK STRUGGLES OF SOME OF US EXPERIENCE IN PRESENTATION AND PUBLICATION OF FINDINGS. SO WE THINK VERY HARD ABOUT HOW TO INTEGRATE DATE THE FROM THESE LARGE STUDIES WHICH NUMBER OF YOU HAVE DESCRIBED THESE LARGE STUDIES, MASSIVE AMOUNTS OF DATA. QUANTITATIVE AND QUALITATIVE HOW CAN WE POSSIBLY ACCOMPLISH INTEGRATION IN THIS WAY. AND A LITTLE POND THAT WE SWIM IN IS HEALTH SERVICES RESEARCH. AND BIOMEDICAL, THIS AREA OF LITERATURE. AND I HAVE TO SHARE THAT IT IS EXTRAORDINARILY DIFFICULT TO PUBLISH A MIXED METHOD STUDY IN THESE JOURNALS, THIS IS A A POND MIKE AND I KNOW WELL. 3,000 WORD LIMITS, NO QUESTION. BUT ON TOP OF THAT A LOT OF FAMILIARITY WITH THE METHODS AMONG THE LEADERS WITHIN THE JOURNAL, THE LACK OF ADULT TO HAVE QUALIFIED, CREDIBLE PEER REVIEW AND SPACE CONSTRAINTS. THERE ARE A NUMBER OF CHALLENGES THAT WE'RE HITTING UP AGAINST. I DON'T KNOW, I GUESS I WAS TRYING TO STAY OPTIMISTIC KNOWING PEOPLE HERE WHO HAVE ROLES OF PEER REVIEWERS AND ON EDITORIAL BOARDS, THIS IS THE NEXT WAVE TO MOVE THIS FORWARD BECAUSE WE CAN DO ALL THIS PHENOMENAL WORK. THIS GREATLY RIGOROUS THAT WE FEEL GOOD AND FRANKLY IF IT DOESN'T HIT THE PEER REVIEW LITERATURE WE ARE CONSTRAINED. AT LEAST THAT'S OUR ORBIT ANYWAY AND OUR EXPERIENCE. OUR WORK IS TYPICALLY PUBLISHED IN SERIES, IN STAGES. THIS IS NOT AS GRATIFYING AS IT COULD BE IF WE HAVE COMPILATIONFUL FULLY INTEGRATED PAPER. BUT WE HAVE HAD FANTASTIC EXPERIENCE IN THIS INSTANCE WITH THE ANNALS OF INTERNAL MEDICINE. THIS QUALITATIVE PAPER WAS PUBLISHED JUST ABOUT A YEAR AGO, THAT IS THE BASELINE REPORT OUT OF THE QUALITATIVE DATA. WE'RE NOW DOING MULTIPLE SUBANALYSES ON THAT. ONE LOOKING AT THE ROLE OF EMERGENCY MEDICINE SERVICES, ANOTHER LOOKING AT DISCHARGE PLANNING, THERE ARE LOTS MORE TO BE DONE ON THAT FRONT. SECOND PAPER IS ONE THAT I MENTIONED THAT WAS JUST RELEASED TUESDAY, THIS IS THE QUANTITATIVE STUDY, THIS IS THE RESULTS OF THE CROSS SECTIONAL SURVEY OF THE 537 HOSPITALS. I WANT TO LINGER HERE JUST A MOMENT. AGAIN AS INVITATION TO THOSE IN THESE LEADERSHIP ROLES, THAT WE HAD A VERY POSITIVE EXPERIENCE WITH THE EDITORIAL STAFF AND PEER REVIEWERS AT THE ANNALS OF IN II FALL MEDICINE, I'M EXTREMELY GRATEFUL IT REALLY PUSHED US VERY HARD, I GUESS I SHOULD SAY, GAVE US OPPORTUNITY, TO DO A LOT OF CROSSWALKING BETWEEN QUALITATIVE AND QUANTITATIVE DATA. TO BE VERY EXPLICIT ABOUT FULL SCOPE OF THE STUDY WHAT WE LEARNED IN THE FIRST PHASE, WHAT THEY PUBLISHED. AND THEN WHAT WE LEARNED IN THE SECOND PHASE HOW THOSE TWO THINGS WORKED TOGETHER. AND I AM GRATEFUL FOURNEAU KIND OF THOUGHTFULNESS THAT DELIBERATE ATTENTION THAT WAS FOCUSED ON BUILDING THE PAPER. SOME OF THAT WAS WOVENC PRIMARY PAPER THAT'S IN THE JOURNAL. THEN ALSO INVITED US TO SUBMIT A NUMBER OF ONLINE APPENDICES THAT DO MORE THAT HAVE KIND OF CROSSWALKING. BUT AS GREG AND I WERE TALKING ABOUT NOBODY READS THOSE. ANOTHER STRATEGY. WE CAN THINK ABOUT BUILDING THESE SUPPLEMENTARY FILES. IN ONE TALK EARLIER A REFERENCE TO SOMETHING BEING DONE IN APPENDIX. THAT'S ANOTHER WAY TO REPORT IT OUT. ANOTHER WAY TO INFILTRATE, ANNALS INVITED -- FROM THE INSTITUTE OF HEALTH CARE IMPROVEMENT. THE EDITORIAL POINTS BOTH TO THE QUALITATIVE AND QUANTITATIVE DATA THIS IS ANOTHER WAY TO MOVE THE FIELD ALONG BY RECOGNIZING THE IMPORTANCE OF MIXED METHODS. MAYBE EVEN THROUGH THESE EDITORIAL PIECES. I MENTIONED WE'RE HARD HAT WORK ON THIS NATIONAL QUALITY INITIATIVE. GOING TO BE LED BY THE AMERICAN COLLEGE OF CARDIOLOGY. RUN THROUGH PLATFORM OF NCDR WHICH IS THE NATIONAL CARDIOVASCULAR DATA REGISTRY. THERE IS SUBREGISTRY CALLED ACTION THAT S DATABASE REGISTRY THAT HELPS HOSPITALS WITH REALTIME DATA ON QUALITY IMPROVEMENT, IN THIS CASE AROUND AMI CARE. THE P.I. OF THE CAMPAIGN WILL BE HENRY FROM THE MAYO CLINIC. WE AT YALE ARE DOING THE EVALUATION WORK. WE'RE CURRENTLY ENGAGING OUR WIDE RANGE OF DIVERSE PARTNERS AND WE'RE VERY EXCITED TO HAVE, FOR EXAMPLE, THE NATIONAL PHARMACIST ASSOCIATION ON BOARD BECAUSE OF THE KEY ROLE OF FARM, EMS AND MERGE SEE MEDICINE THEY HAVE STROLL ROLE TO PLAY WE'RE REACHING OUT BUILDING THESE PARTNERSHIPS NOW FOR THE CAMPAIGN. TO ME WHAT'S NEXT EXCITING THING, ALTHOUGH CHALLENGING THAT THE CHANGE PACKET OR TOOL KIT, ADULT LIKE THOSE WORDS FOR THIS PURPOSE, WILL HAVE BOTH QUALITATIVE AND QUANTITATIVE DATA IN THEM. WE FOUND WITH THE FIRST PAPER A LOT OF QUESTIONS, WHAT DOES THAT MEAN, CREATIVE PROBLEM SOLVING. IT WILT BE VERY HELPFUL TO HAVE RICH VIGNETTES TO I AM STRAIGHT THE COMPONENTS OF THE MULTI-VARIABLE MODEL. THEN LASTLY BECAUSE WE CAN'T HELP OURSELVES WE'RE DOING MIXED METHODS EVALUATION OF THIS QUALITY INITIATIVE, THIS IS FUNDED BY THE COMMONWEALTH FOUNDATION AND DONOHUE FOUNDATION WILL BE LOOKING AT AS THE CAMPAIGN ROLLS OUT A PREPOST TEST LOOKING AT ADOPTION OF THESE FIVE RECOMMENDED EVIDENCE-BASED STRATEGIES, ONES THAT JUST CAME OUT IN THE ANNALS PAPER TUESDAY. THOSE HOSPITALS THAT HAVE GREAT SUCCESS IN ADOPTION AND UPTAKE OF THESE FIVE STRATEGIES WILL DO A DEEP DIG USING QUALITATIVE METHODS TO UNDERSTAND HOW. HOW THEY GOT THERE. WE'RE EXCITED ABOUT THIS NEXT PHASE. THAT'S ALL I HAVE TO SAY, I'M GOING TO STOP AND SAY, THANK YOU. I WANT TO THANK YOU FOR ALL THAT I'VE LEARNED TODAY. I'LL PAUSE THERE. [APPLAUSE] >> GOOD AFTERNOON, I'D LIKE TO ECHO MY APPRECIATION TO ALL OF YOU AND FOR OBSSR FOR SPONSORING THE WORKSHOP TODAY. I JUST FEEL LIKE LEARNING SO MUCH MYSELF IT'S REALLY GREAT TO BE HERE. ALSO ALWAYS FUN TO TALK ABOUT SOME OF THE WORK THAT WE'RE DOING AS WELL. I'M GOING TO TALK ABOUT MIXED METHODS INTEGRATION IN A A PROJECT WHICH IS EYE VALUE WAITING THE PROCESS OF ADAPTIVE CLINICAL TRIAL DEVELOPMENT. THIS CAME UP A LITTLE BIT EARLIER IN ONE OF THE QUESTIONS. I'M A PHYSICIAN, FAMILY PHYSICIAN BY BACKGROUND. AND SO I'M VERY INTERESTED IN PROBLEMS THAT OCCUR IN DAY-TO-DAY PATIENTS' LIFE AND TRAINED IN EPIDEMIOLOGY WAS VERY INTERESTED IN PREVENTIVE SERVICES AND EPIDEMIOLOGY. YET I'VE ALSO SPENT A LOT OF TIME IN INTERNATIONAL SETTINGS AND WORKING PARTICULARLY WITH JAPANESE PATIENTS AND CROSS CULTURAL ISSUES, THAT LED ME TO REAL INTEREST AND APPRECIATION FOR INFLUENCE OF CULTURE ON MEDICAL DECISION MAKING WHICH LED OF COURSE TO AN INTEREST IN QUALITATIVE RESEARCH. FOR A LONG TIME MY BACKGROUND HAS BEEN IN THINKING ABOUT ISSUES FROM A QUANTITATIVE AND EPIDEMIOLOGICAL PERSPECTIVE WHILE ALSO THINKING ABOUT THINGS FROM ANTHROA LOGICAL SPECIAL TECH I HAVE THAT LED ME TO THE VERY PASSIONATE ABOUT MIXED METHODS RESEARCH. I'M GOING TO START OUT WITH A LITTLE BIT OF BACKGROUND ABOUT THE PROBLEM WHERE IT MIGHT BE NECESSARY TO HAVE MIXED METHODS PROJECT. RANDOMIZED CONTROLLED INTERESTED BY MANY TO BE GOLD STANDARD OF EVIDENCE. LARGELY THIS IS WHAT NIH LIKES TO SUPPORT. ADAPTIVE CLINICAL TRIALS ARE A NEW TYPE OF TRIAL WHICH HAVE BEEN DEVELOPED AS INNOVATIVE ALTERNATIVE WHICH HAVE POTENTIAL BENEFITS IN EFFICIENCIES THAT ARE NOT SEEN IN TRADITIONAL KINDS OF TRIALS. WHAT THIS LEADS TO IS REALLY INTENSE DEBATE AND CONTROVERSY CURRENTLY AMONG TRIALISTS REGARDING THE ASSUMPTIONS AND VALIDITY THAT ARE BEING MADE IN THE CONDUCT OF THESE KINDS OF NEW ADAPTIVE CLINICAL TRIALS. THIS IS THE NAME OF THE OVERARCH KNOWLEDGE PROJECT THAT WE'RE WORKING ON CURRENTLY CALLED ADAPT IT, WHICH IS ADAPTIVE DESIGNS ACCELERATING PROMISING TRIALS INTO TREATMENT. YOU HAVE TO HAVE A GREAT ACRONYM. WE'RE GRATEFUL TO BE FUNDED THROUGH THE NIH COMMON FUND. AND NUMEROUS COLLABORATORS ON THE PROJECT ARE AT THE BOTTOM. THIS IS ACKNOWLEDGEMENT OF THE OTHER MEMBERS OF OUR TEAM INCLUDING THE PRINCIPAL INVESTIGATORS. AND THEN GROUP OF US AT THE UNIVERSITY OF MICHIGAN, CONSULTANTS AS WELL AS MEDICAL UNIVERSITY OF SOUTH CAROLINA. SO, OVERARCHING OBJECTIVE FOR THE PROJECT IS TO ELEVATE AND EXPLORE HOW BEST TO USE ADAPTIVE CLINICAL TRIAL DESIGNS TO IMPROVE THE EVALUATION OF DRUGS AND MEDICAL DEVICES AND TO USE MIXED METHODS TO CHARACTERIZE AND UNDERSTAND THE BELIEFS, OPINIONS AND CONCERNS DURING AND AFTER THE DEVELOPMENT PROCESS. IF YOU'RE INTERESTED IN MORE DETAILS OF THE VERY LARGE STUDY IT HAS BEEN RECENTLY PUBLISHED IN ANNALS OF EMERGENCY MEDICINE. JUST TO ORIENT YOU THERE ARE BASICALLY THREE SPECIFIC TASKS OF THE GRANT WHICH IS DESIGNED FOR CLINICAL TRIALS. IN FACT GROUP SO ENTHUSIASTIC WE DESIGNED FIVE. THESE ARE ADDRESSING STATUS OF THESE LISTED. THAT'S NOT WHAT I'M GOING TO TALK ABOUT TODAY. THAT'S ONE OF THE TASKS. SECOND TASK IS ACTUALLY LEARNING ABOUT THE PROCESS OF ACT DEVELOPMENT AND FOR THAT WE'RE USING VARIETY OF DATA COLLECTION STRATEGIES INCLUDING SURVEYS, FOCUS GROUPS, OBSERVATIONS AND KEY STAKEHOLDER INTERVIEWS. THAT'S GOING TO BE THE MAIN FOCUS OF MY PROJECT. THE THIRD COMPONENT OF THE PROJECT TASK IS TO EDUCATE CLINICIANS AND STATISTICIANS. WRONG WAY. THE LABORATORY FOR THIS WORK IS IN THE NEUROLOGICAL EMERGENCY TREATMENT TRIALS GROUP WHICH IS A NATIONAL LABORATORY WHICH IS FUNDED BY NIH, THE NATIONAL INSTITUTES OF NEUROLOGICAL DISORDERS AND STROKE. VARIETY OF SITES ACROSS THE COUNTRY. FUNDAMENTALLY IN THE ADAPTIVE TRIALS HAVE TO BE HIGH STAKES, TREATMENTS HAVE TO START IN AMBULANCE OR IN THE EMERGENCY DEPARTMENT AND IT IS AN OPEN NETWORK. BELIEVE IT OR NOT FIRST TRIAL FINISHED EARLY. JUST TO ILLUSTRATE WHY THIS IS SUCH A CONTROVERSIAL AREA, I WANT TO GO THROUGH WHAT THE ADAPTIVE PROCESS IS INVOLVED. ONE BEGINS PROJECT BY CONDUCTING THE DATA COLLECTION WITH INITIAL ALLOCATION AND SAMPLING ROLES, THERE IS ITERATIVE POINT WHEN DATA ARE ANALYZED. ONCE A CERTAIN AMOUNT OF DATA BECOME AVAILABLE AND THEN THERE'S AN ASSESSMENT MADE AS TO WHETHER OR NOT STOPPING ROLE HAS BEEN MET. IF IT HAS NOT BEEN MET THE ALLOCATION AND SAMPLING ROLES ARE CHANGED ACCORDING TO THE ADAPTIVE ALGORITHM. AT THAT POINT THEN THE NEW RULES DATA COLLECTION IS CONTINUED. THERE IS THEN ANOTHER SET POINT FOR ANALYZING AVAILABLE DATA. THIS PROCESS CAN GO ON THROUGH SEVERAL IT OPERATIONS UNTIL STOPPING RULE IS MET. WHEN THAT HAPPENS, ONE THEN CAN STOP THE TRIAL AND BEGIN THE NEXT PHASE IN A SEAMLESS DESIGN FOR THOSE TRIALS THAT ARE MOVING FORWARD. THIS IDEA OF ITERATIVELY BASICALLY CHANGING THE RANDOMIZED CONTROL TRIAL IS VERY CONTROVERSIAL AMONG TRIALISTS, TO IMPLEMENT THIS NEW TYPE OF TRIAL YOU NEED TO KNOW WHERE THE STAKEHOLDERS ARE FALLING IN ON THIS PROCESS. THAT LEADS US TO OUR MIXED METHODS AIM OF THIS PROJECT WHICH IS TO STOT AUTO THIS COLLABORATIVE PROCESS USING MIXED METHODS APPROACH. IT'S VERY LARGE PROJECT YOU'LL SEE IN A MOMENT IN THE INTEREST OF TIME AND DISCUSSION TODAY I'M GOING TO FOCUS ON ONE COMPONENT OF THAT WHICH IS BASELINE ASSESSMENT OF KEY STAKEHOLDER PERSPECTIVES ABOUT ADAPTIVE CLINICAL TRIALS INCLUDING ETHICAL ISSUES. FROM THAT WE'LL THEN GO THROUGH THE ISSUES OF INTEGRATION. FAMILY MANAGE AND ALSO RESEARCHER ON MULTIPLE MIXED METHODS PROJECTS AND AS MANY OF THE PRESENTERS TODAY, I FEEL LIKE I'M IN THE TRENCHES A LOT, IF I HAD ONE COMPLAINT ABOUT PRESENTATIONS TODAY EVERYTHING LOOKS SO CLEAN. THE REALITY OF THIS IS THAT IT'S NOT ALL SO CLEAN AND PRETTY AND EVERYONE ALLUDES TO THAT. BUT WHAT YOU SEE WHEN YOU COME TO THESE PRESENTATIONS OFTENTIMES IS SOMETHING LOOKS VERY POLISHED IT LOOK LIKE IT MUST HAVE BEEN SO EASY. I'M GOING TO FOCUS IN GREAT DETAIL ON THE ACTUAL STEPS THAT WE WERE USING IN THE INTEGRATION ITSELF TO FRY TO THINK ABOUT, WHAT'S THE BEST WAY THAT WE CAN CONVEY THIS INFORMATION THE WAY THAT PEOPLE UNDERSTAND. THIS IS A PROJECT OVERVIEW, I'M NOT GOING IN GREAT DETAIL OTHER THAN TO SAY THAT IT IS A THREE-YEAR PROJECT. THIS IS ILLUSTRATING WHAT THE TRIALS WERE, THIS IS IN THE PROPOSAL THERE. IS ACTUALLY A FIFTH WHICH IS NOT ON HERE. BUT THIS PROJECT INVOLVED SERIES OF FACE TO FACE MEETINGS, CONCEPT TELECONFERENCES, PERFORMANCE TELECONFERENCES, KEY INFORMANT INFER VIEWS AND GRANT REVIEW RESPONSE ANALYSIS. WE'RE GOING TO LOOK AT THE PINK SHEETS WHEN THEY COME BACK TO SAY WHAT IS IT THAT WE CAN LEARN FROM THAT AS WELL. IN THIS ANALYTIC PROCESS. THESE VARIOUS POINTS IN THE PROJECT ALL HAVE PROVIDED US OPPORTUNITIES TO COLLECT QUALITATIVE AND QUANTITATIVE DATA. THIS IS A VERY BUSY SLIDE, YOU'RE NOT GOING TO BE ABLE TO READ IT IN YOUR OWN NOTES, IT'S REALLY JUST TO SHOW YOU CONCEPTUALLY WHAT THE OVERALL PROJECT IS ABOUT AND AMOUNT OF DETAIL THAT WE'RE COLLECTING. THIS APPROACH ILLUSTRATING THE PURPOSE, THE APPROACH THAT'S BEING USED, DATA COLLECTION PROCEDURES AND EXPECTED OUT COME IS ACTUALLY A PROCESS THAT WE'VE HAD A LOT OF LUCK WITH IN AUTO LOT OF OUR GRANT PROPOSALS TO EXPLAIN THESE COMPLEX PROJECTS TO REVIEWERS. AND HELPFUL WHEN YOU'RE CONDUCTING THE PROJECT YOURSELF AND ALL THESE DIFFERENT MEMBERS OF DIFFERENT GROUPS TO SAY, THIS IS REALLY WHAT WE'RE TALKING ABOUT, WE'RE ALL ON ONE PAGE HERE. AMONG ALL OF THE COLLECTION PROCEDURES THIS IS ONE PIECE I'LL FOCUS ON TODAYM IS ON THE NEXT SLIDE. AND WE WERE USING IN THIS APPROACH A CONCURRENT DESIGN. WE'VE BEEN GOING OVER THIS EARLIER TODAY THAT THE QUAL EMPHASIZED THAT THIS IS MORE FOCUSED PROJECT ON THE QUALITATIVE PIECE AND THE QUANTITATIVE PIECE BEING A BIT SMALLER THAN THAT. THE PURPOSE OF THIS PARTICULAR COMPONENT IS TO UNDERSTAND CONCERNS AND STRATEGIES OF PERSONNEL PARTICIPATING IN THE FACE TO FACE MEETINGS PRIOR TO OUR DESIGN ACTIVITIES. THEN OUR DATA SOURCES WERE ATTITUDES ABOUT TRADITIONAL TRIALS AND ACTS. DESCRIPTION OF WHAT WE GOT FROM THIS DESCRIPTION OF THE ACTUAL DATA COLLECTION INSTRUMENTS WE USE SURVEY WITH 21 QUESTIONS. WITH QUANTITATIVE VISUAL ANALOG SCALES AS WELL THERE WERE QUALITATIVE SHORT ANSWER FIELDS, DEMOGRAPHIC QUESTIONS IN ADDITION TO THIS SURVEY WE ALSO CONDUCTED MANY FOCUS GROUPS AMONG STAKEHOLDERS AND THIS THEN PROVIDED FOR US QUALITATIVE DATA SUCH AS TRANSCRIPTS OF THE MINI FOCUS GROUPS AS WELL AS OBSERVATIONS BY THE FOCUS GROUP FACILITATORS. THESE TWO COMPONENTS ALL ADDRESSING THIS QUESTION LED TO THESE KIND OF RESULTS SO WE COULD LOOK AT ATTITUDES ABOUT ADAPTIVE CLINICAL TRIALS AND TRADITIONAL CLINICAL TRIALS AT THE VERY BASELINE OF OUR PROJECT. ALSO DETAILS REGARDING THEIR EXPERIENCES AND BELIEFS ABOUT ADAPTIVE CLINICAL TRIALS AND CLINICAL DESIGNS. WHAT WE SHOWED AT THE BOTTOM HERE IS EXAMPLE OF ONE OF THE 21 QUESTIONS WITH THE VISUAL ANALOG SCALES, ADAPTIVE CLINICAL TRIAL DESIGNS POSE ETHICAL QUESTIONS. THEN ONE WOULD MARK ON VISUAL ANALOG SCALE THE LEVEL OF THE AGREEMENT WITH THIS. THAT THEN GIVES US A QUANTITATIVE NUMBER THAT WE CAN USE, INDIVIDUALS ALSO GIVEN OPTION FOR QUALITATIVE EXPLANATION. THIS IS HOW WE ANALYZE. THIS IS GETTING IN TO THE NITTY-GRITTY DIRTY WORK OF WHAT ONE WOULD DO FOR ONE SPECIFIC COMPONENT. FIRST WITH THE QUALITATIVE ANALYTIC WE USE USUAL APPROACH WITH IMMERSION IN THE DATA. IDENTIFICATION OF EMERGING THEMES, DEVELOPING A CODING SCHEME, CONDUCTING THEMATIC SEARCHES. WE'RE USING AT LICK T.I. FOR. THIS THE QUANTITATIVE. THE MEANS, SCORES ON THE VISUAL ANALOG SCALES. WE DEVELOPED BOX PLOTS TO ILLUSTRATE DISTRIBUTION OF THOSE ATTITUDINAL SCORES. WHAT I'M GOING TO DO IS AIR MY DIRTY LAUNDRY. THIS IS US STRUGGLING THROUGH HOW WOULD WE PUT THIS DATA TOGETHER IN A WAY, HOW WOULD WE INTEGRATE IN WAY THAT REALLY LEVERAGES THE STRENGTH OF BOTH OF THOSE TYPES MUCH DATA. WHAT I WANT YOU TO FOCUS ON MOSTLY IS THE STRUCTURAL ISSUES HOW THE TABLE IS SET UP. WE'LL GO THROUGH FOUR TABLES THAT WE DEVELOPED -- I DON'T KNOW WHAT THAT NOISE IS COMING FROM. ACROSS THE TOP INITIALLY IS HE WED LET'S PUT THESE QUANTITATIVE SCORES IN FOR THE GROUPS THAT WE'RE INTERESTED IN. THERE WERE -- THIS IS SHOWING THE VISUAL ANALOG SCORES, MEAN ON THIS, FOR THIS GROUP, THE SCORE FOR ANOTHER GROUP OF STATISTICIANS, CLINICIANS AND OTHER PARTICIPANTS THEN BELOW THIS THE DATA THAT WAS RELEVANT TO THAT FROM THE QUALITATIVE DATA COLLECTION WERE ALL GROUPED TOGETHER. THIS WAS A LITTLE DENSE, DIDN'T REALLY SEEM LIKE WE WERE GETTING AS MUCH OUT OF IT. OUR NEXT STEP, MAJOR POINT OF THIS SLIDE TO SAY THAT WE THEN FOCUSED ON CONVERTING THOSE QUANTITATIVE DATA IN TO BOX PLOTS. BUT WHAT WE FOUND WITH THIS TYPE OF PRESENTATION THE HORIZONTAL PRESENTATION MADE IT STILL HARD FOR INDIVIDUALS TO REALLY UNDERSTAND THIS, HAVE QUALITATIVE DATA BELOW, AS YOU CAN SEE THERE ARE PARENTHESIS HERE ALLOWING US TO ILLUSTRATE WHERE THE ATTRIBUTIONS CAME FROM BUT ONE WOULD STILL HAVE SOME DIFFICULTY UNDERSTANDING THAT. THE NEXT ITERATION WAS TO MOVE THIS IN TO A VERTICAL FASHION WHERE NOW YOU CAN START GETTING A BETTER PERSPECTIVE ABOUT WHERE THE FOUR STAKEHOLDERS ARE FALLING OUT ON THESE POINTS AND THE QUALITATIVE DATA THAT GOES WITH THEM IS MATCHED UP TOGETHER. THEN THIS IS THE CURRENT DEVELOPMENT, AGAIN YOU CANNOT REALLY SEE THESE COLORS IN YOUR OWN SLIDES BECAUSE THEY'RE BLACK AND WHITE. BUT HERE WE'VE ACTUALLY INTEGRATED COLOR IN TO THE BOXPLOT PRESENTATION. THEN PURPOSE MATCHING UP WITH ANOTHER GROUP. SO NOW WE'VE GOT EVERYONE'S TEXT FROM ALL THE STAKEHOLDERS. THE PEACE THAT REALLY STANDS OUT ARE VERY -- TWO VERY IMPORTANT STAKEHOLDERS IN THE PROCESS OF DEVELOPMENT. PEOPLE BRINGING VERY DIFFERENT VIEWS. THE PROJECT IMPLICATION OF THE INTEGRATION THAT QUALITATIVE DATA DESCRIPTIVE DATA THAT WE WERE INTERESTED IN IN TERMS OF BREADTH OF INTEREST BUT THAT WAS SERIES THE QUANTITATIVE DEMONSTRATED THE GAP ESPECIALLY OF TWO KEY STAKEHOLDER GROUPS. THROUGH THIS REORGANIZATION AND SORT OF WASHING OF THE DIRTY LAUNDRY A MORE -- THE OUTCOME OF THIS WAS THROUGH OUR INTERIM ANALYSIS OF EVALUATION PROCESS THIS CONTRIBUTED TO THE ACTUAL IMPLEMENTATION OF AN INTERVENTION TO PROMOTE MORE DIALOGUE BETWEEN THE GROUPS THAT HAD THE WIDEST GAPS IN TERMS OF WHERE THEY WERE AT. THE MYTHOLOGY IT DOES PROVIDE INTEGRATION AND QUAL PLUS DESIGN. IT ILLUSTRATES ORGANIZATION OF QUANTITATIVE DATA IN A FORMAT CONDUES SIEVE FOR COMPARISON INTEGRATION, FOR EXAMPLE, USING SCORES IN THE BOX PLOTS. AND ALSO PRESENTS QUALITATIVE DATA IN FORMAT CONDUES SIEVE TO COMPARISON AND INTEGRATION. AGAIN WE HAVE THEMATIC ANALYSIS AND ORGANIZATION BY THE STAKEHOLDERS. THEN THE OTHER ISSUE I'M GOING TO STOP THERE AND LET THE NEXT SPEAKER GO. THANK YOU,. [APPLAUSE] >> THANKS FOR ORGANIZING FASCINATING CONFERENCE. I'M FROM COLUMBIA UNIVERSITY. I'M GIVING THIS PRESENTATION TOGETHER WITH PETER NEW MAN FROM UNIVERSITY OF TORONTO AN ILLUSTRATION OF A BROADER FLESH INTEGRATION OF THE RESEARCH TEAM. PETER AND I WILL TALK TO YOU ABOUT MIXED METHOD STUDY WE CONDUCTED OF PROJECT VIBE, IT'S A STUDY THAT AIMS TO PREPARE FOR FUTURE HIV VACCINES. HIV AND AIDS HAVE BEEN AND WILL CONTINUE TO BE FOR QUITE AWHILE THAT IS HARMING MANY PEOPLE. ONE OF THE BEST HOPES TO MITIGATE THAT PANDEMIC IS TO HAVE GOOD VACCINE. AND HOWEVER DESPITE MANY YEARS OF HARD WORK WE -- ANY VIABLE VACCINE BE AVAILABLE FOR IMPLEMENTATION. AT THE SAME TIME WHEN WE DO HAVE HOPEFULLY SOMEWHERE DOWN THE FUTURE THE VACCINES AVAILABLE, IT IS LIKELY ONLY TO BE PARTIALLY EFFICACIOUS. NOT GOING TO BE CLOSE TO 100% EFFICACIOUS. WE DO HAVE THE VACCINE READY FOR IMPLEMENTATION IN THE FUTURE, WE WILL FACE MONUMENTAL CHALLENGE. THE FIRST CHALLENGE IS COMMON TO ALL IMPLEMENTATION PROGRAMS. WHERE THEY HAVE ACCESS TO IT. AND LITERATURE FOR VACCINES TELL US THAT HAVING A GOOD VACCINE IS NOT A GUARANTEE THAT IT WILL BE IMPLEMENTED. AT THE SAME TIME THE HIV VACCINE IMPLEMENTATION ALSO POSES A BIG CHALLENGE OF RISK COMPENSATION BECAUSE OF THE PARTIAL EFFICACY. THERE'S A VERY REAL POSSIBILITY THAT SOME OF THE CONSUMERS WHO GET VACCINATED MIGHT BE MORE RELAXED ON THEIR PREVENTION BEHAVIOR. IN THE END, ANY BENEFIT OF THE VACCINE MIGHT BE MORE THAN COMPENSATED BY THE INCREASE RISK BEHAVIOR. MIGHT ACTUALLY BE ELEVATION IN THE TRANSMISSION RISK. EVEN THOSE CHALLENGED WE BELIEVE ARE VALUABLE HIV VACCINES ARE STILL YEARS AWAY FROM BEING READY FOR IMPLEMENTATION. THE TIME TO DEVELOP THE SUCCESSFUL VACCINE IS AN OPPORTUNITY TO DO THE PREPARATION WORK FOR FUTURE IMPLEMENTATION. WE CONDUCTED THIS PROJECT VIBE, VACCINE INTEREST AND BENEFIT EYE VALUATION. STUDY TO TRY TO PREPARE FOR THE FUTURE IMPLEMENTATION. IN THE SENSE OF PREPARING FOR THE IMPLEMENTATION IN THE FUTURE. I THINK THIS IS WIDELY -- LIKE IN CONSUMER PRODUCT DEVELOPMENT. THE DEVELOPERS CONDUCT PREPARATION, CONSUMER RESEARCH TO TRY TO UNDERSTAND WHAT IS THE MARKET LIKE THEN TRY TO DESIGN THE PRODUCT AND TRY TO DESIGN THE IMPLEMENTATION PROGRAM THAT WILL ACCOMMODATE WHAT IS IN THE MARKETPLACE. THIS STUDY IS A MIXED METHOD, CONSUMER RESEARCH STUDY AND WE ASK POTENTIAL CONSUMERS WHO MIGHT BE ELIGIBLE OR APPROPRIATE FOR HIV VACCINE IN THE FUTURE ABOUT THEIR ATTITUDES AND BARRIERS ABOUT HYPOTHETICAL HIV VACCINES. WE HAD THREE AIMS. THE FIRST ONE TO EXPLORE THE ATTRIBUTES OF HIV VACCINE LIKE HOW EFFICACIOUS IT IS AND WHAT ARE THE SIDE EFFECTS AND WHAT KIND OF -- TO UNDERSTAND WHAT IS IMPORTANT FOR THE CONSUMER POPULATION. THE SECOND AIM IS TO TRY TO ASSESS THE POTENTIAL IMPACT OF THOSE ATTRIBUTES AS PREDICTORS FOR VACCINE ACCEPTABILITY. THE THIRD AIM IS TO EXPLORE AND ASSESS THE RISK COMPENSATION IN RESPONSE TO THE UPTAKE OF VACCINES. THOUGH THE OVERALL OBJECTIVE FOR THE STUDY TO TRY TO INFORM THE PROPER DESIGN AND DEVELOPMENT OF PUBLIC HEALTH EDUCATION THAT WILL BE NEEDED UP FRONT TO HELP ENHANCE FUTURE IMPLEMENTATION. WE ATTENDED TO A NUMBER OF INTEGRATION ISSUES IN THE DESIGN FIRST PART IS TEAM WORK. STUDY IS LED BY ME, SOMETIMES CONSIDERED TO BE FROM THE DARK SIDE. AND PETER NEW MAN SOCIAL PSYCHOLOGIST WITH EXPERTISE IN MIXED METHODS AND QUALITATIVE RESEARCH. THEN INCLUDED DIVERSE RANGE OF EXPERTISE IN THE COLLABORATION. INDUSTRIAL ENGINEER WITH SPECIALIZATION OF METHODOLOGIES TO STUDY HEALTH SYSTEMS. ENGINEER IS VERY IMPORTANT WAY TO THINK ABOUT IMPLEMENTATION. BUT ALSO ENGINEERING HOW TO GET THINGS TO WORK AND WU WAS CONTRIBUTED TO OUR STUDY. SUNG-J ARCE LEE IS EPIDEMIOLOGIST AND KATHLEEN ROBERTS IS A GOOD SOCIOLOGIST. AND ABOUT STREET FIGURESS LATER TO MAKE SURE WE HAVE ENOUGH TIME. WE ALSO ATTENDED TO THE INTEGRATION IN THE STUDY PROTOCOL. WE IDENTIFIED DIVERSE RANGE OF METHODS. WE FOCUS ON TWO METHODS. FOCUS GROUP AND ANALYSIS AS TWO OTHER ELEMENTS OF THE PROJECT VIBE TO ELEVATE INTEGRATION OF THE QUALITATIVE AND QUANTITATIVE METHODS. PETER, YOU READY? >> OUR PROJECT LARGELY CAN BE CALLED AN EXPLORATORY SEQUENTIAL DESIGN IN TERMS OF REALLY SOME OF THIS LOOKS NEATER THAN IT IS, FRANKLY I DIDN'T SIT THERE WITH THIS MAP AT THE BEGINNING OF THE PROJECT. I THINK THE FACT THAT WE HAD JOHN ON OUR TEAM TO DISCUSS TO RETROACTIVELY SUPERIMPOSE A CLEARER CONCEPTUALIZATION FRANKLY OF EVEN WHAT WE DID IN AIRING SOME OF THE DIRTY LAUNDRY. BUT LARGELY THIS IS BY DESIGN. WE HAVE NO IDEA AT THE OUTSET WHAT THE FIELD OF ATTRIBUTES MIGHT BE IMPORTANT TO PEOPLE, PEOPLE AT HIGH RISK FOR HIV IN TERMS OF WHAT WOULD MAKE VACCINE -- WE USED FOCUS GROUPS AT THE BEGINNING TO EXPLORE A RANGE OF FACTORS WITH SEMI STRUCTURED INTERVIEW GUIDE. SOME OF WHICH ROSE IN THE GROUPS THAT WAS PART OF THE PROCESS. BUT WE USE THESE THEN TO BUILD, I'LL TALK A LITTLE BIT ABOUT CONJOINT ANALYSIS. BUT ROSE OUT OF DATA FROM THE FOCUS GROUP. BUT SECONDLY THERE WAS ASPECT THIS THAT WAS MIXED MESSAGE DESIGN. IT WAS INTEGRATION THEN IN THIS EARLY DESIGN BUT THEN INTEGRATION AT THE POINT OF DATA ANALYSIS AND INTERPRETATION. SO THAT THE CONCEPTUALIZATION WASN'T NEARLY SMALL QQUAL THEN GO TO BIG QQUAINT. WE HAVE TWO FIELDS OF DATA HOW DO THEY INTER-ACT WITH EACH OTHER. TALK ABOUT DIFFERENT WAYS THAT THE DATA COLLABORATED OR NOT ACROSS THE DIFFERENT PHASES. JUST BRIEFLY THIS IS MOST AT-RISK POPULATIONS IN L.A. COUNTY FOR HIV. WE USE VENUE-BASED SAMPLING OF TWO TO THREE OF EACH COMMUNITY HEALTH CENTERS THAT WORKED WITH SPANISH SPEAKING POPULATIONS. THOSE FOCUS GROUPS WERE CONDUCTED IN SPANISH. NEEDLE EXCHANGE PROGRAMS WHICH WAS WHOLE OTHER STORY OF PROGRAMS THAT ARE UNDER FEDERAL -- THEY COULD BE CLOSED AT ANY TIME. HAD TO STRIKE A NEW SAMPLING TIME EVERY CREEK BECAUSE SAME PROGRAMS WEREN'T EVEN THERE. YOU NEED A STATISTICIAN ENGAGED THROUGHOUT THE PROJECT. THAT'S JUST THE BEGINNING. LGBT CENTER, SOME PUBLIC STD CLINICS. THIS IS BY WAY OF SHOWING, YOUTH WHO HAD WORKED TOGETHER IN A PROGRAM WAS DONE TOGETHER. WE AUDIO TAPED THE GROUPS, MORE STANDARD WAYS, DATA WAS ANALYZED USING NARRATIVE, CONSTANT COMPARATIVE METHOD. OUR SOCIOLOGIST BROUGHT IN THIS ETHNOGRAPH SOFTWARE WHICH WE USED TO EXPLORE THE DATA. THEN WE MOVED ON TO THE SECOND PHASE OF CONJOINED ANALYSIS WHICH IS STATISTICAL SORT OF TECHNIQUE AND MOSTLY FROM THE DATA WE HAD AT THAT TIME WAS PUBLICATIONS IN MARKETING RESEARCH. SO, FOR EXAMPLE, WHEN GENERAL MOTORS DECIDES TO PUT OUT A NEW CAR, THEY'RE NOT GOING TO INVEST THE TENS OF THOUSANDS AND MORE DOLLARS TO BUILD PROTOTYPES THEN TO START MASS PRODUCTION WITHOUT HAVING SOME SENSE THAT AT LEAST SEGMENT OF THE MARKET IS GOING TO BE INTERESTED IN BUYING THAT PRODUCT. THEY MIGHT MODEL, IT'S A HYPOTHETICAL CAR AT THAT POINT IT DOESN'T EXIST YET. WELL WHAT ARE THE MILES PER GALLON YOU WANT, HOW FAST DO YOU WANT IT TO GO, HOW MANY PEOPLE DO YOU WANT TO GET IN THE CAR. SOME OF THOSE ARE NOT -- IF YOU ARE GOING TO TRADE OFF. ASK THE QUESTIONS ONE BY ONE, PROBABLY WE'D ALL SAY BEST MILES PER GALLON, CHEAPEST PRICE, I WANT A SPORTY CAR, ET CETERA. WE CAN'T MAX OUT EVERYTHING. THERE'S GO TO BE SOME GIVE PROBABLY IN MILES PER GALLON AND HOW FAST THE CAR WILL BE ABLE TO GO, ET CETERA. WE USE THIS PROCESS THEN TO LOOK AT WHAT ARE SOME OF THE TRADE OFFS THAT PEOPLE MIGHT HAVE TO MAKE IN DECIDING ON WHETHER OR NOT TO GET AN HIV VACCINE. WE LOOKED AT SEVEN DICHOTOMOUS HIV ATTRIBUTES. IT'S EASIER TO SEE THIS ON A COUPLE OF THE CARDS THAT SHOW THESE HYPOTHETICAL VACCINES. WHEN WE HAVE SEVEN DICHOTOMOUS ATTRIBUTES ACROSS EIGHT VACCINES THERE'S 128 DIFFERENT POSSIBILITIES AND PERMUTATIONS, I DON'T BELIEVE ANYBODY IN THIS ROOM, MYSELF CERTAINLY INCLUDED ARE GOING TO SIT THERE RANKING 128 DIFFERENT PATTERNS. WE TAKE OUT OF THAT TWO TO THE THIRD. SO EIGHT HYPOTHETICAL VACCINES USING AN ALGORITHM THAT REPRESENT THE 128 DIFFERENT POSSIBILITIES. THIS IS GIVING YOU AN EXAMPLE OF WHAT WAS PRESENTED TO PARTICIPANTS. SET OF EIGHT CARDS. 95% EFFECTIVE VERSUS 50%. PROTECT YOU FOR TEN YEARS VERSUS ONE YEAR. SOME INTEREST IN TERMS, WILL THIS PROTECT AGAINST DIFFERENT SUBTYPES OF HIV AROUND THE WORLD OR ONLY THOSE CIRCULATING IN THE U.S. BY INJECTION OR ORALLY, ALSO IF THEY'RE GOING TO BE THREE OR ONE DOSE. AT THE TIME WE WERE WORKING WITH VAC IS INOLOGIST WHO WAS INTERESTED -- GOING TO BE DEAL BREAKER FOR PEOPLE IS THAT GOING TO MAKE HIM SAY, NO. I'M JUST NOT GOING TO TRY THIS. WE MADE SURE TO MODEL THAT IN TO OUR DISCUSSION OF SIDE EFFECTS. THEN ALSO THE COST OF THE PRODUCT. THIS IS A PARTICIPANT, ACTUALLY ONE OF INVESTIGATORS, FIRST TASK WAS TOt HYPOTHETICAL VACCINES FROM BEST TO WORST. THEN SECOND TASK WAS PEOPLE GET BACK THE SAME CARDS AND A SKILL, I'M HIGHLY LIKELY TO GET THIS TO FIVE, HIGHLY UNLIKELY. THEY WOULD REORDER. OF COURSE THERE IS POSSIBILITY THAT SOMEBODY MAY SAY WELL, YOU KNOW WHAT, YES, THIS IS HOW I RANK THEM, FRANKLY I'M NOT SUCH HIGH RISK I'LL TRY ANY OF THESE. THEY COULD HAVE ALL BEEN IN ONE SET IN TERMS OF THE SCALE WE IMAGINE THIS MIGHT HELP US TO BREAK SOME OF THE TIES IN TERMS OF THE VACCINES. MOST PEOPLE THERE WAS SOME DISTRIBUTION. ACCEPTABILITY, DEN CANNED VARIABLE WAS TRANSFORMED ON TO 100 POINTED SCALE. AND THEN BY WAY OF EXPLANATION, FOR EXAMPLE IN THE SET OF EIGHT CARDS HOLDING ALL OTHER VALUES CONSTANT, ALL OTHER CHARACTERISTICS, FOUR VACCINES AND 95% EFFECTIVE AND 50%. BASICALLY JUST A NOVA FOR EACH PERSON ON EACH ATTRIBUTE THEN ANALYZED ACROSS EACH PARTICIPANT TO DERIVE IMPACT SCORE. THIS IS JUST AN OVERVIEW OF THE SAMPLE OF ROUGHLY IN THIS SAMPLE FOR SURVEY WITH THE ANALYSIS, HALF OF THE LGBT, GOOD ETHNIC DIVERSITY, STUDY IN LOS ANGELES, YOU WANT TO MAKE SURE YOU HAVE A GOOD CONTINGENT IN TERMS OF LATINO POPULATION THAT'S WHY ALSO MAKING THE FOCUS GROUPS ACCESSIBLE IN SPANISH ON SITE TO THE CLINICS WAS QUITE IMPORTANT. ALSO NOTABLY THAT OVER THIRD OF THESE PEOPLE HAVE NO HEALTH INSURANCE. TALKING ABOUT THE COST, MADE IT CLEAR THIS IS OUT OF POCKET COST TO YOU. THEN INTEGRATION IN TERMS OF LOOKING AT THE INTERPRETATION OF THE RESULTS FROM THESE TWO DIFFERENT METHODS, I'M GOING TO FOCUS ON EFFICACY, CROSS CLADE PROTECTION. BEING PROTECTED AGAINST DIFFERENT TYPES OF HIV AND INTEREST OF POSSIBLE INCREASE IN RISK BEHAVIOR. BY AM COMPEL FROM THE FOCUS GROUPS, PRETTY TOP A FAULT LIKE ACROSS EVERY FOCUS GROUP IT WAS PRETTY CLEAR THERE WAS EXPECTATION THAT VACCINE MEANS IT'S 100% EFFECTIVE. THAT'S JUST WHAT VACCINES ARE. MUCH COURSE WE EXPECT THAT, IF THIS ISN'T 100% I'M NOT TAKING IT. WITH CROSS-CLADE PROTECTION THERE WAS MORE VARIABILITY. FEW OF THE GROUPS TOOK THAT UP REALLY WENT WITH IT TALKED HOW THAT WAS QUITE IMPORTANT. WE HAD A SIZABLE PORTION OF THE SAMPLE WERE NOT EVEN BORN IN THE U.S., MADE SOME SENSE BUT IT DIDN'T APPEAR ACROSS ALL THE GROUPS. THEN IN CONJOINED ANALYSIS WHEN WE ANALYZED THE RESULTS FROM THIS QUANTITATIVE TASK, IT VERIFIED VERY CLEARLY TO US, THIS IS THE IMPACT ON 100 POINT SCALE OF EFFICACY FROM 50 TO 95%. EFFECTIVELY HOLDING ALL OTHER VALUES CONSTANT IF WE HAD 40% -- AROUND 40 OUT OF 100 POINT SCALE ACCEPTABILITY OF 50% EFFICACY VACCINE, WHEN IT CHANGES TO 95% THAT'S 63 OUT 1060 POINT SCALE. IT'S A VERY, VERY SIZABLE AFFECT IN TERMS OF ACCEPTABILITY OF THIS VACCINE. AND WE HAD HEARD OF LESS CLADE PROTECTION BUT IT WAS MORE IMPORTANT THAN WE HAD THOUGHT IN TERMS OF LOOKING ACROSS PARTICIPANTS. LASTLY IN TERMS OF FOCUS DRUPES THEY RUN WITH THIS IDEA. TALK ABOUT HOW PEOPLE MIGHT CHANGE THEIR BEHAVIORS THEY SAY, MORE THAN HALF OF MY FRIENDS, PEOPLE IN MY PEER GROUP WILL INCREASE THEIR RISK BEHAVIOR. PEOPLE ARE GOING TO GET LOOSE AND FREE. WE GAVE PEOPLE PIPE THETICAL VACCINE WHAT WOULD YOU DO, WHAT MIGHT YOU DO. WE LOOKED AT DIFFERENT OUTCOMES. 10% SAID THEY WOULD DECREASE CONDEMN USE. ABOUT 15% TALKED ABOUT POTENTIALLY INCREASING THE NUMBER OF PARTNERS. JUST BY BRIEFLY IN TERMS OF THE INTEGRATION, THE FOCUS GROUPS AND CONJOINT ANALYSIS CORROBORATED EACH OTHER ON THE EFFICACY. IN FACT QUANTITATIVE GAVE US A CLEAR PICTURE HOW IMPORTANT EFFICACY WAS IN TERMS OF ACCEPTABILITY OF VACCINE. MORE IMPORTANTLY THAN WE MIGHT THINKvq@ THAT COULD BE DUE TO FATALISM, BUT I'M GOING TO KEEP DOING WHAT I'M DOING WE WANTED TO UNDERSTAND WHAT IS THE MECHANISM OF THAT AND FOCUS GROUPS HELPED US TO GET BEHIND IT, PEOPLE WERE TALKING ABOUT WHAT SOUNDED LIKE COMPENSATION. ONE DOESN'T WANT TO USE FOCUS GROUPS TO ESTIMATE MAGNITUDE. IT WAS HELPFUL TO HAVE THE TWO TOGETHER IN THE DESIGN. AS OTHER SPEAKER HAVE TALKED ABOUT, EXPECTATION OF UNIFORMITY, DOING MIXED METHODS STUDY THIS IS STANDARD OF SUCCESS CAN BE VERY PROBLEMATIC. IT'S HELPFUL TO BE OPEN TO THE FACT THAT YES, THERE ARE DIFFERENT QUESTIONS SOMETIMES WE LEAVE WITH MORE QUESTIONS ABOUT CERTAIN VARIABLES THAN IF I HAD JUST HAD MY QUANTITATIVE DATA. MAYBE I'D RATHER NOT KNOW THIS DIRTY LITTLE FACT UNDERNEATH. IT DRIVES YOU TO MOVE TO THE NEXT STAGE DO MORE RESEARCH ON IT. LARGELY HELPED US VERY MUCH TO UNDERSTAND THE PHENOMENON SEVERAL PAPERS EMERGED FROM THIS. >> THANK YOU. APPRECIATE THE PUBLICATIONS. THIS WAS A PILOT STUDY I THINK WE HAD TOTAL OF 90,000 FUNDING WE HAD MORE THAN TEN PAPERS PUBLISHED. I'D LIKE TO GIVE A SIDE BAR ON INTEGRATION. I THOUGHT MOVE THIS TO THE END SO WE DON'T BREAK UP THE FLOW OF THE PROJECT VIBE. THIS ECHOES WITH THE COMMENT KATHLEEN MADE EARLIER THAT IT WAS CHALLENGING TO FIND ABLE TO COMMUNICATE WITH QUALITY RESEARCHERS. I HAVE WORKED AND ENJOYED MIXED METHODS RESEARCH. I TALKED TO MY STATISTICIAN COLLEAGUES ENCOURAGED THEM GET INVOLVED IN THIS EXCITING NEW AREA OF RESEARCH. THE VAST MAJORITY OF MY STATISTICIAN COLLEAGUES HAVE NEVER HEARD ABOUT MIXED METHODS RESEARCH. I GUESS THERE ARE LOTS OF BARRIERS TO OVERCOME. I WOULD LIKE TO ENCOURAGE ALL OF US TO THINK ABOUT MAYBE AN EFFORT. TO TRY TO HELP US MAYBE ENGAGE STATISTICIANS TO PARTICIPATE IN MIXED METHODS OF RESEARCH LIKE MAYBE GIVE PRESENTATIONS OF MIXED METHODS STUDIES. THE POTENTIAL PAY OFF FOR THE OUTREACH FOR THE FIELD COULD BE VERY VALUABLE. STATISTICIANS ARE NOT ONLY CAN BE VERY HELPFUL IN TERMS OF ANALYZING MODELING OF THE DATA BUT ALSO IN CONTRIBUTING STUDY DESIGN AND IMPLEMENTATION. MOST IMPORTANTLY I THINK TO HELP WITH THEIR ANALYTIC SKILL AND THINKING TO HELP FORMULATE THE STUDY AIMS AND STUDY PROTOCOLS. THAT'S MY SCHPEEL. [APPLAUSE] >> HAVE ALL THE PANELISTS COME ON UP. THEY WILL BE PREPARED TO ANSWER SOME QUESTIONS HERE. MEANWHILE, WHILE THEY'RE COMING UP I'LL TELL YOU THAT I'VE BEEN DRAFTING JOURNAL ARTICLE WHILE THEY HAVE BEEN SPEAKING. THE TITLE OF THIS ARTICLE WILL BE SOMETHING LIKE "MULTIPLE PERSPECTIVES ON INTEGRATION AND MIXED METHODS" IN THE IMPLEMENTATION RESEARCH. I WOULD START WITH SOME OF THOSE WAYS OF INTEGRATION THAT I MENTIONED, MIXING METHODS WITH DESIGNS, PHILOSOPHIES, RESEARCH QUESTIONS. MIXING IN DATA COLLECTION PHASE. MIXING IN DATA ANALYSIS. MIXING IN DATA INTERPRETATION. THEN I WOULD ADD SOME OF THE INNOVATIVE PROTOCOLS THAT EMERGED. LESLIE TALKED ABOUT HOW TO INTEGRATE THROUGH THE PUBLICATION PROCESS. HOW YOU MIGHT EVEN SEPARATE YOUR QUANTITATIVE PAPER FROM YOUR QUALITATIVE PAPER FROM MORE MIXED METHODS PAPER. AND MIKE, YOU RECALL THAT TABLE THAT HE HAD THAT HAD PURPOSE, APPROACH AND DATA COLLECTION. YOU'RE INTEGRATING IN A PLANNING DOCUMENT HOW TO LAYOUT YOUR STUDY. I THOUGHT THAT WAS VERY INTERESTING. THEN OUR LAST TWO SPEAKERS INTRODUCED YET ANOTHER INTERESTING INTEGRATION COMPONENT, THEY TALKED ABOUT INTEGRATING IN TERMS MUCH THE COMPOSITION OF THE TEAM AND MORE SPECIFICALLY HOW THEY WERE INTEGRATING DIFFERENT STATISTICIANS WITH DIFFERENT PERSPECTIVES. NOW I'M LEFT TO THINK ABOUT INTEGRATION WAY BEYOND RESEARCH PROCESS TO MORE OF EVEN THE BROADER CONTEXT IN WHICH INQUIRY OCCURS. WE ALL NEED TO WRITE THIS PAPER ON MULTIPLE PERSPECTIVES ON INTEGRATION. GOING WAY BEYOND THE METHODS TO OUTLINE ALL THESE DIFFERENT POSSIBILITIES. WHAT QUESTIONS DO YOU HAVE? >> THAT WAS GREAT. JOE GALLOW FROM HOPKINS. I'M INTERESTED IN THE CONJOIN. THAT IS REALLY INTERESTING BECAUSE IT'S AN INHERENTLY MIXED PROCEDURE. BECAUSE YOU HAVE THE CONJOIN WHICH IS A VERY COMPLEX STATISTICAL MODEL. YOU HAVE TO UNDERSTAND WHAT ATTRIBUTES ARE IMPORTANT TO PEOPLE FOIN CORPORATE. WHAT I DIDN'T HEAR WAS, JUST WAS WONDERING, I OFTEN THINK OF QUALITATIVE COMPONENT ON THE OTHER END, SO, FOR EXAMPLE, WHEN PEOPLE ARE DOING THE SORTING THEY'RE TALKING TO THEMSELVES OR TO THE INTERVIEWER, THE PERSON WHO IS HELPING THEM AND THAT BECOMES DATA THAT THEN HELPS YOU UNDERSTAND WHAT YOU'RE FINDING. I WAS JUST WONDERING ABOUT THAT. >> THANK YOU FOR THAT QUESTION. I DO AGREE BECAUSE YOU CAN DO THIS VERY HIGH FALLUTIN STATISTICS, BUILT WHAT ASPECTS DO YOU PUT IN. ACTUALLY WHAT WE DID DO, THIS WAS SMALL SCALE STUDY WE HAD QUAINT TAKETIVE COMPONENT WHERE WE ASKED PEOPLE AFTER THE CONJOIN ANALYSIS WE HAD INITIALLY BEEN TOLD YOU CAN'T DO THIS, PEOPLE AREN'T GOING TO UNDERSTAND IT AND BY GOD, LOW INCOME PEOPLE IN LOS ANGELES AND ETHNIC MINORITIES, YOU KNOW THE REST OF THE STORY ARE NOT GOING TO BE ABLE TO DO THIS. HOW EASY OR DIFFICULT THAT WAS FOR YOU, WE KNEW HOW LONG IT HAD TAKEN THEM. BY AND LARGE WE FOUND THAT WAS MOST INTERESTING PART OF THE STUDY FOR THE PARTICIPANTS BECAUSE IT IS RESPONDING TO QUESTION AFTER QUESTION THEY GOT TO ACTIVELY ENGAGE. [ NOT AUDIBLE ] >> I THINK THAT IS A GOOD POINT THAT COULD BE USED IN A STUDY. I HAVE DONE ONE STUDY WHERE I IMPLEMENTED CONJOINED IN A FOCUS GROUP. THAT WENT UP AND DOWN. PEOPLE WERE TOO CLOSE TOGETHER SEEING WHAT THE OTHERS WERE DOING AND THE LIKE. BUT WE DID GET VERY INTERESTING DATA. THANK YOU FOR THAT. >> JUST TO MENTION PEOPLE ALREADY SAID IT THAT ATTRIBUTES THAT WE SELECTED FOR THE CONJOIN ANALYSIS WAS DERIVED FROM THE PREVIOUS STUDY IN SEQUENTIAL. THAT WAS VERY IMPORTANT. >> I THINK YOU'RE ALLUDING ONE NAME IS TRANSPARENCY. HAVE PEOPLE BECOME TRANSPARENT WITH THEIR THOUGHTS WHILE THEY'RE DOING A PROCESS LIKE THAT. THE TECHNIQUE THAT YOU CAN USE IS VERY EFFECTIVE, WITH JUST A LITTLE BIT OF EXPLANATION PEOPLE CAN GET IT VERY QUICKLY AND THINK OUT LOUD REALTIME YOU CAN RECORD, DON'T EVEN HAVE TO ASK THEM. IT WORKS VERY WELL. >> HI, I'M KATIE FROM UNIVERSITY OF PENN. COULD THE PRESENTERS AND ANY EDITORS IN THE ROOM PARTICULARLY FROM HEALTH JOURNALS DISCUSS AT WHAT POINT WE SHOULD DEAL WITH THE WORD ISSUE. WE RECENTLY SUBMITTED TRYING TO KEEP TO THAT WORD LENGTH, GET REVIEWS BACK THAT THERE WASN'T ENOUGH DEPTH. WE HAD THE DEPTH JUST COULDN'T PUT IT IN THAT AMOUNT OF WORDS. SHOULD WE BE WRITING EDITORS BEFORE WE EVEN START THE WRITING PROCESSOR WHAT IS YOUR EXPERIENCE. >> THAT'S REALLY GOOD QUESTION. GIVES ME AN OPPORTUNITY PUT IN A PLUG FOR THE JOURNAL OF MIXED METHODS RESEARCH. I'M ONE OF THE EDITORS, ASSOCIATE EDITORS FOR THE JOURNAL OF MIXED METHODS RESEARCH. WHAT I LIKE TO ENCOURAGE PEOPLE TO SUBMIT THEIR METHODS TO THE JOURNAL, WORD LIMIT IS 10,000 WORDS. WE'RE SPECIFICALLY LOOKING FOR METHODS ALSO PREMIERE JOURNAL IN THE FIELD AND IF YOU PUBLISHED YOUR MIXED METHOD IN THE JOURNAL YOU CAN THEN CITE THAT IN YOUR HEALTH SERVICES JOURNAL ONLY JOURNALS ARE OUTSIDE OF OUR OWN. ARE OFTEN 2,000 WORDS OR 25000 WORDS. ONE OF THE REAL RISKS WHEN YOU TRY TO PUBLISH IS THAT SOMEONE WHO IS PRIMARILY QUALITATIVE PERSON, THERE'S NOT ENOUGH DETAIL HERE. I CAN'T UNDERSTAND. THIS THE STATISTICIAN READS SAME PAPER, THERE'S NOT ENOUGH DETAILS HERE. I DON'T UNDERSTAND WHAT YOU DID. BY HAVING YOUR METHODS PUBLISHED ELSEWHERE, CREATES PUBLICATION. BUT YOU CAN USUALLY WRITE OFF YOUR DISSERTATION OR RESEARCH AND INCLUDE YOUR TABLES PUT ALL THAT INFORMATION OUT THERE. THEN CITE YOURSELF AND SAY FURTHER DETAILS CAN BE FOUND IN THE JOURNAL. AS FAMILY PHYSICIAN I'M MOST FAMILIAR WITHIN MY FIELD, FOR EXAMPLE, FOR THE ANNALS OF FAMILY MEDICINE AND VERY INTERESTED IN METHODOLOGY PAPERS AS IS FAMILY MEDICINE. THOSE ARE SOME OPTIONS AT LEAST WITHIN FAMILY MEDICINE. IF YOU LOOK WITHIN WHATEVER FIELD YOU'RE WORKING IN EXPECT YOU CAN FINAL JOURNALS WHICH WILL TAKE METHODOLOGY PAPERS. PEOPLE ARE VERY INTERESTED IN MIXED METHODS. ALSO OPTION FOR JOURNAL MIXED METHODS RESEARCH. >> GREG, WOULD YOU LIKE TO SAY FEW WORDS ABOUT -- GREG? [ NOT AUDIBLE ] >> JUST ONE OTHER STRATEGY FROM LIVING IN THIS ORBIT TRYING TO FIGURE OUT HOW TO PUBLISH IN THESE JOURNALS WHERE IT'S ABSOLUTELY 3,000 WORDS NOT SYLLABLE MORE. THIS IS ONE SET OF STRATEGIES, CARVING METHODS OUT AND PUBLISHING ELSEWHERE. WE HAVE HAD MICKED SUCCESS, IT DEPENDS ON THE RELATIONSHIP WITH THE JOURNAL IN PRODUCING IN THE COVER LETTER THIS IS AN -- WE'VE WRITTEN AS PRECISELY AS POSSIBLE BUILT IT EXCEEDS BY 750 WORDS, BLAH, BLAH, THAT'S ONE WAY THAT USUALLY DOESN'T GO VERY FAR. WHAT WE FIND WITH THE REVISED, RESUBMIT PROCESS REVISION LETTERS ARE TWICE AS LONG AS THE MANUSCRIPT AS AN OPPORTUNITY, AN EDUCATIONAL OPPORTUNITY. SOME OF THOSE CAN BE PERSUASIVE DOCUMENTS. IN THE END ULTIMATELY, FOR INSTANCE, WITH THE MOST RECENT SET OF PUBLICATIONS WE WERE GIVEN THE OPPORTUNITY TO BUILD THE WEB ONLY APPENDICES. GOT A LITTLE BIT MORE LIMITED IN THE MAIN PAPER. COUPLE OF DIFFERENT STRATEGIES. I FIND THIS ONE OF THE HARDEST THINGS ABOUT DOING THIS WORK. >> THANK YOU VERY MUCH. I'M A DENTIST AND ORAL HEALTH RESEARCHER FROM USCF AND ALSO WITH INDIAN HEALTH SERVICE. I GUESS SOME OF THE THINGS THAT SORT A LITTLE FURTHER BEHIND WITH GENERAL MEDICINE RESEARCH AND SOME OF MY QUESTIONS HAVE TO DO WITH BEFORE YOU EVEN GET TO DISSEMINATION AND IMPLEMENTATION HOW DO YOU, LET'S SAY, TALK TO YOUR IRB. YOUR COMMUNITY -- HOW DO YOU TRANSLATE IN TO THE VERNACULAR. >> IN OUR WORLD, YES, WE HAVE MULTIPLE IRBs AT YALE WE HAVE FICK SAKES WITH IRBs. SOME ARE MORE RECEPTIVE. MID SCHOOL WE HAVE TRAINED IT'S TAKEN US EIGHT YEARS THEY'RE OKAY. EVEN GETTING FAMILIAR WITH THE MIXED METHODS. WE FRANKLY I SHOULD BE SAY END RUN. WE'LL SUBMIT WE DID PHOTO VOICE PROJECT THAT WE WERE TALKING ABOUT EARLIER. WHICH WAS VERY FOREIGN TO THE MED SCHOOL IRB THEY IN FACT SUGGESTED OUR CONTACT SAY WE PUT IT THROUGH GRADUATE SCHOOL. YOU MAY HAVE MULTIPLE OPPORTUNITIES TO FIND IRBs THAT ARE MORE OR LESS FRIENDLY, YOU FEEL LIKE SYS FOOS WITH IRBs AND EDUCATION THAT HAS TO HAPPEN AND REASSURANCE. >> I WAS IMPRESSED WITH YOUR 91% SURVEY RESPONSE RATE FROM LARGE HOSPITAL SYSTEMS. THAT WAS REALLY WHERE THIS QUESTION STARTED AT THE BEGINNING OF THIS SESSION. HOW DO YOU SORT OF REVEAL TO THE GROUPS OR WHAT POINT DO YOU ENGAGE IN THAT. OBVIOUSLY THE IRB BUT EVEN ON INFORMED CONSENT, DO YOU HAVE DIFFERENT INFORMED CONSENT THAT YOU USE FOR THE DIFFERENT PHASES OF THE PROJECT. AND JUST FUNNY BECAUSE I LONG SUSPECTED SOME OF THE PROJECTS I'VE BEEN INVOLVED WITH WERE MIXED METHODS. I THINK AFTER TODAY THEY REALLY WERE. IT MAKES ME REALLY START TO THINK ABOUT HOW I CAN TAKE MY OWN EXPERIENCE, PART OF IT ESPECIALLY WORKING WITH NATIVE AMERICANS IN ALASKA INDIANS AND ALASKA NATIVES IS GETTING PAST SOME OF THEIR TRIBAL CONCERNS, THEY'RE TRIBBLE ENTITIES IN TERMS OF, YOU DO HAVE TO DESCRIBE WHAT KIND OF RESEARCH YOU'RE GOING TO DO HOW IT WILL IMPACT THE PEOPLE. I THINK IT WOULD SPEAK TO THEM IN MANY WAYS THAT THERE IS -- I GUESS THAT WAS PART OF MY THOUGHT. >> CAN I ADD ONE THING. I WOULD JUST SAY YOU NEED TO SUBMIT YOUR PROJECTS ITERATIVELY, THAT'S THE WAY IT'S WORKED OUT. IF YOU'RE LEARNING SOMETHING IN ONE PHASE YOU'RE DESIGNING NEW INSTRUMENTS FROM IT YOU CAN'T DEVELOP ALL OF YOUR INSTRUMENTS AT THE SAME TIME. STARTING OUT VERY BASICALLY DEVELOPING THE FIRST PHASE, WHAT IS GOING TO BE THE COMPLETE PHASE KNOWING WHAT THAT IS GOING TO INVOLVE BEING VERY EXPRESS IT IS IMPORTANT. YOU HAVE TO -- ESTIMATE MAXIMUM NUMBER. USUALLY QUALITATIVE RESEARCHERS, YOU TRY TO SET THAT NUMBER AT THE MAX SO THAT YOU CAN FALL SHORT OVER RECRUITING IS BAD NEWS. BECAUSE THE MINDSET AND LOT OF MAJOR INSTITUTION FALL SETTINGS, DID YOU OVER RECRUIT FOR SOMEONE IN A TRIAL AND THEN IF THAT IS THE CASE THEN SOMEONE WAS DEPRIVED OF A TREATMENT. SO IT'S VERY DIFFERENT MENTALITY FROM QUALITATIVE RESEARCH WHERE YOU MIGHT SAY WE JUST NEED ENOUGH IS ENOUGH YOU SAY, WE JUST WENT OVER BY TWO PEOPLE BECAUSE WE FOUND THIS REALLY INTERESTING THAT CREATES SIGNIFICANT PROBLEMS FOR IRBs. YOU ALSO WANT TO FOCUS YOUR STUDIES WHEN IT'S POSSIBLE TO SAY THIS THIS IS REALLY A LOW RISK STUDY. BECAUSE IT'S AMAZING SOME TIMES REVIEWERS THAT ARE USED TO RANDOMIZED CONTROLLED TRIALS WITH VERY DANGEROUS CHEMO THERAPEUTIC AGENTS WILL SAY, THIS STUDY, I DON'T GET IT. I DON'T DO THIS SORT OF STUFF, I'M NOT A TRIALIST. THIS STUFF FOUND REALLY COMPLICATED. KEEPING A FOCUS IN THE APPLICATION UP FRONT IF IT IS LOW RISK RESEARCH TO SAY, THIS IS REALLY LOW RISK RESEARCH MAY NOT BE FAMILIAR TO THE REVIEWERS. ALSO YOU WANT TO MAKE SURE YOU CONCLUDE -- INCLUDE A COVER LETTER EXPLAINING SORT OF THE BASICS OF IT. IF YOU HAVE SOME DIAGRAMS THAT CAN HELP THE REVIEWERS AND FOCUS THEM IN THE APPLICATION, THIS IS WHAT WILL BE MOST INTEREST AND WALK THEM THROUGH IT. THEY WANT TO HELP YOU, THEY'RE OFTENTIMES OVERWHELMED WITH VERY COMPLICATED MIXED METHODS PROJECT. [ NOT AUDIBLE ] >> KAISER CENTER FOR -- >> FOR QUESTION ABOUT -- BUT ALSO VERY IMPORTANT QUESTION WHETHER DISSEMINATION IMPLEMENTATION IS RESEARCH OR QUALITY IMPROVEMENT. RESEARCH THAT REQUIRES THE USUAL IRB PROCEDURE IS ACTIVITIES THAT AIM TO PRODUCE GENERALIZABLE KNOWLEDGE. WE STUDY SOME PATIENTS IN THE CLINICAL TRIAL THEN LEARN SOMETHING AND APPLY THAT TO FUTURE PATIENTS. THOSE PATIENTS ARE PAYING ETHICAL PRICE FOR THE SOCIETY TO BENEFIT OTHERS. THERE IS TREMENDOUS AMOUNT OF PROTECTION WE NEED TO MAKE SURE THEY'RE NOT. IMPLEMENTATION STUDIES ARE REALLY QUALITY PROVING IN NATURE ARE HELPING HOSPITAL OR NETWORK TO TRY TO IMPROVE SERVICES TO THE PATIENT. THERE IS A RATIONAL TO APPLY AS ONE EXCELS FROM THE IRB. THIS IS NOT HUMAN SUBJECT RESEARCH. THIS IS QUALITY IMPROVEMENT. THESE ARE VERY CONSERVATIVE. MIGHT NOT BE EASY TO GIVE EXEMPTION BUT SOME SITUATIONS THAT MIGHT BE SUCCESSFUL. IT USED TO BE THAT MOST IRBs INTERPRET IF YOU PUBLISH THAT IS CONSIDERED GENERALIZABLE KNOWLEDGE SO YOU DO QUALIFY FOR THE EXEMPTION. IT BREAKS THROUGH WHAT IS MADE -- THE V.A. I THINK DETERMINED THAT JUST PUBLICATION DOES NOT NECESSARILY MEAN THIS IS NOT QUALITY IMPROVEMENT. IN A SENSE THEY ARE ALLOWING QUALITY IMPROVEMENT STUDIES TO PUBLISH THEIR FINDINGS FOR OTHER USE. THAT'S OPTION. THOSE OF US WHO WORK IN DISSEMINATION AND IMPLEMENTATION SHOULD LOOK FOR AND MAYBE TRY TO WORK WITH IRBs TO TRY TO HELP THEM APPRECIATE THAT THIS STREET REALLY VERY DIFFERENT FROM TRADITIONAL TYPE CLINICAL TRIAL. >> I'LL MAKE IT QUICK. I HATE TO DO THIS BUT I WANT TO GO BACK TO THE PUBLICATION ISSUE PARDON ME? [ NOT AUDIBLE ] I'LL WAIT. >> NOW WE HAVE BREAK SCHEDULED. 20 OF A A TORE 3:00. WE'LL RETURN AT 3:20. >> VERY GOOD.lp >> I THINK WE'RE GOING TO GET STARTED. WE'RE A LITTLE BEHIND SCHEDULE. LOOKS LIKE WE LOST EVERYBODY TO THE COOKIES OUT THERE. I THINK WE'LL HAVE A LITTLE BIT MORE TIME FOR DISCUSSION IN THIS PANEL. WE'RE GOING TO BE TALKING ABOUT THE CHALLENGES OF MIXED METHODS FROM BASICALLY APPLYING FOR THE GRANT APPLICATION RNVIEW AND THEN PUBLICATION AND SOUNDED LIKE PEOPLE HAD A LOT OF QUESTIONS AND ISSUES TO DISCUSS ABOUT THAT. JUST KEEP THOSE IN THE BACK OF YOUR MIND AND WE'LL HOPEFULLY GET TO ALL OF THEM. OUR PANELISTS ARE ANN KLASSEN FROM DREXEL UNIVERSITY. JACINTA BRONTE-TINKEW FROM NIH. SHE'S SCIENTIFIC REVIEW OFFICER FOR THE DISSEMINATION AND IMPLEMENTATION RESEARCH AND HEALTH STUDY SECTION. WE'RE LUCKY TO HAVE HER HERE. THEN HEAR AGAIN FROM GREGORY AARONS TO TALK ABOUT PUBLICATION ISSUES. I'LL TUSH IT OVER TO ANN. >> GOOD AFTERNOON. I JUST WANT TO SAY THAT IN CASE YOU COME AWAY WITH THE IMPRESSION THAT I'M SAYING DO AS I SAY NOT AS I DO, I'M NOT GOING TO TALK ABOUT MY RESEARCH TODAY BUT ONE OF THE PROJECTS I'M DOING AT DIRECT ELSE INVOLVES PHILADELPHIA PUBLIC HOUSING SMOKE FREE POLICIES. IN PHILADELPHIA LIKE IN MANY JURISDICTIONS WE'VE RECEIVED FUNDING FROM CDC THROUGH OUR HEALTH DEPARTMENT TO HELP OUR PUBLIC HOUSING AUTHORITY POLL OUT SMOKE FREE POLICIES IN ALL THEIR RESIDENCES. AS WE PARTNER WITH THE PHILADELPHIA HOUSING AUTHORITY AND PHILADELPHIA DEPARTMENT OF PUBLIC HEALTH TO CREATE AN EVALUATION THAT MEASURES NOT JUST THE OUTCOME BUT THE PROCESS WE'RE DRAWING ON A MIXED METHODS DESIGN AND AS WELL WE'RE WORKING WITH ALL THE STAKEHOLDERS TO TRY TO FIGURE OUT WHAT THEY'RE COMFORTABLE MEASURING, WHAT THEIR METRICS WOULD BE HOW WE CAN BEST CAPTURE FOR THEM THE RICH CONTEXTUAL PROCESS THAT IS URBAN PUBLIC HOUSING AND IS THE DEVELOPMENT AND IMPLEMENTATION OF POLICIES THAT CHANGE PEOPLE'S LIVES AND CHANGE OUR CITY. ALL THE VARIOUS THINGS THAT HAPPEN WHEN YOU SIT OUT TO DO THAT. TODAY WHAT I'M GOING TO TALK TO YOU ABOUT PART OF OUR GUIDE THAT FOCUSED ON HOW TO WRITE A PROPOSAL SUCCESSFULLY THAT PROPOSES MIXED METHODS RESEARCH. IN THE GUIDE WE FRAME THIS AROUND THE ELEMENTS OF A CONVENTIONAL NIHR IN THE NEW FORMAT BUT I THINK WHAT WE'RE TALKING ABOUT WORKS FOR ANY PROPOSAL. TO TELL THEM WHY YOU WANT TO APPROACH A QUESTION IN A MIXED METHODS FASHION, WHETHER YOU ARE DEVELOPING YOUR THESIS PROPOSAL. WHETHER YOU'RE TRYING TO DEVELOP A PROCEEDSAL TO IRB IT'S IMPORTANT TO HAVE CERTAIN FRAMEWORK ON CERTAIN POINTS OF CLARIFICATION THAT HELP YOU PERSUADE YOUR AUDIENCE THAT YOU'RE APPROACH TO YOUR RESEARCH IS WELL THOUGHT THROUGH. AGAIN I WANT TO GIVE ANOTHER SHOUT OUT TO HELEN ISNER FOR SUPPORTING REALLY FOR AT LEAST THE DEVELOPMENT OF WORK IN THIS AREA AND THE WORK I'M SPEAKING FROM COMES FROM OUR MIXED METHODS GUIDE WHICH WE DEVELOPED WITH OBSSR ON THEIR WEBSITE. THERE ARE A A LOT OF STRENGTHS AS WE'VE TALKED 'A BOUT FOR MIXED METHODS APPROACHES. REALLY A TRULY STRONG MIXED METHOD APPROACH CAN ADD CONSIDERABLE STRENGTH. YOU USUALLY SEE MIXED METHODS PROPORTIONED WHEN THERE'S A COMPLEX COMPELLING SCIENTIFIC QUESTION. OFTEN THE QUESTION ITSELF IS A QUESTION THAT REVIEWERS OR AUDIENCES FIND REALLY COMPELLING, BECAUSE IT'S MULTI-LEVEL, BECAUSE THERE ARE LOT OF PARTS TO IT AND BECAUSE IT'S REALLY DEALING WITH SOMETHING THAT'S COMPLEX. YOU START WITH ONE ADVANTAGE. BUT YOU'RE DEVELOPING A SET OF TOOLS THAT WILL BE USE TO HAVE OTHER SCIENTISTS FUNDERS CAN USUALLY SEE VALUE IN THAT. IT'S HARD TO GET THEM FUNDED BECAUSE THEY REQUIRE DETAIL. WE'RE CONSTRAINED TO THE SAME LENGTH AS EVERYBODY ELSE THAT YOU'VE HEARD. DIVERSE EXPERTISE IS NEEDED FROM YOUR AUDIENCE, YOUR IRB AND REVIEWERS IN STUDY SECTION YOU NEED A SYNERGY AMONG THE PEOPLE IN THE ROOM SO THAT THEY END UP WHERE YOU WANT THEM TO BE. THIS ARE COMPLEX, ASK FOR MORE TIME, MORE RESOURCES, HAVE LOGISTICS THAT CAN BE DIFFICULT TO PULL OFF AND THEY REQUIRE A COMPLEX DIVERSE TEAM OF INVESTIGATORS. AND THEREFORE, THE BENEFITS OF USING MIXED METHODS IN A PARTICULAR INQUIRY VERSUS THE COST ARE NOT ALWAYS PERSUASIVELY COMMUNICATED TO THE DIVERSE STAKEHOLDERS. EVEN THOSE WHO GET MIXED METHODS. NOT ALWAYS A QUESTION OF WHETHER THEY BUY MIXED METHODS IS GOOD WAY TO DO THINGS THEY NEED TO BE PERSUADED THAT MIXED METHODS ARE WORT IT FOR WHAT YOU'RE PROPOSING. AND THE NEED FOR INDUCTIVE AND DEDUCTIVE INQUIRY IN TANDEM. WE OFTEN SAY, WHY DIDN'T THEY GET AN R21 OR EXPLORATORY GRANT TO DEVELOP THESE INSTRUMENTS THEN COME IN FOR THE RO1. MIXED METHODS STUDIES IF IT'S GOING TO HAVE BOTH INDUCKTIVE AND DEDUCTIVE PIECES HAVE TO ARGUE THAT THEY MUST BE COMPLEMENTARY AND MUST OCCUR WITH THE SAME TEAM UNDER THE SAME SCOPE OF FUNDING. THAT IS YOUR HARDEST SELL, I THINK. THAT SAID WHEN YOU CAN PERSUADE REVIEWERS OF THAT SYNERGY, OF THAT -- SUM IS MORE THAN JUST -- WHOLE IS MORE THAN JUST THE SUM OF THE PARTS YOU ARE PRESENTING SOMETHING THAT CAN'T BE DONE ANY OTHER WAY. AS IN ANY STUDY METHODS MUST FLOW FROM THE SPECIFIC AIM THAT THEY'RE BEING ASKED TO DIRECT. A GOOD MIXED METHOD. BUT SHOULD HAVE AT LEAST ONE AIM A, ONE SPECIFIC AIM THAT CAN ONLY BE'S DRESSED IN THE METHODS. AIMS BE BE PARSED OUT, IN TO YOUR QUAL AND QUANT AND DON'T HAVE A NAME INTEGRATION YOU'RE NOT PROPOSING MIXED METHODS TO ADDRESS THE RIGHT SPECIFIC AIM. WE'VE TALKED A LOT TODAY ABOUT CONCURRENT AND EMBEDDED DESIGNS AND DIFFERENT DESIGNS. BUT I THINK IMPORTANT TO NOTE THAT YOU'RE SPECIFIC AIM SHOULD BE LISTED IN SOME KIND OF ORDER THAT MAKES SENSE. WHAT IF YOU'RE DOING, QUALITATIVE AND QUANT HAPPEN AT THE SAME TIME NOT SEQUENTIALLY. IT MIGHT DECIDE THAT IN OFFERED FOR YOUR AIMS TO BE MOST CONGRUENT WITH YOUR RESEARCH STRATEGY YOU'RE GOING TO PRIORITIZE THEM YOU BY IMPORTANCE. YOU WANT TO ASSESS TREATMENT OUTCOMES USING AN EXPERIMENTAL DESIGN. AND THAT'S YOUR MAIN ACTIVITY. YOU'RE GOING TO RANDOMIZE PATIENT TWO DIFFERENT ARMS OF THE TRIAL AND GOING TO ASSESS THE TREATMENT OUTCOMES AND USE FAIRLY RIGOROUS QUANTITATIVE ANALYSIS TO DO THAT. BUT THEN WITHIN THAT, DIFFERENT SAMPLES YOU MIGHT USE QUALITATIVE DATA TO EMBED WITHIN THE EXPERIMENT AND UNDERSTAND THE PROCESS OF PAIN MANAGEMENT. UNDERSTAND THE PROCESS OF RECOVERY FROM SEVERE MENTAL ILLNESS OR TRAUMA. THEN TO UNDERSTAND YOUR OUTCOMES WITHIN THE CONTEXT OF THE TREATMENT PROCESS, YOU WOULD INTEGRATE THE DATA. THAT WOULD BE YOUR MIXED METHOD. YOU MIGHT HAVE AN EASIER TIME IF YOUR AIMS ARE SEQUENTIAL TO DESCRIBE IN CHRONOLOGICAL ORDER. YOU WANT TO UNDERSTAND THE CULTURAL CONTEXT OF PHENOMENON OF INTEREST IN QUALITATIVE MEASURE. DEVELOP QUALITATIVE CONSTRUCTIONALLY. THEN YOU WANT TO USE THAT SAME INSTRUMENT TO ASSESS THE PHENOMENON OF INTEREST IN A TARGET POPULATION. AND CONDUCT QUANTITATIVE STATISTICAL ANALYSIS OF YOUR SURVEY DATA THAT YOU COLLECTED WITH THE INSTRUMENT. THEN YOU WANT TO MIX YOUR RESULTS. I INTERPRET THE RESULTS OF STRUCTURED ANALYSIS IN THE QUALITATIVE FINDINGS. AND CONTRIBUTE MIXED METHODS. SO AGAIN, BY COMING AT THE END OF THE DAY I'M REINFORCING THINGS I'M NOT SAYING ANYTHING NEW BUT YOU'D BE SURPRISED HOW OFTEN THE SPECIFIC AIMS IN A MIXED METHODS PROPOSAL WALK YOU THROUGH THESE VERY LOGICAL STEPS. SOMEONE MAY READ IN YOUR PROPOSAL MIXED METHODS BUT CANNOT MAP YOUR METHODS BACK TO EACH OF YOUR AIMS IN AN ORGANIZED WAY. UP WANT TO REVIEW YOUR FIELD TO DATE AND METHODS USED TO DATE ESPECIALLY FOR MIXED METHODS STUDY IT'S IMPORTANT TO TELL REVIEWERS WHERE SOME OF THE STRONGEST FINDINGS HAVE COME FROM AND WHAT THE METHODS HAVE BEEN THAT HAVE YIELDED THOSE FINDINGS. BECAUSE WHAT YOU'RE DOING IS INTRODUCING THE REASON THAT YOU'RE GOING TO ADDRESS A GAP IN THE KNOWLEDGE AS WELL AS THE REASON YOU'RE GOING TO USE A MIXED METHOD APPROACH TO ADDRESS THAT GAP. COMBINING BOTH MUCH DISCUSSION OF WHAT IS KNOWN AS WELL AS WHAT HAS BEEN USED TO ACHIEVE THAT KNOWLEDGE, WILL LEAVE REVIEWERS PERSUASIVELY TO YOUR USEFUL OF MIXED METHODS IF YOU'RE SUCCESSFUL. YOU WANT WANT TO EMPHASIZE WHILE YOUR STUDY WILL ADD NEW TOOLS AS WELL AS INFORMATION. IN THE SPECIFIC SCIENCE OR HEALTH AREA YOU'RE ADDRESSING. WHAT IS THE STATUS OF MIXED METHODS IN THE FIELD OF INQUIRY YOU'RE INTERESTED IN. HOW WILL YOU TAKE IT TO THE NEXT STEP. HOW WILL FUTURE RESEARCHERS BENEFIT FROM YOUR METHODS AS WELL AS YOUR DISCOVERIES ON THE SCIENTIFIC QUESTION. AGAIN, YOU'RE PROPOSING BY VERY NATURE OF USING MIXED METHODS YOU BECOME A METHODOLOGIST. YOU MOVE THE FIELD FORWARD BUT ALSO METHODOLOGICALLY. ONE CAUTION. IT IS NO LONGER ENOUGH TO SAY MIXED METHODS IS INNOVATIVE. BY ITSELF IS NOT LIKELY TO BE SEEN AS A INNOVATIVE BY REVIEWERS ANY MORE, THAT'S THE PRICE OF ALL POPULARITY IT'S A GREAT THING. BUT YOU'LL GET YOURSELF IN TROUBLE IN STUDY SECTION IF YOU SAY, UNDER INNOVATION, WE'RE USING MIXED METHODS. HOWEVER, IN A SPECIFIC PROBLEM AREAS I SAID USING MIXED METHODS MAY BE UNCOMMON OR MAY NOT HAVE BEEN USED MAC MALLLY TO YIELD WHAT WOULD BE YIELDED. MIXING SPECIFIC TYPES OF METHOD IN PROPOSED WAY MAY BE NEW. A NOVEL APPROACH THAT IS USEFUL BY PREVIOUSLY UNTRIED AND JUST RIGHT WAY TO MOVE THAT FIELD FORWARD. CAN BE ARGUED TO BE INNOVATIVE. YOUR CHOICE AT THAT POINT IN TIME IS AN INNOVATIVE CHOICE. IN ADDITION METHODOLOGICAL CONTRICK -- CONTRIBUTIONS MAY BE INNOVATIVE IF THEY USE METHODS IN SPECIFIC WAY AND DEVELOP NEW TOOLS OR TECHNIQUES. WHEN JOHN RICH TALKED ABOUT TAKING ORAL HISTORIES OF TRAUMA AND USING WORD ANALYSIS TO CREATE BLOCKS OF MEANING, THAT'S INNOVATIVE. I'M NOT SURE THAT PEOPLE HAVE SEEN THAT USED TO YOU NEED TO INTRODUCE WAYS THAT ARE Z SAW ST. BUT VERY COMPELLING FOR REVIEWERS OF YOUR PROPOSAL AND YOUR PROCEEDS POSED SET OF ACTIVITIES. SO PROVIDE BRIEF DEFINITION OF MIXED METHODS. YOU'D BE SURPRISED -- YOU PROBABLY WOULDN'T BE SURPRISED HOW OFTEN THAT CAN HELP. IT REALLY CAN DIRECT REVIEWERS AT THE VERY BEGINNING BEFORE THEY READ WHAT YOU'RE GOING TO DO TO WHAT YOUR DEFINITION OF MIXED METHODS IS. SITE STUDIES IN THE AREA OF WORK RELATED TO WORK IS POSSIBLE WHICH IF USED. PROOF THROUGH THIS QUICKLY. DON'T LAUNCH IN TO A METHODS TEXTBOOK BUT IT IS VERY IMPORTANT TO GROUND THE REVIEWERS FROM THE START IN YOUR METHODOLOGICAL AREA. NAME YOUR DESIGN BEING USED. REFERENCE THE TERMINOLOGY. CERTAINLY JOHN CRESWELL'S TERMINOLOGY ONE OF THE MOST WIDELY USED CERTAINLY YOU CAN USE THAT IN REFERENCE AND OTHERS. USE THAT ONE TERMINOLOGY CONSISTENTLY THROUGHOUT YOUR PROPOSAL. YOU CAN SAY THERE ARE OTHER TERMINOLOGY BUT THIS IS TERMINOLOGY I'M USING. STATE YOUR REASONS FOR YOUR CHOICE OF DESIGN AND LINK IT BACK TO YOUR OVERARCHING PROBLEM OR QUESTION. WHY DID YOU THINK SEQUENTIAL DESIGN WAS A GOOD ONE? WHY ARE YOU USING BIG QUANT LITTLE QUAL. WHY DOES THAT HELP YOU ANSWER YOUR QUESTION? ORGANIZE YOUR APPROACH TO MATCH YOUR AIMS. THAT'S NOT JUST A QUESTION OF ORDER, ALTHOUGH THAT'S VERY IMPORTANT. YOU WANT TO TALK ABOUT RIGOR OF YOUR QUANTITATIVE METHODS. DATA COLLECTION AND ANALYSIS, ESTABLISH THE RIGOR OF YOUR QUALITATIVE RECORDS. DON'T JUST SAY YOU'LL LOOK FOR THEMES IN QUALITATIVE DATA. DON'T JUST SAY YOU'LL DO ITERATIVE ANALYSIS. REFERENCE METHODOLOGY OF YOUR QUANTITATIVE ANALYSIS. TACK ABOUT SOFTWARE, VALIDITY AND HOW YOU'RE GOING TO LOOK AT YOUR RESULTS INDITE THEM. BALANCE YOUR SECTIONS BY THE PRIORITY OF YOUR AIM. TAKE THE BLOCK OF TEXT THAT IS YOUR APPROACH SECTION AND WHEN YOU'RE DONE MAKE SURE THAT YOU'VE GIVEN EQUAL TIME TO EACH OF YOUR METHODS IN PROPORTION TO THEIR IMPORTANCE TO YOUR DESIGN. AGAIN, PEOPLE TEND TO SPEND A LOT OF TIME ON THEIR SAMPLE. MAYBE A LOT OF TIME ON THEIR EXPERIMENTAL DESIGN, MAYBE LOT OF TIME ON HOW THEY'RE GOING TO COLLECT THEIR QUALITATIVE DATA. AND THEN THEY RUN OUT OF TIME WHEN THEY WANT TO TALK ABOUT QUALITATIVE ANALYSIS THEN HAVE EVEN LESS SPACE WHEN THEY WANT TO TALK ABOUT META INFERENCE AND INTEGRATION AND THE MIXING. ESPECIALLY TALK ABOUT HOW AND WHEN INTEGRATION WILL OCCUR. REALLY IDEA OF MIXED METHOD STUDY INTEGRATE DATA WHEN DO YOU A WORD COUNT AND YOU'VE ONLY USED MAYBE HALF A PARAGRAPH ON YOUR INTEGRATION YOU NEED TO GO BACK TRY TO GIVE THAT MORE AIR TIME IN YOUR PROPOSAL BASSINETS REALLY WHERE REVIEWERS HAVE TO BE SOLD THAT YOUR MIXED METHOD APPROACH WILL YIELD IMPORTANT FINDINGS. INCLUDE A CLEAR TIMELINE OF ALL ACTIVITIES THAT GOES WITHOUT SAYING. BUT OFTEN THAT INTEGRATION, THAT META INFERENCE. THAT IMPORTANT PIECE THROUGHOUT AND ESPECIALLY AT THE END WHERE IT ALL COMES TOGETHER ISN'T GIVEN ENOUGH TIME. SPECIFICALLY PUT ON THE CALENDAR. TODAY IF WE'VE LEARNED ANYTHING I THINK WE'VE LEARNED THAT MIKE FETTERS' SCHEMA FOR PLANNING OUT A STUDY IS ONE OF THE BEST TOOLS THAT YOU CAN HAVE. SO THIS IS A STUDY THAT I'M DOING NOW WITH C.A.T. LYNN AT JOHNS HOPKINS, IT'S NCA STUDY LOOKING AT DIETARY BEHAVIORS IN LONG TERM CANCER SURVIVORS. THIS MIXED METHOD STUDY HAD THREE AIMS TO EXPLORE ADULT CANCER SURVIVORS EATING PRACTICES. AND HOW CANCER SHAPES THEIR SELF CONCEPT AND READINESS FOR DIETARY BEHAVIOR CHANGE IN ORDER TO GENERATE HYPOTHESES FOR EFFECTIVE AND SUSTAINABLE DIETARY INTERVENTIONS. AGAIN, I THINK SOMETIMES UNFORTUNATELY THE SCHEMAS CAN'T BE READ ON SLIDES. BUT WHAT WE DID WAS BROKE IT DOWN BY OUR AIM A, FROM OUR AIMS COME THE RESEARCH QUESTIONS. FROM EACH OF THOSE QUESTIONS COMES AN APPROACH TO DATA COLLECTION. AN AUDIENCE, SAMPLE SIZE, A TIMELINE, ANALYTICAL APPROACH AND THEN OUR PRODUCTS. WHAT WE EXPECT TO KNOW, WHAT WE EXPECT TO DISSEMINATE AND WHAT WE EXPECT TO TAKE US TO THE NEXT STEP. SO THAT'S A VERY HELPFUL TOOL FOR OUR TEAM, I THINK, BECAUSE AGAIN IT TAKES US BACK TO OUR FRAMEWORK. WHAT ARE WE PUTTING OUR EFFORT IN, WHERE ARE WE. AND I THINK AGAIN FOR REVIEWERS IT GIVES THEM CONCRETE OUTCOMES, CONCRETE ACTIVITIES AND TELLS WHAT OUR ROAD MAP IS FOR THIS STUDY. IN MIXED METHODS ESPECIALLY I THINK IT'S IMPORTANT TO TALK ABOUT THE UNEXPECTED. THIS IS OBVIOUSLY A DELICATE BALANCE IF THERE'S TOO MUCH THAT'S UNKNOWN IT FRIGHTENS REVIEWERS. HOWEVER I THINK IF YOU ARE UP FRONT ABOUT THE FACT THAT INFORMATION WILL BUILD AS YOU GO AND DECISIONS WILL HAVE TO BE MADE ACROSS THE LIFE COURSE OF THE PROJECT, THAT YOU'RE PREPARED TO DO THAT AND YOU HAVE THE RIGHT TOOLS AND PLANS FOR DOING THAT, THEN I THINK THAT'S VERY MUCH A SIGN OF STRENGTH. IS YOUR TIMELINE REALISTIC, WE'VE HEARD A LOT TODAY ABOUT HOW LENGTHY THE PROCESS CAN BE FOR REALLY GOOD MIXED METHOD RESEARCH HOW WILL YOU HANDLE DELAYS. WHAT CAN YOU DO IN ONE AREA IF THERE'S DELAY IN ANOTHER. HOW WILL YOU VALIDATE YOUR ACTIVITIES AND DECISIONS THAT YOU GO ALONG WHAT ARE YOUR TIME POINTS FOR RE-EVALUATION. WHAT ARE THE CRITICAL TIME POINTS AT WHICH WHOLE TEAM WILL GET BACK TOGETHER AND EVALUATE WHERE THEY ARE AND CHASE COURSE IF NECESSARY. WHO IS THE PRIMARY RESPONSIBILITY FOR EACH PIECE AND HOW WILL THE KEY PLAYERS INTER-ACT AND WORK TOGETHER. AGAIN, DR. CURRY HAS WONDERFUL ARTICLE ABOUT BUILDING AND MAINTAINING A MIXED METHODS TEAM AND ASSUMING THE ROLES THAT ALSO MAKING SURE THAT WHOEVER LEADS THOSE ROLES WORKS WITH THE PEOPLE WHO ARE PART OF THEIR TEAM. HOW DO THE PIECES BUILD HOW VULNERABLE ARE LATER PIECES TO EARLIER UNEXPECTED EVENTS. AGAIN, HOW WILL YOU DEAL WITH THAT. WHAT DO YOU HAVE IN PLACE TO MAKE SURE THE PROJECT KEEPS GOING. BEYOND RESEARCH PLAN THERE ARE SOME OTHER REALLY CRITICAL ELEMENTS IN RO1 PROPOSAL. WHEN WE GET AN RO1 PROPOSAL TO REVIEW THEY ARE USUALLY 70 PAGES, SOMETHING LIKE THAT. THEY'RE ONLY 12 PAGES OF THE RESEARCH PLAN. EVERYTHING ELSE IS COMMUNICATION ABOUT WHAT YOU'RE GOING TO DO. DON'T IGNORE THOSE PAGES. I THINK IT'S A TOSS UP IF YOU WANT TO NAME YOUR METHODOLOGICAL APPROACH AND TITLE ONLY HAVE 81 CHARACTERS. MIXED METHODS USES UP MORE THAN TEN. IF YOU CAN'T USE IT IN THE TITLE THEN THROUGHOUT THE ABSTRACTS. CONNECT THE METHODS TO THE AIMS, AGAIN, REFER TO THE WAYS THAT THE INTEGRATION WILL OCCUR, PUT THAT IN YOUR ABSTRACT. TELL PEOPLE IN YOUR ABSTRACT HOW YOU WILL INTEGRATE YOUR DATA. FROM HUMAN SUBJECT STANDPOINT WE'VE TALKED ABOUT THAT AS WELL. BE PREPARED TO EDUCATE IRBs AND REVIEWERS ON HUMAN SUBJECTS ISSUES AS WELL AS THE ISSUES THAT COME ABOUT THROUGH MIXING. MIXING OFTEN AS FOR REPEATED CONTACT WITH PARTICIPANTS WHEN EITHER A QUAL OR QUANT WAVE MIGHT BE DONE. ANONYMITY IS LOST. INDEPTH MULTI-FACETED DATA PROTECTION ISSUES MAY COME UP SO AGAIN THE PHOTOGRAPHY BY ITSELF COULD BE ONE TYPE OF HUMAN SUBJECTS RISK. A SURVEY BY ITSELF MIGHT BE ANOTHER. A SURVEY THAT ASKS ABOUT INDEPTH SENSITIVE ARE USE ABOUT SEXUAL BEHAVIOR, HEALTH, INCOME, THAT STREET LINKED TO PHOTOS YOU CAN TELL WE FEEL THAT THAT IS A MORE SENSITIVE INSIGHT IN TO THAT PERSON'S LIFE THERE FOR BRINGS UP MORE ISSUES OF PROTECTING THEM. I WANT TO MENTION AT DREXEL WE USE A DATA SECURITY SYSTEM THAT IS AVAILABLE THROUGH VANDERBILT FOR A LOT OF INVESTIGATORS, THE RED CAP SYSTEM. IT WAS CREATED FOR CLINICAL DATA, IT HAS A LOVELY CAPACITY TO BE ABLE TO ADD VISUALS, DOCUMENTS, VIDEOS, FILES, ART, ALL KINDS OF THINGS IN TO IT. AND PROTECTED IN THE SAME WAY. WHEN YOU ARE THINKING OF BUILDING A A MIXED METHODS DATABASE WITH ALL THESE DIFFERENT TYPES OF INFORMATION LOOK FOR SECURE DATA SYSTEM THAT ALLOWS ALL THE INVESTIGATORS SHARE ALL THE DATA AND STILL PROTECT THE PARTICIPANTS AT THE LEVEL THAT YOU'RE INTERESTED IN PARTICIPATING. AT DREXEL OUR CREATIVE ARTS THERAPY PROGRAM USES MIXED METHODS VERY INNOVATIVELY AND COLLECTS ALL KINDS OF ARTISTIC EXPRESSION INFORMATION AND DATA FROM PARTICIPANTS WHICH AGAIN HAVE TO BE PROTECTED. AND INTRODUCES VERY INNOVATIVE WAYS OF HUMAN SUBJECT DATA UNDERSTANDING. YOUR INSTITUTION FALL ENVIRONMENT. THIS IS OFTEN, I HAVE TO SAY ONE OF THE MOST OVERLOOKED PARTS OF AN APPLICATION. IT'S WHERE YOU HAVE FREE WORD TOE TALK ABOUT YOUR CAPACITY. TANGIBLE, SUCH AS SOFTWARE, COMPUTERS, MEETING ROOMS, STRUCTURAL SUPPORT FOR TEAM SCIENCE. WHEN YOU LOOK AT WHAT DAN AND SOME OF THE OTHERS WRITE ABOUT THE ART OF TEAM SCIENCE, WHAT DOES IT TAKE TO REALLY HAVE GOOD TEAM SCIENCE MIXED METHODS IS CERTAINLY PLACE WHERE THOSE RESOURCES CAN REALLY SING OUT IN A PROPOSAL IF YOU'RE READY TO DO MIXED METHODS ESPECIALLY ACROSS INSTITUTION. DO YOU HAVE VIDEO CONFERENCING. DO YOU HAVE WAYS TO SHARE VISUALS IN AN INTERFACE THAT BRINGS INVESTIGATORS FROM MULTIPLE INSTITUTIONS PERHAPS TO LOOK AT PHOTOGRAPHS. HOW CAN YOU DO THAT. THOSE KIND OF THINGS WILL TELL PEOPLE THAT YOU'RE READY TO DO A HIGH QUALITY MIXED METHODS TEAM EFFORT. AS WELL INTANGIBLES AND YOUR CULTURE. IS YOUR INSTITUTION OR PARTNERSHIP INSTITUTIONS ARE THEY PLACES WHERE MIXED METHODS CAN THRIVE. DO YOU HAVE JOURNAL CLUBS, DO YOU HAVE STUDENTS YOU'VE TRAINED WHO HAVE USED MIXED METHODS. ARE THERE TRAINING GRANTS, COURSE WORK, PRIOR LEADERSHIP. WHAT HAVE YOU DONE TO CONVINCE US THAT YOU REALLY ARE A PLACE OR GROUP THAT MIXED METHODS WILL REALLY THRIVE UNDER YOUR CARE. IF THOSE ARE PLACES WHERE YOUR INSTITUTIONS WANTS TO BUILD ITS CAPACITY THAT'S OKAY, TOO. HOW WILL YOUR GRANT LEVERAGE. THAT WORK THAT WASN'T„ THERE BEFORE. DON'T BE AFRAID TO TALK ABOUT HOW THAT IS GOING TO BE SOMETHING YOU'RE GOING TO ENRICH YOU PUT TOGETHER A TEAM THAT CAN DO THIS PARTICULAR POOJECT. SO IF YOU HAVE COLLEAGUE AND YOU'RE RUSHING TO WRITE THE GRANT YOU ASK THEM FOR A FAVOR AND CAN THEY PLEASE BE ON IT SEND ANY OTHER BIOSKETCH THEY HAVE. YOU DON'T WANT TO BOTHER THEM DON'T PUT THAT BIOSKETCH IN THE GRANT. HAVE THEM OR WORK WITH THEM TO WRITE AN INTRODUCTORY ONCE YOU HAVE ALL THOSE BIOSKETCHES WHAT THE P.I. NEED TO DO MAYBE TEAM AS GROUP NEEDS TO DO IS LOOK AT ALL THE BIOSKETCHES IN CONCERT. IS? TEAM PROCESS THAT CAN BE CLEARLY MAPPED OUT. ARE THERE LEADERS AN TEAMS FOR EACH TASK. IF I'M SAYING WE'RE SPENDING 20% OF OUR TIME DOING THIS TASK DO I SEE THE PEOPLE THAT WILL CONTRIBUTE THAT HUMAN CAPITAL TO THAT TASK. HOW DO THEY WORK TOGETHER. WHAT EVIDENCE CAN I GIVE REVIEWERS THAT WE'VE DONE THIS BEFORE AND IT WILL WORK. THIS IS MORE OF THE SAME, BUT LOOK FOR ISSUES IN PERSONNEL ROLES. WITH MEMBERED METHODS THAT ARE VERY IMPORTANT. ARE YOU ASKING FOR SUFFICIENT RESOURCES TO HAVE TIME TOGETHER TO ENSURE THAT MIXING IS MAXIMIZED. AGAIN WHETHER YOU'RE GOING TO COME TOGETHER OR DO IT REMOTELY HAVE YOU REALLY BUILT IN THE RESOURCES THAT YOU NEED TO MIX THE DATA AND MAXIMIZE THE OUTPUT FROM THE MIXED DATA. AND OBVIOUSLY NO GAPS. IN CONCLUSION I HAVE A A COLLEAGUE WHO SAID THAT GRANT WRITING IS ALL ABOUT REDUCING THE TRUST-ME FACTOR. GRANT PROPOSAL IN MIXED METHODS IS VERY HARD. BECAUSE YOU'LL OFTEN FEEL THAT REVIEWERS WHO MAY BE DON'T KNOW ENOUGH ABOUT MICKED METHODS FELT THAT THEY WOULD READ YOUR GRANT AND LEARN WHAT MIXED METHODS AREMENT I THINK A A STRONG PROPOSAL CAN CONVINCE THAT SAME PERSON WHO MAY NOT BE MIXED METHODS EXPERT THAT YOU ARE AN EXPERT. THAT YOU ARE USING ACCEPTED STANDARDS AND BEST PRACTICES. YOU CAN DO THAT BY REFERENCING BY BEING VERY CLEAR IN YOUR TERMINOLOGY AND BY INTRODUCING TO THAT REVIEWER THAT YOU ARE PART OF A NETWORK OF SCHOLARS THAT HAS PERFECTED THESE METHODS THAT YOU WILL BE USING THE BEST PRACTICES. SO THAT'S THE DIFFERENCE BETWEEN HAVING TO EXPLAIN WHAT YOU'RE DOING AND EXPLAINING THAT YOU'RE DOING WHAT IS KNOWN IN THE FIELD TO BE THE RIGHT THING TO D. THE WAYS YOU CAN DO THAT IS TIE ALL THE ELEMENTS OF YOUR APPROACH BACK TO YOUR GOALS IN QUESTION. PROVIDE REFERENCES FOR ALL YOUR METHODOLOGICAL STATEMENTS AND DECISIONS. ESPECIALLY WITH SEQUENTIAL DESIGNS LAYOUT WHEN THE KEY DECISIONS WILL BE MADE AND HOW. LIMIT THE UNKNOWN. DESCRIBE NONSTATIC, FLEXIBLE INFRASTRUCTURE OF BOTH HUMAN AND ENVIRONMENTAL RESOURCES THAT SHOWS HOW SCIENCE WILL BE PRODUCED AND FULLY REALIZE IT'S POTENTIAL CONTRIBUTION TO THE FIELD YOUR SCIENCE WILL BE ABLE TO ACHIEVE ITS FULL IMPACT. AGAIN WANT TO POINT YOU TO THE MOST DOWNLOADED DOCUMENT FROM THE OBSSR WEBSITE, AGAIN THANK HELEN MEISSNER FOR HER VISION AND LEADERSHIP IN CREATING THIS CAPACITY. [APPLAUSE] >> GOOD AFTERNOON, I'M JACINTA. SCIENTIFIC REVIEW OFFICER AT THE CENTER FOR SIGNTIVE CAN REVIEW OF NATIONAL INSTITUTES OF HEALTH. I AM ALSO THE SCIENTIFIC REVIEW OFFICER FOR THE DISSEMINATION. AS WELL AS COMMUNITY LEVEL HEALTH PROMOTIONS. TODAY MY PRESENTATION WILL COVER REGARDING PEER REVIEW PROCESS. AS YOU MAY BE AWARE, NIH IS LARGEST SOURCE OF FUNDING FOR MEDICAL RESEARCH IN THE WORLD. AND IT'S MADE UP OF 27 INSTITUTES AND CENTERS EACH WITH A SPECIFIC RESEARCH 'GENERAL D. OFTEN FOCUSING ON A PARTICULAR DISEASE OR BODY SYSTEM. MORE THAN 80% OF THE NIH'S BUDGET GOES TO MORE THAN 300,000 RESEARCH PERSONNEL AT OVER 3,000 UNIVERSITIES. IN ADDITION ABOUT 6,000 SCIENTISTS WORK ON THE NIH CAMPUS WHICH IS WHERE WE ARE HERE TODAY. THE CAMPUS IS ALSO HOME TO THE NIH CLINICAL CENTER WHICH IS LARGEST HOSPITAL IN THE WORLD DEDICATED TO CLIP 8:00 RESEARCH. IT IS CENTER IN GREEN IS ONE OF THE 27 INSTITUTES IS PRIMARILY RESPONSIBLE. ARE LEGISLATIVELY BE REQUIRED EVALUATED BY PEER REVIEW PROCESS. AT CSR WHICH IS INSTITUTE THAT DOES THE BULK OF REVIEW OF NIH, THERE ARE FIVE DIVISION. THERE ARE SUBDIVISIONS WHICH ARE KNOWN AS A IRGs WITHIN EACH OF THESE IRGs THERE ARE SEVERAL STUDY SECTIONS, FOR EACH STUDY SECTION THERE'S A DESIGNATED SCIENTIFIC REVIEW OFFICER RESPONSIBLE FOR THAT PARTICULAR REVIEW GROUP. THIS DIAGRAM AS MANY OF YOU KNOW SHOWS TYPICAL REVIEW PROCESS OF RESEARCH GRANT AT NIH. IT'S EITHER ASSIGNED TO INSTITUTE FOR REVIEW OR SCIENTIFIC REVIEW FOR REVIEW. THE INSTITUTE OR CSR IS ASSIGNED TO STUDY SECTION WHERE IT'S EVALUATED FOR SCIENTIFIC MERIT AND THIS CONSTITUTES FIRST LEVEL OF REVIEW. THEN SENT ADVISORY COUNCIL AND BOARD FOR SECOND LEVEL OF REVIEW AND RECOMMENDED ACTION FOR FUND OR NOT FUNDING AS NEEDED. IT IS INSTITUTE DIRECTOR THAT ULTIMATELY TAKES FINAL ACTION REGARDING WHETHER OR NOT GRANT APPLICATION IS FUNDED. AS YOU ALL KNOW AS WELL TAKES ABOUT A YEAR FROM THE SUBMISSION OF APPLICATION TO ITS ACTUAL FUNDING. IF YOU USE STANDARD DUE DATES FOR RECEIPT OF APPLICATION. AS WE MENTIONED WITHIN AN IRG YOUR APPLICATION IS ASSIGNED EITHER TO STANDING STUDY SECTION WHEN SUBJECT MATTER OF THE APPLICATION MATCHES REFERRAL GUIDELINE FOR THAT STUDY SECTION OR MAY BE ASSIGNED SPECIAL EMPHASIS PANEL. OR WHERE THERE MAY BE CONFLICT. WE ALSO USE SPECIAL EMPHASIS PANELS FOR THE REVIEW OF SMALL BUSINESS APPLICATIONS. FOR THOSE OF YOU WHO ARE NOT FAMILIAR WITH SPECIAL EMPHASIS PANEL IS, HAS FIXED MEMBERSHIP A SPECIAL EMPHASIS PANEL COMMITTEE HAS A GROUP OF SCIENTISTS ARE CONSULTANTS IS BROUGHT TOGETHER GENERALLY FOR ONLY ONE MEETING TO REVIEW ONE OR MORE APPLICATION. EACH CSR STANDING STUDY SECTION TYPICALLY HAS 12-25 REGULAR MEMBERS PLUS TEMPORARY REVIEWERS WHO MAY BE BROUGHT IN FROM TIME TO TIME DEPENDING ON THE FOCUS OF THE APPLICATION, IS THAT ARE RECEIVED. AND CSR STANDING STUDY SECTIONS TYPICALLY CONVENE IN FACE TO FACE MEETINGS. TYPICALLY THESE MEETINGS MAY LAST ONE TO TWO DAYS. AS I MENTIONED BEFORE THE SRO, MANAGER OF THESE STUDY SECTIONS ARE TYPICALLY DOCTORAL A LEVEL, WHOSE SCIENTIFIC BACKGROUND RELATIVELY CLOSE TO THE FOCUS OF THE STUDY SECTION. YOU MAY HAVE LOTS OF QUESTIONS ABOUT HOW WE SELECT REVIEWERS THE STUDY SECTION. I MYSELF WHEN I RELY VERY HEAVILY ON PROGRAMS, STAFF AND NIH PROGRAM OFFICIALS SEVERAL TIMES THEY CAN IDENTIFY SCIENTISTS WHO WOULD BE OF GREAT VALUE IN TERMS OF THE STUDY SECTION. BUT BEYOND THAT WE HAVE A NUMBER OF DIFFERENT METHODS THAT WE USE. WE HAVE SEVERAL NIH REVIEWER DATABASES WITH NAMES OF HUNDREDS OF THOUSANDS OF REVIEWERS. WE ALSO RELY ON SUCCESSFUL APPLICANTS WHO HAVE BEEN FUNDED THROUGH THE STUDY SECTION. AS I MENTIONED WE TAKE RECOMMENDATIONS FROM REVIEWERS AND OTHER NIH STAFF. GENERALLY WE LOOK FOR MATURE INDIVIDUAL SCIENTISTS WHO DEMONSTRATED OBVIOUSLY SCIENTIFIC EXPERTISE AS WELL AS SUPPORT. THESE FOLKS ARE OFTEN DOCTORAL LEVEL OR EQUIVALENT HAVE BEEN KNOWN TO EXERCISE MATURE JUDGMENT, CAN WORK EFFECTIVELY IN A GROUP CONTEXT VERY IMPORTANTLY THEY HAVE A BREADTH OF PERSPECTIVE, THEY SHOULD BE IMPARTIAL. WE ALSO STRIVE TO ENSURE THAT THE REVIEW GROUPS ARE WELL REPRESENTED IN TERMS OF WOMEN AND MINORITY SCIENTISTS, THERE IS SOME GEOGRAPHIC BALANCE IN TERMS OF WHERE REVIEWERS ARE RECRUITED. BEFORE THE STUDY SECTION MEETING EACH APPLICATION IS TYPICALLY ASSIGNED TO THREE OR FOUR REVIEWERS, FIVE TO SIX WEEKS IN ADVANCE OF THE MEETINGS. REVIEWERS ASSESS EACH APPLICATION BY PROVIDING A PRELIMINARY OVERALL IMPACT SCORE AS WELL AS CRITERION SCORES FOR EACH OF THE FIVE CORE REVIEW CRITERIA. WE ALSO ASSESS THE APPLICATIONS, THIS IS ALLUDED TO BY THE PRIOR SPEAKER IN ADDITION TO THE OVERALL IMPACT SCORE THEY ARE FIVE CORE REVIEW CRITERIA THAT ARE USED IN TERMS OF ASSESSING APPLICATIONS. SIGNIFICANCE, INVESTIGATOR TEAM INNOVATION, APPROACH, ENVIRONMENT. THIEF OF THESE PROTECTION OF HUMAN SUBJECTS. INCLUSION OF WOMAN, MINORITIES AND CHILDREN. APPROPRIATE USE OF VERTEBRATE ANIMALS AS WELL AS MANAGEMENT OF BIOHAZARDS. REVIEWERS TYPICALLY DISCUSS ABOUT TOP HALF OF APPLICATIONS USING THE ORDER OF REVIEW. ORDER OF REVIEW IS DETERMINED BASED ON THE AVERAGE OF THE PRELIMINARY OVERALL IMPACT SCORE FROM THE ASSIGNED REVIEWERS. FOR THE DISCUSSION USUALLY STARTS WITH APPLICATION WITH THE BEST AVERAGE PRELIMINARY SCORE AND GO DOWN THE LIST TO DISCUSS THAT. THE APPLICATIONS ARE ALSO CLUSTERED SO NEW INVESTIGATOR APPLICATIONS ARE TYPICALLY DISCUSSED ONE WITHIN CLUSTER AND OTHER MECHANISMS SUCH AS R21s, R15s MAY ALSO BE CLUSTERED. THE PANEL THAT ALSO DISCUSSED ANY APPLICATION, REVIEWER WANTS TO DISCUSS OR RESCUE IF IT DOES NOT AUTOMATICALLY FALL IN TO THE TOP HALF OF APPLICATIONS. AFTER THE MEETING SUMMARY STATEMENTS WERE PROVIDED, THESE ARE HONORED AT THE CRITIQUES WITH SCORES OF EACH OF THE CRITERION. IF YOUR APPLICATION IS DISCUSSED ADDITIONAL FEEDBACK IS GIVEN IN TERMS OF A SUMMARY OF REVIEW DISCUSSION AND OVERALL IMPACT PRIORITY SCORE AND PRESENTS RANKING AND BUDGET RECOMMENDATIONS. WHICH ARE TYPICALLY USED BY PROGRAM APPLICATION IS EVENTUALLY GOING TO BE FUNDED. I WANT TO TALK ABOUT SUBMITTING. BEFORE YOU SUBMIT A PROGRAM OFFICER AT NIH INSTITUTE OR CENTER OR SCIENTIFIC REVIEW OFFICER LIKE MYSELF CAN CHAT WITH YOU ABOUT THE APPLICATION IN TERMS OF DETERMINING THE BEST FIT FOR A STUDY SECTION. AFTER YOU SUBMIT YOUR SCIENTIFIC REVIEW OFFICER IS THE PERSON YOU SHOULD CONTACT. AFTER YOUR REVIEW YOU ARE ASSIGNED PROGRAM OFFICIAL IS THE PERSON THAT SHOULD YOU SPEAK WITH. THERE'S MORE INFORMATION ABOUT THIS AT -- THE GRANTSINFO.GOV PAGE. SOMEBODY THERE CAN ANSWER YOUR QUESTIONS. BY NOW YOU MAY BE THINKING THIS IS A LOT TO NAVIGATE. WE MAY NEED TO KNOW HOW TO GET YOUR APPLICATION TO THE RIGHT STUDY SECTION. ONE OF THE THINGS THAT WE ALWAYS ENCOURAGE IS TO HAVE POTENTIAL APPLICANTS REVIEW THE CSR INTEGRATED REVIEW GROUP AND SCIENTIFIC REVIEW GROUP OR STUDY SECTION GUIDELINES TO IDENTIFY HOME FOR APPLICATION. AS I MENTIONED YOU MAY ALSO WANT TO CHAT WITH A PROGRAM OFFICIAL TO DETERMINE, THEY CAN GIVE YOU GUIDANCE. ALWAYS BE SURE TO SUBMIT A COVER LETTER. THE COVER LETTER SHOULD BE USED FOR A NUMBER OF IMPORTANT PURPOSES TO SUGGEST INSTITUTE ASSIGNMENT, TO SUGGEST REVIEW ASSIGNMENT, TO IDENTIFY POTENTIAL CONFLICT, TO IDENTIFY ANY AREAS OF EXPERTISE THAT YOU THINK WOULD BE NEEDED TO EVALUATE YOUR APPLICATION. OF COURSE DISCUSS ANY SPECIAL SITUATIONS. IT'S NOT APPROPRIATE TO USE THE COVER LETTER TO SUGGEST SPECIFIC REVIEWERS. WHEN PREPARING THE APPLICATION, YES, INSTRUCTIONS ARE PRETTY STRAIGHT FORWARD. BUT IT'S SURPRISING HOW OFTEN WE GET APPLICATIONS THAT ARE MISSING CRITICAL INFORMATION OR THAT HAVEN'T FOLLOWED THE SUBMISSION RULES FOR -- IMPORTANT TO READ THE INSTRUCTIONS. NEVER ASSUME THAT REVIEWERS WILL KNOW WHAT YOU MEAN, BE SURE TO REFER TO LITERATURE, STATE CLEARLY RATIONAL OF YOUR PROPOSED INVESTIGATION, INCLUDE WELL DESIGNED TABLES AND FIGURES, PRESENT AN ORGANIZED LUCID WRITE UP OF CORE IT'S ALWAYS A GOOD IDEA TO OBTAIN PEER REVIEW FROM COLLEAGUES AT YOUR ORGANIZATION. IN ADDITION IT'S IMPORTANT TO SUBMIT BEFORE THE DUE DATE TOO TO ALLOW YOURSELF TIME TO CHECK THAT YOU SUBMITTED YOUR APPLICATION AND TO MAKE ANY NECESSARY CORRECTIONS. YOU ALSO WANT TO CHECK THAT ALL OF THE SECTIONS OF YOUR APPLICATION HAVE UPLOADED CORRECTLY. IN THINKING ABOUT WHAT YOUR PEERS MIGHT GENERALLY LOOK FOR AS REVIEWER OF APPLICATION, I WANT TO BUILD ON WHAT WAS MENTIONED BEFORE. KEEP PRACTICAL IDEAS IN MIND. OF COURSE REVIEWERS ARE LOOKING FOR APPLICATIONS HIGHLY SIGNIFICANT AND HAVE HIGH IMPACT. YES, THEY ARE LOOKING FOR EXCITING IDEAS. MORE OVER THEY'RE LOOKING FOR CLARITY AND IDEAS THAT THEY CAN IDEA DON'T ASSUME TOO MUCH. BREVITY WOULD THINK THAT EVERYBODY KNOWS. WE WANT REALISTIC AIMS AND TIMELINES, DON'T BE OVERLY AMBITIOUS. BE SURE TO NOTE LIMITATIONS OF YOUR STUDY. AND IN GENERAL, A CLEAN, WELL-WRITTEN APPLICATION. WE ASK THAT YOU DO NOT ANTAGONIZE YOUR VIEWERS HAVING THEM HAVE TO HUNT THROUGH THE APPLICATION FOR LETTERS OF SUPPORT, BIO SKETCHES, OTHER SUCH DOCUMENTS. BE SURE THAT WHEN YOU DO YOUR STATEMENT IT IS SPECIFIC TO THIS APPLICATION THAT YOU HAVE PRESENT. MANY TIMES WE HAVE APPLICATION THAT FOCUSED ON OBESITY AND MEMBERS OF THE TEAM HAVE -- REFERRING TO SUBSTANCE USE, SUBSTANCE ABUSE APPLICATION. BE SHORT AS WELL TO UPLOAD THE CORRECT VERSION OF YOUR APPLICATION NOT THE VERSION THAT WAS DONE TWO WEEKS BEFORE SUBMISSION. SO BE SURE THAT THESE THINGS HAPPEN. BE SURE WHATEVER TABLES AND TEXT THAT REVIEWERS CAN SEE CLEARLY. WE HEAR LOTS OF TIMES, IT LOOKS INTERESTING BUT I COULDN'T QUITE SEE MANY OF THE NUMBERS OR THIS FIGURE IS SO TINY, IT'S SQUISHED ON THE SIDE OF THE PAGE. THESE THINGS REALLY ANTAGONIZE REVIEWERS. WHAT HAS HAPPENED NOW THE STRUCTURE OF YOUR APPLICATION IS MORE -- IS ALIGNED WITH THE WAY THE APPLICATION IS REVIEWED. YOU HAVE A SIGNIFICANT SECTION, YOU WERE -- WRITE ABOUT THAT AT LENGTH IN RESEARCH. INVESTIGATIVE SECTION, YOU WILL MAKE SURE THAT YOU HAVE VERY DETAILED AND TIGHT BIO SKETCHES AND THAT'S INCLUDED FOR YOUR PERSONAL STATEMENT. FOR YOUR PERSONAL STATEMENT. IN TERMS OF THE INNOVATION WE WOULD WRITE ABOUT THAT IN THE RESEARCH STRATEGY. THE APPROACH, AGAIN, PROVIDE THOSE DETAILS IN THE RESEARCH STRATEGY. THE ENVIRONMENT, RESOURCES AND ENVIRONMENT THOSE ARE SEPARATE SECTIONS. THERE'S NOT ALIGNMENT WITH THE WAY THE APPLICATION IS SCORED AND ACTUAL STRUCTURE OF THE APPLICATION. SO BE SURE THAT THE TWO THINGS ARE ALIGNED. SOME OF THE COMMON PROBLEMS THAT WE FIND IN APPLICATIONS ARE LACK OF NEW OR ORIGINAL ITEMS, ASH SENSE OF ACCEPTABLE SCIENTIFIC RATIONAL. LACK OF EXPERIENCE IN THE ESSENTIAL METH METHODOLOGY. THE EXPERIMENTAL APPROACH. WITH THE LACK OF SUFFICIENT EXPERIMENTAL DETAIL. LACK OF KNOWLEDGE OF CURRENT PUBLISHED RELEVANT WORK. AND UNREALISTICALLY LARGE AMOUNT OF WORK BEING PROPOSED. THEN UNCERTAINTY CONCERNING FUTURE DIRECTIONS. WE WANT TO ASK THAT APPLICANTS TRY TO AVOID THESE. AS YOU KNOW YOU CAN SUBMIT YOUR GRANT APPLICATION AS A NEW OR EARLY STAGE INVESTIGATOR. THE P.I. WHO HAVE NOT YET COMPETED SUCCESSFULLY FOR A SUBSTANTIAL NIH RESEARCH GRANT. EARLY STAGE INVESTIGATORS ARE THOSE WHO QUALIFY WHO ARE WITHIN TEN YEARS OF COMPLETING A RESEARCH DEGREE OR TEN YEARS OF COMPLETING MEDICAL RESIDENCY OR EQUIVALENT. AS YOU MAY ALL KNOW ONLY SMALL PERCENTAGE OF APPLICATIONS GET FUNDED ON INITIAL SUBMISSION. SUMMARY STATEMENTS PROVIDE VERY INSTRUCTIVE FEEDBACK FOR REVISING YOUR APPLICATION, TAKE ADVANTAGE OF THIS FEEDBACK. YOU CAN ALSO VOLUNTEER TO REVIEW IF YOU HAVEN'T DONE SO, IF YOU HAVE NOT SERVED ON STUDY SECTION THROUGH EARLY CAREER REVIEWER PROGRAM. YOU MUST HAVE ACTIVE RESEARCH PROGRAM AND OF COURSE HAVE PUBLISHED HIGH IMPACT JOURNALS AND YOU DO NOT HAVE TO HAVE HAD PREVIOUS NIH FUNDING. BUT THIS WOULD GIVE YOU AN OPPORTUNITY TO SERVE THE SCIENTIFIC COMMUNITY BY PARTICIPATING IN THE NIH PEER REVIEW PROCESS. IT WILL HELP YOU DEVELOP YOUR OWN CRITIQUE WRITING SKILLS YOU WILL LEARN WHAT DRIVES REVIEW DISCUSSION HOW IMPACT IS EVALUATED. I WANT TO LEAVE WITH YOU A NUMBER OF IMPORTANT RESOURCES. NIH GUIDE IS VERY IMPORTANT RESOURCE AVAILABLE WEEKLY ALONG WITH NIH SCIENTIFIC, ADMINISTRATIVE INFORMATION, SUPPLIES LINKS TO APPLICATIONS FORMS, AVAILABLE ON NIH WEBSITE. THERE ARE SEVERAL VIDEOS THAT YOU CAN VIEW THAT ARE RESOURCES DUE TO CENTER FOR SIGNTIVE CAN REVIEW. ONE OF THEM THE USE OF NIH PEER REVIEW PROCESS REVEALED. PROVIDES TIPS FOR APPLICANTS. FINAL RESOURCES AS I MENTIONED CSR WEBSITE. AS WELL AS NIH OFFICE OF EXTRAMURAL RESEARCH. THAT'S IT. THANK YOU. [APPLAUSE] >> FROM THE BEGINNINGA VERY GOOD PLACE TOH I'LL MAKE THIS PRETTY QUICK. NOT SURE WHY I'M DOING THIS EXCEPT THAT I'VE BEEN THROUGH ALL THE PROCESS THAT A NORTH CAROLINA TALKED ABOUT AND JACINTA TALKED ABOUT FROM APPLICANT TO REVIEWER ON THE DIRH STUDY SECTION. TO SUCCESSFUL AWARD OF GRANTS AND PUBLICATION ALSO ASSOCIATE EDITOR OF IMPLEMENTATION SCIENCE. JUST WANTED TO TALK ABOUT FEW THINGS THEN GET DISCUSSION GOING BECAUSE I THINK DISCUSSION WILL BE THE MOST HELPFUL THING. WHEN I HAVE A PAPER THAT I NEED TO PUBLISH OR RESULTS I THINK ABOUT WHERE DO I WANT TO PUBLISH, WHAT JOURNAL. THERE ARE VERY FEW JOURNALS FOCUSED ON MIXED METHODS, FOCUSED ON IMPLEMENTATION EARLY ON I WENT PRIMARILY TO MENTAL HEALTH SERVICES JOURNAL BECAUSE I PUBLISH IN MENTAL HEALTH. I AM NOT TYPICALLY GOING TO SEND MIXED METHOD IMPLEMENTATION STUDY TO MOLECULAR GENETICS JOURNAL JUST BAD POLICY. TRY TO UNDERSTAND WHY THE JOURNAL S. HAVE THEY PUBLISHED MIXED METHOD STUDIES WE'RE STARTING TO SEE MORE OF THIS, BOTH IN TERMS OF SUBSTANTIVE PAPERS AND IN TERMS OF METHODOLOGICAL GUIDANCE ON PAPERS, RECENTLY PSYCHIATRIC SERVICES PUBLISHED A PAPER THAT WAS MORE ABOUT MIXED METHODS THAN A MIXED METHOD STUDY BUT ALSO PUBLISHED THEM, KATIE'S WORK WHO WAS PRESENTING EARLIER. ALSO JOURNAL PUBLISHED IMPLEMENTATION STUDIES IN THE PAST. AND FOR UNSURE JUST BY DOING REVIEW BECAUSE KEY WORDS MAY NOT APPEAR IN THE TITLE. BUT WE MAY WANT TO CORRESPOND WITH EDITOR IN ORDER TO JUST GET A SENSE OF WHAT WE WANT TO SUBMIT IF THAT IS GOING TO BE APPLICABLE FOR THAT JOURNAL. IN TERMS OF THE JOURNAL CHARACTERISTICS, WE WANT PEER REVIEWED JOURNALS OF COURSE, AND THIS BLIND VERSUS OPEN REVIEW IS SOMETHING THAT IS EMERGING WITH OPEN ACCESS JOURNALS ESPECIALLY IMPLEMENTATION SCIENCE, BIOMED CENTRAL IT'S AN OPEN REVIEW. ARTICLES ARE NOT BLINDED FOR REVIEWERS AND PERSON WHO SUBMITTED THE AUTHORS WHO SUBMITTED KNOW WHO THE REVIEWERS ARE. SO IT'S A VERY OPEN PROCESS I'LL SHOW YOU LATER THAT ACTUALLY THE REVIEWS AND RESPONSE TO REVIEWS WHOLE EDITORIAL PROCESS IS ALSO PUBLISHED ALONG WITH THE ARTICLES IF IT'S ACCEPTED. BUT ANOTHER THING THAT I LOOK AT IS THE QUALITY OF REVIEWS. AND BY QUALITY, THERE'S TWO KINDS OF QUALITY. ONE IS THAT IT'S GOOD QUALITY IN TERMS OF RIGOR THE OTHER IS GOOD QUALITY IN TERMS OF BEING HELPFUL, BEING USEFUL AND WITH INTENT TO IMPROVE THE PRODUCT RATHER THAN PEJORATIVE I'VE HAD SOME ARTICLES THAT I'VE SUBMITTED TO JOURNAL, IS THAT GOT SCATHING REVIEWS. WE THOUGHT THE WORK WAS PRETTY GOOD SO WE WENT TO ANOTHER JOURNAL WHERE THE REVIEWS WERE HELPFUL AND ACTUALLY END UP GOING TO BETTER IMPACT JOURNAL. I LOOK FOREREVIEWS -- LOOK FOR REVIEWS AND JOURNAL TO BE SUPPORTIVE. ALSO THERE'S LOT OF DISCUSSION ABOUT LENGTH OF ARTICLES AND I WAS SURPRISED JOURNAL OF NM IS 10,000 WORDS. THANK YOU FOR NOT INVITING ME FOB A REVIEWER. THAT IS QUITE GENEROUS. ALSO WHETHER YOU WANT TO REACH INTERNATIONAL AUDIENCE OR NATIONAL AUDIENCE DISSEMINATION IMPLEMENTATION RESEARCH TRADITIONS ARE GLOBAL. A LOT FROM THE U.K., A LOT FROM CANADA. CANADA KNOWLEDGE EXCHANGE AND TRANSLATION ARE THE WORDS THAT WE HEAR ABOUT. THINK ABOUT WHAT AUDIENCE YOU WANT TO REACH FROM THAT PERSPECTIVE. THEN BREADTH VERSUS TARGET AUDIENCE. WE JUST HAD PAPER ACCEPTED IN JOURNAL OF AMERICAN ACADEMY OF ADD LESS TENT. WE WANTED TO RAISE THE ISSUE OF ORGANIZATIONAL ISSUES AND LEADERSHIP IN IMPLEMENTATION FOR THIS BROAD AUDIENCE. SO ENDED UP GOING TO THAT JOURNAL. INTERESTINGLY THAT ARTICLE WAS REJECTED AT A POLICEMAN HI MEN FAKES SCIENCE. LEGS SEE WE TALKED ABOUT ACCESS QUALITY. IMPACT FACTOR. I DON'T KNOW ABOUT IN YOUR DEPARTMENTS BUT WHEN MY PORTFOLIO IS REVIEWED EVERY TWO TO THREE YEARS THEY LOOK AT JOURNALS WHAT YOU PUBLISH IN, IMPACT FACTORS, CITATIONS, SOMETHING TO CONSIDER AND COST. OPEN ACCESS JOURNALS ARE PUBLISHED DON'T HAVE SUBSCRIPTIONS BUT ARTICLE PROCESSING CHARGE. AFTER CONVERSATIONS WITH PROGRAM STAFF AT NIH YOU CAN INCLUDE ARTICLE PROCESSING CHARGES IN YOUR BUDGET. AS A LINE ITEM. YEAR FOUR OR FIVE OF FIVE-YEAR GRANT YOU CAN HAVE SOME FUNDS IN THERE TO SUPPORT THAT. ARTICLE PROCESSING CHARGES THAT IMPLEMENTATION SCIENCE NOW ARE RUNNING U.S. DOLLARS ABOUT $1800 TO $2,FOR ARTICLES. YOU DON'T WANT TO PAY FOR THAT ON YOUR POCKET. IN TERMS OF THE MANUSCRIPT CHARACTERISTIC I THINK A LOT OF WHAT ANN WAS TALKING ABOUT IN TERMS OF APPLICATIONS, APPLIES TO MANUSCRIPTS. FRAMING AN APPROPRIATE THEORY, CLEAR RATIONAL FOR HOW THE WORK WILL ADVANCE IMPLEMENTATION SCIENCE. CLEARLY DESCRIBE METHODS, MAKE SURE THAT YOUR METHODS ARE RUG RUSS YOU DESCRIBE THEM AND WHY MIXED METHODS. THE PURPOSE FOR MIXED METHODS TALK ABOUT IN YOUR METHODOLOGY SECTION HOW YOU'RE GOING TO INTEGRATE THEM THEN IN YOUR RESULTS SHOW THEM, HAVE IT CLEARLY FOLLOW FROM -- HOW YOU LAYOUT METHODOLOGY TO THE RESULTS SO IT'S JUST VERY NICE LEAD IN TO RESULTS. THEN IN THE DISCUSSION INTEGRATE THEORY WITH THE RESULTS, RAISE AND ADDRESS ISSUES THAT STUDY COULDN'T ADDRESS OR DIDN'T ADDRESS AS WELL AS YOU MIGHT HAVE LIKED. IDENTIFY RESEARCH GAPS AND OPPORTUNITIES FOR FURTHER RESEARCH. WHILE I WAS SITTING IN THE BACK, LET'S DO COUPLE SEARCHES SEE IF I PUT IN THE SEARCH TERMS, MIXED METHOD FOR MIXED METHODS, WITH A LITTLE WILD CARD THERE ON PSYCH INFO, HOW MANY TITLES HAVE BOTH MIXED METHODS AND IMPLEMENTATION IN THE TITLES, HOW MANY JOURNALS WILL TURN UP. WE FOUND -- I FOUND SEVEN PEER-REVIEWED PUBLICATIONS THAT HAD BOTH DARE VISION AND IMPLEMENTATION IN THE TITLE. IT WAS JUST ONE PER JOURNAL. THESE ARE SOME OF THE JOURNALS, HEALTH PROMOTION AND PRACTICE, I HADN'T -- HOMICIDE STUDIES, INTER-DISCIPLINARY JOURNAL, HADN'T HEARD THAT HAVE ONE. GENERAL HOSPITAL PSYCHIATRY. JOURNAL OF HEALTH CARE FOR THE POOR AND UNDER SERVED. TO ME I WAS SURPRISED THAT THERE WAS SUCH A SMALL UNIVERSE OF TITLES IN JOURNAL, IS THAT WERE CARING MIXED METHODS RESEARCH. I THINK WE NEED TO GET OUT THERE, BE SUBMITTING MORE WIDELY AND MORE BROADLY. INTERESTINGLY PSYCH INFO WILL NOT -- DOES NOT INDEX IMPLEMENTATION SCIENCE. I HAD CONTACTED THE FOLKS THAT MAKE THE DECISIONS ABOUT THE PSYCH INFO DATABASE AND ARGUED WITH THEM ABOUT WHY IMPLEMENTATION SCIENCE SHOULD BE INCLUDED HAD COAGENT ARGUMENTS FROM MYSELF AND EDITOR IN CHIEF THEY STILL TURNED US DOWN. e SIDE NOTE WITH OTHERS WERE ACTUALLY MOVING TO TRY AND HAVE A DIVISION AT THE AMERICAN PSYCHOLOGICAL ASSOCIATION FOR IMPLEMENTATION SCIENCE THEN HAVE A JOURNAL THAT WILL GO ALONG WITH THAT OR IMPLEMENTATION SCIENCE IN PSYCHOLOGY WE'LL SEE IF WE'RE SUCCESSFUL THERE, MAYBE WE CAN GET THEM TO ALSO INDEX IMPLEMENTATION SCIENCE. ALSO DID SAME SEARCH ON PUBMED WE CAME UP WITH 12 PEER REVIEWED PUBLICATIONS, THREE WERE IN IMPLEMENTATION SCIENCE SOME OVERLAP IN TITLES. IT WAS INTERESTING THINGS LIKE PRIMARY CARE, RESPIRATORY JOURNAL, JOURNAL OF EYE VALUATION CLINICAL PRACTICE AND TRIALS. ALL IMPACT FACTORS OF THESE JOURNALS TEND TO BE OKAY BUT NOT STELLAR. SOME OF THEM ARE PRETTY LOW. SOMETHING TO CONSIDER WHEN YOU'RE SUBMITTING. ALSO PUT TOGETHER AN AD HOC LIST OF SOME JOURNALS WHERE I'VE PUBLISHED WHERE I KNOW THAT SOME IMPLEMENTATION STUDIES AND -- AND/OR MIXED METHOD STUDIES HAVE BEEN PUBLISHED WE HAVE PRETTY GOOD LISTING. ALSO WHICH WAS THE ONE A ANNALS OF INTERNAL MEDICINE, DR. CURRY WAS TALKING ABOUT 16.7 WAS IMPACT. WE CAN GET SOME OF THESE MIXED METHODS TYPE OF STUDIES IN TO THESE HIGHER IMPACT JOURNALS, I PUT JAMA AS FRAME OF REFERENCE. WE NEED TO PUBLISH MORE IN JAMA AND NEW ENGLAND JOURNAL. THERE HAVE BEEN THINK PIECES. THOSE HELP SPREAD THE WORD ABOUT IMPORTANCE OF T3 TRANSLATION. JUST WANT TO TALK FOR A FEW MINUTES ABOUT JOURNAL IMPLEMENTATION SCIENCE. IT'S SIX YEARS OLD. JUST HAD OUR FIRST IN-PERSON MEETING IN LONDON OF ASSOCIATED EDITORS AND EDITORS IN CHIEF. IT IS A GOOD OUTLET FOR IMPLEMENTATION RESEARCH. FOR MIXED METHOD STUDIES ON EDITORIAL BOARD. WE HAVE SOME GOOD QUALITATIVE RESEARCHERS AND MIXED METHODS RESEARCHERS. I THINK IT'S A GOOD OUTLET LOOKS AT METHODS TO PROMOTE THE UPTAKE OF RESEARCH FINDINGS AND ROUTINE CARE, INFLUENCES ON HEALTH CARE PROFESSIONAL AND ORGANIZATIONAL BEHAVIOR. INTERVENTIONAL DEVELOPMENT. NOT NECESSARILY JUST INTERVENTION BUT HOW ARE THOSE AFFECTS ACHIEVED HOW DOES CONTEXT PLAY A ROLE IN THAT. THEN ROLE OF THEORY AND IMPLEMENTATION, RESEARCH. I'M NOT SURE HOW MANY JOURNALS ARE DOING THIS BUT WE USE PLAGIARISM DETECTION FOR THEIR SOFTWARE NOW TO LOOK FOR PLAGIARISM IN SUBMISSIONS TO JOURNAL ARTICLES. OPEN ACCESS, I THINK THIS IS GOOD MODEL FOR THE TWITTER-VERSE AND FACEBOOK GENERATION WHICH WE EXPECT -- ESPECIALLY SCIENTIFIC RESOURCES AND KNOWLEDGE EXCHANGE AND TRANSFER TO BE FREELY AND OPENLY BASEBALL. THAT'S ARTICLE PROCESSING CHARGES ARE THERE. SOME INSTITUTIONS ARE SUBSCRIBERS OR MEMBERS OF BIOMED CENTER. YOU EITHER GET A DISCOUNT, VARYING LEVELS OR FREE DEPENDING ON THAT. CHECK WITH YOUR INSTITUTION, IT'S ONLINE ONLY. PRETTY RAPID TURN AROUND PUBLICATION TIME. TYPES OF MANUSCRIPTS INCLUDE RESEARCH REPORTS WHICH CAN BE UP TO 6,000 WORDS. AND IF YOU APPEAL TO THE EDITOR IN CHIEF CAN BE UP TO 8,000 WORDS. HELPS DEAL WITH SOME OF THAT CONCERN ABOUT VERY SHORT LIMITS ON SOME OF OUR MEDICAL JOURNALS AND SHORT REPORTS ARE ABOUT 2500 WORDS. DIFFERENT TYPES OF ARTICLES, RESEARCH, METHODOLOGIES, STUDY PROTOCOLS. I JUST PUBLISHED BY FIRST STUDY PROTOCOL WHICH I NEVER REALLY HEARD OF. IF IT IS FUND THE YOU HAVE IRB APPROVAL THEN IT DOES NOT GO OUT FOR PEER REVIEW. IT GETS IN HOUSE EDITORIAL REVIEW. IT'S GOOD WAY TO GET YOUR STUDY DESIGN AND INFORMATION ABOUT CONCEPTUAL MODELS AND STUDY DESIGNS AND APPROACHES IN TO THE LITERATURE THAT YOU CAN THEN ALSO REFER TO. SYSTEMATIC REVIEWS ARE ALSO DONE, WE'VE JUST ADDED SOME MEMBERS TO OUR EDITORIAL BOARD TO FOCUS ON SYSTEMATIC REVIEWS, FOLKS THAT HAVE PARTICULAR EXPERTISE. I TALKED ABOUT INDEXING AND OPEN REVIEW. WITH THAT OPEN REVIEW THIS IS AN EXAMPLE OF THE BRADLEY CURRY, ET A AL ON POSITIVE DEVIANTS. GREAT ENDORSEMENT. YOU CAN LOOK AT THOSE. IF YOU'RE THINKING OF SUBMITTING TO IMPLEMENTATION SCIENCE FIND A SIMILAR ARTICLE YOU CAN ACTUALLY GO AND LOOK AT THE REVIEW PROCESS AND REVIEW HISTORY WHAT REVIEWERS SAID, THERE'S NO GUARANTEE GET THE SAME REVIEWERS BUT GIVE YOU INDICATION OF WHAT ARE SOME OF THE CONCERNS THEY CAN PREPARE. THIS TELLS YOU WHO THE REVIEWERS WERE. THE DATES SO YOU CAN ALSO SEE HOW LONG THE PROCESS TAKES WHERE THE HOLD UP WAS. THIS ONE AT A GOOD PACE I THINK. FINAL THOUGHTS, DECIDING WHETHER YOUR MANUSCRIPT TO BE SUBSTANTIVE OR INSTRUCTIVE IS IMPORTANT. I MEAN YOU'RE ASKING A RESEARCH QUESTION AND ANSWERING WITH MIXED METHODS VERSUS INSTRUCTIVE. WHERE IDENTIFYING YOUR APPROACH TO MIXED METHODS EXPLICATING THOSE METHODS IS PRIMARY. CAN BE TWO DIFFERENT TYPES OF ARTICLES, IN STRUCK TOUGHER. GO TO JOURNAL OF MIXED METHODS OR OTHER JOURNAL THERE IS OFTEN A NEED TO HAVE INSTRUCTIVE TYPE ARTICLES GO TO JOURNALS WHERE MIXED METHOD STUDIES DON'T TYPICALLY GO TO ACQUAINT THE READERSHIP WITH MIXED METHODS. YOU CAN DO SERVICE TO THE FIELD BY TRYING TO PUBLISH THOSE IN YOUR AREA OF EXPERTISE. DETERMINE WHAT OUTLET WILL BEST MEET YOUR NEEDS. WHAT AUDIENCE DO YOU WANT TO REACH, IS IT ADMINISTRATORS OR POLICY MAKERS, GOING TO ADMINISTRATIVE SCIENCE QUARTERLY, GOING TO GO TO MORE CLINICAL TYPE OF JOURNAL. BEST FIT FOR RESEARCH THAT YOU'RE DOING AND EXPECTATIONS OF YOUR INSTITUTION THIS ALL HAS TO BE WEIGHED IN TERMS MUCH YOUR CAREER DEVELOPMENT HOW YOU SUPPORT YOUR CV. WHERE ARE YOU LIKELY TO GET REVIEWERS THAT UNDERSTAND MIXED METHODS AND IMPLEMENTATION RESEARCH QUESTIONS. THAT'S NO SMALL TASK BUT HOPEFULLY THAT WILL BE GROWING AS WE GO. THE WORD AND PAGE LIMITS APPROPRIATE FOR YOUR MANUSCRIPT. I DO URGE FOLKS TO TRY TO BE CONCISE AND TEAM IT BRIEF. IT'S EASIER ON EDITORS AND REVIEWERS IF POSSIBLE. AND REMEMBER, AS I SAID, IF YOUR MANUSCRIPT IS REJECTED JUST PART OF THE PROCESS. TWICE IN THE LAST TWO YEARS WHERE I WENT TO HIGHER IMPACT JOURNAL VERY SUCCESSFULLY SO WE USED TO DO THE STEP DOWN, TRY HERE, DIDN'T GET IN THERE. I'M GOING THE OTHER WAY NOW. I'M TRYING TO BUCK THE SYSTEM. WITH THAT I'LL JUST END. I THINK WE CAN OPEN IT UP FOR BRIEF DISCUSSION. [APPLAUSE] CARL YOU HAD A QUESTION OR A COMMENT? >> CARLA GREEN, I HAD QUESTIONS AND COMMENTS AND EXPERIENCE TO SHARE, TOO. I THINK THIS PUBLICATION ISSUE IS A SIGNIFICANT ONE AND OBVIOUSLY PUTTING LOT OF TIME TALKING ABOUT IT BECAUSE IT'S TRUE FOR A LOT OF US. COMMENT THAT I'VE HAD SEVERAL CIRCUMSTANCES WHERE I'VE HAD INTEREST FROM HIGHER IMPACT JOURNALS BUT THEY ALSO I THINK HAVE THE STRICTEST PAGE LIMITS BECAUSE THEY CAN PICK AND CHOOSE THEY'RE TRYING REALLY HARD TO GET AS MANY ARTICLES IN AS THEY CAN. IN ONE CASE THEY:XK)U‡U SAID, NO, YOU HAVE TO CUT IT TO 3,000 WORDS AND I SAID, WELL, THERE'S NO WAY TO DO THAT. ANOTHER ONE SAID YOU CAN PUT ALL QUALITATIVE DATA ON -- IN AN APPENDIX ON THE WEB. I SAID, IT WAS PAPER I PRESENTED TO YOU TODAY ACTUALLY. I JUST COULDN'T FIGURE OUT HOW TO DO THAT WITHOUT COMPLETELY GUTTING THE RESULTS. I WENT TO A LOWER IMPACT JOURNAL THAT WAS WILLING TO GIVE US THE WORDS WE NEEDED. I HATE MAKING THAT KIND OF DECISION. IF WE CAN FIND WAYS TO ADDRESS THAT WOULD BE REALLY GOOD. I HAVE HAD -- I WAS INTERESTED IN -- I ASKED LESLIE, DID ANNALS PUBLISH THE QUALITATIVE AND QUANTITATIVE PAPER IN THE SAME ISSUE. I'VE HAD SOME SUCCESS WITH THAT. WE ACTUALLY DID -- GOT A SPECIAL SECTION IN JOURNAL, I MAILED AN EDITOR SAID, HERE IS WHAT WE'VE GOT. COMPLICATED MIXED METHOD STUDY, THREE PAPERS, WE THINK YOU MIGHT BE INTERESTED. THEY SAID, SEND THEM TO US WE'LL LOOK AT THEM. THEY TOOK THEM. GOT COMMENTARY WITH THEM, IT WAS REALLY GREAT. I'VE ACTUALLY APPROACHED ANOTHER EDITOR WITH THE SIMILAR PROPOSAL WHO ALSO SAID, SURE, WE'LL TAKE A LOOK AT THAT KIND OF APPROACH. I WOULD URGE PEOPLE TO TRY THAT WHEN WE HAD SPECIAL SECTION PUBLISHED WE SUBMITTED THREE PAPERS, STAND ALONE PAPERS FOR REVIEW THEN WHEN THEY ACCEPTED THEM THEY SAID, PLEASE PUT THE COMMON METHODS IN THE FIRST PAPER. THAT WORKED REALLY QUITE NICELY. I THINK THAT IS ONE APPROACH BUT AGAIN LIMITED SPACE IN THESE JOURNALS. I REALLY LIKE MIKE'S IDEA OF PUBLISHING YOUR METHODS IN THE JOURNAL OF MIXED METHODS RESEARCH BUT IF YOU ARE GOING TO A BLINDED REVIEW PROCESS THEN YOU CAN'T REFERENCE THAT PAPER. SO THAT'S A PROBLEM FOR JOURNAL, IS THAT REQUIRE BLINDED REVIEW. NOT SURE WHAT TO DO ABOUT THAT. IF IT IS NOT BLINDED THEN YOU'RE OKAY. I'M JUST REALLY CURIOUS TO HEAR ABOUT OTHER APPROACHES PEOPLE HAVE USED TO TRY TO GET THESE PAPERS PUN LUSHED. AND ALSO I WONDER IF WE COULD, AS A GROUP, WHO IS INTERESTED IN THIS MAYBE WRITE A JOINT LETTER OR SOMETHING THAT WOULD GO TO A VARIETY OF EDITORS TO ADDRESS THIS ISSUE. NIH IS FUNDING THESE COMPLICATED PROJECTS. THEY ARE AT THEIR BEST WHEN THE PAPERS REALLY DEAL WITH ALL THE METHODS THAT ARE USED AND SO YOU CAN SEE THE COMPLEXITY OF THE TOPIC. THERE'S A REASON THEY WERE FUNDED AS MIXED METHODS STUDY, TO NOT BE ABLE TO PUBLISH THEM I THINK LOSES IMPACT OF SPENDING MONEY ON RESEARCH. I'M CURIOUS WHO OTHER PEOPLE HAVE DONE, WHAT SUGGESTIONS PEOPLE HAVE. >> I THINK THE SERIES APPROACH, SOMETHING THAT IMPLEMENTATION SCIENCE DOES AS WELL. IT'S NEGOTIATION WITH EDITORS AT SOME POINT HAVE BEEN ABLE TO GET SOME WAIVER TO GO FROM 3500 TO 4,000 WORDS. THAT MAY NOT BE SUFFICIENT, I LIKE YOUR IDEA OF SOME KIND OF CONCERTED EFFORT, JUST AS JOURNALS HAVE DIFFERENT PAGE LENGTHS OR SPECIFIC TYPES OF ARTICLES MAYBE WE COULD FOSTER AN INITIATIVE WORK WITH EDITORIAL BOARDS AROUND MIXED METHOD STUDIES TO HELP THEM UNDERSTAND THAT COMPLEXITY AND NEED FOR SOME ALLOWANCES. >> NATIONAL INSTITUTE OF -- >> BEFORE WE GO -- I'M SORRY. BEFORE WE CHANGE TOPIC, I JUST WANT TO ADD EXTENSION SPECIAL ISSUE IS OFTEN LESS COSTLY THAN YOU THINK ESPECIALLY 23 YOU'RE LOOKING AT A PER-PAPER CHARGE UP AROUND $1800. FOR EXAMPLE, IF YOU CREATE A TEAM OF ASSOCIATE EDITORS AND COME UP WITH FUNDING ACROSS SEVERAL INSTITUTIONS, JOURNAL MIGHT BE WILLING TO LET YOU SHAPE THE WORD LENGTH OF THAT SPECIAL ISSUE. THEY HAVE LOT OF IMPACT BECAUSE ONE STOP SHOP FOR BODY OF WORK THAT COMPLEMENT. SOUND LIKE LOT OF WORK FOR INDIVIDUAL AUTHOR TO CREATE SPECIAL ARE EUJUST FOR A PAPER BUT IF YOU KNOW OF OTHER GRANTS IN YOUR AREA WANT TO DO IT I THINK GOES BEYOND JUST NEGOTIATING. >> NATIONAL INSTITUTE OF MENTAL HEALTH. I WORK IN AIDS, CLEARLY THIS DEMONSTRATES THAT WE'RE ALL IN SILOS HERE BECAUSE WE'VE BEEN DOING MIXED METHODS RESEARCH BUT WE DIDN'T CALM IT THAT. IT DIDN'T COME UP IN ANY OF YOUR SEARCHES. THAT IS MY POINT, WHEN WE'RE THINKING ABOUT WHERE WE'RE GOING TO PUBLISH SOMETHING AND WE'RE OFTEN THINKING OF IMPACT ON THE FIELD. IF YOU PUBLISH IN AN AIDS JOURNAL PEOPLE AREN'T GOING TO READ IT. BUT IF YOU PUBLISH IT IN ANOTHER KIND OF JOURNAL WHERE IT HAS JUST AS MUCH RELEVANT AND APPLICABILITY YOU OFTEN GET A LETTER BACK SAYING, THIS IS PROBABLY MORE SUITABLE FOR AIDS JOURNAL. AND I THINK THIS IS THE REAL PROBLEM THAT WE DON'T LOOK ACROSS DISEASES AND AREAS TO FIND OUT WHAT INNOVATIVE THING ARE BEING DONE IN OTHER DISEASE AREAS. DO YOU HAVE ANY INNOVATIVE SOLUTIONS TO HOW WE MIGHT DEAL WITH THAT? >> WELL, I THINK SEVERAL DIFFERENT SETS OF PEOPLE HAVE DONE MIXED METHOD PUBLICATION ANALYSIS. THE TYPES OF JOURNALS AND ARTICLES NOT PARTICULARLY DISSEMINATION, BUT JOURNAL, IS THAT ARE MOST OPEN TO MIXED MESSAGE WE'VE SEEN JOURNAL, IS THAT ADDRESS SPECIAL POPULATIONS. THAT SORT OF -- BUT ANY TIME THAT THERE IS AN UNDERSTANDING AN ISSUE OR POPULATION OR CULTURE OR COMMUNITY REQUIRES MORE CONTEXTUAL UNDERSTANDING THEN I THINK YOU'LL SEE MIXED METHODS IN JOURNAL, IS THAT CHAMPION THOSE HEALTH ISSUES AND THOSE COMMUNITIES, CERTAINLY WOMEN'S HEALTH, DIVERSE POPULATION HEALTH JOURNAL TEND TO UNDERSTAND. NOW THAT DOES -- YOU THINK ABOUT AFFECTED GROUP THAT IS WHERE YOU'LL FEND TO SEE JOURNALS STEP UP BECAUSE WE KNOW THAT'S THE WAY TO ADDRESS DISPARITIES IN OUR GROUP. I DON'T WANT TO CRITICIZE BIOMEDICAL JOURNALS BUT WHEN THE TOPIC OF THE THE JOURNAL IS DISEASE TOPIC AND HAS TO CROSS ALL THE WAY FROM THE LABORATORY TO THE COMMUNITY THAT IS WHEN MIXED METHODS BECAUSE A HARDER SELL. >> RECRUITMENT ANNOUNCEMENT AND QUESTION. IF YOU WOULD BE INTERESTED IN REVIEWING MANUSCRIPTS FOR THE JOURNAL OF MICKED METHODS RESEARCH, GIVE ME YOUR CARD WITH A NOTE ABOUT YOUR AREAS OF EXPERTISE BECAUSE WE'RE ALWAYS LOOKING FOR REVIEWERS. >> TIME FOR ONE MORE QUESTION. JOHNS HOPKINS UNIVERSITY. KIND OF MORE OF COMMENT TO GREG THAT NEW ENGLAND JOURNAL DOES PROVIDE THOUGHT. THERE WAS ONE I THINK JUST LAST WEEK THAT WAS PUBLISHED ABOUT TRANSLATION OF EVIDENCE-BASED PRACTICES IN CLINICAL SETTINGS IT WAS PUBLISHED BY THE DIRECTOR -- WHO OBVIOUSLY THAT WOULD BE MAYBE SOMETHING WE WOULD THINK ABOUT I WAS THINKING, HOW CAN WE LOOK AT USING MIXED METHODS, IT SEEMS TO MARRY NICELY WITH PATIENT-CENTERED OUTCOME. >> JUST WANTED TO BRING UP ONE COMMENT THAT YOU WERE TALKING ABOUT NEW QUALITY IMPROVEMENT THINGS, V.A. IS GROUND BREAKING IN TERMS OF THIS EXEMPTION. I JUST REJECTED AN ARTICLE THAT CAME FROM AN ORGANIZATION THAT HAD THAT EXEMPTION AND IT SEEMED LIKE IT WAS USED AS A LICENSE TO NOT DESCRIBE THE METHOD VERY WELL. FOR EXAMPLE, THE SAMPLE WASN'T DESCRIBED. THIS IS WHERE INTERVIEWS, OBSERVATIONS OF GROUP MEETINGS AND ANALYSIS OF E-MAILS WAS UNDERTAKEN WITH NO CONSENT, NO HUMAN SUBJECT REVIEW. IF IT'S A RESEARCH STUDY. BUT I THINK WE NEED TO THINK WHAT THE MEANING OF QUALITY IMPROVEMENT IS WHAT THAT MEANS FOR RIGOR OF OUR METHOD THAT WERE UTILIZING IT. I JUST THINK SOMETHING THAT WILL BE ONGOING ISSUE OVER NEXT -- LEVELS OUT. >> PLEASE THANK THE PANEL. [APPLAUSE] I WANT TO THANK EVERYBODY FOR HANG IN THERE FOR OUR CLOSING. WE HAVE DR. BOB KAPLAN WILL MAKE REMARKS ABOUT MIXED METHODS AND IMPLEMENTATION RESEARCH AND THE CONTEXT OF THE BROADER BEHAVIORAL AND SOCIAL SCIENCES RESEARCH AT NIH. >> THIS IS A VERY SENSITIVE MOUSE. THANK YOU. USUALLY THE LAST SPEAKER SAYS, I KNOW I'M THE ONLY THING THAT'S BETWEEN YOU AND THE COCKTAIL HOUR, WELL, WE DON'T DO COCKTAIL HOURS, USED TO BE THAT MAYBE I WAS THE ONLY THING YOU AND THE JUG WINE AND A COOKIE. BUT WE DON'T DO THAT EITHER. IN FACT WE DON'T EVEN DO COFFEE ANY LONGER. BUT I THINK THIS MEETING WAS GOOD EVIDENCE THAT IN FACT EVEN WITHOUT SERVING COFFEE YOU ACTUALLY -- SO THERE. [ NOT AUDIBLE ] ANYWAY, THIS IS -- I HAD TO GO AWAY FOR PART OF IT DURING THE MIDDLE OF THE DAY BUT THE PARTS I HEARD I THOUGHT WERE TERRIFIC I DID HAVE CHANCE TO REVIEW THE SLIDES. JUST WANTED TO SUM THIS UP BY SAYING THAT, I THINK IT IS IMPORTANT TO THINK ABOUT WHAT YOU'RE DOING IN MIXED METHODS RESEARCH IN RELATION TO NIH MIX STATEMENT. THE NIH AS PART OF THEIR MISSION SAYS THAT THE NIH IS STEWARD OF BIOMEDICAL AND BEHAVIORAL RESEARCH AND SCIENCE PURSUIT OF FUNDAMENTAL KNOWLEDGE ABOUT NATURE AND BEHAVIOR OF LIVING SYSTEMS AND APPLICATION THAT HAVE KNOWLEDGE TO EXTEND REDUCE BARENESS OF ILLNESS AN DISABILITY. IN ORDER TO ACHIEVE THIS OBJECTIVE WE NEEDED A WIDE RANGE OF METHODS ALL FOCUSED ON THIS PROBLEM OF HELPING PEOPLE LIVE LONGER AND LIVE MORE PRODUCTIVE LIVES. JUST WANTED TO BRIEFLY SAY IN THE OFFICE OF BEHAVIOR AND SOCIAL SCIENCES RESEARCH WE COVER TREMENDOUS AMOUNT OF TURF, I WAS NOT AWARE OF THIS UNTIL I CAME TO THE NIH A YEAR AGO WE ARE THE PORTAL TO THE NIH FOR WHOLE VARIETY OF ACADEMIC DISCIPLINE. LISTED ON THE LEFT SIDE, ALL OF WHICH DO BOTH QUALITATIVE AND QUANTITATIVE RESEARCH AND PROVIDE INTERFACE FOR ALL KINDS OF PROFESSIONAL SCHOOLS LIKE CLINICAL PSYCHOLOGY, GERONTOLOGY, BUT ALSO WITHIN MEDICINE, AREAS LIKE PEDIATRICS, GENERAL INTERNAL MEDICINE, FAMILY MEDICINE, FEELS THEY HAVE STRONG CONNECTION TO BEHAVIORAL AND SOCIAL SCIENCES RESEARCH. I HAD A LOOK AT WHAT WE THINK NIH IS SPENDING IN DIFFERENT AREAS. THIS IS ACTUALLY VERY GOOD EXAMPLE OF WHY WE NEED MIXED METHODS PRESEARCH THAT THE ACCESS TO THE TOOLS THAT WE HAVE AT THE NIH DON'T NECESSARILY GIVE US THE RIGHT NUMBERS. OFTEN YOU HAVE TO SORT OF GO THROUGH THIS BY HAND TRYING TO GET A SENSE OF WHETHER OR NOT GRANTS IN THESE AREAS ARE REALLY WHAT YOU THINK THEY ARE. BUT IF YOU DO THE GENERAL SEARCH WHAT WE CALL NIH REPORTER, IT TELLS US THAT POPULATION HEALTH, NIH BELIEVES THEY'RE FUNDING ABOUT 25,000 PROJECTS IN THE YEAR 2011 AND THEY'RE SPENDING ABOUT $11 MILLION -- BILLION IN THAT AREA. MORE THAN THIRD OF THE NIH BUDGET. I DID DRILL DOWN LOOK AT SOME OF THOSE PROJECTS I DON'T BELIEVE THAT MANY OF THEM ARE REALLY POPULATION HEALTH AS WOULD YOU THINK. QUALITATIVE RESEARCH, I LOOKED AT AND THAT ACTUALLY MAY BE MORE ACCURATE. IT DOES SUGGEST THAT STUDIES THAT IDENTIFY THEMSELVES EITHER IN THE ABSTRACT OR SOMEWHERE IN THE TEXT OF THE GRANT DOING SOMETHING RELATIVE TO QUANTITATIVE RESEARCH ABOUT A THOUSAND GRANTS, ABOUT $350 MILLION EXPENDITURE. IF YOU PUT IN MIXED METHODS COME UP WITH 799 GRANTS ABOUT 271 MILLION DOLLARS INVESTMENT. AGAIN WHEN I LOOK AT THE SORT OF GRANT BY GRANT IT WAS CLEAR THAT A LOT OF THOSE MIXED METHODS WERE SORT OF -- THIS ASSAY VERSUS THAT. PROBABLY NOT VERY ACCURATE. WHEN I LOOKED THROUGH THE ACTUAL TEXT OF THESE, LOOKED MORE ACCURATE. ABOUT 404 PROJECTS, ABOUT $246 MILLION. THEN COMBINATION IF YOU JUST PUT IN MIXED METHODS AND SOCIAL SCIENCE, JUST 2 1 PROJECTS AND $8 MILLION IN EXPENDITURE NOT VERY MUCH. IN THE POPULATION HEALTH YOU CAN SEE THIS IS SPREAD OUT VERY WELL ACROSS NIH. IF YOU HAVE THIRD OF THE ENTIRE NIH BUDGET YOU WOULD EXPECT. THAT THESE 21 PROJECTS YOU CAN SEE, I HAVE TO GET MY GLASSES ON TO SEE THIS. THOSE 21 PROJECTS SEND TO BE NIMH, NICHD, NIDDK, NCI OR THE BULK OF THEM. THEN OTHER INSTITUTES TO SMALLER EXTENT. I'D LIKE TO JUST TAKE A FEW MINUTES IF I CAN TO TELL YOU ABOUT SOME OF THE NEW DIRECTIONS WE'RE THINKING ABOUT IN THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCES RESEARCH HOW THEY RELATE TO THIS. I'M -- CAME TO THE NIH I MET WITH DIRECTOR COLLINS, HE WANTED TO KNOW WHAT I WAS INTERESTED IN I HAD THIS LONG SLIDE SHOW OF 80 SLIDES HE SAID, YOU CAN'T DO THAT. YOU HAVE TO ORGANIZE ALL THINGS YOU'RE DOING AROUND THREE TOPICS. TWO THE TOO FEW. JUST THREE BASIC IDEAS. THAT'S A LOT. MENTIONED BEFORE WE COVER A LOT OF THESE DISCIPLINES. HERE IS WHAT WE CAME UP WITH. WE ARGUING THAT WHAT WE'D LIKE TO DO IS ORGANIZE ACTIVITIES AROUND THESE THREE MAJOR THEMES. FIRST NEXT GENERATION OF MEASUREMENT AND DATA. SECOND IS DELIVERY OF HEALTH SERVICES IN REFORMING HEALTH CARE SYSTEM. THIRD IS NEXT GENERATION OF EDUCATION AND TRAINING. WITH REGARD TO THE NEXT GENERATION OF DATA AND ANALYSIS METHODS WE WANT TO KNOW WHAT METHODS WILL BE REQUIRED TO ADVANCE OUR UNDERSTANDING OF HEALTH AND HOW WILL THESE NEW METHODS CHANGE BASIC ACADEMIC DISCIPLINES. HOW IS SCIENCE OR OUR SCIENCE GOING TO LOOK DIFFERENT. I BORROWED THIS SLIDE WHICH IT WAS A TOUR DE FORCE HAD CHANCE TO LOOK AT THE SLIDES BUT I DIDN'T GET YOUR PERMISSION, I HOPE THAT'S OKAY. I WAS VERY IMPRESSED THAT THERE IS EVOLUTION IN THE METHODOLOGY BUT ALSO IN CATEGORIZATION OF THESE METHODS. HIS PAPER JUST DOES GREAT JOB OF HELPING US UNDERSTAND HOW METHODS WORK TOGETHER, HOW CLASSIFICATION SYSTEMS FOR THIS METHODS ARE EVOLVING BUT ALSO ANOTHER SLIDE IN HIS TALK, THIS ONE I DO APOLOGIZE THIS DIDN'T COPY WELL, NOT ONLY DO I NOT HAVE PERMISSION BUT I DON'T HAVE THE SLIDE RIGHT. WHEN YOU SEE -- THIS IS WHAT HAPPENS WHEN YOU GO FROM MAC TO PC IT LOOKS GREAT ON THE MAC THERE WAS BOX AROUND THE MIXED METHODS IN SOCIAL SCIENCE, HE HAD A BOX AROUND QUANTITATIVE THEN ROUGHLY EQUALLY SIZED BOX WHERE IF I HAVE IT CORRECTLY IN THE HEALTH SCIENCES THERE WAS REALLY LARGE BOX AROUND QUANTITATIVE. THEN MUCH SMALLER BOX AROUND QUALITATIVE. I THINK THAT AGAIN THINKING ABOUT NEXT GENERATION OF METHODS AND ANALYSIS THAT WE NEED TO BE THINKING ABOUT HOW TO BRING OUT BEST IN THESE METHODS. GET BEST BALANCE BETWEEN QUALITATIVE AND QUANTITATIVE. I'VE BEEN THINKING ABOUT SOME OF THE OTHER THINGS WE'RE DOING IN OBSSR. ONE OF OUR BIG AREAS IS IN THE M-HEALTH AREA. THEY'RE DOING ALL SORTS OF NEW THINGS. THIS WAS MENTIONED TODAY, BUT I DO THINK THIS IS INTERESTING, SO, FOR EXAMPLE, THERE ARE TINY LITTLE CAMERAS LIKE THIS ONE, IN FACT THEY'RE EVEN SMALLER ONES NOW THEY'RE LITTLE PIECES OF JEWELRY THAT YOU CAN WEAR THAT HAVE MINIATURE OUT-FACING CAMERA. THAT TAKES ABOUT 5500 PHOTOS PER DAY ALLOWS TO YOU CHARACTERIZE OVER THE COURSE OF A DAY THE QUALITATIVE EXPERIENCES THAT PEOPLE HAVE. IN FACT THERE NEW SOFTWARE THAT'S AVAILABLE THAT ACTUALLY ANALYZES THESE AREAS SO IT COUNTS GREEN AREAS, SOFTWARE TAUGHT TO RECOGNIZE THINGS, ALL KINDS OF NEW DEVELOPMENTS IN TEXT ANALYSIS THAT JIMMY -- JAMIE PENNYBAKER AND OTHERS ARE USING WHERE YOU CAN CAPTURE TEXT THEN YOU CAN ACTUALLY ANALYZE TEXT IN NEW WAYS THAT WE HAVEN'T HAD BEFORE. I THINK THAT THESE REPRESENT NEW TECHNOLOGICAL DEVELOPMENTS THAT MIGHT GIVE US A BETTER UNDERSTANDING OF QUALITATIVE BUT ALSO ALLOW US TO CAPTURE INFORMATION FROM MUCH LARGER SAMPLES OF PEOPLE. SECOND THEME THAT WE'RE THINKING ABOUT IN OUR OFFICE IS BEHAVIORAL AND SOCIAL SCIENCES IN EVOLVING HEALTH CARE SYSTEM. USE THE WORD EVOLVING BECAUSE WE SEE HEALTH CARE SYSTEM CONTINUING TO CHANGE REGARDLESS OF WHAT HAPPENS IN THE SUPREME COURT IN THE NEXT COUPLE OF MONTHS. ONE OF THE IDEAS THAT I'VE BEEN INTRIGUED BY COMES FROM NEIL, MY COLLEAGUE AT UCLA AND LESTER, HE WAS REMARKABLE PERSON WHO CONTRIBUTED TO THE FIELD OF PUBLIC HEALTH OVER REMARKABLE PERIOD OF TIME. HE PASSED AWAY JUST A FEW WEEKS AGO AT AGE 97. HE WAS ACTUALLY PRETTY ACTIVE UNTIL ABOUT A YEAR AGO. WHEN I FIRST CAME TO UCLA LESTER HAD WE OVER TO DINNER ONE NIGHT, HE LIKES TO MAKE MARTINIS AND TELL STORIES, HE WAS SAYING, WHEN I WAS THE HEAD OF THE STATE HEALTH DEPARTMENT IN CALIFORNIA MY GOVERNOR -- MY BOSS, THE GOVERNOR CALLED ME IN ONE DAY EARL SAID TO ME -- TURNS OUT LESTER WAS THE HEAD OF THE STATE HEALTH DEPARTMENT IN CALIFORNIA WHEN EARL WARREN WAS THE GOVERNOR OF CALIFORNIA. HE CONTINUED IN THIS ACTIVE CAREER ALL THE WAY UP TO THE END AND WROTE A BOOK, PUBLISHED IN 2005 ABOUT EVOLUTION OF PUBLIC HEALTH. ONE OF THE THINGS THAT NEIL AND LESTER SAY THAT THERE HAVE BEEN THREE ERAS IN DELIVERY OF HEALTH CARE. FIRST WAS -- THESE WERE GOVERNED BY THE DOMINANT SCIENTIFIC UNDERSTANDING AT THE TIME. FIRST HEALTH CARE SYSTEM, BEGINNING OF LAST CENTURY FROM THE BEGINNING OF THE DELIVERY OF MEDICINE WAS REALLY VERY MUCH BASED ON THE GERM THEORY OF DISEASE. THAT WE THOUGHT THAT ALL DISEASE WAS CAUSED BY INFECTIONS AND GERMS. WE -- LOOK AT THE WAY HOSPITALS ARE CONSTRUCTED TODAY THE WAY WE TRAIN RESIDENTS MOST IS BASED ON THIS NOTION THAT YOU FIND IT AND YOU FIX IT. THAT HEALTH CARE ABOUT FINDINGS THINGS AND FIXINGS THINGS DEALING WITH THE OUTBREAKS. BUT IN FACT SCIENTIFIC UNDERSTANDING MOVES ALONG WE HAD FRAMINGHAM HEART STUDY AND OTHER STUDIES THAT HELPED US UNDERSTAND THAT THERE WAS MORE THAN GERMS. GERMS WERE STILL THERE BUT IN FACT MOST PEOPLE THAT WERE USING HEALTH CARE SYSTEM HAD CHRONIC ILLNESSES ABOUT 70% OF THEM. WE STARTED THINKING ABOUT MULTIPLE RISK FACTORS AND INTERVENTIONS IN MID LIFE WHEN PEOPLE STARTED GETTING HIGH BLOOD PRESSURE AND HIGH CHOLESTEROL WE HAD TO REORIENT HEALTH CARE SYSTEM TO DEAL WITH PEOPLE WHO MIGHT APPEAR TO BE RELATIVELY HEALTHY AT THE TIME BUT ADD A THESE TICKING -- TICKING TIME BOMBS, MY CHOLESTEROL, HIGH BLOOD PRESSURE, SOFTER. OR UNDETECTED CANCER. THIS IS PRETTY MUCH HEALTH CARE SYSTEM THAT WE CURRENTLY HAVE. WE BUILT THE SYSTEM THAT DEALS WITH BOTH GERMS AND RISK FACTORS. BUT IN FACT AS ALL OF YOU PEOPLE HAVE BEEN SHOWING OUR SCIENTIFIC UNDERSTANDING HAS MOVED ON. WE NO LONGER THINK THAT IT'S JUST GERMS AND RISK FACTORS, THERE ARE COMPLEX SYSTEMS THAT THERE ARE THINGS HAPPEN THROUGHOUT LIFE SPAN AND THERE ARE MULTIPLE SOCIAL DETERMINANTS OF DISEASE AND ILLNESS. OUR SCIENTIFIC SYSTEM -- UNDERSTANDING NOW HAS GONE BEYOND SYSTEM THAT WE CURRENTLY HAVE. THEY SAY YOU CAN THINK OF THIS LIKE SYSTEMS, DOS SYSTEMS. WE HAD DOS 1.0 THEN DOS 2.0. WE HAD HEALTH CARE SYSTEM THAT WAS BUILT TO DEAL WITH GERMS AND WE REBUILT IT TO DEAL WITH BOTH GERMS AND RISK FACTORS. PROBLEM IS WE NOW HAVE ALL THIS NEWING THAT IS COMING FROM MIXED MED ODDS AND OTHER PRESEARCH WE HAVE OUR HEALTH CARE SYSTEM THAT DOESN'T REALLY RESPOND. LIKE HAVING AN APP THAT RUNS WELL BUT YOU TRY RUN ON DOS 2.0. ARGUMENT THAT WE NEED TO REBUILD HEALTH CARE SYSTEM IN WAY THAT DEALS WITH THIS COMPLEXITY AND MORE COMPLEX UNDERSTANDING. JUST GIVE YOU ONE KELP PELL, I'M PRETTY MUCH OUT OF TIME. EARLIER THIS WEEK WE SPONSORED MEETING OR SHARED MEDICAL DECISION MAKING. SOMETHING THAT EVOLVED OUT OF MIX HEALTH ODD RESEARCH. THIS ONE RATHER SHOCKING EXAMPLE. SURVEY CONDUCTED IN 1997-98 THEY ASKED PEOPLE WHO HAD ALREADY BEEN CONSENT TO GO -- UNDERGO PCI, ANGIOPLASTY AND PERHAPS STINTING, THEY ASKED WHETHER OR NOT THIS PROCEDURE WOULD MAKE THEM LIVE LONGER AND WHETHER OR NOT IT WOULD PREVENT RECURRING MYOCARDIAL INFARCTION. 75% SIGNED CONSENT FORM SAID, YES, MAKE THEM LIVE LONGER. 75% THOUGHT IT WOULD PREVENT RECURRENT HEART ATTACK. 71% BELIEVED IT WOULD MAKE THEM LIVE LONGER. THEN AFTER THAT, BETWEEN -- NEXT TEN YEARS AFTER THAT, LOTS OF STUDIES CAME OUT THERE WAS METAANALYSIS OF 61 TRIALS INVOLVING 25,000 PATIENTS. THEY SHOWED IN FACT DOESN'T MAKE YOU LIVE LONGER AND DOESN'T PREVENT RECURRENT HEART ATTACK. THEY REDID THE STUDY. IN 2007-8 AT THE TIME WHEN MOST OF THESE CARDIOLOGISTS WERE QUITE AWARE OF STINTING DIDN'T MAKE YOU LIVE LONGER OR PREVENT RECURRENT M.I. STILL 88% OF THOSE CONSTONED PCI THOUGHT IT WOULD PREVENT RECURRENT M.I., 76% THOUGHT WOULD MAKE THEM LIVE LONGER. BASIC NOTION OF INFORMED CONSENT IS REALLY WRONG. SOMETHING AS DRAMATICALLY GONE AWRY HERE. ONLY MIXED METHODS RESEARCH CAN DEAL WITH THESE SORTS OF PROBLEMS. FINAL ISSUE THAT WE'RE DEALING WITH NEXT GENERATION OF BEHAVIORAL AND SOCIAL SCIENTISTS JUST SAY THAT, WE THINK THAT THIS IS REALLY GETTING SHAKEN UP. THAT WE THINK THAT THE PH D PROGRAM OF THE FUTURE HAS TO LOOK DIFFERENT BECAUSE MOST OF THE PH D PROGRAMS ARE NOT DEALING WELL WITH AREAS LIKE MIXED METHODS AS WELL AS OTHER THINGS WE'RE DEALING WITH. MASSES OF DATA AS WELL. NEW PEOPLE PROBABLY BE COLLECTING LESS ORIGINAL DATA USING DATABASES MORE WE THINK THAT YOU CAN'T SORT OF DO THAT WITH THE WAY WE'RE TEACHING STUDENTS NOW. WE HAVE TO CHANGE THAT. THE AAMC, AMERICAN ASSOCIATION OF MEDICAL COLLEGES CAME OUT WITH REPORT IN NOVEMBER SAYING THAT SOCIAL AND BEHAVIORAL SCIENTISTS NEED TO BE BIGGER PART OF THE MEDICAL SCHOOL CURRICULUM IN 2015 THE MCA IS BEING REVISED TO HAVE ABOUT 20% OF THE CONTENT IN BEHAVIORAL AND SOCIAL SCIENCES. WE HAD PIECE ABOUT THIS WHICH I ENCOURAGE TO YOU LOOK AT. BY WAY OF SUMMARY, THERE ARE LIMITATIONS TO QUANTITATIVE METHODS. THERE ARE LIMITATIONS TO QUALITATIVE METHODS. BUT TREMENDOUS POTENTIAL FOR MIXED METHODS THEY CAN YIELD-CYTES THAT NEITHER QUALITATIVE OR QUANTITATIVE METHODS CAN PROVIDE BY THEMSELVES. BUT DON'T FORGET THAT THEY'RE GOING TO BE IMPORTANT CHALLENGES, YOU JUST TALKED ABOUT CHALLENGES AND REVIEW PROCESS, I SERVED AS JOURNAL EDITOR FOR TWO DIFFERENT JOURNALS OVER LAST 12 YEARS. I'M AWARE OF ALL THE ISSUES THAT WERE DISCUSSED IN THAT LAST PANEL. ALSO BIG ISSUES IN RESEARCH VOICE DEALT. THIS IS FIELD THAT HAS TREMENDOUS POTENTIAL BUT HAVE TO PULL TOGETHER AS FIELD TO GET THE ATTENTION THAT YOU NEED FOR THIS RESEARCH TO PERSUADE THESE CENTERS AT THE NIH THAT THEY NEED TO DEVOTE LARGERS PORTIONS OF THEIR PORTFOLIO OF THIS FORGETTING RIGHT PEOPLE ON REVIEW PANELS AND WRITING REALLY GOOD GRANTS BUT GET REVIEWERS EXCITED. I HAVE GONE OVER BY A MINUTE. I APPRECIATE YOU INVITING ME TODAY. I'M VERY EXCITED ABOUT WHAT YOU GUYS ARE DOING. AGAIN, I'M VERY SORRY THAT THERE ARE NO COCKTAILS, NO JUG WINE AND NO COFFEE. BUT THANKS AGAIN FOR COMING. [APPLAUSE]d >> THANKS EVERYBODY FOR HANG IN. ACTUALLY, PLEASE FILL OUT YOUR EVALUATIONS AND LEAVE THEM AT THE DESK AS YOU LEAVE. THE WHOLE CONFERENCE WAS VIDEO COST ARE WEBCAST, IT WILL BE AVAILABLE IN A FIVE DAYS ON THE OBSSR WEBSITE. AS WILL SLIDES WITH THE PERMISSIONS FROM THE AUTHORS,