>> WELCOME EVERYONE TO THE NATIONAL CHILDREN'S ADVISORY COMMITTEE MEETING. THE AGEND IS BEFORE US. WE'RE A BIT LATE BUT WE'LL GET STARTED. I'M RANDY OLSEN, FROM OHIO STATE UNIVERSITY. I'M OUR ACTING CHAIR FOR THE DAY. THANKFUL SO I WOULD LIKE TO WELCOME EVERYONE AND I GUESS WE SHOULD START WITH INTRODUCTIONS SO THAT YOU KNOW WHO EVERYBODY IS, SO LET'S JUST ZIP AROUND THE ROOM AND KEEP IT SHORT, STEVEN. >> STEVEN HIRSCHFELLED FROM THE UNIVERSITY OF MARYLAND. >> MARIAN BALL, ACTING FEDERAL AFICIAL FOR THE ADVISORY COMMITTEE. YVONNE MADDOX, DEPUTY DETECTED DIRECTOR OF NICHD. YOLANDIA, UNIVERSITY OF TEXT AT AUSTIN SOCIAL WORKS. >> CARMEN, AT STANFORD YESTERDAY. >> JENNIFER MADDEN ASSOCIATE DIRECTOR FOR SCIENCES AND NATIONAL CENTER FOR HEALTH STATISTICS. >> HI, JOSLYN, QUALITY IMPROVEMENT COACH WITH THE NCS COLLABORATIVE IMPROVEMENT AND INFORMATION NETWORK. >> GOOD MORNING I'M MARK HUDYAK, FROM UNIVERSITY OF FLAIR FLAIR IN JACKSONVILLE, ONE OF THE SPEAKERS TODAY AND SERVED AT THE LOCAL PRINCIPLE INVESTIGATOR FOR BAKER COUNTY IN NCS. >> HELLO I'M ADDA, GRIMBERG. >> I'M LAURA BESKOW. >> JOE KONSTAN, COMPUTER SCIENTIST FROM THE UNIVERSITY OF MINNESOTA. >> ALMA KUBY, SURVEY METHODOLOGIST CHICAGO. >> ALAN GIEWT--GUTTMACHER, DIRECTOR OF NICHD. >> JESSICA GRUBER. >> NANCY. >> DAVE HUBEL. >> SARAH GOLD STEIN BRANCH CHIEF IN OFFICE AND ACQUISITIONS OF NICHD. >> GOOD MORNING, JACK FROM THE NCS PROGRAM OFFICE. MARY ELEN MORTEN SEN ILLEGALSEN, CENTER FOR DISEASE CONTROL AND NATIONAL CENTER FOR ENVIRONMENTAL HEALTH. >> RICH BOOZE ALLEN. >> BRADLEY WITH BOOZALLEN. >> EMILY WITH BOOZE ALLEN. >> ELAINE, WITH BOOZE ALLEN HAMILTON. >> AMY BRANEM FROM THE NATIONAL CENTER FOR HEALTH STATICS. >> MELANIE YOUNG USCPA. >> GOOD MORNING MARIA LOPEZ, NCS PROGRAM OFFICE. >> BRIANA WOODSEN NCS PROGRAM OFFICE. >> HI, NED, UNIVERSITY OF CHICAGO. >> GOOD MORNING LAURA COLLFIELD, PI OF THE NATIONAL CHILDREN'S STUDY MONTGOMERY COUNTY MARYLAND CENTER. THANK YOU. >> JOHN OFFICE OF DISEASE PREVENTION AT NIH. >> LISA NICHD. >> JOHN EXECUTIVE OFFICER NICHD. >> GOOD MORNING RUTH BENNER AND THE NATIONAL CHILDREN'S STUDY PROGRAM OFFICE. >> JAMES BROMBERGER, AMERICAN ACADEMY OF PEDIATRICS. >> CHRISTINA PARKS, NCS PROGRAM OFFICE. >> GOOD MORNING I'M KATAN, FROM THE NCS PROGRAM OFFICE. >> GOOD MORNING, I'M MICHAEL DELARK O, I'M WITH THE NCS PROGRAM OFFICE. >> MARRING RETE CLARK SON, PRICE WATER HOUSE COOPERS. >> CHRISTIAN, WITH PRICE WATER HOUSE COOPERS. >> ZACH, WITH BOSTON CHILDREN'S. >> DAVID JONES NCS PROGRAM OFFICE. >> TINA ERV, NICHD. >> LESLIE COOK, NCS PROGRAM OFFICE. >> DANIEL CLUDIA PROGRAM OFFICE. >> OKAY, THANK YOU SO LET'S TURN IT OVER TO DR. GUTTMATCHER AND DR. HERSCHFELLED WHO GIVE THEIR WELCOME AND FIRST OF ALL LET ME JUST THANK EVERYONE FOR TAKING YOUR TIME TO BE HERE, PARTICULARLY THOSE WHO HAD TO FLY IN FROM PLACES TO GET HERE COMING TO WASHINGTON IN THE MIDDLE OF THE SUMMER IS USUALLY NOT SOMETHING ONE WOULD DO FOR THE HECK OF IT. SO WE APPRECIATE YOUR DOING THIS. I'M GOING TO SAY NOTHING ELSE OF A WELCOMING NATURE BECAUSE I WANT TO GET THE WORK DONE SO I'LL TURN IT OVER TO STEVE WHO WE HAVE A TAG TEAM SLIDE SHOW FOR YOU. >> THANK YOU DR. GUTTMATCHER, SO WE WANTED TO INTRODUCE A FEW TOPICS AND THE FIRST IS THAT WE WILL GIVE YOU AN UPDATE ON THE STAT US OF THE NCS VANGUARD STUDY WHICH BEGAN IN 2009 AND THIS SLIDE IS A BIT BUSY BUT WE TRY TO MAKE A FEW POINTS HERE AND THE FIRST IS THAT WE HAD SEVERAL TYPES OF RECRUITMENT STRATEGIES THAT WE HAVE EXAMINED SYSTEMATICALLY OVER THE PAST FEW YEARS ON THE COLUMN, ONE OF THESE MIGHT BE A POINTER, I DON'T KNOW. ANYWAY, THERE'S A COLUMN THAT'S RATHER DARK AND JUST TO THE RIGHT OF THAT IS A COLUMN THAT SAYS INITIAL HOUSEHOLD BASE, THAT'S WHERE WE STARTED IN JANUARY 2009, IT'S SEVEN LOCATIONS AND FOR ABOUT A YEAR AND HALF, WE EXAMINED THE KINETICS PARAMETERS OF RECRUITING STUDY PARTICIPANTS BY GOING DOOR TO DOOR IN SPECIFIC GEOGRAPHIC AREAS AND WE RECRUITED ABOUT 15, CLOSE TO 2000 WOMEN AND WE HAVE ABOUT 1500 CHILDREN IN THERE AND WHAT I'LL CALL YOUR ATTENTION TO IS ON THE BOTTOM ROW THERE IS SOME RATIOS AND THE FIRST RATIOS THAT NUMBER OF WOMEN THAT WE CONTACTED TO THE NUMBER OF WOMEN ENROLLED AND THAT'S ROUNDING OFF TO BE ABOUT 18. AND THEN ON THE BOTTOM ROW IS A NUMBER WHICH SAYS THE ROUNDING OFF ABOUT 23 WOMEN WERE CONTACTED FOR EVERY CHILD ENROLLED. SO SUBSEQUENTLY WE WANTED TO COMPARE GOING DOOR TO DOOR BUT WITH USING ALTERNATE MEANS OF ENGAGING THE GENERAL POPULATION AND SEE HOW THEY WOULD ENROLL INTO THE STUDY AND I'LL JUST FOCUS NOW ON THE BOTTOM ROW, TWO ROWS, SO, THE COMPARA TOR IS THE MIDDLE COLUMN, IT'S CALLED ENHANCED HOUSEHOLD BASE. WE ENHANCE BECAUSE WE--ON TOP OF OUR RECRUITMENT EFFORTS, A DIRECTED MEDIA CAMPAIGN, AND THERE THE VERY LAST CELL IN THAT COLUMN SHOWS THAT THE NUMBER OF WOMEN CONTACTED TO THE NUMBER OF CHILDREN ENROLLED ROUNDED OFF IS 23 WHICH IS ALMOST IDENTICAL TO WHAT OUR PRIOR EXPERIENCE WAS SO WE HAD SOME CONFIDENCE IN WHAT THE SYSTEM COULD GENERATE USING THAT APPROACH. WE THEN COMPARED THAT TO AN APPROACH WHERE WE WENT DIRECTLY TO THE PUBLIC AND USED ADVERTISEMENT AND MEDIA VARIOUS FORMS, EVENTS TO BRING ATTENTION TO THE STUDY AND HERE AGAIN GO TO THE TWO BOTTOM ROWS AND IN THAT COLUMN, THE NUMBER IS 8.4 WOMEN WERE CONTACTED FOR EVERY WOMAN ENROLLED AND GROUNDED SLIGHTLY UP. 18 WOMEN WERE CONTACTED FOR EACH CHILD ENROLLED AND THEN WE LOOKED IN HEALTHCARE PROVIDER CLINICS AND OFFICES AND THAT'S THE COLUMN THAT IS LABELED PROVIDER BASED ON THE SLIDE. IT'S UNDER THE S IN STUDY IN THE TITLE AND IF YOU GO DUNE TO THE BOTTOM YOU'LL SEE THERE, THAT FOR ENROLLING ONE WOMAN,w3 WE CONTACTED 2.6 WOMEN AND THEN FOR EVERY CHILD THAT WAS ENROLLED WE HAD A CONTACT 3.5 WOMEN, SO THIS IS IMPRESSIVELY DIFFERENT IN TERMS OF EFFICIENCY THAN THE OTHER APPROACHES, SO WE THEN BEGAN TO REFINE THAT APPROACH. IT'S IN THE FIELD NOW, RECRUITMENT WILL BE COMPLETED THIS MONTH SO THAT'S WHY AT THE BOTTOM OF THAT COLUMN WE HAVE IT IN PROGRESS BUT WE HAVE ENOUGH DATA NOW TO KNOW THAT WE'VE ENROLLED ABOUT 750 WOMEN AND THAT THE‡  RATIO HERE IS QUITE CONSISTENT WITH OUR EARLIER EXPERIENCE THAT WE CONTACT ABOUT TWO AND HALF WOMEN TO GET ONE ENROLLED. SO OVERALL WE HAVE APPROXIMATELY 4700 CHILDREN WHO ARE ENROLLED IN THE STUDY AND WITH THE NUMBER OF CHILDREN STILL AWAITING BIRTH FROM THEIR PREGNANT MOTHERS THAT ARE ENROLLED IN THIS STUDY, WE ESTIMATE WE'LL HAVE 5000 CHILDREN ENROLLED IN THE VAN GUARD STUDY WHEN WE COME PLETE THIS RECRUITMENT--COMPLETE THIS RECRUITMENT. I ALSO WANTED TO-- >> AND JUST TO BACK UP, THOSE NUMBERS ARE FOR THE PROVIDER BASED--MAYBE A LITTLE BIT PESSIMISTIC BECAUSE YOU ALL ALMOST CERTAINLY PICK UP--IN A STRETCH THAT MIMICS YOUR FULL IMPLEMENTATION, YOU'LL PICK UP SUBSEQUENT BIRTHS FROM SOME OF THOSE SAME WOMEN SO YOUR YIELDS PER WOMAN CONTACT WILL BE EVEN BETTER THAN THIS, IS THAT FAIR? >> RIGHT, IN FACT WE INTEND TO DO THAT IN THE VANGUARD STUDY AND PROBABLY A YEAR FROM NOW WE WOULD BE ABLE TO SHARE WITH YOU THOSE DATA. BUT-- >> AND IF MEMBERS OF THE ADVISORY COMMITTEE HAVE QUESTIONS, AS TO DISCUSSION GOES, JUST TICK UP THE HAND AND WE'LL STOP, THAT'S WHY WE'RE HERE. ON THE TOPIC OF SHARING DATA. WE ARE KEEN ON SHARING DATA. WE HAD BEEN PLANNING FOR THIS FOR SEVERAL YEARS AND I WANTED TO UPDATE THE COMMITTEE ON OUR RECENT ACTIVITIES SINCE WE LAST MET IN FEBRUARY 2013 SO WE HELD HELD A DATA USERS WORKSHOP, A FULL DAY PUBLIC EVENT TO IS LICIT PRACTICES AND EXPECTATIONS THAT WOULD INFORM THE POLICIES, AND THE--HOW WE MIGHT GO ABOUT SHARING DATA. WE THEN PUBLISHED IN MARCH 2013 ON OUR PUBLIC WEB SITE A CONCEPT OF OPERATIONS THAT DOCUMENT IS STILL OPEN FOR COMMENT. IF ANYONE WISHES TO PROVIDE INPUT, THEN WE HELD AN INVITATION ONLY WORKSHOP WITH OUR FEDERAL PARTNERS TO DISCUSS SPECIFICATIONS FOR ESTABLISHING ENCLAVE AND OUR THINKING IS THAT WE WILL PARTNER WITH SOME AGENCY OR ANOTHER AS A PRELIMINARY STEP BEFORE WE DECIDE WHAT OUR LONG-TERM APPROACH WILL BE AND USE THAT AS AN EXPLORATORY PERIOD AND WE THEREFORE ANTICIPATE HAVING A RELEASE FOR SPECIFICATIONS FOR ENCLAVE THIS YEAR AND WE THROUGH WHOEVER WE END UP PARTNERING WITH, WE EXPECT TO HAVE THE VANGUARD DATA AVAILABLE NEXT SPRING. SO THOSE ARE OUR ACTIVITIES AND TIME LINE FOR DATA SHARING. MORE TO COME ON THAT. >> I HAVE A QUICK QUESTION, YOU USE THE TERM DATA RENNE CLAVE,--ENCLAVE USUALLY THOSE ARE USED FOR CONFIDENTIAL DATA WHERE THERE'S A RISK OF REIDENTIFICATION. HOW ABOUT DATA SHARING AS A PUBLIC--PUBLIC USE DATA WHEREVER YOU'RE THERE. WELL, WE WOULD LIKE TO HAVE ONE INTEGRATED SYSTEMS SO THERE WOULD BE DIFFERENT LEVELS OF ACCESS, DIFFERENT POLICIES BUT WE ESSENTIALLY, ALL DATA WOULD BE ACCESSED EXCEPT FOR PERHAPS SOME HIGH LEVEL AGGREGATE DATA. ALL DATA WOULD BE ACCESSED THROUGH A DATA ACCESS COMMITTEE AND THEN, THE RESTRICTIONS ON THE ACCESS WOULD BE A FUNCTION OF THE PURPOSE OF THE USE AND THE NATURE OF THE DATA. BUT IT VIEW IT AS A CONTINUUM AND THEN ONCE SOMEONE ENGAGES WITH THE NCS, FOR ACCESSING DATA, THEN IT BECOMES A CASE BY CASE SOLUTION. WE DON'T INTEND TO HAVE SEPARATE DATA SOURCES IN WAYS OF ACCESSING THE DATA. SO THERE WOULD BE NO PUBLIC USE DATA EITHER FROM THE VANGUARD STUDY OR FROM THE FULL STUDY, IS THAT CORRECT. THERE CERTAINLY WILL BE PUBLIC USE DATA BUT IT WOULD ALL BYE-BYE--THERE WILL BE BUT WE HAVE TO FIRST ESTABLISH OUR SEE WHAT WE HAVE, SEE THE SPECIFICATIONS, MAKE SURE WE RESPECT THE PRIVACY OF THE INDIVIDUALS THAT WE FOLLOW DISCLOSURE POLICIES THAT ARE CONSISTENT WITH OTHER FEDERAL AGENCIES AND AND THAT THERE WILL BE OPPORTUNITIES. NOW PUBLIC USE, AT LEAST I HOPE WE'RE USING THE TERMS CONSISTENTLY WOULD MEAN THAT SOMEONE SAYS I'M INTERESTED IN LOOKING AT DATA AND THEN THEY INDICATE THEIR INTERESTS AND THEY FILL OUT A REQUEST TO ACCESS OUR DATA, AND THEN THEY GET THAT REQUEST APPROVED AND THEN THEY WOULD BE ABLE TO ACCESS, BUT WE AT THIS POINT DON'T FORESEE A SITUATION WHERE THERE'S JUST A WEB SITE WHERE ANYBODY IN THE WORLD WITHOUT CREDENTIALS OR IDENTIFICATION CAN GO IN AND START ACCESSING NCS DATA. >> HMM, WELL THAT'S BEEN DONE WITH MOST OTHER STUDIES, I THINK? WELL, SAY THAT WE ARE IN DISCUSSION WITH OTHER AGENCIES AND OTHER AGENCIES HAVE A SPECTRUM OF STRINCHENCYS ABOUT, AND WE HAVE TO BE CAREFUL AND PRECISE ABOUT THE TERMS WE'RE USING BECAUSE IF ONE WANTS TO ACCESS FOR INSTANCE THE DATA FROM THE NATIONAL CENTER FOR HEALTH STATISTICS, THERE ARE SOME AGGREGATE DATA NAYOU CAN JUST GO ON AND GET AND WE WOULD HAVE SOME FACILITIES, SOME OPTION LIKE THAT, BUT YOU HAVE TO MAKE A REQUEST AND THE--TO ACCESS DATA FROM-- >> OF COURSE IT WOULD BE RESTRICTED. I MEAN THERE WOULD BE RESTRICTED AND PUBLIC DATA SO THE RESTRICTED DATA WOULD HAVE MORE GEOGRAPHICAL DATA AND STUFF LIKE THAT, BUT THE PUBLIC DATA IS USUALLY JUST AVAILABLE TO ANYONE. RIGHT. SO AGAIN, THE PRECISION OF THE TERM SYSTEM WE WOULD HAVE AGGREGATE, SOMEONE WANTS TO SAY, YOU KNOW HOW MANY PEOPLE WERE ENROLLED OR OTHER QUESTIONS LIKE THAT, THOSE WOULD BE FREELYY AVAILABLE, BUT ANYBODY THAT WANTS LINE ITEM DATA OR SPECIFIC ANALYSIS OR WANTS TO DO ITEM SPECIFIC ANALYSIS, WOULD APPLY. >> IT BOILS DOWN TO THAT WHICH IS PERSONALLY IDENTIFIABLE AND THE GOALPOST ON THAT CONTINUES TO BE MOVING PARTICULARLY FOR STUDIES THAT HAVE GENOMIC DATA IN THEM BUT THOSE THINGS WHICH WERE PERSONALLY IDENTIFIABLE THEY WILL BE RESTRICTED. THOSE ARE NOT IDENTIFIABLE WILL BE GENERALLY AVAILABLE. >> RIGHT. AGAIN,FD’ SPECTRUM, WHAT WE'RE VIEW SUGGEST WE WOULD HAVE ONE FACILITY THAT WOULD HAVE A CONTINUUM OF ACCESS DEPENDING ON THE KIND OF DATA AND THAT SOME DATA WOULD BE GENERALLY AVAILABLE AND OTHERS WOULD REQUIRE APPLYING THROUGH A COMMITTEE TO ACCESS AND I'LL ASK DR. MENSK, CAN SOMEONE GO IN AND GENERATED THE DATA FROM ENHANES. >> YEAH, THE VAST MAJORITY OF DATA ARE AVAILABLE AND THE GENERAL USAGE OF THAT TERM MEANS IT'S POSTED ON A WEB SITE, ANYBODY CAN CAN GO IN AND GET IT BUT IT HAS BEEN VETTED FOR DISCLOSURE REVIEWS SO THAT THERE ARE BASED ON THAT REVIEW WHICH IS IN OUR CASE FAIRLY STRINGENT BUT IN ALL CASES VERY STRINGENT, IT IS DEEMED THAT IT'S NOT IDENTIFIABLE. NOW WE DO NOT PUT THE GENOMIC DATA ON THE WEB SITE, SO WE ONLY PUT A SUBSET OF INFORMATION FROM ALL OF OUR SURVEYS HOPEFULLY ON THE WEB SITE AFTER IT HAS BEEN VETTED AND OTHER INFORMATION IF GOTTEN IN OTHER WAYS THROUGH THE ENCLAVE OR SOMETHING ELSE, BUT I THINK THAT IN %#m EXPERIENCE THE TERM PUBLICLY AVAILABLE, PUBLIC USE, MEANS ACCESS TO THE INDIVIDUAL UNIT RECORDS WITHOUT ANY RESTRICTIONS. BUT I THINK WE ASK PEOPLE TO--YOU KNOW THEY HAVE TO CHECK OFF, THEY WON'T DO BLAH, BLAH, BLAH, BUT THAT'S IT. WE ASSUME THAT'S SAFE AND NO ONE CAN BE IDENTIFIED. THINGS ARE TAKING OFF THAT WOULD ALLOW THE IDENTIFICATION, IT TAKES TIME AND EFFORT BUT THAT'S THE GENERAL TERMINOLOGY. >> AND THAT'S A GENERAL APPROACH WE WANT TO TAKE, TOO. SO SOME DATA WILL BE READILY AVAILABLE AND OTHERS BECAUSE WE COLLECT A LOT OF DATA AND WE HAVE SO MANY KINDS OF DATA THAT CAN BE CROSS WALKED WITH GEOGRAPHIC DAT AGENOMIC DAT ADEMOGRAPHIC DATA AND SO FORTH. WE WANT TO BE CIRCUMSPECT ABOUT HOW WE'RE GOING FORWARD. DOES THAT ADDRESS-- >> YEAH, I THINK YOUR WILLINGNESS TO PUT OUT UNIT RECORD DATA CERTAINLY NOT ALL OF IT, YOU KNOW I DON'T THINK ANYONE WOULD EVER SUGGEST THAT THAT WOULD BE APPROPRIATE BUT THERE ARE WAYS AS JENNIFER SUGGESTS TOXIC EFFECTS COMPACTIFY THE DATA SO THAT IT'S MORE SUMMARY LEVEL AND MORE RESISTANT TO REIDENTIFICATION. >> RIGHT AND SO WE'RE USING THE TERM AGGREGATED BUT I THINK WE'RE CIRCLING AROUND THE SAME CONCEPT. SO TO SEGUE TO THE NEXT SECTION I JUST WANTED TO REFRESH OUR COLLECTIVE MEMORY THAT THERE WAS REVIEW OF THE NATIONAL CHILDREN'S STUDY AS IT WAS CONCEIVED IN 2007, BY A PANEL FROM THE NATIONAL RESEARCH COUNCIL, INSTITUTE OF MEDICINE. WE HAVE IN THE AUDIENCE REPRESENTATIVE OF THE COMMITTEE ON NATIONAL STATISTICS, DR. NANCY KIRKENDALL WHO RECALLS THAT REVIEW ALSO AND I WANTED US TO UNDERSTAND COLLECTIVELY WHERE WE WERE AND WHAT WE HAVE DONE IN THE INTERIM TO BROACH SOME OF THE POINTS THAT WERE RAISED IN THAT REVIEW. SO THE REVIEW NOTED FIVE STRENGTHS AND UNDER EACH OF THE POINTS THAT WERE RAISED IN THE REVIEW WHICH ARE IN BOLD FONT, THE CURRENT STAT STATUS OF THE STUDY IS IN A REGULAR FONT. SO, THE FIRST IS SET, IT WAS RESPONSIVENESS TO THE CHILDREN'S HEALTH ACT OF 2000 THE STUDY--WE HAVE BEEN EVOLVING IT CONTINUES TO RESPOND TO EACH COMPONENT OF THE CHILDREN'S HEALTH ACT. THE LARGE NUMBER OF BIRTHS TO BE INCLUDED AND AFTER MUCH DISCUSSION, WE HAVE RETAINED A TARGET OF ABOUT A HUNDRED THOUSAND LIVE BIRTHS INTO THE STUDY, THE LONGITUDINAL DESIGN STRETCHING FROM BEFORE BIRTH UNTIL AGE 21 AND BOTH THE VANGUARD AND THE PROPOSED MAIN STUDY DOES HE SIGN WILL CONTINUE TO FOLLOW PARTICIPANTS UNTIL AGE 21. THE MANY VARIABLES TO BE GENERATED ON BOTH OUTCOMES AND EXPOSURES AND SINCE THAT REPORT, THE POTENTIAL TO COLLECT DAT HAS INCREASED OVER THE PAST DECADE WITH NEW TECHNOLOGIES AND WHAT WE BELIEVE TO BE A MORE FLEXIBLE VISIT STRUCTURURE TO INCREASE THE NUMBER OF PRECISION AND EXPOSURES AND OUTCOMES AND THE WELL DESIGNED NATIONAL PROBABILITY SAMPLE AND PROPOSAL AS WE'VE DISCUSSED THIS COMMITTEE FOR THE LAST TWO YEARS, MAINTAINS A NATIONAL PROBABILITY SAMPLE WITH WE BELIEVE GREATER EFFICIENCY THAN IT IS PRIOR DESIGN. NEXT SLIDE PLEASE, I'M HAVING A LITTLE SENSITIVITY ISSUE HERE WITH THE REMOTE. SO THERE WERE NINE WEAKNESSES THAT WERE IDENTIFIED. THE ABSENCE OF INADEQUATE PILOT PHASE AND WE'VE BEEN DISCUSSING IN THIS FORUM, THE VANGUARD STUDY FOR THE PAST FOUR YEARS AND THAT VANGUARD STUDY IS ROBUST, WE'LL HAVE ABOUT 5000 CHILDREN ENROLLED AND WE'LL CONTINUE FOR 21 YEARS. THE DECENTRALIZATION OF DATA COLLECTION AND WE HAVE ALTERED THE BUSINESS MODEL SO THAT DATA COLLECTION IS CURRENTLY CONSOLIDATED INTO FOUR REGIONAL OPERATION CENTERS, A TRANSITION THAT BEGAN ALMOST A YEAR AND HALF AGO AND WILL BE COMPLETED BY OCTOBER 1st, 2013. AND WHAT WE WILL DO FOR THE MAIN STUDY WILL BE INFORMED BY THE VANGUARD STUDY EXPERIENCE. INADEQUACY TO MAXIMIZE RESPONSE RATES AND THAT'S PART OF OUR DISCUSSION TODAY BUT WE FOUND THAT HAVING A PROVIDER BASED RECRUITMENT AND RETENTION MODEL, SEEMED TO DEMONSTRATE NOT ONLY THE BEST EFFICIENCY AND RESPONSE RATES IN THE RECRUITMENT PHASE OVER ALTERNATES, BUT WE HAVE A COLLABORATIVE IMPROVEMENT NETWORK WHICH YOU'LL HEAR ABOUT IN GREATER DETAIL LATER THAT'S ACTIVE IN IMPROVING RETENTION. THE WEAKNESS OF A CONCEPTUAL MODAND HE WILL WE'RE PREPARED TO DISCUSS ALTHOUGH WE DIDN'T ELABORATE IN THE BRIEFING DOCUMENT ALL OF THE DETAILS BUT WE HAVE A FRAMEWORK THAT IS GROUNDED IN THE CONCEPTS OF HEALTH AND DEVELOPMENT NEXT SLIDE PLEASE. >> BEFORE WE GO ON TO THE NEXT SLIDE, CAN YOU GO BACK AND JUST CHECK-IN WITH THE ADVISORY COMMITTEE TO SEE WHETHER THEY HAD ANY--WHETHER THEY HAD ANY COMMENTS ON ANY OF THESE--ANY OF THE WEAKNESS ISSUES AND WHERE THE STUDY IS GOING IN THAT REGARD. I KNOW I HAD ONE QUESTION ABOUT ONE AND THAT WAS THAT THE PILOT PHASE IS FAIRLY--HAS BEEN FAIRLY ASSIDUOUS ABOUT THE ENTIRE ENROLLMENT PROBLEM WHICH IS OBVIOUSLY THE MOST IMPORTANT ISSUE AT HAND. BUT IN TERMS OF WHERE THE VARIOUS VANGUARD SITES ARE WITH THEIR PILOT STUDY AND THE CONTENT, THE SCIENTIFIC CONTENT OF WHAT'S GOING OUT, WHERE ARE WE THERE? SO THAT'S A SPECTRUM OF DIFFERENT STATUSES. WE HAVE THE MOST COMPLETE AND ROBUST DATA FROM THOSE LOICATIONS THAT WERE--STARTED EARLIEST AND THE LEAST DATA FROM THOSE THAT STARTED AT THE END. AND WE HAVE BEEN EXAMINING AND WILL PROVIDE FOR YOU NEXT TIME ON SPECIFIC DOMAIN AND ITEM ANALYSIS OF THE EARLY EXPERIENCE. BUT THE FRAMEWORK IS IN PLACE TO ADDRESS--AND HERE AGAIN WE REVOLVE AROUND DEFINITIONS AND TERMS. BECAUSE THERE'S A SCIENCE TO PROCESS AND WE ARE VERY FOCUSED ON PROCESS IN THE VANGUARD STUDY AND WE ARE EMPHASIZING THE FEASIBILITY THAT IS THE TECHNICAL PERFORMANCE, THE ACCEPTABILITY THAT IS THE IMPACT ON THE PARTICIPANTS ON THE--ON THE STUDY INFRASTRUCTURE AND THE COST AS VARIED BY MANY PARAMETERS OF EVERYTHING THAT WE DO. SO, WE HAVE BEEN EXAMINING THOSE DATA, AS WELL AS THE RESPONSES TO THE VARIOUS DATA DIRECTIONS AND INSTRUMENTS THAT WE HAVE MOVED THROUGH THE VANGUARD STUDY. WE BEGAN WITH AN APPROACH WHICH WAS TO COLLECT MANY TYPES AND A LARGE RELATIVELY LARGE BURDEN OF ITEMS ON EACH PARTICIPANT. I A KIND OF ONE-SIZE-FITS-ALL APPROACH AND THE VERY FIRST EXPERIENCE WAS THAT WHEN OUR STAFF WENT INTO THE FIELD, A HOME VISIT COULD TAKE MORE THAN HALF A DAY. THAT THEN BECAME A LITTLE MORE EFFICIENT BUT IT WAS STILL A THREE-FOUR HOUR ENTERPRISE INSTEAD OF A SIX TO EIGHT HOUR ENTERPRISE AND WE'VE THEN MOVED TO A PHASE OF THE STUDY WHERE WE EMPHASIZED THE RECRUITMENT KINETICS. SO WE PAIRED DOWN THE DATA COLLECTION INSTRUMENTS SO THAT WE WERE USING INSTRUMENTS THAT WERE USED IN OTHER STUDIES, BUT WE DIDN'T HAVE THE BREDTH OF DATA COLLECTION THAT WE HAD INITIALLY. SO WE HAVE A DISCONTINUITY OF WHAT WE WERE COLLECTING IN THE EARLY PHASE AND WE WERE COLLECTING A SUBSEQUENT PHASES. NOW THAT WE THINK WE HAVE A MUCH BETTER HANDLE ON RECRUITMENT, WE ARE REFOCUSING AND THAT'S PART OF THE DISCUSSION TODAY ON THE CONTENT OF WHAT WE WILL COLLECT BUT WE WANTED TO FOCUS THAT USING CRITERIA AND PRINCIPLES THAT WILL ALLOW US TO EMPHASIZE ASPECTS OF HEALTH AND DEVELOPMENT AND BEFORE THE FRAMEWORK FROM WHICH MANY OF THE INSTRUMENTS WERE CHOSEN WAS ESSENTIALLY SCREENING FOR DIFFERENT KINDS OF DISEASES AND CONDITIONS. SO THAT'S THE TRANSITION ALSO AND THAT'S GOING TO BELET INTENSE WORK OVER THE COMING YEAR TO YEAR AND HALF TO US EVOLVE CONTENT. THERE'S GOING TO BE SCIENTIFICALLY MEANINGFUL, INFORMATIVE AND FEASIBLE. >> I GUESS AS ALMA AND PROBABLY OTHER COMMITTEE MEMBER WHO IS HAVE BEEN INVOLVED IN PRIMARY DATA COLLECTION ARE ALL TOO PAIN EMPLOYLY FAMILIAR, ONE OF THE MAJOR PREMISES OF PRETESTS OR PILOT STUDIES IS TO GET A GRIP ON THE TIMING OF SPECIFIC STRINGS OF QUESTION AND EXACTLY HOW MUCH IN THE WAY OF TIME THEY TAKE. HOW WELL PROVISIONED ARE WE IN TERMS OF BEING ABLE TO LOOK AT MODULES OF QUESTIONS THAT HAVE APPEARED IN THESE VARIOUS PILOTS AND THE VARIOUS SITES--KIND OF THE SORT OF SEMIFACTORRIAL DESIGN THAT WOULD ALLOW US TO LOOK AT, WELL IN THIS SITE THEY ASK THESE QUESTIONS AND THIS IS HOW LONG IT TOOK AND SO FORTH. >> SO I CAN RESPOND TO THAT, WE TOOK THAT WHAT I WOULD CALL AN OPERATIONAL PERSPECTIVE QUITE SERIOUSLY AND IT ALL STARTED ONE DAY WHEN WE RECONSTRUCT OUR MEMORY RESPONSE--MEMORYMEMORYMEMIORSS THAT WE SPENT ON IT, AND TRYING TO DESIGN OPERATIONAL ELEMENTS N. SOME CASES THESE ARE CALLED PARADAT BAH WE WANT TO FOCUS ON STUDY OPERATIONS SO THAT WE COULD ADDRESS EXACTLY THOSE QUESTIONS YOU'RE RAISING DR. OLSEN. AND THAT IS, WHAT'S INVOLVED, HOW LONG DOES IT TAKE, WHAT'S THE ITEM RESPONSE RATE RATE. HOW MUCH EQUIPMENT DID YOU HAVE TO BRING IN, WHAT WAS THE LOGISTICS AND SUPPLY CHAIN TO BRING THOSE IN, WHAT HAPPENED TO THE DATA AFTER IT WAS COLLECTED ONSITE, WAS IT ARCHIVED, WAS IT TRANSMITTED, WHAT WAS THE QUALITY? SO THOSE ARE SPECIFICALLY WHAT WE DEVELOPED AND WE HAVE ACTUALLY SEVERAL HUNDRED OF THESE OPERATIONAL DATA ELEMENTS TO SYSTEMATICALLY CAPTURE ALL THOSE OPERATIONAL COMPONENTS OF IMPLEMENTING THE STUDY SO IN ADDITION TO WHATEVER THE RESPONSE MAY BE TO THE QUESTION, DID YOU FEED CEREAL THAT RED, GREEN, OR BLUE THIS SOUTH MORNING, WE WILL HAVE A LOST DATA AROUND THAT SPECIFIC INQUIRY. SO YEAH, THE TIMINGS ARE PROBABLY THE NUMBER ONE. >> I APOLOGIZE JEFFREY THAT ARE YOU ON THE PHONE AND WE NEGCOLLECTED TO DAK YOUR INTRODUCTION. YOU ARE THERE? >> YES I AM, I HOPE CAN YOU HEAR ME, I'M JEFF CRUSHER AND I'M IN VOLVE WIDE STUDY OF BIOSTATISTICS AT THE UNIVERSITY OF SOUTH FLORIDA. >> THANK YOU. >> AND JUST SHOUT OUT WHEN YOU HAVE ANY QUESTIONS. >> SO I WILL GIVE CREDIT TO DR. BOLSOM, OUR DESIGNATED FEDERAL AFICIAL. JEFF, NOT OUT OF SIGHT, OUT OF MIND, OKAY? >> THANK YOU, I'M LISTENING. >> OKAY, SO NEXT SLIDE THEN? >> DOES ANYBODY HAVE ANY QUESTIONS ON ANYTHING? ALMA? PLEASE? >> I HAVE A QUESTION ABOUT THE IOM POINT OUTS DESENSITIZATION OF DATA COLLECTION. IN THE LONGER VERSION THEY HAD MADE A COMMENT ABOUT METHODS USED IN THE NATIONAL LONGITUDINAL STUDY OF ADOLESCENT HEALTH AND I WAS WONDERING IF IN TERMS OF THE IMPROVEMENT THAT WE'RE GOING TO BE ADDRESSING SOME OF THE IOM REPORT IF THAT'S BEEN CONSIDERED? >> SO THE SHORT ALMOST GLIB ANSWER IS YES. WE HAVE BEEN MEETING REGULARLY WITH REPRESENTATIVES FROM MULTIPLE AGENCIES WHO HAVE CONDUCTED LONGITUDINAL STUDIES. WE HAVE A FEDERAL CONSORTIUM THAT HAS MET THREE TIMES IN THE LAST EIGHT MONTHS, BUTEE ALSO HOLD WORKSHOPS WHERE WE BRING PEOPLE TOGETHER, AND NOT ONLY FROM THAT STUDY, BUT OTHERS, AND WE KEEP LOOKING FOR NOT ONLY BEST PRACTICES BUT WE WANT TO KNOW ALL PRACTICES. BECAUSE YOU DON'T KNOW HOW SOMETHING'S GOING TO IMPORT OR BE ADAPTED AND WE WOULD LIKE TO HAVE THE COMPLETE PICTURE THAN THE BOTTOM LINE. WELL, YEAH, I WAS GOING TO SAY, NOW WE'RE IN PARTICULAR TRYING TO ENGAGE IN A STUDY CALLED TEDDY WHICH IS A LONGITUDINAL STUDY ON THE ENVIRONMENT AND DEVELOPMENT TYPE ONE DIABETES AND YOUTH, THAT'S THE WHY. AND THERE ARE MANY OVERLAPS IN THEIR METHODS, TOO. SO WE TRY TO BE INFORMED. OKAY, SO THE NEXT TWO POINTS THIS, IS FIVE AND SIX OUT OF NINE IS THAT THERE WAS WEAKNESS IN CERTAIN DATA INSTRUMENTS AND WE HAVE INSTITUTED SEVERAL METHODOLOGIES, I WILL SAY BEFORE WE GO INTO THE FIELD, I WANT TO GIVE CREDIT TO OUR PARTNERS AT WEST STAT WHO HAVE EVOLVED A CHOREOGRAPHY TEAM AND THEY SPECIFICALLY TYME THE VISITS AND TRY TO GO THROUGH THE MOTIONS AND WORK THROUGH WHAT THEY CONSIDER TO BE SEQUENCES WHICH FUNCTION WELL. SO IN ADDITION TO THE EXECUTION OF THE STUDY, WE HAVE TO FOCUS ON THE CONTENT AND WE'RE FOCUSING ON CONTENT IN MULTIPLE WAYS. SO WE HAD A ROBUST RESEARCH PROGRAM WHICH WE NOW SCALED BACK BUT I THINK EVERYONE WHO WAS INVOLVED IN THE STUDY IN THE PAST FEW YEARS HAD AN OPPORTUNITY TO DO TO DO SPECIFIC METHOD LOGIC QUESTION SO THAT WE COULD FIND OUT WITH OUR OWN DATA WHAT WAS GOING TO BE MOST APPROPRIATE AND FUNCTIONAL FOR THE NATIONAL CHILDREN'S STUDY, AND OUT OF THAT GREW A HEALTH MEASUREMENTS PROGRAM WHICH IS NOW INTEGRATED WITH OUR REGIONAL OPERATIONS CENTERS WHERE WE EXAMINE SYSTEMATICALLY, AVAILABLE INSTRUMENTATION, AVAILABLE ASSESSMENTS AND THEN WE ASK, IS IT APPROPRIATE FOR THESE AGENCIES, VISITS, THESE POPULATIONS, IF NOT, HOW CAN IT BE ADAPTED? AND IF THERE'S NOTHING AVAILABLE, IS THERE SOMETHING THAT COULD BE DEVELOPED? AND WE WORK CLOSELY WITH SOME TRANSNIH INITIATIVES IN IN REGARD, INCLUDING THE PROMISE INITIATIVE, THE PATIENT REPORTED OUTCOMES AND WITH THE NIH TOOL BOX FROM THE NATIONAL INSTITUTE OF NEUROLOGIC DISEASES AND STROKE AND AND WITH SOME OF THE OTHER NIH INITIATIVES TO EXAMINE WHAT IT IS THAT CAN BE STANDARDIZED CAN BE USED AT MULTIPLE AGES AND THEN FITS THE TRIFECTA WITH BETTER, CHEAPER, FASTER. THE INSUFFICIENT RACIAL AND ETHNIC DISPARITIES SO IN OUR PROPOSED PLANS WE TEND TO US HOSPITALS AND BIRTHING CENTERS WHERE OVER 98% OF CHILDREN ARE BORN WHICH IS RELATIVELY UNBIASED POPULATION. AND THEN YOU'LL HEAR FROM DR. GUTTMATCHER TO ADDRESS SPECIFIC OPPORTUNITIES FOR SPECIAL POPULATIONS AND THEN WE INVESTED INFORMATIVE RESEARCH TO DEVELOP TOOLS AND METHODS TO ENGAGE SPECIFIC POPULATIONS SUCH AS BUT NOT LIMITED TO AMERICAN INDIANS. >> ON FIVE, DID THE IOM SPECIFY WHICH OF THESE CERTAIN DATA INSTRUMENTS WERE IN DETAIL OR--OR NOT? >> IN THEIR REPORT THEY DTHEY MADE RECOMMENDATIONS, WE DID OUR OWN ANALYSIS BECAUSE THE PORTFOLIO OF DATA INSTRUMENTS THAT WAS PROPOSED TO THE IOM IN 2007 WAS NOT THE PORTFOLIO THAT WENT INTO THE FIELD AND BETWEEN THIS REPORT AND GOING INTO THE FIELD THERE WERE OTHER ADJUSTMENTS, BUT WE TOOK THE SYSTEM TO HEART AND REVIEWED THE PROCESS. >> AND AT THAT TIME DID THE PORTFOLIO THINGS THEY REVIEWED INCLUDE THE DEVELOPMENTAL, SOCIAL COGNITIVE AND DEVELOPMENTAL EFFORTS OR WERE THEY PRIMARILY ASSESSING THE APPROPRIATENESS OF THE CLINICAL MEASURES. >> OH NO, THEY INCLUDE VERY MUCH THE FORMER CATEGORY.Ž AND THEN ON SIX CAN YOU SAY MORE ABOUT THAT IN TERMS OF OVERSAMPLES OF DIFFERENT GROUPS IN TERMS OF ATTENTION TO RACIAL AND ETHNIC DISPARITIES. I'M GOING TO GIVE YOU TWO PART ANSWER. THE FIRST IS THAT WE WERE VERY CONCERNED ABOUT HOW OUR RECRUITMENT METHODS MIGHT IN ONE WAY OR ANOTHER INFLUENCE THE PEOPLE THAT WOULD COME INTO THE STUDY. SO WE USED AS A REFERENCE FRAME, BOTH THE AMERICAN COMMUNITY SURVEY DATA OF THE COUNTIES WHERE WE WERE RECRUITING IN AS WELL AS NATALITY DATA WERE AVAILABLE FROM THE SAME LOCATIONS AND THE GOOD NEWS IS THAT WE DID NOT DETECT IN GENERAL SYSTEMATIC BIAS BUT WE DID FIND THAT WHEN WE USED HEALTHCARE PROVIDERS, WE WERE ACTUALLY ABLE TO HAVE BETTER RECRUITMENT OF WOMEN WHO HAD LESS EDUCATION, WERE LOWER SOCIOECONOMIC STATUS AND WERE NOR DIVERSE. SOIME GOING TO CALLIFY--MORE DIVERSE SO I'M GOING TO QUALIFY BY SAY HANDWRITING YOU RECRUIT PREGNANT WOMEN, YOU DON'T RECRUIT THEM ALL THE SAME STAGE OF PREGNANCY AND WE'VE HAD DISCUSSIONS FOR A COUPLE OF YEARS NOW ON HOW WE CAN RECRUIT WOMEN EARLY IN PREGNANCY AND AND WE FIND THAT ALTHOUGH RAY--A MAJORITY OF WOMEN HAVE SOME PRENATAL CARE, THE DURATION OF THAT PRENATAL CARE, THE LOCATION OF THAT PRENATAL CARE IS HIGHLY VARIABLE AND SO, WHAT WE THEN HAVE TO LOOK AT IS WE HAVE TO RELATE, THE TIME, THE WOMEN CAME INTO THE STUDY, DURING THE CONTINUUM OF THEIR PREGNANCY WITH OTHER SOCIOLOGICAL DEMOGRAPHIC FACTORS AND THE LATER THEY COME INTO PREGNANCY, THE LESS BIAS THERE IS. EARLIER IN PREGNANCY WE TEND TO FAVOR WOMEN WHO EITHER HAVE ACCESS TO HEALTHCARE OR PREEXISTING MEDICAL CONDITIONS OR OTHER FACTORS WHICH WOULD FAVOR THEM SEEKING EARLIER PRENATAL CARE. SO, NOW IN TERMS OF OVERSAMPLING, WE'VE HAD EIGHT WORKSHOPS ON THE PROBABILITY DESIGN AND EVERY TIME WE HAVE A DISCUSSION WENT UPPED MORE OR LESS THAT WE DON'T WANT TO BALLETTER THE NATIONAL PROBABILITY DESIGN BY OVERSAMPLING ANY SPECIFIC POPULATION FOR A VARIETY OF REASONS. SO THAT'S WHY WE HAVE IT SUPPLEMENTAL RECRUITMENT SO THAT WE CAN THEN ADDRESS SPECIFIC SCIENTIFIC QUESTIONS WHICH MAY BE OF INTEREST AND WOULD REQUIRE THE NATIONAL CHILDREN'S STUDY TO EXAMINE THEM. THERE ARE MANY QUESTIONS AND ONE OF THE POINTS WE WILL MAKE IS THAT THE NATIONAL CHILDREN'S STUDY CAN'T DO EVERYTHING FOR EVERYONE, BUT WE CERTAINLY TAKE VERY SERIOUSLY THAT WE SHOULD ENROLL A DIVERSE POPULATION AS THE LAW SAYS AND WE SHOULD EXAMINE HEALTH DISPARITIES AS THE LAW SAYS. SO WE HAVE THIS METHODOLOGY TO APPROACH THOSE QUESTIONS. SO TO SUMMARIZE A A LITTLE BIT MORE WORDS THAN I INTENDED, OUR DATA UP TO NOW DO NOT INDICATE SYSTEMATIC BIAS IN THE APPROACHES WE'RE USING AND WE HAVE MECHANISMS IN PLACE TO ADDRESS SPECIFIC QUESTIONS FOR SPECIFIC POPULATIONS. >> I JUST WANTED TO MENTION THE ISSUE OF DISPARITIES AND SUBPOPULATIONS WILL ALSO RECEIVE ATTENTION THIS AFTERNOON BECAUSE IN ADDITION TO RECRUITMENT, THE ISSUE OF RETENTION OF THESE GROUPS IS CRITICALLY IMPORTANT. WE WANT TO UNDERSCORE THAT AND WE KNOW THERE'S DIFFERENTIAL ATTRITION. >> AND IT'S VERY ENCOURAGING THAT THE PROVIDER BASED SAMPLING ROUTINE NOT ONLY HAS THESE FAR MORE FAVORABLE COST METICS BUT DOES A BETTER JOB IN SWEEPING UP THE LESS ADVANTAGED SO THAT'S REALLY GREAT NEWS. >> YOU SAID YOU WERE LOOKING UNDER SIX FOR--HAD SOME IDEAS TO AUGMENT YOU'RE AT 90,000 FOR THE PROBABILITY, IS THAT WHERE WE WERE, YOU KNOW? SO WHAT SORTS OF SUBSAMPLES WERE YOU CONSIDERING FOR THE OTHER 10,000 DISCRETIONARY OBSERVATIONS. >> WELL THE ONLY ONE I THINK WE HAVE A CLEAR COMMITMENT TO IS A PRECONCEPTION COHORT, PARTICULARLY EMPHASIZING NOLA-PARIS WOMEN BUT THE OTHERS WE'RE QUITE OPEN TO VARIOUS OPTIONS AND I THINK AS THE TIME DRAWS NEAR AND AS WE GET FEEDBACK AND RECOMMENDATIONS FROM THE IOM REPORT THEN WE WILL BRING THEM BACK TO THIS COMMITTEE AND WE WILL THEN COLLECTIVELY DISCUSS OPTIONS AND OPPORTUNITIES. BUT WE DON'T--WE DON'T HAVE A SPECIFICS MENU OR AGENDA THAT WE'RE COMMITTED TO. >> WELL, YES, WHAT YOU SUGGEST IN TERMS OF NOLA-PARIS WOMEN AND THE PRECONCEPTION DATA, THAT'S THE ONE PIECE THAT A A LITTLE BIT--THAT'S MISSING SO I THINK THAT YOU'RE REALLY--YOU REALLY--YOU'RE PURSUING THE RIGHT LEAD THERE. >> OKAY, SO SEVEN WAS FAIL TOWER ADEQUATELY INTEGRATE DATA FOR MEDICAL RECORDS AND THAT'S STILL A BIG CHALLENGE. WE KNOW THAT MEDICAL RECORDS ARE KEPT NOT ONLY IN VARIOUS MODEL MODALITIES, ALTHOUGH THERE'S CLEAR TRENDS TOWARD ELECTRONIC MEDICAL RECORDS BUT THOSE ELECTRONIC MEDICAL RECORDS TEND TO USE PROPRIETARY PLATFORMS, THE PROPRIETARY PLATFORMS ARE INCOMPATIBLE WITH EACH OTHER AND THE NIH ISSUED SEPARATELY THE INITIATIVES THAT WOULD INTEGRATE HEALTHCARE DELIVERY RECORDS WITH RESEARCH. THERE ARE PLATFORMS THAT ATTEMPT WITHIN SOME HEALTHCARE SYSTEMS TO DO THIS TYPE OF INTEGRATION, NOTABLY IN THE PARTNER SYSTEM, THERE'S AN ITWO BTWO THAT WAS NIH SUPPORTED, BUT THERE ARE OTHERS AROUND THE COUNTRY, ARE WORKING ALL THE WAY UP TO THE OFFICE OF THE NATIONAL COORDINATOR FOR HEALTH INFORMATICS IT'S TODD PARK AND WE'VE BEEN WORKING WITH OTHER COMPONENTS NOT JUST IN HHS, BUT ACROSS THE FEDERAL GOVERNMENT TO ALINE THE WAY THAT WE CAN ACCESS ABSTRACT INFORMATION. WHERE WE ARE NOW AND I WON'T GET MORE TECHNICAL THAN THIS, I'M LESSED, IS THAT WE ARE LOOKING AT WAYS WE CAN USE METADATA TAGS WHICH CAN OVERCOME SOME OF THE LIMITATIONS OF INCOMPATIBLE PLATFORMS. FAILURE TO PLAN ADQUITLY FOR DISCLOSURE OF RISK TO PARTICIPANTS. WE'VE STRENGTHENED OUR OVERSIGHT THROUGH THE FORMATION OF AN INDEPENDENT STUDY MONITORING AND OVERSIGHT COMMITTEE. WE'VE HAD SEVERAL DISCUSSIONS ON HOW WE SHOULD EVOLVE POLICIES AND THEN WE INVESTED INFORMATIVE RESEARCH FOR TOOL WHERE IS WE CAN GO DIRECTLY TO PARTICIPANTS AND ASCERTAIN THE KINDS OF DATA SHARING AND THE KINDS OF PERCEPTIONS THAT THEY HAVE ABOUT THEIR INVOLVEMENT IN THE STUDY. AND THEN WE HAVE THE DISCUSSIONS THAT WE MENTIONED EARLIER ABOUT DATA SHARING AND DATA ENCLAVES. ANY QUESTIONS HERE. ALL RIGHT AND THE LAST ONE IS FAILURE TO PLAN FOR RAPID DISSEMINATION OF DATA AND I THINK WE TOUCHED ON THAT BEFORE SO UNLESS THERE ARE FURTHER QUESTIONS, I WILL TURNOVER THE DISCUSSION LEAD TO DOCTOR GUTTMACHER. ALAN? >> THANK YOU AND SHALL I EVEN TRY THE GIZMO, IT WILL MAKE ME FEEL GOOD TO TRY IT. >> SOMETIMES IT WORKS. >> OKAY. >> SO I'M GOING TO TALK TOXIC EFFECTS YOU SOMEWHAT ON THE PROPOSED, WE HAD MUCH LESS TIME WITH THE AGENDA BECAUSE WE HAD A RICHER DISCUSSION WE ANTICIPATED FOR THE PRIOR PART BUT I HOPE THE ACTING CHAIR WILL GIVE ME A LITTLE EXTRA TIME BECAUSE I SUSPECT YOU GUYS WILL WANT TO TALK ABOUT THIS, WE HOPE YOU DO AND THE CONTEXT FOR THIS OF COURSE IS THE 2013 PROARMSS LAW, WE WILL PARDON ANYONE FROM THE BELT WAY, IF YOU COME UP FROM YOUR WORK STATION EACH MORNING BUT ONE OF THE MOST INTERESTING PARTS OF THIS OF THE 2013 APPROPRIATIONS LAW WAS IN TERMS OF THE FUNDING FOR THE NIH WHERE THERE THIS SPECIFIC PROVISION INTRODUCED AND I'M THROWING IT IN THERE SO HAVE YOU TIME TO READ IT IN DETAIL AND WHAT IT ASKS IS THAT WE HAVE ANOTHER IOM NATIONAL RESEARCH COUNCIL REVIEW OF THE PROPOSED MAIN STUDY PLAN. THE TIMES THOUGHT TO BE PARTICULARLY APPROPRIATE BECAUSE EVERYONE RECOGNIZES THAT YOU KNOW THE LAUNCH OF THE MAIN STUDY WHILE CLEARLY BEATEN BASED UPON THE VANGUARD EXPERIENCE WE WILL CONTINUE TO REFINE THE MAIN STUDY DESIGN THROUGHOUT THE COURSE OF IT. THE LAUNCH WAS WITH THE COURSE, PARTICULARLY THE QUESTION WAS OBVIOUSLY ENROLLMENT, ET CETERA, BUT THE EARLY PART OF THE STUDIES PARTICULARLY IMPORTANT SO A DESIRE ON BEHALF OF THE CONGRESS TO MAKE SURE THAT THAT HAD BEEN FULLY VETTED, FULLY REVIEWED, ET CETERA, GET EVEN WIDER INPUT INTO THAT, ET CETERA AND I'LL TELL YOU A LITTLE BIT MORE ABOUT THE TIME SCHEDULE FOR THAT AND IN A MOMENT, I JUST WANT TO GO THROUGH THE BASICS OF THE PROPOSED DESIGN. WE ARE STILL SORT OF REFINING THE EXACT FINAL DOCUMENT TO THE IOM AS WE SPEAK. BUT THESE ARE THE ESSENTIALS OF IT, MUCH OF WHAT IS FAMILIAR TO ALL OF YOU, SOME OF WHICH HAS ALREADY BEEN DISCUSSED THIS MORNING, SO AGAIN THE IDEA OF THE NATIONAL PROBABILITY SAMPLE FOR ABOUT 90,000 LIVE BIRTHS, PRIMARY HOSPITALS AND BIRTHING CENTERS. THAT APPROXIMATELY HALF THE SAMPLE, HALF OF THE 90,000 WOULD BE RECRUITED THROUGH HOSPITAL AND BIRTHING CENTERS AND THE OTHER HALF OF THE SAMPLE TO ACHIEVE A SAMPLE DURING PREGNANCY TO REFER WOMEN TO THE HOSPITALS AND BIRTHS CENTERS. SO IT'S THE SAME GEOGRAPHIC SAMPLING SCENARIO BUT IT'S GETTING THESE FOLKS TO A VERY DIFFERENT VENUE BECAUSE WE THINK THERE ARE BOTH EXPENSE BUT ALSO SCIENTIFIC ADVANTAGES TO BOTH A PREGNANCY COHORT OR AS WE TALK ABOUT A PREGNANCY STRATEGIC PLAN UMPIRES AND--STRAYSTRATUM, AND THEN THE BIRTHS OF 10,000 WE TALKED ABOUT MORE. I PREDICT THIS WILL BE THE NEXT FREQUENT TOPIC OF DISCUSSION WITH THIS GROUP BECAUSE IT WILL BE IMPORTANT TO GET THIS RIGHT. IT'S CLEAR TO ME THE 10,000 ASSUMED TO THE IOM PANEL SAID YES, THAT'S ABOUT THE RIGHT NUMBER FOR THIS, WHATEVER THAT NUMBER WILL BE, IT'S KIND OF LIKE THE TAX REFUND CHECK YOU GET AND IT'S VERY TEMPTING TO SPEND IT ABOUT SIX DIFFERENT WAYS SO YOU END UP ACTUALLY LOSING MONEY FROM YOUR TAX REFUND. WE WILL ONLY BE ABLE TO SPEND THESE 10,000 FOLKS ONE WAY BUT THERE WILL BE MANY COMPELLING THINGS WE MIGHT WANT TO DO TO REACH THEN THOUSAND SO THAT WILL BE SOMETHING WE WILL CLEARLY WANT TO BE DISCUSSING IN DETAIL TO BE GETTING YOUR ADVICE TO HOW WE OPTIMALLY UTILIZE THAT STRATEGIC PLAN UMKC. THE IDEA TO GET OTHER PREGNANCY AND DATA IN A SUFFICIENT WAY TO ENROLL SUBSEQUENT BIRTHS THAT IS THREE YEARS WITHIN THE ORIGINAL ENROLLMENT OF MOTHERS AND CHILDREN IN THE STUDY. ALL OF THOSE BUT FOR AT LEAST MANY OF THEM TO BE ABLE TO FOLLOW FUTURE BIRTHS TO GET THAT PRENATAL DATA, ET CETERA. NOW CLEARLY IT'S FOR ANOTHER PREGNANCY BUT IT WILL GIVE US IMPORTANT INFORMATION INDEED BY DEFINITION THESE WILL ALL BE WOMEN--OR NONE OF THESE WOMEN WILL BE NOLIPEROS, SO WE DO WANT TO MAKE SMORE WE HAVE SOME OF THESE WOMEN, AND LOADING DATA COLLECTION OF THE INTENDED ASSESSMENTS OF THE CHILD DONE, 13 OF THEM DONE IN THE FIRST FIVE YEARS. CORE QUESTIONNAIRE AND TOPIC AND SPECIFIC MODELS. THIS IS THE IDEA THAT THERE WILL BE A CORE QUESTIONNAIRE WHICH WILL BE USE INDEED ALL 100,000 BUT THAT THERE WOULD ALSO BE SPECIFIC MODULES FOR SOME SUBSETS THAT COULD EITHER BE RANDOM SUBSETS OR COULD BE SUBSETS DETERMINED BY SOME PARTICULAR CHARACTERISTIC OF THE PREGNANCY OF THE CHILD OR OTHER GEOGRAPHY, WHATEVER MIGHT BE SO THAT TO AVOID PARTICIPANT RECOIL, WE WOULD ALSO FOR EXPENSE REASONS AND PRIMEAR ILLEGALS NEUROECTODERMAL TERMS EVER PARTICIPANT--PRIMARILY IN TERMS OF PARTICIPANT BURDENING THERE ARE SOME THAT WOULD NOT BE DONE TO ENTIRE POPULATIONS BUT CERTAINTY SUBSETS. AND A CLEAR EMPHASIS IN THINGS THAT CAN BE DONE IN TERMS OF RETENTION, WE HAVE THE ADVANTAGE OVER CHILDREN'S STUDY, OVER LARGE STUDIES OF ADULTS IN THAT OF COURSE, MUCH OF WHAT WE'RE INTERESTED IN HAPPENS IN THE FIRST YEARS OF LIFE. BUT THERE'S ALSO A LOT WE'RE INTERESTED IN THAT'S NOT HAPPENING THE FIRST YEARS OF LIFE SO RETENTION THROUGHOUT THE 21 YEARS WILL BE IMPORTANT TO THE AVAILABILITY TO GET CODE AND INFORMATION PARTICULARLY IN THE LATER YEARS. SO I THOUGHT WE HAD A COUPLE MORE SLIDES HERE, MAYBE I'VE JUST EXITED THE PROGRAM OR SOMETHING. CAN YOU SEE IF YOU CAN GET ME BACK? WE--YEAH, A LITTLE BIT FURTHER UP. KEEP GOING BACK UP. WELL, IF YOU JUST START ME THERE, I'LL LET US THROUGH. THERE WE GO, GOOD. SO IN TEMPLES OF PREEXCEPTION PREGNANCY, EXPOSURE, DATA COLLECTION, THESE JUST GO FROM GOOD ESTIMATES THAT WE HAVE, THE ESTIMATE IS ABOUT 50,000 WOMEN RECRUITED AT VARIOUS STAGE OR PREGNANCY TO ENROLL ABOUT 45,000 LIVE BIRTHS SO OBVIOUSLY NOT EVERYONE THAT YOU ORIGINALLY RECRUIT FOR VARIOUS REASONS PRODUCE A LIVE BIRTH INTO THE STUDY, WHICH IS JUST ABOUT 15 TO 2013 BIRTHS WITHIN THE THREE YEAR FOLLOWING WINDOW THE SUPPLEMENTAL STRATUM, OF 5000 NONLIPEROUS WOMEN AND OF COURSE COLLECTION AND DATA SAMPLES OF THOSE CHILDREN ENROLLED AT BIRTH, CAN YOU STILL GET OBVIOUSLY SOME DATA ABOUT EXPOSURE DURING PREGNANCY, ET CETERA, NOT THE SAME KIND OF RICH DATA YOU CAN GET FROM THE PRENATAL COHORT. >> COULD YOU CLARIFY THE ACCOUNTING HERE SO WE HAVE 45,000 LIVE BIRTHS SWEPT UP IN THE PROVIDER SAMPLE THEN WE HAVE SUBSEQUENT BIRLS WE HAVE ABOUT 20,000 SIBLINGS THAT'S 65,000 SWEE HAVE 5000 SO-- >> RIGHT. >> SO WHERE ARE THE OTHER 30,000. SO THERE WILL BE I THINK IT'S A QUESTION OF WHAT WE'RE--WE'RE USING THE FOLLOW UP BIRTHS PRIMARILY TO GET PRENATAL EXPOSURE DATA. SO THESE ARE NOT SHOWING THAT WE WOULD FOLLOW THEM THROUGH 21 YEARS OF LIFE, WE WOULD BE ABLE TO GET ON A SIMILAR, NOT THE SAME BUT SIMILAR ENVIRONMENTAL DATA, ET CETERA AND PREGNANCY. SO THAT THERE ARE LIMITS TO IT BUT IT'S BETTER, IT'S ANOTHER DATA POINT OR SET OF DATA POINTS TO HELP UNDERSTAND THE ENVIRONMENTAL EXPOSURE, AND UNTIL THE TIME OF BIRTH. >> IF I MAY ADD WE HAVE TO TAKE EACH BULLET SO THERE'S 45,000 IN THE PREGNANCY, 45,000 WHICH IS THE BOTTOM BULLET IN THE HOSPITAL AND THAT'S 90,000 THEN WE'RE ESTIMATING ANOTHER 1,520,000. SO IT'S GOING TO COME OVER A HUNDRED THOUSAND. >> SO IT'S 45,000 INTERCEPTED DURING THE CARE STAGE AND THEN 5000 AT BIRTH AND THEN THIS OTHER 10,000 WHICH HERE WE'RE GUESSING ABOUT 5000 WHICH WOULD BE TO NOLIPEROUS, WOMEN, ANOTHER 5000 POPULATIONS ARE OF B SO TOP OF THAT THAT REACHES A HUNDRED THOUSAND BUT ON TOP OF THAT THERE WOULD BE 15-20,000 BIRTHS OF SIBLINGS TO THOSE ENROLLED AT BIRTH OVER THE NEXT THREE YEARS, NOT ALL OF WHICH WILL DEFINITELY FOLLOW BUT AT LEAST SOME OF WHICH WE WILL FOLLOW TO GATHER IN PREGNANCY INFORMATION THAT WOULD GIVE US BY IMPLICATION SOME MORE INFORMATION WE WOULD HAVE OTHERWISE ABOUT THE PREGNANCY THAT PRODUCED THAT ORIGINAL 45,000. >> YEAH AND I APOLOGIZE THAT THE SLIDE IS CONFUSING. WE WERE TRYING TO SHOW A PARTICULAR POINT HERE. THIS ISN'T TOTAL ENROLLMENT, THIS IS THE MECHANISMS BY WHICH WE WILL GET REALTIME PREGNANCY AND INFORMATION AND EXPOSURE AND IT'S NOT EVERYBODY IN THE STUDY, SO THE FIRST THREE BULLETS ARE OPPORTUNITIES TO GET REALTIME PREGNANCY EXPOSURE INFORMATION AND THEN THE LAST BULLET WILE WE CAN'T GET REALTIME WE CAN GET SOME REGINANCE SCHEEXPOSURE INFORMATION FROM THE HOSPITAL BIRTH POPULATION, THROUGH THE USE OF QUESTIONS WHICH ARE SEEKING EXPOSURE HISTORY AND WHO THE COLLECTION FROM THE COLLECTION IS EXPERIMENTS FROM THE ENVIRONMENT FOR WHICH THERE'S STABILITY AND WE CAN INFER. >> SO A COUPLE OF QUESTIONS RELATED TO THIS, FOR THE 15-20,000 SUBSEQUENT BIRTHS, IS THE INTENT TO COLLECT DATA UNTIL THEY'RE ENROLLED, OLDER SIBLING AND 21 SINCE YOU'RE MAKING VISITINGILATES ANYWAY, BUT--VISITS ANYWAY, BUT NOT NECESSARILYOT TIME CYCLE BUT ON THE TIME YOU WOULD VISIT FOR THE OLDER SIBLING OR ARE SOME OF THESE GOING TO BE FOLLOWED FOR A YEAR AND ABANDONED. >> SOME OF IT WILL BE ON A COST ANALYSIS, BUT WE'RE PARTICULARLY INTERESTED IS THE PREGNANCY DATA SO THEY WILL CLEARLY FOLLOW THE PREGNANCY, BECAUSE WE'RE TRYING FIND THAT OUT TPHAOEUGZINIZEATION--ORGANIZATIO N IT'S A SECOND HAND, NOT AS GOOD AS HAVING THEM--YOU KNOW ALL OF THIS, THE QUESTION IS TO ANSWER THE QUESTION, GHEE, WE CAN UNDERSTAND FINANCIALLY THAT YOU CAN'T AFFORD TO ENROLL AND ALSO IN TERMS OF TIMES INVOLVED YOU CAN'T AFFORD TO ENROLL ALL THAT INFORMATION AND THIS IS WHAT IT WILL LOOK LIKE. >> IT SEEMS LIKE THERE ARE AT LEAST TWO EQUALLY VIABLE HYPOTHESIS FOR ANY CONCLUSION THAT COMPARES TO SUBSEQUENT PREGNANCY WITH THE EARLIER ONE. ONE OF THEM IS THAT THE PROCESS OF BEING ENROLLED AND HAVING ALL THESE THINGS MEASURED, LEADS SOMEONE TO CHANGE BEHAVIOR, IN FACT THE PROCESS OF HAVING A CHILD MAY LEAD SOMEONE TO CHANGE BEHAVIOR. THE OTHER IS THAT YOU'RE GETTING A BETTER AND MORE ACCURATE MEASURE OF WHAT YOU HAD BEEN MEASURING BEFORE. 5000 PEOPLE THAT A A PRECONCEPTION COHORT, SEEMS LIKE A SMALL NUMBER TO DISAMBIGUATE THOSE TWO HYPOTHESIS IN THESE OTHER CASES. HAVE YOU DONE THE MODELING TO SEE, DO WE HAVE A MECHANISM AND THE STATISTICAL POWER WE NEED TO COME TO ANY MEANINGFUL CONCLUSION WITH THIS OR WILL ALL OF THIS STILL LEAD US TO SAY, WELL, EITHER PEOPLE CHANGE THEIR BEHAVIOR OR IT TURNS OUT THE DATA THAT WE GOT, THAT THE SELF-REPORT IS NO GOOD. >> I THINK IT'S--CLEARLY YOU ARTICUMENTED THE LIMITATIONS OF THIS. IT HAS LIMITATIONS NO QUESTION, BUT AGAIN IT'S PART OF--IT IS THIS PLUS--YOU KNOW WE WILL HAVE APPROXIMATELY WE THINK 45,000 ARE ENROLLED PRENATALLY SO WE WILL HAVE OTHER DATA ABOUT THE PREGNANCY EXPERIENCE AS L. SO IT'S NOT SIMPLY COMPARING TO THE 5000 NO-LIPS, BUT THERE IS CLEARLY ANOTHER APPROACH. >> I'LL SAY IN THE MODELING WE CAME UP WITH THESE NUMBERS AND AS ALAN POINTED OUT IT'S COST AND RESOURCE ISSUE, TOO, AND WE'D LOVE TO BE ABLE TO FOLLOW EVERYBODY FOR 21 YEARS, WE DON'T KNOW IF WE CAN OR WE CAN'T AND AND FROM MY PERSONAL PERSPECTIVE WE'RE REMAINING A LITTLE BIT AMBIGUOUS ABOUT THIS PENDING WHAT THE IOM REPORT WILL WILL RECOMMEND SO THIS IS A PROPOSAL AND WE ANTICIPATE THAT THE POINTS YOU RAISE AND AND THEN DISCUSSIONS AND THEN AM BE REFLECT INDEED WHAT COMES, AND WILL BRING IT BACK TO YOU ALL AND WE'LL SEE WHERE YOU GO FROM THERE. >> THANK YOU BOTH FOR THE VERY NICE SEGUE TO THE NEXT SLIDE IN FACT, BEFORE WE SEGUE. >> OH, OKAY? >> MAYBE FOLLOW UP TO JOE'S QUESTION THAT THAT THE SIB AND WE WON'T HAVE IT NEARLY AS WELL BUT THE DOWN SIDE IS HE WAS SUGGESTING THAT IF WE DON'T FOLLOW THE SIBLINGS, THE ABILITY TO RELATE THE EXPOSURES DURING THIS PERSPECTIVE PHASE AND RELATE THAT TO SUBSEQUENT OUTCOMES IS GOING TO BE LOST WHEREAS IF--I DON'T WANT TO PUT WORDS IN YOUR MOUTH, JOE BUT IF YOU FOLLOW THOSE 15-20 ON, YOU WOULD BE ABLE TO HAVE YOUR CAKE AND EAT IT, TOO, AS IT WERE. >> WELL THERE WILL BE SOME LIMITATIONS AGAIN, BECAUSE OBVIOUSLY NOT ALL OF THEM, FIRST PREGNANCIES, YOU COULD--I THINK IF WE WERE GOING TO TAKE THAT APPROACH FROM THE GET-GO, YOU WOULD SAY WHY HAVE ONLY 35,000 TO ENROLL IN PREGNANCY, WHY DON'T YOU GET 60,000 INSTEAD WOULD BE A MORE EFFICIENT WAY TO DO THAT. >> NOW THESE ARE THE QUESTIONS, THE IOM WILL ADDRESS THESE AS WELL, BUT THERE'S SPECIFIC QUESTIONS WE WANT THE INPUT ABOUT, SO ONE OF THEM IS THE USE OF LATER SIBLINGS ENROLLED ON CHILDREN AND PRECONCEPTION AND EARLY PEG NANCY EXPOSE AND YOU ARE THESE ARE GOOD WEB CONNECTEDS YOU ALL ARE ASKING AND WE RECOGNIZE THAT & THAT'S WHY IT'S SPECIFICALLY ON THIS LIST OF OF QUESTIONS WE INTEND TO ASK THEM TO MAKE SURE THEY DO FOCUS ON, THOUGH THEY'RE OTHER ONES IN ONE CAN CERTAINLY IMAGINE, AGAIN, ALMOST ALL OF THESE ARE THINGS THAT WE HAVE TALKED ABOUT AROUND THIS TABLE SOME OF THEM IN GREAT DETAIL, THE USE OF THE HEALTHCARE PROVIDERS AND WE LOOKED AT OTHER OPTIONS FOR THAT. THE USE OF HEALTHCARE PROVIDERS, THE REFERRAL MECHANISM FOR PARTICIPANTS, THE VARIOUS ADVANTAGES OF DIFFERENT PROPORTIONS, THIS IDEA OF 45 AND 45,045,000 IS NOT SIMPLY BECAUSE--45,000. AND 45,000 IS NOT SIMPLY BECAUSE OF A SWEET SPOT OR DO THEY AGREE AND CONSIDERING COST LIMITATIONS AND OTHER KINDS OF THINGS, IT'S VERY IMPORTANT THAT IT CAN BE 60,000 PRENATAL WHATEVER. , THAT'S THE DURATION OF THE ENROLLMENT PERIOD, THAT'S THE DISADVANTAGE OF THE SHORTER AND LONGER PERIODS, WE ARE INTERESTED IN WHETHER THEY SEE THAT THE SAME WAY IN TERMS OF OVER HOW LONG A PERIOD WE SHOULD BE SEEKING TO ENROLL CHILDREN FOR THE MAIN STUDY. TALKED ABOUT LATER SIBLINGS, THE PROPOSAL TO CHARACTERIZE HEALTH DISPARITIES AND OPTIONS FOR COLLECTING THOSE AND OTHER SPECIFIC DATA. AND WE ARE COLLECTING INFORMATION ON PREGNANCY AND EARLY CHILDHOOD RATHER THAN LATER CHILDHOOD AND ADOLESCENCE. AND THE PROPOSAL ON ADDRESSING PHENOTYPES AND THIS GETS TO THE IDEA THAT IS BEING PROPOSED THAT WHAT WE REALLY TRY TO DO WITH THE PHENOTYPIC ASSESSMENTS IS NOT KIND OF PREJUDGE THE SITES. WE'RE TRYING NOT TO DEFINE THE DISEASE STATES BUT GATHER REALLY RICH PHENOTYPIC DATA SO IT IS END USER THOSE DURING THE ANALYSIS CAN DEFINE HOW TO AND NOW THERE ARE SOME THINGS FROM THESE STATES BUT THERE ARE OTHERS WHERE HUMAN EXISTENCE FROM A SPECTRUM OF HEALTH AND DISEASE AND RATHER THAN PRESUPPOSING THAT ONCE YOU GET PASSED IN TERMS OF FEV THAT EQUALS ASTHMA OR SOMETHING BEABLE TO PROVIDE AS RICH AS YOU CAN PHENOTYPIC DATA SO THAT PART OF THE ANALYSIS WILL BE DIFFERENT PEOPLE APPLYING DIFFERENT DEFINITIONS OF DISEASE OR HOWEVER YOU WANT TO DEFINE THAT, TO LOOK AT THE DATA IN A RICHER WAY AND THEN WE PRESUPPOSE AND PRELABELLED ALL CASES, YOU ARE EITHER A CASE OR NOT A CASE, OBVIOUSLY WOULD REDUCE THE SCIENTIFIC WORTHS WE THINK. ANY OTHER QUESTIONS ABOUT THAT STUFF WHICH YOU THINK WILL BE A COUPLE OF OTHER QUESTIONS THAT WILL GO ON THERE BUT THAT'S THE LIST WE HAVE AT THE MOMENT. ARE THERE ANY QUESTIONS THAT YOU THINK, THE ANSWERS TO WHICH ARE SO OBVIOUS THAT WE'RE WASTING THE TIME OF THE PANEL TO ASK THEM TO LOOK AT? >> I WONDER IF THERE COULD BE AN EFFECT OF DROPPING ONE CHILD FROM A STUDY AT A CERTAIN POINT, JUST AGAIN THINKING ABOUT RETENTION THAT WE'VE GOT THE TRUST OF THE PARENT, THE CHILDREN, THEY'RE INVOLVED IN THE STUDY AND THEN AT SOME POINT YOU DROP THE SECOND CHILD FROM OBSERVATION. >> CLEARLY IF WE DID THAT, THAT WOULD BE EXPLAINED AT THE VERY BEGINNING OF THE PREGNANCY, SO NO IT STILL COULD HAVE AN IMPACT BUT CLEARLY THAT'S SOMETHING THAT THE PARTICIPANTS WILL BE VERY WELL AWARE OF EXACTLY WHAT THE PLANS WERE BUT YOU'RE RIGHT, IT DOES UNDERSCORE THE POINT THAT WE CAN'T KIND OF DECIDE FIVE YEARS INTO IT WHAT WE WILL DO WITH THE SECOND CHILDREN, IT NEED TO BE DONE BEFORE AND NOT AFTERWARDS. >> THIS IS JEFF, CAN I ASK A QUESTION. >> PLEASE? >> WITH RESPECT TO THE ISSUES BROUGHT TO THE IOM, I THINK IT MIGHT BE HELP WILLFUL IF YOU WERE TO A BIT MORE SPECIFIC--HELPFUL IF YOU WERE TO BE A BIT MORE SPECIFIC WITH REGARD TO THE ORIGINAL 45,000 DURING PREGNANCY, YOU ADDRESS THE QUESTION OF A BIRTH ORDER OR PAROFFICE OF DIVERSITYY IN THAT POPULATION WHETHER THAT WAS STRATIFIED IN ANYWAY, YOU WILL TAKE WHAT COMES MIGHT BE INFORMATIVE IN TERMS OF THE SECOND PREGNANCY COHORT THAT YOU'RE ADDRESSING IN THIS AND MIGHT HELP THEM ADDRESS THAT SECOND POINT, HOPEFULLY WE'RE SPECIFICALLY BECAUSE WE'LL UNDERSTAND WHETHER THERE ARE SIMILAR PREGNANCIES IF YOU WILL IN TERMS OF PARITY BEING INVOLVED IN THE FIRST COHORT. SO THE--WE WILL GIVE THEM AS MUCH INFORMATION AS THEY REQUEST BASICALLY, SO THE INITIAL DOCUMENTS FOR THEM, WE WILL PROBABLY NOT GO INTO THE LEVEL OF DETAIL THAT YOU'RE SUGGESTING, BUT WE WILL MAKE IT CLEAR AS WE HAVE IN EARLIER CONVERSATIONS WITH THEM THAT WHATEVER KIND OF BACKGROUND DOCUMENTATION, ET CETERA, THEY WOULD LIKE, THEY WILL CERTAINLY BE PROVIDED WITH. WE'RE TRYING NOT TO DROWN THEM WITH TOO MUCH TO BEGIN WITH BUT BE READY WITH THE FACT THAT THEY WILL WANT MORE INFORMATION WHEN PROVIDED THE PRELIMINARY DOCUMENTATION. >> JEFF, WERE OF GETTING AT THE POINT--WERE YOU GETTING AT THE POINT WHERE YOU HAVE DATA ON MULTIPLE SIBLINGS WITHIN A HOUSEHOLD YOU HAVE MORE ANALYTIC POWER AND THAT'S A ARGUMENT FOR RETAINING THE SUBSEQUENT BIRTHS? >> WELL WHAT I WAS GETTING TOO REALLY THINKING ABOUT IN THIS FIRST COHORT, I DIDN'T HEAR ANY DISCUSSION ABOUT PARITY OR BIRTH OR WHATEVER AGO I'M SURE THE DATA WILL BE COLLECTED SO THE IDEA IS JUST INSTEAD OF 5000 COHORT, STRATIFY ANYWAY OR RECRUELTY TARGETS THAT REFLECT NOLA-PARIS WOMEN VERSES WOMEN WHO HAD MULTIPLE PREGNANCY AND THEIR OUT COMES? AND THEN THAT MIGHT HELP US FIGURE OUT THIS OTHER COHORT OF SIPPLINGS MIGHT BE SIMILAR TO, TO THE EXTENT OF WHICH THEY REALLY REPRESENT AN OPPORTUNITY TO AUGMENT THE FIRST GROP AS WAS SUGGEST FEDERAL THEY WERE GOING TO BE FOLLOWED OR EXACTLY HOW THEIR DATA ON PREGNANCY MAY BE EARLY OUTCOMES WOULD BE USED WHEN COMBINED WITH OTHER COHORT. >> RIGHT. I THINK I UNDERSTAND THE POINT. SO RIGHT NOW WE'RE BEING, I THINK ON THE SPECTRUM OF ACTIVE TO PASSIVE IN THAT REGARD, WE WILL COLLECT THE DATA BUT WE HAVE NOT EVOLVE INDEED AN APPROACH BASED OFFICE OF DIVERSITY WHAT WE WANT FOR PARITY FOR THE PURPOSES OF LINKING AND STRENGTHENING THE ASSOCIATION BETWEEN THESE VARIOUS STRATA. WE WOULD BE OPEN TO FURTHER DISCUSSION BUT PRELIMINARY DISCUSSIONS DIDN'T INDICATE A NEGLIGENT THAT WE FELT WE COULD LIMP ELEMENT AND THAT WE WOULD BE ABLE TO UTILIZE IN A WAY THAT WAS GOING TO END UP ACHIEVING THAT GOAL. MAYBE IT'S A FUNCTION OF THE WAY WE APPROACHED IT, BUT IT ENDED UP BEING--INTRODUCING A LEVEL OF COMPLEXITY THAT WE WEREN'T SURE WHAT WE WERE GOING TO GAIN USING THAT APPROACH. SO, AT THIS POINT WE'VE OPTED TO BE MORE PASSIVE, COLLECT THE DATA AND THEN EXAMINE POTENTIAL, THE POTENTIAL STRENGTH OF THE ASSOCIATIONS AFTER THE FACT. BUT LET'S KEEP THAT IN THE PARKING LOT BECAUSE WE'RE NOT LOCKED INTO ANY DESIGN NOW. OKAY. >> GEN FOR, I THINK IT WOULD BE GOOD ARE IF YOU PANEL TO LOOK AT THE SAMPLES AND HOW THEY INTERACT AND HOW THEY CAN BE BEST BROUGHT TOGETHER TO ANSWER A SET OF QUESTIONS BECAUSE IT MIGHT LEAD YOU TOWARDS SLIGHTLY DIFFERENT ALLOCATION AND I THINK THOSE KINDS OF ISSUES THAT JOE BROUGHT UP AND ISSUES ABOUT THIS ONE, NONE OF THESE ARE PERFECT, THIS ONE GIVES YOU THIS AND THAT ONE GIVES YOU THAT PIECE AND YOU MIGHT BE ABLE TO PUT THEM TOGETHER IN A BETTER SPA I THINK THAT COMMITTEE WOULD BE ENOUGH--THERE ARE ENOUGH PEOPLE ON THAT COMMITTEE THAT ADDRESS THAT THE OTHER IS, IS THERE A FORMAL CHARGE FOR THE COMMITTEE YET OTHER THAN WHAT'S IN THE BILL OF THE APPROPRIATIONS AND IT HAS SOMETHING SPECIFIC OTHER THAN THE QUESTIONS. >> THERE WILL BE A FORMAL CHARGE DELIVERED TO THEM, I WON'T GO TO THE NEXT SLIDE YET BUT AS YOU COME IN THE SLIDE, THE FIRST MEETING WILL BE AUGUST 16th AND IT WILL BE PART OF WHAT HAPPENS THEN. >> YES, I THINK THE POINT ABOUT THE WHOLE DESIGN SUBSEQUENT SIBS, ORIGINAL SIBS, DESIGN OF THE FAMILY, THESE ARE IMPORTANT ISSUES AND THE ABILITY TO SEPARATE OUT VARIOUS ENVIRONMENTAL AND GENETIC FACTOR CANS BE IMPROVED WITH MORE COMPLEX FAMILY STRUCTURE. ON THIS LAST POINT, YOUR PROPOSAL WILL ASSESS, HELP PHENOTYPES PRESUMABLY AND THAT'S HELP AND DEVELOP, IT'S UNDERSTOOD, I THINK. YOU'RE RIGHT IT SHOULD BE MADE EXPLICIT. >> UNLESS I HESITATE, ON THE ORIGINAL SIBS, SOUNDS LIKE ORIGINAL SIN TO ME, BUT SO ANYWAY UPCOMING DATES OF NOTE. FIRST OF ALL REMEMBER THERE WAS AN IOM WORKSHOP HELD BACK IN JANUARY THE PREPRINT FROM THAT WILL BE AVAILABLE VERY SOON. FINAL COPY AVAILABLE IN SEPTEMBER BUT IN SOME WAYS NOW THAT IS, I THINK OVERTAKEN BY THE CONGRESSIONAL MANDATE REVIEW, WE'VE BEEN TALKING ABOUT THE CURRENT TIME LINE WOULD BE THAT THE FORMAL RELEASE OF THE REPORT TO THE PUBLIC WOULD BE BY DIGITAL IN A DIGITAL FORM AT IN AN EXTRA LINE WITH THE FORMAT TO FOLLOW MONTHS LATER. IT IS THE DIGITAL FORMAT THAT WE WILL ALL BE DIESES CUSSING AND RACKETING TOO AND ET CETERA AND USING THE BASES FOR FUNDING OPPORTUNITIES SOMEWHAT LETTER. IN TERMS OF MEETINGSINGS THERE IS THE MEETING IN AUGUST 16th, IT'S AN OPEN MEETING, ET CETERA AND THAT IS A MEETING IN WHICH THE CHARGE WILL BE DELIVERED, ET CETERA. THE NEXT SCHEDULED MEETING IS ON HALLOWEEN, I THINK YOU MAY DRESS HOWEVER YOU LIKE WHERE THEY WILL BE LOOKING AT NCS METHODOLOGY AGAIN WE'RE BE HERE IN D. C., MORE DETAILS PARTICULARLY ABOUT THAT ONE AS WE GET CLOSE TORE THE DATE AND ANY OF YOU WHO HAPPEN TO BE AVAILABLE ON ANY OF THOSE DATES AND WOULD LIKE TO SIT IN THESE ARE LARGELY OPEN MEETINGS AND I THINK THAT BRINGS US TO THE END UNLESS THERE ARE OTHER FOLKS, POINTS THAT STEVE AND I DID NOT TALK ABOUT THAT YOU'RE ANXIOUS TO HEAR ABOUT FROM. >> OKAY. SO LET'S OPEN IT UP FOR DISCUSSION, POINTS PEOPLE WOULD LIKE TO MAKE? >> YES, ONE OF THE POINTS YOU WANTED ANSWERED BY THE INSTITUTES OF MEDICINE, WAS THE ENROLLMENT TIME LINES AND I'M WONDERING IF WE COULD ALSO HELP YOU WITH PROVIDING FEEDBACK. YES OF COURSE. THAT'S WHY YOU'RE HERE. IT SEEMS I'M INTERESTED IN THE DISCUSSION ON RETENTION LATER ON THIS AFTERNOON, BUT IN GENERAL FOR ENROLLMENT, IT SEEMS THAT THE TIME IN THAT, I'M NOT SEEING ANYTHING NEW HERE BUT ANY TIME ALLOTMENTS YOU PREPARE FOR IT, IT'S A BIT LONGER SO WITHIN THE CONTEXT OF VARIABILITY, AND WITHOUT ANY FURTHER DELAY FOR CHILDREN'S HEALTH STUDY WOULD--IN TERMS OF PROS AND CONS HAVE THE DISCUSSION BEEN WAVED MORE TOWARDS ENROLLMENT WITHIN EYE FINITE PERIOD OF TIME--WITHIN A FINITE PERIOD OF TIME TO GET TRACTION RELEASE BUT NOT ANY LONGER THAN THAT. SO THE CURRENT PROJECTION IS THREE-FOUR YEARS AS THE TIME WINDOW AND DEPENDING ON WHAT HEAR NEXT SUMMER FROM THE IOM, WE MIGHT MAKE ADJUSTMENTS THERE, THERE'S DIFFERENT PERSPECTIVES WE'VE HEARD, MORE COMPACT IS BETTER AND THEN THE OTHER PERSPECTIVE IS MORE EXTENDED IS BETTER. SO I WILL JUST SAY THAT THERE'S NO CONSENSUS ON IT BUT WE HAVE TO PROJECT ON WHAT WE THINK IS LOGISTICALLY FEASIBLE. WE NOW HAVE A FAIR AMOUNT OF DATA TO RECRUITING THE ALMOST 5000 CHILDREN WE HAVE NOW AND WE HAVE SOME IDEAS SPECIFICALLY OF THE EFFICIENCY OF GOING INTO CLINICS AND PRACTICES AND WE'RE GETTING BETTER AND BETTER, BETTER AND BETTER MEANING MORE PRECISION ABOUT THE DATA AND RECRUITING IN HOSPITALS, AND SO WE'VE MODELED THAT UP AND WE THINK THAT BASED SOLELY ON THE DATA WE HAVE, THAT SOMEWHERE LONGER THAN THREE YEARS BUT NOT QUITE FOUR YEARS IS PROBABLY GOING TO GIVE US ABOUT WHAT WE NEED. BUT THAT COULD CHANGE. >> OTHER POINTS? QUESTIONS? ALL RIGHT WELL THEN WE'VE--WE'VE GONE A LITTLE BIT FURTHER INTO THE SCHEDULE THAN WE THOUGHT TIMEWISE, BUT THAT'S PERFECTLY OKAY. SO THAT ENTERING US US TO THE NEXT ORDER OF BUSINESS WHICH IS THE DISCUSSION OF CONCEPTUAL FRAMEWORK AND CONTENT DEVELOPMENT AND THESE ARE OF COURSE CLEARLY REALLY IMPORTANT BECAUSE THIS IS WHERE ALL THE SCIENCE OR A WHOLE LOT OF THE SCIENCE IS GOING TO BE PUT AND SO, MAYBE LET'S START THAT DISCUSSION AND I GUESS, AGAINST THAT THERE'S THE BACK DROP OF WEAKNESS NUMBER FOUR IDENTIFIED BY THE PREVIOUS IOM REVIEWOT CONCEPTUAL FRAMEWORK SO WE WANT TO MAKE SURE WE TALK ABOUT THAT SO HOW WOULD YOU LIKE TO PROCEED? >> I THINK WHAT FOR US WE'RE SEEKING IS A DISCUSSION OF THE LARGER FRAMEWORK SO WE RECOGNIZE THAT WE CAN'T ASK EVERYTHING OF EVERYBODY AND WE WOULD LIKE TO HAVE THEN SOME DISCUSSION ABOUT OUR CONCEPTS ON THREE POINTS. ONE IS THE FRONT LOADING BY WHICH WE MEAN EARLY YEARS, GET MORE DATA COLLECTIONS THAN LATER YEARS. OUR APPROACH OF USING A CORE QUESTIONNAIRE AND THEN TOPIC SPECIFIC MODULES. AND THEN AS DR. GUTTMACHER NOTED OUR EMPHASIS ON CAPTURING ROBUST PRIMARY PHENOTYPIC DATA RELATED TO EXPHELGT DEVELOPMENT, THOSE ARE THE POINTS WE'RE SPECIFICALLY SEEKING INPUT ON. DOES THAT HELP? >> WELL, LET'S GO IN THAT ORDER AND I THINK THE SUGGESTION THAT OVER 20 YEARS, 21 YEARS PLUS OF DATA COLLECTION ON A PARTICULAR CHILD DEVOTING 13 OF THE VISITS OR DATA COLLECTION INSTANCES TO THE TIME FROM ZERO TO FIVE IS OF COURSE A REMARKABLE FRONT LOADING, BUT PERHAPS THE PLACE TO START IS IS THAT APPROPRIATE FRONT LOADING DEGREE, IS THERE REASON TO DO IT OTHERWISE. THAT IS PUT MORE OF THE RESOURCES LATER IN LIFE. I GUESS BY WAY OF BACKGROUND, ONE OF THE THINGS PEOPLE HAVE LEARN FRIDAY A VARIETY OF STUDIES ON THE DEVELOPMENT SIDE IS THAT A TREMENDOUS AMOUNT OF A CHILD'S SOCIAL EEMOTIONAL AND COGGATIVE DEVELOPMENT IS MANIFEST BY AGE FIVE BEFORE THEY EVEN ENTER SCHOOL AND I THEN IS BEING--THIS IS BEING REFLECTIVE IN MANY PROPOSALS TO DO THINGS LIKE GO TO ALL-DAY KINDERGARTEN AND THINGS OF THAT SORT. THESE ARE--THESE ARE REFLECTIONS OF THE RESEARCH AGENDA, MUCH OF IT SUPPORTED BY NICHD, THAT HAS REALLY POINTED OUT THE IMPORTANT CORE DEVELOPMENT TO FOCUS ON SO THOSE EARLY YEARS. SO, ARE THERE ANY--FROM MY POINT OF VIEW, FROM THE DEVELOPMENT SIDE THAT REALLY SOUNDS SPOT ON ON THE CLINICAL SIDE, THOSE OF YOU WHO ARE MORE EXPERT IN THERE SHOULD--MIGHT LIKE TO HEAR WHAT YOU HAVE TO SAY. >> AS A PEDIATRICIAN, ABSOLUTELY I AGREE, FOR FIVE YEARS, ON THE TABLE THREE THERE'S LOOKING LIKE 14-16 MONTHS AND I'M WONDERING THERE MUST HAVE BEEN SOME ASSUMPTION AS TO HOW MANY SAMPLES YOU WOULD GET BEFORE 20 WEEKS VERSES PREGNANCY AFTER WEEKS, SO I'M WONDERING IF THAT WILL SOMEHOW BE CONTROLLED IN THE ENROLLMENT SINCE IT'S IMPORTANT TO GET ENVIRONMENTAL EXPOSURE IN THE FIRST TRIMESTER. >> I'M NOT QUITE SURE I GOT ALL THE NUANCES OF THAT BUT IF I COULD PARAPHRASE AND YOU SEE IF THIS IS WHAT YOU'RE REQUESTING: HOW ARE WE GOING TO GET EARLY EXPOSURE DATA? SO THE SLIDE THAT DR. GUTTMACHER THAT SHOWED AUGHT THE VARIOUS APPROACHES THAT WE'RE CONTEMPLATING, A PRECONCEPTION COHORT AND USING THE SIBLINGS AND THEN DOING THE BEST WE CAN WITH THE WOMEN THAT ARE ENROLLED DURING PREGNANCY AND EVEN AT THE HOSPITAL, WE'RE USING WHAT WE HOPE ARE COMPLIMENTARY AND HOPEFULLY CONVERGENT APPROACHES TO GET THE DATA IN THAT REGARD. WE ALSO HAVE PARTNERED WITH OTHER STUDIES SO WE'RE NOT OPERATING IN ISOLATION AND WE'RE PARTNERING WITH--AT THIS POINT PRIMARILY INTERNATIONAL BIRTH COHORT STUDIES TO SEE WHAT WE COLLECTIVELY CAN BRING UP. EVERYONE HAS--EVEN IN SYSTEMS WHERE A HUNDRED% OF THE POPULATION HAS FREE HEALTHCARE AND THEY CAN GET AND ARE ENCOURAGED TO GET PRENATAL CARE, IT'S STILL A STRONG TENDENCY TO GET THE LATER EXPOSURES AND EXPERIENCE AND NOT THE EARLY ONES. SO IF YOU HAVE SUGGESTIONS WE'RE STILL QUITE OPEN BUT WE'VE BEEN THINKING THAT THERE'S NO SINGLE APPROACH AND SO WE'RE USING THESE COMPLIMENTARY APPROACHES. >> I THINK THAT'S GREAT. I JUST WANT TO POINT OUT THAT IN TABLE THREE OF THE BRIEFING REPORT, IT DOES HAVE 14 VISIT POINTS, NOT 13 FOR THAT, SO THAT WAS IMPLICITY SUSMGZ THAT WOULD BE OBTAINED AND AFTER THERE'S JUST 14 POINTS, BUT THAT'S OKAY. >> I CONTINUING HAD TO DO WITH THIS PERENNIAL TOPIC THAT COMES UP WITH CLINICAL RESEARCH IN GENERAL, WHAT'S DAY ONE AND WHAT'S DAY ZERO AND I THINK SOME PEOPLE WERE COUNTING THE BIRTH ON ONE SIDE AND SOME ON THE OTHER SIDE AND WE'LL HAVE TO BE CARE EXCLUSIVELY BE CONSISTENT IN THAT, OKAY? >> FOR ME UPLOADING THE CONCEPTS MAKES SENSE FROM A PHYSIOLOGICAL PERSPECTIVE BUT MY REACTION TO THAT IS IF WE'RE GOING TO UPLOAD IN THE FRONT, ARE WE MISSING ANYTHING BEHIND? SO I APPRECIATION MINIMIZING THE BURDENSOT PATIENTS BUT WHAT ELSE CAN WE GET FROM OTHER SUPPLEMENTS AND LET US GET AWAY FEWER VISITS LATER AND IS THERE ANYTHING THAT'S A PRIMARY OUTCOME THAT WE WOULD BE JEOPARDIZING BY RELYING ON OTHER SOURCES OPPOSE TO OUR PRIMARY VISIT. AND MY ONE RED FLAG IN THIS IS GROWTH AND DEVELOPMENT LATER ON, YOU KNOW NOT BEING MEASURED AND ASSESSED BY THE TRAINED STUDY PERSONNEL AND IT SEEMS LIKE THAT'S AN IMPORTANT PRIMARY OUTCOME. >> WE WERE COUNTING ON YOU TO RAISE THAT POINT SO WE HAVE ESSENTIALLY DOUBLED THE NUMBER OF VISITS FROM THE PRIOR DESIGN AND AFTER FIVE YEARS, WE INTONED HAVE AN IN-PERSON VISITS ABOUT EVERY OTHER YEAR SO YEAH, THERE WILL BE 24 MONTH GAP BETWEEN VISITS BUT WE ACTUALLY EXPECT IT TO HAVE A SPREAD ACROSS THE VISITS SO WE DON'T EXPECT THE VISITS TO BE--TO CONSIDER ON THE BIRTHDAY OF THE CHILD BUT PLUS OR MINUS MANY MONTHS ON EITHER SIDE AND THAT'S MY INTENT BECAUSE WE ACTUALLY THINK THAT JUSTICE IN THE FIRST YEAR, WE HAVE FOUR DATA COLLECTION OPPORTUNITIES, BUT SO MANY THINGS ARE HAPPENING AND TYPICALLY PEOPLE DON'T GO FOR A 10th MONTH VISIT OR AN EIGHT MONTH VISIT FOR HEALTHCARE DELIVERY OR SOMETHING. IT TENDS TO BE COORDINATED WITH VACCINE SCHEDULES AMONG OTHER THINGS SO WE'RE--SO WE'RE DOING IS HAVING A FLEXIBLE APPROACH KNOWING THAT WE'LL GET PEOPLE REALLY AT MANY DIFFERENT AGES IN THERE, SO THE POST FIVE EVERY OTHER YEAR, THAT'S STILL RELATIVE TO ANY OTHER STUDY WE'VE SEEN OR KNOW ABOUT, PROBABLY TWICE AS MANY VISITS AS OTHERWISE AND WE HAVE, LET'S SAY QUESTIONS ABOUT THE CONSISTENCY AND RELIABILITY OF THE HEALTHCARE DELIVERY RECORDS. SO WE WOULD NOT WANT TO RELY ON THEM FOR ADDRESSING KEY HEALTH AND DEVELOPMENT QUESTIONS AND IN FACT, THEY TEND NOT TO COLLECT THE KIND OF INFORMATION WE'RE SEEKING. SO EVERYTHING'S COMPROMISE, BUT BUT WE BRING IT UP FOR QUESTIONING AND IF THERE'S ONE CHALLENGE IS NONAPOPTOTIC KNOW AND AS YOU SO ELOQUENTLY PROVIDED CONSULTATION TO US AND OUR INTERNATIONAL PARTNERS IN MAY WHEN WE HAD A CONFERENCE DISCUSSING THESE POINTS, WHERE DO YOU START? BECAUSE DO YOU START WITH ADENAR, MENARK? SOME PEOPLE, SOME POPULATIONS, SOME GEOGRAPHIC AREAS, HAVE EARLIER ONSET, SOME HAVE LATER ONSET AND SO FORTH. SO YOU END UP WITH PROBABLY CLOSE TO A DECADE OF DATA COLLECTION UNTIL YOU FEEL THAT YOU'VE COVERED MOST OF THE POPULATION IN THERE SO WE'RE OPEN TO IT, BUT FOR THE TIME BEING, EVERY OTHER YEAR SCHEDULE SEEMS TO BE A PLAUSIBLE WAY FORWARD. WE RUN THE VANGUARD SPECIFICALLY SO WE CAN INFORM THE MAIN STUDY SO IF WE FIND IN THE VANGUARD THAT INDEED WE HAVE DATA GAPS OREE LEARN THINGS THAT'S OUR OPPORTUNITY TO MAKE THE ADJUSTMENTS IN THE MAIN STUDY AND WE WILL COME BACK TO YOU WITH THE GANNA GARD DATA AND ASK YOU TO ASSESS IT AND SAY, IS THIS DATA WHICH YOU FEEL IS INFORMATIVE AND DATA WHICH CAN BE SCALED UP AND WILL LEAD US TO ADDRESS A BROAD RANGE OF QUESTIONS, SO THOSE ARE THE TWO KINDS OF APPROACHES WE'RE TAKING. >> JOE? >> SO TWO QUESTIONS. ONE SMALL, IT JUST SEEMS WEIRD TO ME IN A SEQUENCE TO GO TO ONE GAP OF 12 MONTHS FROM 24-36 AND THEN GO BACK TO SIX MONTHS. SO IT'S THREE, SIX, NINE, 12, 18, 24, 36. 42, 48, 54, 60. >> OH THERE'S A TYPO THERE. >> OKAY, WELL THAT'S IN TABLE THREE, IF IT'S A TYPO I WON'T WORRY ABOUT IT, BUT THAT'S AN EXTRA VISIT THEN. BECAUSE THE THING THAT WORRIES ABOUT THAT IS THAT RETENTION. >> IT'S EVERY SIX MONTHS. >> OKAY, I THINK NOT BEING THERE PERIODICALLY IS AN ISSUE. >> UNTIL AGE FIVE. >> AND THE OTHER ONE I WORRY ABOUT AND THIS DEPENDS ON WHAT YOUR DEFINITION OF CHILDREN'S HEALTH IS, THERE'S A WHOLE BUNCH OF THINGS WHERE THE STIMMULE I WHERE IT COMES LATER, SCHOOL NUTRITION, BULLYING, SEXUAL ISSUES THAT COME UP, ANYTHING SIEBER BUT ANYTHING RELATED TO MEASURING ONLINE ACTIVITY, AND I'M NOT SURE THIS IS SO MUCH AN ISSUE OF WHETHER EVERY OTHER YEAR IS ENOUGH TO CATCH THOSE. IT MIGHT BE BUT I WORRY THAT THE AMOUNT THAT YOU CAN PACK INTO ONE VISIT EVERY OTHER YEAR WOULD MAKE IT VERY DIFFICULT TO GET INTO ANY OF THOSE THAT GET INTO CHILD INITIATED AND CHILD-DIRECTED BEHAVIORIAL HEALTH. AND IT MAY BE THAT THE DESIGN EVENTUALLY FINDS OUT, IT'S OKAY YOU CAN MAKE THE VISITS LONGER ONCE THE CHILDREN ARE OLDER BUT IF THAT'S NOT THE CASE, THAT WORRIES ME A BIT AND IT MIGHT ALLAY MY FEARS WHERE WE MAY SEE WHAT'S COLLECTED BUT WITH A MODEL THAT SAYS, WELL, YOU KNOW ROUGHLY HALF OF THIS IS THE STANDARD CORE QUESTIONNAIRE, THAT'S VERY LITTLE TIME TO GET INTO NUTRITION AND BULLYING AND AND EVERYTHING ELSE THAT KIDS DO WHEN THEY START ACTUALLY CONTROLLING THEIR OWN BEHAVIORS. >> RIGHT. SO NUTRITION IS PART OF THE CORE, WE'LL START THERE BUT NONETHELESS, I UNDERSTAND THE CONCERN WE CERTAINLY SHARE IT BECAUSE YOU DON'T KNOW WHAT IT IS THAT YOU'RE LOOKING FOR NECESSARILY. YOU CAN JUST BEGIN TO GATHER WHAT WE HOPE WILL BE ROBUST HEALTH RELATED PHENOTYPIC DATA AND THEN MAKE THE ANALYSIS THAT SEEM TO ARISE AS TIMES AND CONDITIONS DICTATE. THERE ARE QUESTIONS THAT WERE BURNING A DECADE AGO WHICH ARE LESS BURNING NOW AND NEW QUESTIONS, PARTICULARLY IN YOUR FIELD DR. KAHN'S MEDIA, EXPOSURE AND COMPUTERS AND SO FORTH WHICH WEREN'T EVEN IMAGINABLE A DECADE AGO, IN TERMS OF BEING REQUESTED SO, ONE OPTION WE'VE WE'VE DISCUSSED WHICH WE DECIDED NOT TO BRING FORWARD BUT IT WAS PART OF OUR DISCUSSION AND COULD BE REDISCUSSED AGAIN IS TO TAKE SOME OF THE WHOLE COHORT AND SEE THEM ON THE EVEN YEARS AND SOME ON THE ODD YEARS, BUT I'M NOT SURE WHAT WE GAINED. SO WE DISCUSSED THAT AND WE COULDN'T COME TO A DEFINITIVE STRATEGY BUT THE IDEA IS WHAT HAPPENS IN THE GAPS AND SO OTHER THAN ATTACH MONITORS THAT PROVIDE YOU KNOW WHERE'S MY CELL PHONE WHERE'S MY CHILD AND THEN YOU CAPTURE EVERYTHING. WE HAVE TO SAY WHAT IS IT THAT'S FEAS AND I BELIEVE WHAT IS IT WE'RE LIKELY TO DO, AND AGAIN I WANT TO EMPHASIZE THE NCS CAN DO SOME THINGS BUT IT CAN'T DO EVERYTHING AND IT'S UP TO US COLLECTIVELY TO PRIORITIZE WHAT IT IS WE THINK WE CAN DO AND WHAT IS FEASIBLE, AND WHAT'S GOING TO BE INFORMATIVE AND THEN SAY, WELL, WE MADE A CHOICE AND WE'LL LEVERAGE IT AS BEST AS POSSIBLE. >> I THINK WE DO HAVE ONE ABILITY THOUGH AT THE LATER AGES THAT YOU DON'T HAVE EARLIER. YOU COULD SAY THAT ON THE EVEN YEARS THERE'S GOING TO BE AN ONLINE SURVEY START OF AGES 12, 14 AND SO FORTH. THAT WOULD BE LOW COST COMPARATIVELY IF YOU DON'T SPEND SOMEBODY TO DO MEASUREMENTS AND YOU'RE SIMPLY GATHERING THE KIND OF DATA YOU CAN GATHER THROUGH SURVEY, SOME OF THAT CAN BE MORE FREQUENT AND THERE ARE ENOUGH RESEARCHERS OUT THERE THAT HAVE EXPERIENCE DOING THINGS LIKE LOOKING AT SOCIAL NETWORKS, IT'S REALLY EASY TO GO AND TALK TO SOMEBODY ABOUT THEIR FRIENDS. IT ACTUALLY TAKES A LONG TIME TO PROBE THROUGH WHEN YOU WANT--WHEN SOMEBODY'S TALKING ABOUT ONLINE CONNECTIONS AND FRIENDS, BASEDOT PEOPLE WHO HAVE BEEN DOING IT. SO SOME OF THIS MIGHT BE ECONOMICALLY DEALT WITH AND I THEN IS ONLY GOING TO BE MORE TRUE BY THE TIME WE HIT THIS THAT IT MAY BE THIS MAKES SENSE AT AGE EIGHT. AND COULD BE TESTED IN SOME OF THE VAN GUARD SAMPLES TO LOOK AT WHAT YOU DO WITH THIS POPULATION WITHOUT THE BUDDEN AND COST OF VISITS, BUT TO STILL MAINTAIN CONNECTION WHICH IS IMPORTANT FOR RETENTION AND GATHER THE KIND OF DATA THAT'S PARISHABLE. >> SO YOU BRING UP WHAT WE VIEW AS THE OPPORTUNITY OF THE NATIONAL CHILDREN'S STUDIES AS A PLATFORM AND WE WOULD BE THRILL FEDERAL PEOPLE LOOK AT WAWE HAVE AS A FRAMEWORK AND SAY WOW, I COULD THEN INTEGRATE ANOTHER STUDY ON TO THIS, USE THE COHORT, AND ASK SPECIFIC QUESTIONS ABOUT I'M GOING TO MAKE THIS UP, OKAY. THE ACQUISITION AND DEVELOPMENT OF MUSICAL TALENT OR PEOPLE'S EXPOSURE, INTEREST AND DEVELOPMENT OF CERTAIN KINDS OF ACTIVITIES ASSOCIATED WITH HOBBIES, LET'S SAY, OR THAT THEY'RE LOOKING AT--AND YOU CAN START TO GENERATE LOTS LISTS OF WHAT SORT OF THESE FOCUS DO, WE HAVE A PROGRAM SET UP TO DO THAT ADMINISTRATIVELY AND WE WANT TO ENCOURAGE BECAUSE AGAIN, WE HAVE LIMITED RESOURCES BUT WE THINK IF WE PROVIDE THE ROBUST PLATFORM AND A MECHANISM FOR PEOPLE TO USE THAT OTHERS COULD THEN INTEGRATE WHAT WE DO AND ASK THOSE ADDITIONAL QUESTIONS AND THEY MAY NOT NEED THE ENTIRE COHORT TO DO IT, EITHER. AND WE'RE PREPARED FOR THAT KIND OF INTEGRATED APPROACH. >> GOING BACK TO THE FIRST YEARS AND COLLECTION RESOURCES ARE BEING ALLOCATED, THERE IS THIS--YOU HAVE EVERY OTHER--A PERSONAL VISIT EVERY 12 MONTHS, REMOTE EVERY 12 MONTHS, STAGGERED, RIGHT? AND SO WHEN ONE THINKS ABOUT THE IN-HOME VISITS, THOSE ARE THE ONES THAT WILL BE TREMENDOUSLY VALUABLE AND THE QUESTION IS: ARE YOU PREPARED TO SPEND THE TIME OR DO WE THINK YOU CAN SPEND THE TIME IN THE HOUSEHOLD TO ADEQUATELY ASSESS THE PHYSIOLOGICAL PARTS AND ALSO THE DEVELOPMENTAL PARTS AND IT MAY BE THAT THE PHYSIOLOGICAL ARE GOING TO BE MORE IMPORTANT EARLY ON WHEN THEY'RE VERY YOUNG BECAUSE IT'S VERY DIFFICULT TO ASSESS THE CHILD'S SOCIAL AND EMOTIONAL AND COG NIDIFDEVELOPMENT--COGNITIVE DEVELOPMENT WHEN THEY'RE EXTRAORDINARILY YOUNG. BUT LATER ON YOU ABSOLUTELY CAN AND SO THE QUESTION IS, WELL, NOW, HOW ARE WE GOING TO ALLOCATE OUR TIME IN THE HOUSEHOLD OVER THE LIFE CYCLE BETWEEN THE PHYSIOLOGICAL AND DEVELOPMENTAL. SO THAT'S WHAT WE WOULD LIKE TO BE THE SUBSTANCE OF SUBSEQUENT DISCUSSIONS. OUR FOCUS SOPHISTICATED TO GENERATE THE PRINCIPLES AND THE FRAMEWORK WHEN IT GETS INTO WHETHER WE'RE GOING TO ASSESS HEPATIC FUNCTION, PANCREATIC FUNCTION, PULMONARY FUNCTION, FINE MOTOR, EXECUTIVE FUNCTION, SO FORTH, THAT'S WHAT WE HOPE TO BE DRESSING IN THE FUTURE MEETINGS AND SO THE ALLOCATION BETWEEN WHAT WE WILL CALL FOR THE MOMENT DOMAINS, THAT'S WHERE WE THINK THERE'S A LOT OF SUBSTANCE IN THE COMING YEARS AND THAT'S WHAT WE WANT TO NOT ONLY SEEK YOUR INPUT ON BUT TO IMPERICALLY TEST SO THAT WE COME BACK ALL THE TIME WITH WHAT THE VANGUARD STUDY EXPERIENCE IS. WE ARE JUST COMPLETING THE ITEM BY ITEM ANALYSIS OF WHAT HAPPENED FROM 2009 TO 2011. THE LAG WILL GET MUCH SHORTER BECAUSE AT THAT TIME WE HAD DISPARATE DATA SYSTEMS. THEY WERE PROPRIETARY, WE HAD TO INTEGRATE ALL OF THAT, BUT THAT'S THE KIND OF DATA THAT WE WOULD LIKE TO BRING BACK TO YOU AND SAY, THIS IS WHAT WE LEARNED SO FAR IN THE FIELD AND WHERE SHOULD WE GO? BASED ON WHERE WE THINK WE WANT TO BE? AS I'VE SHARED BEFORE BEFORE MY WIFE IS CANADIAN AND THEREFORE I CAN'T GO THROUGH 12 MONTH CYCLE WITHOUT GETTING ENMESHED IN HOCKEY AND THEN BEFORE HE RETIRED WE ABSOLUTELY HAD TO GO DOWN TO WATCH A HOCKEY GAME AND SEE WAYNE GRETSKY SKATE AND HE WAS THE ONE WHO SAID, YOU HAVE TO GO TO WHERE THE PUCK WILL BE. AND SO WE WANT TO SKATE TO WHERE THE PUCK WILL BE AND THAT'S WHY WE KEEP COMING BACK. >> ON A SMALL NOTE IT WAS WAYNE GRETSKY'S FATHER THAT SAID THAT. FOR FATHERS OUT THERE EVERYWHERE,. >> HIS FATHER HAD A-- >> HE REPEATED THE WISDOM OF HIS FATHER. >> HIS FATHER HAD-- >> ALL OF OUR CHILDREN WOULD DO THAT. >> HAD THIS BACK STROKE AND ACTUALLY COULDN'T REMEMBER THINGS AFTER THAT. SO WAYNE GRETSKY, I WANT TO GIVE HIM CREDIT BUT YOU'RE RIGHT. >> WELL THAT PROVIDES AN EXCELLENT SEGUETO MOTHERS AND FOR A LOT OF THE THINGS WE TALK ABOUT IN TERMS OF CHILD DEVELOPMENT MATERNAL AND PATERNAL INFLUENCES ARE VERY, VERY, SIGNIFICANT AND SO, UNDER THE HEADING OF CONCEPTUAL DIFFICULTIES. SO ONE PLACE I MIGHT MAKE A NOMINATION IS THAT THE CHILD STUDY SERIOUSLY CONSIDER WHAT IT'S GOING TO DO IN TERMS OF MEASUREMENT FOR THE MOTHERS BECAUSE THE MOTHER IS THE PERSON WHO'S THERE WITH THE CHILD SO MUCH OF THE TIME. WHAT'S THE NUMBER--I THINK--I THINK I SAW A NUMBER WHERE IS IT THAT FOR EVERY HOUR A CHILD IS IN SCHOOL, THEY SPEND THREE OR FOUR HOURS WITH THEIR MOTHER OR SOMETHING LIKE THAT. SO WE--WE AWIVE PERSEVERE MITRALERATE ON THE IMPORTANCE OF SCHOOLS AND THEY'RE CERTAINLY VERY, VERY, IMPORTANT, YET WHEN IT COMES TO CHILD DEVELOPMENT, THE MOTHER AND OF COURSE, THE FATHER ARE VERY MUCH RIGHT THERE AND SO THE QUESTION IS, TO WHAT EXTENT WE CAN USE THESE--THESE--THESE REMOTE MEASUREMENT PERIODS TO ESSENTIALLY COLLECT RELEVANT DATA ON THE MOTHER AND THE FATHER SO WE CAN QUANTIFY THE SORTS OF RESOURCES THAT ARE BEING BROUGHT TO THE CHILD DEVELOPMENT PROCESS IN THE H0B„ >> SO AS A PRINCIPLE, THAT'S WHAT WE HOPE WE'VE BEEN PRACTICING IN THE FIELD AND WE EMPHASIZE, THE FATHERS AND THE MOTHERS WHEN WE DO OUR DATA COLLECTION BUT THE PRO BAN, THE KEY PERSON IS THE CHILD AND WE'RE VERY FOCUSED ON THE MOTHERS DURING PREGNANCY. WE GET SOME DATA BUT I DON'T WANT TO LEAVE THE IMPRESSION THAT WE PROVIDE A COMPREHENSIVE POSTPARTUM STUDY SORE THAT WE WILL BE RESOURCE--STUDY OR THAT WE WILL BE WITH MEN'S HEALTH OR ANYTHING GOING FORWARD. WE HAVE MATERIAL THAT BASED ON OUR CURRENT WORLD VIEW IS FOCUSED ON THE DYNAMICS OF THE MOTHERS AND THE CHILDREN AND THE FATHERS AND THE CHILDREN AND THE FAMILY UNITS, WE COLLECT SOCIOLOGICAL DEMOGRAPHIC DATA OCCUPATIONAL EXPOSURES BECAUSE PEOPLE BRING THINGS INTO THE HOUSE FROM INDUSTRIAL CHEMICALS TO STRESS, WE WANT TO BE ABLE TO ASSESS THAT BUT THAT'S AGAIN I WANT TO EMPHASIZE, WE CAN'T BE EVERYTHING TO EVERYBODY AND WHILE WE ACKNOWLEDGE AND DO INVEST RESOURCES INTO THAT DATA COLLECTION, THE FOCUS IS ON THE CHILD. >> ALAN DID YOU WANT TO ADD ANYTHING TO THAT? OKAY. >> I HAVE ONE PRETTY FAST PIECE OF DATA THAT MAY PROVE HELPFUL SINCE YOU'LL HAVE THE STAYEDIOMETER DURING THE HOURS DURING THE COURSE OF THE STUDY, SO GRAB THE MEASUREMENTS OF THE MOTHER AND FATHER SO WHEN YOU'RE DETERMINING THE GROAPGHT OF THE CHILD, YOU WILL HAVE THAT DATA. >> YEAH, THE ONE FACTOR THAT I GUESS WE--MOST IN MY MIND WAS WERE MEASURES OF PARENTAL INTELLECTUAL ACUITY WHICH ARE VERY SIGNIFICANT FACTORS IN PREDICTING THE DEVELOPMENT OF A CHILD'S INTELLECTUAL ACUITY AND EVEN MATERNAL AND PATERNAL AJUTMENT. AND THESE ARE--ADJUSTMENT AND THESE ARE NOT PARTICULARLY DIFFICULT TO COLLECT BUT THEY HAVE A PROLONGED IMPACT ON CHILD DEVELOPMENT ALL THROUGHOUT AND SO, MEASURING THOSE INITIALLY AND CARRYING THEM THROUGH SHOULD, I WOULD THINK PAY--PAY ENORMOUS DIVIDENDS NOT ONLY IN TERMS OF CHILD DEVELOPMENT BUT PARENTAL ACUITY TELLS YOU A LOT ABOUT HOW THE PARENTS ORGANIZE MEDICAL CARE FOR THE CHILDREN, AT WHAT POINTS THEY INTERVENE, THEY OBTAIN TRADITIONAL RESOURCES, THEY GET INTERVENTION IN THE SCHOOL, IT'S NOT ONLY WHAT THEY BRING IN THE DIRECT TRAINING OF THE CHILD, IT'S ALSO WHAT THEY BRING IN TERMS OF ASSEMBLING RESOURCES JUST BECAUSE INCOME, JUST THE ABILITY TO ORGANIZE, THIS IS WHAT THIS CHILD NEEDS AND THIS IS WHERE I NEED TO GO TO GET IT. SO THOSE ARE--THOSE ARE THINGS THAT ARE I GUESS MY PERSONAL EXTRAORDINARILY IMPORTANT. WELL, WE CAN TAKE A SHORT BREAK, NO MORE THAN 15 MINUTES OR LESS AND WHEN WE COME BACK, I THINK WHAT WE WOULD LIKE TO PURSUE IS THE OTHER SUBSEQUENT POINTS NAMELY THE ISSUES OF THE CORE AND SUPPLEMENTAL MODULES AND THE WHOLE QUESTION OF THE EXTENT TO WHICH ALL CHILDREN GET ALL MODULES OR SOME CHILDREN GET SOME MODULES ANDu WHAT THAT DOES TO THE ABILITY TO ANALYZE THE DATA, WHAT IT DOES TO ISSUES OF POWER DOWN THE ROAD FOR A STUDY THAT'S--THAT EXISTS PRIMARILY TO DELIVER--TO DELIVER THE POWER, SO THAT'S--IF THAT'S ALL RIGHT WITH YOU LET'S START UP WITH THAT AFTER THE BREAK. >> GREAT SO WE WILL STIPULATE THAT EVERYBODY'S IN THEIR SEAT AND WE WILL MOVE FORWARD WITH THE NEXT DISCUSSION WHICH IS DEALING WITH MODULES, SO MAYBE AS A PREFASTO THIS DISCUSSION, LET'S PUT STEVEN ON THE SPOT ONE MORE TIME TO DISCUSS JUST EXACTLY WHAT IS MEANT BY A CORE ITEM. WHETHER THAT MEANS WHETHER WE'RE TALKING ABOUT IMPORTANCE OR REGULARITY AND WHAT WE MEAN ABOUT MODULES. ARE THESE MODULES THAT ROTATE IN AND OUT, ARE THEY MODULES THAT ARE--THAT ARE SOMETIMES GIVEN TO SOME PEOPLE AND NOT TO OTHERS JUST EXACTLY HOW YOU SEE THIS PROCESS OF PUTTING TOGETHER THIS MASSIVE COLLECTION OF PROJECT IN A MANAGEABLE MANNER AND THEN WE'LL TRY TO PICK, PICK, PICK AT IT. >> THE CONCEPT OF A CORE IS A BLEND OF WHAT IS NEEDED TO BE ASSESSED ON A REGULAR SCHEDULED BASIS THAT'S LIKELY TO BE DYNAMIC AND THEREFORE WE WOULD WANT TO GET MANY DATA POINTS AND WHAT WE FEEL IS, SENTIAL FOR US TO--ESSENTIAL FOR US TO ASCERTAIN THE HEALTH STATUS OF THE CHILD. >> SO THE IRREDUCIBLE MINIMUM THAT ARE REGULARLY REPEATED. >> RIGHT. A LITTLE ENVIRONMENTAL HEALTH SCIENCESITANT TO APPLY THE WORD IRREDUCIBLE BECAUSE EVEN IN A CORE IT MAY BE BILLED IN SOME INQUIRIES THAT PERHAPS DON'T MEET THOSE SPECIFICALLY BUT PERHAPS WE WOULD FIND A VALUE NOW, A CORE NECESSARILY HAS ALL ASPECTS CHANGE AT EACH ENCOUNTER. WE HAVE FOUR ENCOUNTERS IN THE FIRST YEAR, NOT INCLUDING THE BIRTH VISIT AND THEN WE HAVE TWO ENCOUNTERS EVERY YEAR FOR THE FIRST FIVE YEARS AND THEN AT THIS POINT IT'S AN ENCOUNTER EVERY OTHER YEAR SO THE CORE WOULD ALLOW US TO GENERATE IN A CONSISTENT MANNER, THE DATA POINTS TO UNDERSTAND AND THERE'S SOME DEBATE ABOUT THE PROPER TERM BUT I'LL USE THE TERM TRAJECTORY OF WHERE A CHILD WAS, WHERE A CHILD IS HEADED AND WHERE A CHILD IS AT THE MOMENT. MODULES ARE FOCUSED AROUND TOPICS THAT COULD BY RESPONSES TO THE CORE, COULD BE TRIGGERED BY OTHER OBSERVATIONS EITHER FROM THE FAMILY OR FROMLET MEDICAL RECORD--FROM THE MEDICAL RECORD AND THE MODULES ARE OF VARYING LENGTHS, WE HOPE THAT NONE ARE OVERLY COMPLEX BUT SOME ARE A BIT COMPLEX AS THEY TRY TO SORT OUT, LET'S SAY, I'M GOING TO GIVE YOU HYPOTHETICAL BUT IF WE HAD A MODULE ON--STATIC GROWTH, A CHILD IS NOT GROWING AT THE PACE THAT WE MIGHT EXPECT, THEN A MODULE IN THAT AREA WOULD HAVE TO LOOK AT MANY POSSIBILITIES SO IT MIGHT NOT BE NECESSARILY A SIMPLE COMPACT SERIES OF ASSESSMENTS WHEREAS IF WE WERE TO ASK A QUESTION LIKE, AND AGAIN I'LL MAKE THIS UP, BUT LET'S SAY IF WE WANTED TO LOOK AT A AIRING ANISEED SPORTS MODULE JUST ASKING LIKE LIKE HOW ORGANIZED AND HOW MANY HOURS PER WEEK AND WHAT LEVEL OF ACTIVITY THAT MIGHT BE SUFFICIENT TO GENERATE INFORMATION. SO THE MODULES WILL BE QUITE FLEXIBLE. THE MODULES IN OUR CONCEPT WOULD BE ADMINISTERED BASED ON CLUES, TRIGGERS OBSERVATIONS FROM THE CHILD BUT ALSO BECAUSE WE'RE RUNNING A STUDY WHERE WE WANT TO HAVE SOME CREDIBILITY IN THE RESPONSES, SOME MODULES WOULD BE JUST ASSIGNED RANDOMLY TO DIFFERENT PEOPLE SO THAT WE COULD UNDERSTAND THE NEGATIVES AS LAS THE POSITIVES AND--LOS ANGELES AS THE POSITIVES AND WE WOULD WANT TO HAVE CONFIDENCE THAT THE--THAT THERE IS A LINKAGE BETWEEN THE MODULES AND THE CORE IN ADDITION SO WE WOULD HOPE TO BE ABLE TO MAP TO SOME PART OF THE CORE WHAT THE VARIOUS MODULES ARE. THAT'S ON A HIGH RATHER ABSTRACT LEVEL I WOULD RESPOND TO THE FIRST SERIES OF QUESTIONS. >> SO I DON'T AND THINGS THAT MEASURE SUPPORT SUPPORT AND JEFF WE END THIS WELL, IF YOU WOULD AFTER ADA. >> I'LL TRY. >> MY FIRST COMMENT IS TO CONGRATULATE THE TEAM, MY APPROACH FOR GOING FOR TINO TYPIC DATA FOR SPECIFIC DISEASE ESTATES AND SPOT ON IN THE BRIEFING BOOK IT TALKED ABOUT HOW 21 YEARS FROM NOW WE MAY CALL THINGS DIFFERENTLY, WE MAY NEED TO WAIT 21 YEARS NOW DIFFERENT PEOPLE CALL THE SAME THING DIFFERENT THINGS SO I THINK THE DATA WILL BE MUCH CLEANER, MUCH RICHER AND ALSO REMEMBER THAT A LOT OF IT WILL BE GETTING FROM PARENTS PARENTS AND PARENTS MAY THINK OF IT IN DIFFERENT TERMINOLOGIES SO I THINK THE PHENOTYPIC DATA APPROACH IS THE RIGHT ONE SO I CONGRATULATE YOU ON THAT AND ALONG THOSE LINES WHEN I WAS THINK BEING MY OWN CLINICAL EXPERIENCE, IT SEEMS LIKE THERE'S ALMOST A FREEBIE VARIABLE IN THAT YOU DON'T HAVE TO COLLECT IT FROM THE FAMILY BUT IF YOU BUILD IN THE DATA WHEN YOU'RE DOING YOUR DATA, VETTING AND CLEAN UP, YOU CAN CREATE A NEW COLUMN, YOU KNOW FOR ALL THE CLINICIANS WHEN YOU DO THE SYSTEMS THERE'S SOME PEOPLE WHO HAVE WHAT WE CALL POSITIVE REVIEW SYSTEMS OR DO YOU HAVE HEADACHES, YES, COUGH, YES AND THEY GO THROUGH EVERYTHING VERSES THE PEOPLE WHO DON'T, THE DATA ANALYST CAN SCORE THE REPOSITORY DEGREE BECAUSE THAT ALMOST BECOMES ITS OWN VARIABLE, ARE THERE SOPHISTICATED MAT OICIZERS AND YOUR OWN SELF-PERCEPTION OF HEALTH. SO YOU ALWAYS HAVE A FREEBIE BUILT IN. >> I DON'T HAVE ANYTHING TOAD AT THIS POINT, I WOULD LIKE TO HEAR MORE OF THE DISCUSSION. >> DRNCHL. >> ON THE MATTER OF MODULES AND THE DISCUSSION INVOLVING TRIGGERS, THAT ITSELF IS A TRIGGER. ONE THAT CAME TO MIND WAS WHEN WE COLLECT DATABASED ON OBSERVED OUTCOMES, WE CAN GET INTO A PIT OF TROUBLE AGAIN GOING OUT TO A CONCEPTUAL FRAMEWORK, THERE ARE GENETIC INPUTS TO THE PROCESS GENERATING HEALTH AND DEVELOPMENT PHENOTYPES BUT THERE ARE OTHER INPUTSi AS WELL SUCH AS COMMUNITY INPUT, FAMILY INPUT, SIBLING INPUTS WHETHER YOU--WHETHER YOU HAVE ANOTHER SIBLING OR MOW MANY SIBLINGS YOU HAVE IN THE HOUSEHOLD, TELLS YOU SOMETHING ABOUT HOW RESOURCES HAVE TO BE ALLOCATED OR HAVE TO BE REITERATION WITHIN THE HOUSEHOLD WHO GETS WHAT FRACTION OF WHOSE TIME OR COMMODITIES OR MUSIC LESSONS OR HOCKEY LESSONS WHAT HAVE YOU, WORK ON HOCKEY TODAY, RIGHT? THE DANGER DANGER IS WHEN WE USE THOSE TRIGGERS TO GENERATE, THE COLLECTION OF THE DATA, WE ARE, CENTSIALLY CHOOSING TO COLLECT DATA IN PART BASED ON HOW PARENTS INVEST IN THEIR CHILDREN, HOW THEY TAKE CARE OF THEIR CHILDREN, CHOICES THAT PARENTS MAKE THAT MIGHT EXPOSE CHILDREN TO RISKS OF VARYING SORTS. AND SO, NOW, WHEN WE ARE--WE GIVE THE RESEARCHERS THESE DATA THAT ARE COLLECTED VIA THIS TRIGGERING, THE QUESTION THAT COMES TO MIND IN THE DATA ANALYST AHEAD IS WHAT AM I LOOKING AT? AM I LOOKING AT RANDOM OCCURRENCES BASICKED ON CHARACTERISTICS OF THIS HOUSEHOLD AND THIS CHILD OR AM I LOOKING AT THE CONSEQUENCES OF CHOICES THAT HAVE BEEN MADE, ENVIRONMENTS THAT HAVE BEEN CHOSEN SO THAT ESSENTIALLY THE DANGER OF USING TRIGGERS IS THAT ALL THE SUDDEN ONE HAS WHAT IS SOMETIMES REFERRED TO AS A CHOICE BASED SAMPLE WHICH IS NOT A GOOD THING. HAVING HAD THE EARLIER DISCUSSION ON SAMPLING AND HAVING A NATIONAL PROBABILITY DESIGN THE IDEA OF A CONVENIENT SAMPLE IS AN ASSET TO MANY PEOPLE SIMPLY BECAUSE OF A CONVENIENT SAMPLE ITSELF INFLATES ISSUES OF CHOICES AND PREFERENCES WITH THESE--WITH THESE OBSERVATIONS AND SO THE WHOLE--WHICH WAY THAT CAUSES [INDISCERNIBLE] HAPPEN TO POINT AND BECOMES KIND OF MUDDY. SO LET'S START WITH THAT RESERVATION AND SEE IF THAT MUCKS YOU UP ANY. >> HARDLY, SO, AND CERTAINLY OTHER STUDIES I'VE BEEN INVOLVED IN AND OTHER STUDIES THAT PEOPLE HAVE DONE, IF YOU LOOK AT THE CONCEPT OF TRYING TO UNDERSTAND A PARTICULAR PHENOTYPE, THE QUESTION IS HOW MANY CONTROLS DO YOU NEED OR WOULD YOU LIKE AND SOME PEOPLE SAY TWO-ONE AND SOME PEOPLE SAY THREE TO ONE AND THERE ARE DIFFERENT FORMUE LAIC APPROACHES THAT PEOPLE USE, AND WHAT WE WOULD LIKE TO DO IS LOOK AT A RANGE OF OPTIONINGS AND SEE WHAT WE WANT TO DO. BECAUSE DEPENDING ON THE QUESTIONS WE'RE ASKING DIFFERENT, DIFFERENT NEEDS AND DIFFERENT APPROACHES, AND THEN WHETHER WE WANT TO HAVE FOR OUR COMPARATORS WITHIN OUR POPULATION, WE WANT TO DO SOME SELECTION OR WE WANT TO DO SOME RANDOM ASSIGNMENT, I THINK WE REMAIN OPEN AND FLEXIBLE TO WHAT'S GOING TO GIVE US THE MOST CREDIBLE DATA BUT WE DON'T HAVE A FORMULA IN PLACE, WE JUST HAVE THE PRINCIPLE AND THAT IS OUR INTENT THIS MORNING TO LAY OUT THESE PRINCIPLES AND I THINK YOU'VE STATED VERY WELL, THE CAUTIONS ABOUT--I THINK THE TECHNICAL TERM IS CHERRY PICKING YOUR RESPONDENTS WHEN YOU TRY TO COLLECT DATA, WE HAVE TO COLLECT DATA IN SUCH A WAY THAT WE WANT TO COLLECT DATA THAT WE FEEL IS INFORMATIVE ON CERTAIN POPULATIONS AND CERTAIN INDIVIDUALS, BUT ALSO CREDIBLE IN THOSE POPULATIONS AND INDIVIDUALS. SO, IF THERE'S SUCH A THING AS BAKED IN INTO THE, INTO THE APPROACH, WE WANT TO HAVE THAT BAKED IN. >> AND I THINK ONE OF THE SUGGESTIONS YOU'VE MADE CAN IN LARGE MEASURE GET US OUT OF THIS DIFFICULTY AND THAT IS THE IDEA OF SOME MODULES MIGHT BE TRIGGERED BUT THAT OF COURSE FOR A SAMPLE OF THE POPULATION OR A PART OF THE SAMPLE THAT THAT THE MODULE NEEDS TO BE ADMINISTERED RANDOMLY THAT IS THE WAY OUT OF THE PROBLEM OF CHOICE BASED SAMPLES IS TO UNDERSTAND WHAT THE--WHAT THE--WHAT THE PREVALENCE RATE HAPPENS TO BE FOR THESE VARIOUS CONDITIONS IN THE POPULATION AND SO, WITH A LITTLE BIT OF STATISTICAL CLEVERNESS AND CAN TAKE A LOOK AT THE RANDOM SAMPLE OF A SUBSAMPLE OF THE POPULATION, LOOK AT THE PREVALENCE FOR SOME OF THESE THINGS THAT YOU'RE LOOKING ASTERISKS IN THE TARGETED SAMPLE. RELATE THOSE TO THE CHARACTERISTICS OF PEOPLE WHO THEN CONCEPTUALLY UNDO THE CONTAMINATION THAT MIGHT HAVE OCCURRED TO THE EXTENT THAT SOME OF THESE TRIGGERS MIGHT HAVE BEEN IN FACT CHOICE BASED ON ON THE DECISIONS THAT THE FAMILY UNIT HAS MADE AS OPPOSE TO KIND OF BEING PREORDAINED AND EXOGENOUSLY RAIN DOWN ON THEM IN SOME OTHER WAY, SO, I GUESS I WOULD ENCOURAGE TO YOU KIND OF THINK OF IT IN THAT WAY IF YOU ARE GOING TO GO FOR TRIGGERS TO THINK VERY SERIOUSLY ABOUT HAVING A SET ASIDE OF A RANDOM SAMPLELE SO THAT CAN YOU MAKE SURE THAT YOU KNOW WHAT THE PREVALENCE OF THOSE TRIGGERS AND THESE CONDITIONS ARE ASK THAT DOESN'T ALWAYS HAVE TO DO THE WHOLE SAMPLE AND THAT'S THAT THE TRIGGERS ARE IRRELEVANT AND THAT'S ALL RIGHT, BUT THEN, HAVING DONE THIS OTHER STRATEGY WOULD ALLOW TO YOU ARRIVE AT THAT ANSWER. JENNIFER? >> I'M GETTING I'M HAVING TROUBLE WITH THIS TERMINOLOGY BECAUSE IT'S DATA COLLECTION, THE CORE STAYS THE SAME OVER TIME AND EVERYONE GETS TO WHERE IT'S NOT GOING TO STAY THE SAME OVERTIME AND THE THINGS YOU MEASURE AS CHILDREN AGE, CERTAINLY BACK FOR QUESTION AIR YOU USED AT THREE MONTHS, IF YOU'RE GOING TO USE IT 15 YEARS, SO THERE'S GOING TO BE CHANGE OVER TIME AND THESE IDEA OF TRIGGERS AND I THINK HOW YOU DEAL--THERE'S VERY DIFFERENT KINDS OF TRIGGERS, I THINK I'M HEARING. THERE ARE SOME THAT SOUND LIKE SKIP PATTERNS AND YOU KNOW, YOUu+z YOU MIGHT WANT TO GET SENSE WHEN YOU GO DOWN THE LINE UNLESS YOU GET A YES, YOU CAN CHOOSE TO GO DOWN THERE ON THE POPULATION OR NOT BUT THAT'S VERY DIFFERENT THAN LOOKING FOR OTHER WAYS TO KIND OF FOCUS IN ON ASKING PARTICULAR TOPIC AND I THINK THAT'S WHERE YOU REALLY GET INTO THIS CHOICE AND IT'S VERY HARD TO HANALIZE THE DATA. --ANALYZE THE DATA. ALL THIS TO ME IS IN THE A CONTEXT OF LIMITED BURDEN AND YOU WANT TO GET THE MOST YOU CAN OUT OF THE BURDEN BUT AS YOU TAKE LITTLE PIECES AND ADD A LITTLE PARTS ON SOME OF THAT SAMPLE IS GOING TO GET A LOT OF THINGS BECAUSE THEY GET A LOT OF TRIGGERS AND SOME WILL GETTING IN AND THAT I THINK IS A HUGE ISSUE AND THEN AT THE END OF THE DAY, YOU MAY WANT TO LOOK AT THINGS FROM VARIOUS PARTS OF THAT MESSY QUESTIONNAIRE YOU HAVE NOW AND YOU'RE GOING TO WIND UP WITH THREE PEOPLE, I'VE BEEN THERE. SO IT'S VERY HARD TO PROVIDE ANSWERS TO THIS WITHOUT REALLY UNDERSTANDING MORE OF WHAT IS EVERYONE'S GETTING--WHAT THE TRIGGERS ARE, WHERE THEY'RE TAKING YOU, AND HOW YOU MIGHT TRY TO ADJUST FOR THIS CHOICE ISSUE BECAUSE YOU'RE GOING TO HAVE VERY COMPLICATED SAMPLING SCHEMES, TELL BE HARD TO FIGURE IT OUT EVEN THOUGH IT'S COMPUTER ICED ITINGLY BE EXTREMELY HARD FOR THE USER AND I WOR THAT'S THAT'S--WORRY THAT TRYING TO PUSH THIS DATA COLLECTION SO FAR IS GOING TO MAKE IT FALL APART. I HEARD A LOT THIS MORNING ABOUT WE HAVE TO PRIORITIZE, I THINK YOU HAVE TO PRIORITIZE, YOU IT CANNOT BE EVERYTHING, IT'S GOING TO MAKE SOME PEOPLE UNHAPPY BUT TO TRY TO PUSH IT I THINK YOU'RE GOING--YOU HAVE SO MANY THINGS IN THIS STUDY THAT ARE HARD TO DO LIKE RETENTION AND SOME--AND YOU REALLY WANT TO TAKE ADVANTAGE OF THE FACT THAT YOU HAVE A RANDOM SAMPLE OF BIRTHS AND SOME PREBIRTH PREDATA THAT IT COMES BACK TO THAT BECAUSE IT'S TRULY UNIQUE AND ALL OF THESE DECISION VS TO HAVE SOMETHING THAT YOU GO BACK TO--DECISIONS HAVE TO HAVE SOMETHING YOU GO BACK AND I DON'T KNOW--YEAH, IT SOUNDS FINE EXTENT THE DETAILS AND I THINK YOU NEED THAT DETAIL FOR THESE PHYSICIANS. TO GIVE AN EXAMPLE OF HOW THIS REACHES OUT AND BITES YOU IS LOOKING AT THE AREA OF CHILDCARE AND CAN YOU DO A STUDY OF CHILDCARE AND YOU CAN LOOK AT GHEE WHAT, WHAT DID CHILDCARE SERVICES COST AND WHO USES THEM AND WHAT'S THE IMPACT ON THE CHILD, BUT UNDERNEATH THE SURFACE OF DATA FOR PEOPLE WHO PUT THEIR CHILDREN IN CHILDCARE IS THE FACT THAT SOMEBODY'S MAKING A CHOICE, THAT PEOPLE FACE DIFFERENT QUALITIES OF COST AND CHILDCARE AND BASED ON THE OPPORTUNITIES MAY FACE IF THEY FACE HIGH QUALITY LOW COST CHILDCARE THEY WILL DO ONE THING, IF THEY HAVE A SET OF CHILDCARE OPPORTUNITIES THAT ARE HIGH COAST AND LOW QUALITY THEY'LL DO ANOTHER THING AND THAT INFLUENCES WHO THEY HIRE AND WHAT THE OUTCOMES ARE. PUTTING THAT--ALL OF THAT TOOTH PASTE BACK INTO THE TUBE WHEN YOU ONLY COLLECT THE DATA FOR PEOPLE WHO PUT YOUR CHILD INTO CHILDCARE IS HORRIBLY, HORRIBLY DIFFICULT AND SO I THINK THAT'S--THAT'S THE PART WE REALLY WANT TO, YOU KNOW CAUTION. BUT NOW LET'S GO BACK TO JENNIFER'S POINT AND YOU'VE SEEN THAT IN OTHER STUDIES AS WELL THAT PEOPLE SEE THE BURDEN ISSUE COMING DOWN THE PIKE. WE CAN'T ASK ALL THESE QUESTIONS AND SO THIS WILL DRAW A RONDE AMILLIO NUMBER AND DRAW 20%, THESE QUESTIONS WE GIVE 50%, THESE QUESTIONS AND BY THE TIME YOU DOLET--DO THE DIVVYING UP BECAUSE EVERYBODY HAS QUESTIONS THEY UTR TO PUT IN THE STUDY AND THE QUESTIONS WILL BE HORRIFYING IF YOU HASN'T ALREADY BECAUSE JENNIFER'S POINT IS REALLY WELL TAKEN, YOU DON'T KNOW HOW PEOPLE USE THOSE DATA. FOR ONE PERSON THIS IS A CONTROL VARIANT, IT'S A CO VARIANT. FOR ANOTHER PERSON IT'S AN OUTCOME AND NOW THEY'RE LOOKING AT A MODELING EFFORT TO SEE WELL, WHAT--WHAT--WHAT GOVERNS YOU KNOW CHILDHOOD OBESITY AND YOU'VE GOT THESE QUESTIONS PRESENT FOR THESE KIDS AND THESE QUESTIONS FOR THOSE AND ALL OF A SUDDEN, THE INTERSECTION OF THE OBSERVATIONS WHERE YOU HAVE ALL THE DATA YOU REALLY WANT GETS YOU WAY DOWN THE METER FROM EQUALING A HUNDRED OR EVEN 90,000. I THINK THAT'S WHAT EVERYBODY'S GOING TO WORRIED ABOUT WITH THIS SORT OF THING. >> I WANT TO TRY TO SIMPLIFY THIS FOR A SECOND. NOT THAT IT'S EASY, BUT IT SEEMS LIKE WE'RE MERGING TOGETHER A BUNCH OF THINGS THAT ARE FUNDAMENTALLY DIFFERENT SO THERE IS A DIFFERENCE BETWEEN TRIGGERING SOME SET OF THINGS IT'S A FUNDAMENTAL SKIP THAT SAYS IT DOESN'T MAKE SENSE TO ASK THIS QUESTION IF YOU DIDN'T MEET THE TRIGGER. IT SAYS, YOU KNOW, DOES YOUR CHILD GO TO OUT OF HOME CHILDCARE? IF YOU SAY NO, I CAN'T THEN ASK, WELL IS IT A LICENSED CENTER? BECAUSE THERE'S NO MEANING TO THAT QUESTION. WHEREAS YOU CAN COME BACK AND SAY, WELL WE'RE REALLY INTERESTED IN STUDYING A CERTAIN SET OF THINGS IN PEOPLE WHO HAVE ALLERGIES BUT THOSE QUESTIONS WE COULD ASK TO A POPULATION TO PEOPLE WHO DON'T HAVE ALLERGIES BECAUSE ALLERGY ISN'T SOMETHING THAT IS SUDDENLY AS CLEARLY ON AND OFF. YOU CAN STILL ASK PEOPLE HOW OFTEN THEY FEEL THEY GET HEADACHES, HOW OFTEN THEY FEEL UNWELL WHEN THEY'VE BEEN OUTSIDE OR OTHER THINGS TO A POPULATION SO WE DO NEED A TERMINOLOGY TO DISTINGUISH THOSE. WE'RE ALSO DEALING WITH THE POINT THAT THE VOCABULARIES AND THE THRESHOLDS SHIFT, THAT TODAY WE MIGHT THINK WE KNOW HOW TO TAXON MYSELF THAT SOMEBODY HAS ALLERGIES AND I PROMISE IN 2024 THE DEFINITION OF WHETHER IT'S A ALLERGY OR SOMETHING ELSE WILL BE DIFFERENT BECAUSE WE WILL HAVE BETTER UNDERSTANDING OF WHAT THAT MEANS SO THE PERSON DOING RETROSPECTIVE ANALYSIS WON'T WANT TO DEPEND ON THE CLASSIFICATION OF YOUR YES OR NO AND IT MAY BE A PRINCIPLE YOU NEED THAT YOU ALWAYS KEEP THE MOST RAW DATA EVEN IF MOST PEOPLE ARE GOING TO USE ANALYST CODINGS BECAUSE SOMETIME DOWN THE ROAD SOMEBODY'S GOING TO HAVE A REASON TO RECODE AND THAT'S AN EASY PRINCIPLE, DATA IS CHEAP, THE ONLY ISSUE YOU GET INTO IS WHAT YOU CAN RELEASE OR NOT RELEASE BECAUSE OF IDENTIFIABILITY ISSUES THERE BUT ALWAYS KEEPING THE RAW DATA HAS SOME HELP THERE. I THINK THE THING THAT'S REALLY GETTING EVERYBODY IS THE FACT THAT YOU HAVE SIX OR SEVEN SIGNIFY MENTIONS THROUGH WHICH YOU CAN SAMPLE. SO YOU HAVE THE ISSUE OF YEAH, I CAN SAMPLE OVER TIME, I CAN ADD THIS QUESTION AT 42 MONTHS AND SEVEN YEARS EVEN THOUGH WE WOULD LIKE TO ASK IT EVERY TIME BECAUSE THAT'S ALL WE CAN ALLOCATE. YOU CAN HAVE THESE THINGS YOU SAMPLE ACROSS POPULATION BUT WHEN YOU DO THAT, AS YOU POINTED OUT THERE'S THE SPECIAL ISSUE WE HAVE TO DEAL WITH THE FACT THAT POPULATIONS WEREN'T RECRUITED THE SAME AND SO WHEN YOU'RE SAMPLING ACROSS TIME OR POPULATION DO WE TAKE SPECIAL CARE TO MAKE SURE WE'RE SAMPLING EVENLY FROM THE COHORT DONE THROUGH THE PRE-BIRTH PROVIDERS, THE DELIVERY TIME PROVIDERS, THE PRENATAL COHORT, THE VARIOUS THINGS THAT WE GET OR OTHERWISE YOU LOSE CERTAIN OTHER STATISTICAL PROPERTIES AND THEN THERE'S A WHOLE BUNCH OF DIMENSIONS ON WHICH YOU CAN SAMPLE WITH CAUSE AND I THINK THAT'S WHAT YOU'RE SAYING JENNIFER IS THE ONE THAT WORRIES YOU BECAUSE AS YOU COME BACK AND SAY, I'M GOING TO ATTEMPT TO SAY, I WILL LOOK AT THE PEOPLE IN THIS GROUP, AND PART OF WHAT'S CHALLENGING IS I HAVEN'T SEEN THIS LAID OUT SOMEWHERE AND SOMETHING AS SIMPLE AS SUPPOSE YOU SOMETHING IS WORTH STUDYING IF YOU CAN GET A POPULATION OF 20,000 DATA POINTS WHAT ARE THE WAYS IN WHICH YOU CAN GET THOSE 20,000 DATA POINTS ACROSS THE DIMENSIONS OF THIS STUDY IN THE VARIOUS WEIGH WAYS PEOPLE HAVE BEEN RECRUITED AND IN ESSENCE WHAT'S THE FORM OF THE QUESTION THEY WANT TO POS INTERNAL OR OR EXTERNAL I'D RATHER LOSE THE GENERALIZABILITY RATHER THAN LIEUS THE STATISTICAL POWER OR SOMETHING THAT'S THERE, I DON'T THINK WE CAN RESOLVE THAT HERE BUT THAT'S WHY WHERE A LOT OF THE DISCOMFORT IS COMING IS THAT THIS SEEMS REALLY HARD AND THERE'S NO MAP THAT SAYS HERE ARE THE DIFFERENT DIMENSIONS, HERE ARE THE TRADE OFFS YOU CAN MAKE CAN IN THE END WHAT'S GOING TO HAVE TO HAPPEN FOR CONVENIENCE IS HERE ARE THE SIX PACKAGES I ARE GOING TO SAMPLE. WE'RE NOT GOING TO SUPPORT EVERY COMBINATION THAT SOME PEOPLE CAN SAMPLE ACROSS TIME AND OTHERS ACROSS THIS AND OTHERS ACROSS THAT BUT A SMALL SET OF THEM THAT WORK FOR A REPRESENTATIVE SET OF QUESTIONS. ANOTHER PIECE OF WHAT'S MISS SUGGEST HOW MUCH YOU HAVE TO GO TO THE CORE BECAUSE IT SOUNDS TO ME FROM ALL THE CONVERSATIONS I'VE HEARD, THERE'S A LOT IN THAT CORE. SO, ARE WE SPENDING A LOT OF TIME WORRYING ABOUT FIVE MINUTES WORTH OF QUESTIONS. I DON'T KNOW IF THAT IS WORTH IT. I THINK IT--TO REALLY GET BACK TO WHAT--WHAT GIVES YOU THE BIGGEST BANG FOR THE REASON THAT YOU ARE REALLY DOING THIS STUDY AND TAKING ADVANTAGE OF THE HUGE, HUGE INVESTMENT IN GETTING THIS SET OF BIRTHS AND PRE--PREDELIVERY MOTHERS. ONCE YOU DO THAT, CAN YOU LOOK AT THAT AND THEN FIGURE OUT IF IT'S WORTH IT TO DO ONE RESEARCH ACTIVITY THAT NEEDS TO 20,000 FROM THAT PARTICULAR ONE AND I WORRY ALSO ABOUT ADDING ON--THE PLATFORM IS GREAT, BUT AGAIN IT'S YOUR TOTAL BURDEN ALSO I WANT TO WORRY ABOUT AND HOW DOES IT ALL FIT TOGETHER IT'S ALL TRADE OFFS. I WAS GOING TO ASK ADA, GIVEN YOUR CONCERN ABOUT PUBERTY, WHERE WOULD YOU CHANGE THAT, THAT SCHEDULE, IT'S GIVEN YOU ONLY HAVE 21 VISITS, YOU WILL MOVE SOME, LOSE OTHERS BUT THAT'S AT LEAST I THINK AN ANSWERABLE QUESTION, YOU CAN SAY WELL YOU WANT TO DO THAT OR YOU DON'T WANT TO DO THAT BUT IT'S GOING TO COME DOWN TO THAT. IT'S GOING TO COME DOWN TO MAYBE HUNDREDS OF THOSE DECISIONS AND I THINK AT SOME POINT THAT'S WHAT YOU HAVE TO DO SO I WOULD LIKE--DO YOU HAVE AN ANSWER? SO, YOU KNOW IN TERMS OF THINKING OF PUBERTY SCHECHARACTERRIZING PUBERTY SCHEGROWTH SPURT OF PUBERTY LONGITUDE NAT DATA IS BETTER THAN CROSS SECTIONAL, SO EVEN GETTING KIDS AT DIFFERENT AGES IS DIFFERENT THAN HAVING LONGITUDINAL FOR KIDS. IN PRACTICE WE HAVE CONCERNS WE FOLLOW FOR PUBERTY WE SEE THEM GENERALLY IN SIX MONTH INTERVAL, YOU KNOW, WHEN I DO MY REVIEW OF SYSTEMS, I SAY OKAY DO HAVE YOU PUBIC HAIR, WHEN IT DID START, WEEKS OR MONTH SYMPATHETIC NERVOUS SYSTEM EVEN THIS IT'S TOUGH TO GET AN ANSWER, IF YOU'RE ASKING TWO YEARS LATER, I'M AFRAID WITH TWO YEAR INTERVAL FOR EACH GIVEN CHILD YOU WILL HAVE ONE DATA POINT AND YOU WILL TRY TO RECONSTRUCT ALL THE PUKEER EVERYONE IN THE--PUBERTY AND YO U CAN'T RELY ON THAT RELIABLE. >> IF YOU SAY GIRLS FOCUS ON 11 AND BOYS AT 10, I WOULD FOCUS ON 13 BEING THE CRUX, AND THEN THE QUESTION IS CAN YOU GET AWAY WITH SUPPLEMENTAL MEASURES OR QUESTIONNAIRES CAN YOU ASK MAYBE TO FILL IN SOME OF THE PIECINGS. --PIECES. IT'S HARD-- >> IT'S HARD TO IMAGINE A SCALE AND GIVING A CHILD A SHOW CARD THAT SAYS: SO WHAT DID YOU LOOK LIKE 18 MONTHS AGO? DID YOU DO THAT? [LAUGHTER] SO I THINK THAT'S THE PIECES FOR STEVEN TO RESPOND TO, FIRST OF ALL THE WHOLE QUESTION OF OF ADMINISTERING PARTS OF THE PROTOCOL TO PARTS OF THE SAMPLE, IN A SITUATION WHEN THE NUMBER OF PARTITIONS THAT ARE GOING TO BE CONTEMPLATED IS SO VERY, VERY LARGE AND THEN WE'LL GO ON TO THE CORE ITEM OF THINGS THAT CHANGE OR TIME AND IN UNPREDICTABLE WAYS THAT ARE MOST CONVENIENT FOR OUR SCHEDULE. >> SO WE WANTED TO EXPLORE AND I'LL NOTE THAT ALTHOUGH WE HAD WELL-INTENTIONED PEOPLE TELLING US THAT WE SHOULD ASK THE SAME QUESTIONS OF EVERYBODY ALL THE TIME, WE WANT TO HEX MORE THIS MATE--EXPLORE THIS MATRIX APPROACH. AND IN THE MATRIX APPROACH, THERE'S AGAIN VARIANTS, CAN YOU HAVE THIS INCREDIBLY COMPLEX EVEN IF THE CHAIR MAY OR NOT BE APPROPRIATE BAROQUE QUESTIONNAIRE WITH SKIP PATTERNS AND BRANCHES AND ALL KINDS OF THINGS THERE, BUT WE OPTED TO DO SOMETHING DIFFERENT AND THEN WHAT WE WANTED TO SET UP A SYSTEM THAT MIGHT NOT NECESSARILY ADHERE TO ONE OF EINSTEIN VICTIMS MAKE IT AS SIMPLE AS POSSIBLE BUT NOT SIMPLER, BUT WE WANTED TO ASSURE THAT WE HAD A STRATEGY FOR FORLET CONSISTENT LONG NUDETAL DATA SYSTEM, WE THINK THAT'S A VERY VALUABLE ASPECT OF THE STUDY. SO THAT WAS THE GENESIS OF THE CONCEPT OF THE CORE, THE CONTENT OF THE CORE IS STILL EVOLVING. WE HAVE A VERSION, WE SHARED SOME OF IT AT LEAST ON A HIGH LEVEL WITH YOU, BUT THAT'S DEFINITELY A WORK IN PROGRESS. WE HAVEN'T CHAIRED WITH YOU MUCH DETAIL ABOUT THE MODULES EXCEPT THOSE THAT ARE AGE SPECIFIC AND THEY ARE REALLY NOT SO MUCH AGE SPECIFIC, IT'S DEVELOPMENTAL STAGE SPECIFIC SO JUST AS DR. CONTINE SAID, YOU DON'T WANT TO ASK SOMEONE ABOUT THE LICENSE OF THE DAY CARE PROVIDER IF THEY DON'T HAVE DAY CARE, SOME SOME AGES IT'S A BIG DEAL WHETHER YOUR CHILD ROLLS OVER AND AT OTHER AGES IT'S A--IT SEEMS TO BE SOMETHING THAT WOULD PROVOKE AN EMOTIONAL REACTION BUT NOT A SCIENTIFICALLY VALID RESPONSE. SO, SO THAT'S HOW WE BEGAN TO THINK ABOUT WHERE WE COULD KEY IN SOME OF THESE DEVELOPMENTAL STAGE SPECIFIC INFORMATION. THE TOPICS OF THE OTHER MODULES WILL BE--AND OUR THINKING NOW IS TRIGGERED BY EXPOSURES AND OUTCOMES, SO IF WE FIND THAT I'LL BE PERHAPS DRAMATIC BUT THAT THEY--AN AREA WENT THROUGH AN EARTHQUAKE AND A LOT OF HOMES WERE DESTROYED THEN WE MIGHT HAVE A DISASTER RESPONSE TYPE MODULE THAT WOULD PICK UP ON THIS EVENT AND WE WOULD THEN ASK THE SAME QUESTIONS OF OTHER PEOPLE BUT IT'S NOT GOING TO HAVE THE SAME DEGREE OF INFORMATIVENESS. IF WE HAVE OTHER TYPES, AGAIN TO PROVIDE CONTINUITY IN THE CONVERSATION, IF WE HAVE A CHILD THAT SEEMS TO HAVE SYMPTOMS WHERE WE WOULD SAY THERE'S ENVIRONMENTAL TRIGGERS THAT CAUSE ITCHING THAT CAUSE SOME TYPE OF LOCAL IMMUNE REACTIONS THAT CAUSE SOME KIND OF INFLAMMATION AND SO FORTH, WE WANT TO CAPTURE THAT AND FOLLOW THAT LONG NUDEINALLY, SO FOR THAT--LONGITUDINALLY SO FOR THAT CHILD THEY GET THE CORE QUESTIONNAIRE THAT EVERYONE GETS AND THEN THEY WOULD GET THEIR OWN AUGMENTATION BECAUSE WE WANT TO FOLLOW THAT PARTICULAR ASPECT LONG NUDEINALLY IT WOULDN'T DEPEND ANYMORE ON THEM HAVING TO HAVE SYMPTOMS ORDER TO GENERATE THE QUESTION. ONCE THEY HAVE FALLEN INTO A PARTICULAR CLASS OR CATEGORY THEN OF QUALIFYING FOR A MODULE, WE WANT TO CONTINUE ADMINISTRATION OF THAT. THE COMPLEXITY IS HUGE. WE'VE TRIED TO LOOK AT SOME OF THE PROBABILITIES OF LINKING DIFFERENT MODULES AND IF YOU START WITH SOME NUMBER LIKE 20, YOU KNOW YOU CAN GET LARGE NUMBERS OF OPTIONS, SO, I THINK WHAT I'D LIKE TO CONCLUDE WITH SAYING, WE DON'T HAVE YET A SOLUTION, WE JUST HAVE A STRATEGY AND WHAT WE WANT TO DO IS USE THE VANGUARD STUDY TO SEE HOW WELL THIS WOULD PLAY OUT. IT'S KIND OF A BLEND OF AN ADAPTIVE DESIGN AND A--A CORE CONSISTENT DESIGN JUST LIKE OUR OVERALL SAMPLING IS A BLEND OF A NATIONAL PROBABILITY SAMPLE PLUS SOMETHING ELSE BECAUSE WE CAN'T DO WITH EITHER ONE WHAT WE FEEL WE OUGHT TO DO, FEEL THAT WE'RE REQUESTED TO DO SO WE WANT TO TAKE A HYBRID APPROACH SO THIS IS A HYBRID APPROACH AND ITS EXPERIMENT. I DON'T HAVE ANY FURTHER SPECIFIC COMMENTS BECAUSE WE'RE TRYING TO DEVELOP THESE, BUT WHAT WE WANT TO HAVE TODAY WAS THE INPUT AT A HIGH LEVEL WHAT IS IT THAT WE SHOULD BE FOCUSING ON, TECHNICALLY AND ANALYTICALLY AND WHAT ARE SOME OF THE EXPERIENCE AND THOUGHTS THAT OTHERS MAY HAVE, SO THAT'S I THINK ALL I'LL COMMENT. >> THIS IS JEFF, CAN I CONTRIBUTE AT THIS POINT TO THE DISCUSSION. >> PLEASE DO. >> OKAY, SO IN KEEPING WITH STEVE'S COMMENTS AT THIS POINT AND THEIR DIRECTION, I HAVE A FRY RECOMMENDATIONS--FEW RECOMMENDATIONS. SO ONE, IN TERMS OF THE COURSE OF THE QUESTIONS AND THE CORE ISSUES, OBVIOUSLY THERE ARE MANY DOMAINS HERE, BUT I LEARNED TO LOOK TOWARDS SOME OF THE ONCOLOGISTS OF THE DATA. THERE ARE LOT OF PEOPLE WHO ARE WORKING IN THIS AREA AND THEY HAVE DEVELOPED VARIOUS ONTOLOGIES AND THEY MIGHT BE A GOOD INDICATOR OR QUESTIONS THAT WOULD SUGGEST NOR INDEPTH MODULES. SO FOR EXAMPLE, THERE MAY BE SOME SPECIFIC INDICATOR QUESTIONS RELATED TO ASTHMA AND THEN IF YOU GET IN A RESPONSE, THEN THERE'S ALREADY STANDARDS FOR FOLLOW UPS LIKE THE ISAAC FORM WHICH IS VALIDATED INTERNATIONALALLY, AVAILABLE IN MANY LANGUAGES, ET CETERA SO YOU DON'T WANT TO DO THE ISAAC ON EVERYBODY AT EVERY VISIT BUT YOU COULD USE THESE VARIABLES AS A GOOD SOURCE OF THEM ARE FOR THE USE OF EFFORTS ON PHENOTYPIC ONCOLOGYS AS A SUGGESTION. THE OTHER PIECE I WANT TO QUICKLY ADD AND I--IT'S SOMEWHAT OF A HINGE, SO I HESITATE BUT IN THE MATERIALS WE'VE SEEN PRESENTED AND THE DISCUSSIONS WE'VE HAD, NOWHERE DO I FIND THOSE PART OF THE ACKNOWLEDGMENT OF THE MEDICARE AND MEDICAID, YOU KNOW ELECTRONIC HEALTH RECORD INCENTIVE PROGRAM AND IT'S IMPACT ON ON THIS STUDY AND POTENTIAL DATA COLLECTION EFFORTS, SO I THINK THAT THIS STUDY HAS A VERY LONG TIME [INDISCERNIBLE] AND THIS EFFORT WITH RESPECT TO MEDICARE AND MEDICAID, OF COURSE IS FAIRLY NEW TO THE HEALTHCARE REFORM LEGISLATION BUT NONETHELESS, IT CLEARLY POINTS OUT THAT, YOU KNOW HEALTHCARE PROVIDERS, AND REMEMBER WE TALKED ABOUT THIS STUDY BEING PROVIDER BASED IN SOME WAY, HEALTHCARE PROVIDERS ARE--YOU KNOW THE MAR JORRITY OF THEM WILL NOT GET ON BOARD WITH THIS AND THEY'LL INYEAS IN A LARGE FRACTION OF THEM WHICH PROVIDES ACCESS TO A VAST AMOUNT OF DATA AND THAT THE GOAL OF THIS EHR LEGISLATION SO I WOULD RECOMMEND THAT SOMEWHERE IN THE PLANNING DOCUMENT THAT YOU ACKNOWLEDGE THIS THING AS TAKING PLACE, SO IF YOU'RE SAMPLING PROVIDERS, YOU'LL ACTUALLY BUMP INTO THE SYSTEMS AND THE EHRs AND MIGHT PROVIDE YOU WITH A WAY TO CROSS VALIDATE SOME OF THE ABILITY TO DETECT, AT LEAST HEALTH EVENTS THAT MIGHT WARRANT THE APPLICATION OF MORE DETAILED MODULE. SO I'LL STOP THERE BECAUSE IT'S HARD FOR ME TO DO THIS AND BE REMOTE I HOPE I'VE CONTRIBUTED A COUPLE SUGGESTIONS. >> QUESTION: AS THINGS CURRENTLY STAND WHO ARE THE CUSTODIANS OF OF MEDICAID DATA? MEDICARE IS CMS BUT THAT'S NOT THE SPECTRUM WE'RE LOOKING AT. MEDICAID--IS IT--PARDON? >> [INDISCERNIBLE]. >> STILL CMS? STATES ARE NOT CUSTODIANS. >> NO AND FIRST OF ALL, WE THANK YOU FOR THOSE PARTICULARLY FOCUSED COMMENTS, WITH REGARD TO A FEW OF THE POINTS YOU RAISED WE'VE BEEN, I HOPE AT LEAST COGNIZANT OF THE OPPORTUNITIES THERE SO WE HELD A CONFERENCE LAST FALL, ON THE TERMINOLOGY NOMENCLATURE AND WE BROUGHT IN PEOPLE FROM THE NATIONAL LIBRARY OF MEDICINE AND FROM THE DEPARTMENT WHERE WE WERE TRYING TO INTEGRATE THE INCENTIVE PROGRAM WE'RE ALIGNED AND IN DISCUSSION WITH THE NIH, THE COREY PROGRAM LOOKING AT THE LINK AND HEALTH NETWORKS AND WE ESTABLISHED AS ONE OF OUR ACTIVITIES IN THE NCS, A TERMINOLOGY HARMONIZATION WORKING GROUP THAT'S NOW FOCUSING ON THREE AREAS, ADVERSE EVENTS, RHEUMATOLOGY AND PERIINATOLOGY WHERE WE HAVE BROAD INPUT AND REPRESENTATION NOT ONLY FROM THE U.S. BUT FROM AROUND THE WORLD TO TRY TO LINK THESE VARIOUS DATABASES. WE'VE BEEN IN TOUCH WITH CMS, WE'VE ALSO BEEN IN TOUCH WITH TOUCH WITH OTHER DATABASES BECAUSE WE WANT TO SEE IF WE CAN GET THE TERMS HARMONIZED, MAYBE WE CAN GET TAGS WITH LINKAGES, I WANT TO SAY WE HAVE SOLUTIONS BUT WE HAVE INTEREST AND WE PUT RESOURCES INTO THESE VARIOUS ONTOLOGIES AND GLOSSARIES THAT ARE BOTH BEING USED AND SUPPORTED BY HHS AGENCIES AND USED IN DIFFERENT SETTINGS. THE CHALLENGE IS THAT THE FDA ENDORSES ONE SET OF NATIONAL LIBRARY OF MEDICINE ENDORSED ANOTHER SET, CMS ENDORSED ANOTHER SET, CDC ENDORSES ANOTHER SET, DEDEPENDING ON WHICH PARTS OF THE SOURCE THEY'RE BRINGING IT TOGETHER BECAUSE THE GOOD NEWS IS THAT THEY'VE ALL BEEN NEGLECTFUL OF CHILD HEALTH. SO THOSE SPACES--- >> THAT'S THE GOOD NEWS? >> YEAH BECAUSE IT MEANS THE SPACE IS PRETTY WIDE OPEN SO WHATEVER WE CAN CAN GET PEOPLE TO AGREE ON, CAN BE USED BY ALL OF THEM AND IT GIVES AN OPPORTUNITY FOR HARMONIZATION THAT DOESN'T EXIST IN OTHER AREAS, FOR INSTANCE IF WE WERE TRYING TO DO GASTROINTESTINAL DISEASE IN ADULTS, THERE ARE SOME PRETTY FIRM CAMPS IN THERE. WE ALSO TRY TO LOOK FOR GOOD NEWS. AND I THINK THE OTHER PARTS THAT YOU MENTIONED WHICH ARE SORT OF BOTH STRATEGIC AND TECHNICAL APPROACHES, WE HAVE A--AGAIN A WORKING GROUP TRYING TO LOOK AT HEALTH OUTCOMES AND DEFINITIONS OF HEALTH THAT FEEDS INTO OUR TRIEWMENT DEVELOPMENT PROCESS IN A VERY DIRECT WAY AND WE HAVE TRIED TO BRING IN AT LEAST SOME PEOPLE WE'VE BEEN ABLE TO IDENTIFY WHO ARE LEADING THINKERS IN THAT ARENA AND WANT TO LEVERAGE WHAT THEY DO AND IN TURN WE'RE LINKED UP WITH EFFORTS THAT NOT ONLY NIH IS FUNDING BUT HRSA IS, AND WE'LL CONTINUE TO GET IN TOUCH WITH YOU JEFFREY WHILE WE AND WE'RE HAPPY TO SHARE GREATER DETAILS. >> KERRY? >> I WANTED TO ASK TWO THINGS ONE IS GETTING BACK TO YOUR QUESTION ABOUT WHETHER OR NOT DO SPECIFIC TRIGGER QUESTIONNAIRES VERSES RANDOM SPECIFIC QUESTIONNAIRES, I DON'T GET A SENSE THAT WE'VE ANSWERED THAT QUESTION. WHETHER THEY'RE RANDOMLY DONE OR IF THEY'RE TRIGGERS AND WHAT WE AGREED ON AS TO WHAT METHODOLOGY MAY BE BEST USED AND THEN SECONDLY OF THE OTHER NATIONAL STUDIES THAT HAVE OCCURRED, IF THEY'VE USED RANDOM AND VERSES TRIGGERED SPECIFIC QUESTIONNAIRES THAT HAS HAD A BETTER OUTCOME AND HAS BEEN MORE PRODUCTIVE OR MADE A BIGGER IMPACT. THERE MUST BE MORE STUDIES THAT'S WITH THE TRIGGERED MECHANISM VERSES RANDOM. >> I THINK YOU CAN FIND EXAMPLES OF ALMOST ANYTHING, THE TRIGGERING STRATEGY DOES HAVE THIS LITTLE WEAKNESS TO IT AND IF THE TRIGGER IS--HAS A VERY SIGNIFICANT DEHAIRIAL COMPONENT TO IT OZ OPPOSE TO SOMETHING THAT'S JUST RANDOMLY COMES DOWN FROM THE SKY ASK STRIKES THE RESPONDENT, THEN THERE ARE SERIOUS ANALYTIC--ANALYTIC PROBLEMS. >> GOING BACK TO THE CHILDCARE, YOU KNOW JOE TALKED ABOUT WHERE YOU DON'T ASK ABOUT YOU KNOW, WHETHER THE PROVIDER YOU CARE--THE PROVIDER YOU DON'T USE IS LICENSED. BUT ACTUALLY THE WAY PEOPLE WOULD BE MORE LIKELY TO DEAL WITH THAT PROBLEM IS TO--TO LOOK AT THE DISTRIBUTION OF CHILDCARE PROVIDERS IN THE IMMEDIATE VICINITY AND DETERMINE WHAT FRACTION OF THOSE ARE IN FACT LICENSED OR WHAT'S THE DANCE TO THE--DISTANCE TO THE NEAREST LICENSE DAY CARE PROVIDER OR THINGS OF THAT SORT BECAUSE DOING AND HANDLING THE PROBLEM IN THAT WAY DECONTAMINATES THE DATA FROM THIS CHOICE ELEMENT THAT PEOPLE HAVE CHOSEN WHICH--WHICH CHILDCARE PROVIDER THEY'RE GOING TO USE AND IF YOU WANT TO UNDERSTAND THE ROLE OF CHILDCARE, YOU HAVE TO LOOK AT WHAT ARE THE OPPORTUNITIES THEY FACE. SO IN TERMS OF THE RANDOMIZATION, I THINK AN EXAMPLE WOULD BE LOOKING AT DELINQUENCY OR CRIMINAL BEHAVIOR OF INDIVIDUALS WHO HAVE FOR EXAMPLE DROPPED OUT OF HIGH SCHOOL. YOU KNOW? IT'S A PERFECTLY NICE THING TO DO IN TERMS OF UNDERSTANDING DELINQUENT BEHAVIOR, YOU WANT TO LOOK AT THE WHOLE--WHOLE POPULATION AND AT LEAST KNOW, OKAY AMONGST THOSE WHO DROPPED OUT OF HIGH SCHOOL THIS, IS THE DELINQUENCY RATE THAT WE SEE, AMONG THOSE WE DIDN'T, THIS IS THE DELINQUENCY RATE WE SEE AND FROM THAT PARTICULAR TRIGGER, YOU CAN UNDUE. >> RIGHT, SO I UNDERSTAND THOSE SPECIFIC EXAMPLES BUT WHAT HAVE WE DECIDED UPON AND FOR ADVICE. ARE WE SAYING TO DO A MIX TOUR OR ARE WE SAYING TO DO IT TRIGGERED VERSES RANDOM FOR CERTAIN QUESTIONS, WHAT ADVICE ARE WE PROVIDING? >> I THINK THE ISSUE PARTLY IS WE--IS THE TRIGGER. WHAT TRIGGER ARE WE TALKING ABOUT, I THINK MOST STUDIES DO JUST A RANDOM ALLOTMENT OF A SUPPLEMENT, IT'S NOT RELATED TO ANYTHING, SO YOU'RE INTERESTED IN DELINQUENCY AND YOU CAN'T JUST ASK EVERYBODY, OH THIS HALF OF THE SAMPLE IS GETTING THE QUESTION AND THIS SAMPLE IS GETTING WHETHER THEY TAKE AEROBICS. IT HAS NOTHING TO DO WITH ANY CHARACTERISTICS, THAT'S THE WAY IT'S NORMALLY DONE BECAUSE OF ALL THE REASONS THAT RANDY SAID. TRIGGERS LIKE THERE WAS A HURRICANE ARE VERY DIFFERENT AND QUITE RARE. I DON'T THINK WE HAVE TO WORRY ABOUT THOSE. YOU CAN ALWAYS SAY THAT SOMETHING SPECIFIC HAPPEN INDEED THIS AREA AND WE WANT TO DO SOMETHING MORE. YES IT'S A TRIGGER BUT IT'S JUST NOT--IT DOESN'T HAPPEN ALL THAT OFTEN. THE SKIP PATTERN TRIGGER IS EVERYBODY--I DON'T CONSIDER THAT A TRIGGER AND IT'S KIND OF--IF YOU ASK SOMEONE IF THEY TAKE A MEDICATION, MAYBE THIS IS AN EASIER ONE AND THEY SAY NO, I'M NOT GOING TO ASK YOU HOW MANY TIMES YOU TAKE IT, I IT JUST FOLLOWS FROM THE QUESTIONNAIRE, THOSE ARE TRIGGERS BUT THEY'RE VERY DIFFERENT KINDS OF TRIGGERS, SO I'M NOT--I WOULD SAY, YES, DEFINITELY A SKIP PATTERN AND I ASSUMED YOU WOULD, IF YOU ARE GOING TO DO RANDOM ALLOCATION OF MODULE SO YOU GET MORE BREATH IN CONTENT WITHOUT INCREASING BURDEN, THINK HOW THAT MIGHT BE ANALYZE INDEED WHAT YOUR PREVALENCE IS BEING, DO HAVE YOU ENOUGH POWER IN THE END, IF YOU DO MORE THAN ONE, YOU WIND UP WITH THIS QUARTER GOT THIS, THIS 20% THAT IF THERE'S ANY INTERSECTION OF THOSE WHAT POWER DO YOU HAVE, I DON'T REALLY UNDERSTAND THE OTHER TRIGGERS THAT MIGHT PUT YOU INTO A QUESTIONNAIRE AND I WOULD SAY YOU DO NOT WANT TO GIVE AN INTERVIEWER CONTROL OF MAKING THOSE DECISIONS BECAUSE YOU WILL PULLET IN BY IT AND YOU WILL WANT TO HAVE A LOT OF CONTROL CENTRALLY AND YOU WANT TO THINK THROUGH ANYTHING OTHER THAN RANDOM AND SKIP PATTERNS AND SOME OF THESE ODD THINGS LIKE A HUGE--SOMETHING GOES WRONG IN A PARTICULAR AREA, VERY CAREFULLY BEFORE YOU WOULD DO THAT. >> YEAH LET ME GIVE ANOTHER EXAMPLE. IT GOES BACK TO THE STRATEGY YOU'RE LOOKING TO IMPLEMENT IS MODULES THAT ARE GIVEN TO A SUBSET OF THE SAMPLE THEN THERE'S THE QUESTION OF THE CONCEPTUAL FRAMEWORK THAT YOU DECIDE WHEN YOU WANT TO DO THAT. LET ME GIVE YOU AN EXAMPLE AND THAT IS THE DISTINCTION BETWEEN DRIVERS AND OUTCOMES. THAT IF YOU'RE LOOKING AT DRIVERS, THINGS THAT INFLUENCE OUTCOMES OVER A VARIETY OF DOMAINS, THOSE ARE ACTUALLY PRETTY MUCH BAD CANDIDATES TO RANDOMIZE IN AND RANDOMIZE OUT BECAUSE IF YOU REALLY THINK THOSE FACTORS ARE FUNDAMENTAL DRIVERS, THEN BY THOUGHT HAVING THEM IN--NOT HAVING THEM IN YOUR ANALYSIS, YOU HAVE AN OMITTED VARIABLE OF SOMETHING YOU THINK IS A RELEVANT DRIVER FOR HEALTH OUTCOMES. WE ALL KNOW THAT'S A BIG MISTAKE, BUT IF YOU'RE--IF YOU'RE GOING TO RANDOMIZE ON THINGS LIKE OUTCOMES WHERE YOU'RE SAYING, WELL, I'M INTERESTED IN DOING WHAT FRACTION OF PEOPLE GET HEADACHES, GET MIGRAINE HEADACHES, WELL, INSOFAR AS SOMEONE WILL NOT USE THAT AS A DRIVER OF SOME OTHER OUTCOME, THEN FINE, YOU HAVE YOUR--YOUR HEADACHE MODULE BUT AT LEAST THAT GIVES YOU A WAY OF THINKING ABOUT HOW YOU'RE GOING TO DEAL WITH RANDOMIZING THINGS OUT FOR SOME PEOPLE. >> BUT IT'S NEVER SO SIMPLE. IF YOU TAKE THE EXAMPLE OF CHILDCARE AVAILABILITY, AND YOU SAY OKAY, FOR CERTAIN QUESTIONS KNOWING THE DEGREE TO WHICH THERE IS AVAILABLE, OFORDABLE LICENSED QUALITY WHATEVER CHILDCARE IN A COMMUNITY NEARBY NEED IS WHAT YOU WANT TO KNOW AND FOR OTHERS WHAT YOU WANT TO KNOW IS THE CHOICE THAT WAS MADE. THE SAME THING COULD BE ARGUED FOR HEADACHES, THAT FOR CERTAIN THINGS YOU WANT TO KNOW, GHEE DID GROWING UP AROUND A LOT OF BENZEEN CAUSE YOU TO GET HEADACHES LATER. AND ANOTHER THING YOU MIGHT WANT TO FIND OUT IS GHEE, DO KID WHO IS HAVE HEADACHES PERFORM POORLY IN SCHOOL, REGARDLESS OF THE UNDERLYING CAUSE OF THE HEADACHE BECAUSE THAT'S SOMETHING THAT'S AN OPERATIONALLIZABLE QUESTION THAT YOU MAY NOT BE ABLE TO DETECT AT ANY GIVEN CAUSE BUT YOU MAY DETECT AT THAT INTERMEDIATE POINT IF YOU HAVE A MODEL. SO WHILE I COMPLETELY AGREE THAT CONCEPTUALLY THE DRIVERS VERSES OUTCOME DICHOTOMY CAN BE HELPFUL, IN PRACTICE MOST OUTCOMES CAN LATER BE TURNED INTO DRIVERS OF OTHER QUESTIONS AND THAT I THINK WE REALLY GET INTO IS THE CORE DATA THAT LETS YOU CLASSIFY, TAXON MYSELF IS DIFFERENT FROM THE DATA THAT LETS YOU DRILL DOWN AND ONE OF THE DETERMINENERS THAT MIGHT LET YOU FIGURE THAT OUT IS THE SIZE OF THE POPULATION. SO IT MAY TURN OUT THAT OUT OF A HUNDRED THOUSAND PEOPLE WE GET 2000 THAT SUFFER REPEATED HEADACHES. THAT'S A BIG ENOUGH POPULATION TO STUDY A BUNCH OF QUESTIONS ON, BUT IF HAVE YOU A MODEL THAT DRILL INTO'S DETAILS AND YOU TALKS ABOUT, WELL DO YOU GET YOUR HEADACHES MORE IN THE AFTERNOON, BEFORE EATING? AFTER EATING? ALL THESE THINGS BUT BY THE TIME YOU'RE DOWN TO GROUPS OF 50, YOU PROBABLY CAN'T STUDY ANYTHING SIGNIFICANT ABOUT THOSE GROUPS DOWN THE ROAD AND IT MIGHT MAKE SENSE TO SAY WE'RE NOT GOING TO DISTRIBUTE THAT QUESTION ACROSS EVERYONE, WE'LL TARGET THAT TO A GROUP OF PEOPLE THAT LET US TODAY THE HEADACHE POPULATION AND BE DONE WITH IT. THERE'S NOTHING HERE THAT'S AUTOMATIC. THIS IS ALL ABOUT A SET OF IMPORTANT QUESTIONS THAT LET SMART PEOPLE TRY TO DESIGN THIS AND THAT PEOPLE WILL GET WRONG WHICH IS PART OF THE REASON FOR THIS KIND OF ITERATIVE FEEDBACK AND THE REASON EVERYBODY SORT OF EAGER TO SEE THE DETAILS, BECAUSE THE SOONER DETAILS ARE PUT IN FRONT OF US, THE SOONER THEY WILL GET POTENTIAL FOR IMPROVEMENT. >> I WILL NOTE IN RESPONSE TO DR. NADO'S QUESTION, THAT FOR US THE MOST VALUABLE ASPECT OF THE MEETINGS IS THE DISCUSSION AND JUST HEARING THE POINTS AND THE CONCERNS, THAT'S WHAT GIDS US AND THAT'S THE ADVICE WE SEEK. WE'RE NOT NECESSARILY ASKING QUESTIONS THAT HAVE A DICHOT MOUSE OR BINOMIAL RESPONSE, YES OR NO, WE'RE ASKING QUESTIONS THAT WE WANT INPUT ON AND SO GETTING A RANGE OF INPUTS IS FOR US WHAT WE SEEK AND WHAT WE BENEFIT FROM. >> ONE OTHER LITTLE TECHNICAL THING TO KEEP IN MIND THAT YOU HAVE A COMP LITICATED BASE SAMPLE WHICH IS A LOST WAITING ISSUES THAT SOMEONE'S GOING TO HAVE TO DEAL WITH, YOU ARE NOW PUTTING MORE WAITING ISSUES ON TOP OF THAT. SO FROM A DATA USE POINT OF VIEW, OF ALL THESE SUBSAMPLES AND THEY'RE OVERLAPPING, YOU WILL HAVE PROBABLY 25% DATA HERE AND IT'S NOT A REASON NOT TO DO BUT BUT YOU PUT IN A COMPLEXITY. I MEAN HAVE RUN INTO THIS, WE HAVE SOME CROSS SECTIONS OF OUR SAMPLES WE WILL SAY WE CANNOT DO WAITS FOR THAT, FINE, BUT IT'S NOT REPRESENTATIVE OF ANYTHING BECAUSE IT'S TOO COMPLICATED AND YOU ARE STARTING OUT WITH EXAMPLES OF BRINGING IT TOGETHER AND DUPLICATING ITSELF SO I THINK I DO WANT TO WEIGH THE ADVANTAGE YOU'RE GET PROGRESS GETTING ADVANTAGE ON A SUBGROUP OR ADDING SOME MODULE FOR THE SUBGROUP VERSES ALL THE OTHER COMPLEXITIES AND I THINK THE HIGH PROBABILITY THAT EVEN PEOPLE WITH SMART PEOPLE WITH GOOD INTENTIONS, MAKING A DECISION NOW, FOR SOMETHING THAT MIGHT PLAY OUT IN 10 YEARS, WILL GET MOST OF IT WRONG AND YOU'LL DO THE SAMPLING LIKELY TO NOT BE OPTIMAL AT THE END. SO FOR THIS, I WOULD GO KEEP IT SIMPLE EVEN IF IT'S SIMPLER THAN YOU THINK IT SHOULD BE. YEAH, I THINK DATA USE ABILITY, YOU REALLY HAVE TO KEEP THAT IN MIND, YOU KNOW KIND OF COMES OUT OF THE HEADING OF MISTAKES, MANY OF US WHO HAVE MADE--YOU KNOW MADE THAT ONE BEFORE. YOU HAVE REALLY SOPHISTICATED SKIP PATTERNS BECAUSE YOU'RE LOOKING TO EXTRACT INFORMATION AS YOU CAN FROM THE RESPONDENT AND KEEP THE INTERVIEW SHORT AND YOU END UP WITH AN INSTRUMENT THAT PEOPLE SIMPLY DO NOT UNDERSTAND AND THE DAYS OF PAPER AND PENCIL QUESTIONNAIRES, YOU COULD LOOK AT THAT BOOKLET BECAUSE IT HAD TO BE SIMPLE ENOUGH FOR THAT INTERVIEWER TO GO THROUGH AND GO FROM QUESTION TO QUESTION. NOW YOU PROGRAM THAT SUCKER OUT AND THE SKY IS THE LIMIT ON COMPLEXITY, AND YOU GET A DATA SET AND PEOPLE DON'T KNOW WHO GOT ASKED THAT QUESTION? SO EVEN IF YOU COULD WEIGHT IT, BEFORE CAN YOU WEIGHT IT YOU HAVE TO FIGURE OUT WHOAS IN THE UNIVERSE? SO THAT'S--NOTHING IS MORE DEADLY TO THE LONG-TERM VIABILITY OF A PROJECT THAN DATA THAT ARE NOT USEABLE AND PEOPLE WILL NOT STICK UP FOR WHEN YOU HAVE A FINANCIAL PROBLEM. IF YOU HAVE A FINANCIAL PROBLEM YOU NEED TO HAVE USERS WHO WILL STAND UP AND SAY, YES, I USE THESE DATA. THEY'RE EASY TO USE, I CAN GET AT THEM. IT'S ANOTHER PLUG FOR PUBLIC USE DATA. YOU WILL-- >> WE'RE ALL FOR IT. >> YOU WILL NOT REGRET THE DECISION TO PUT AS MUCH OF THIS DATA OUT WITHOUT PEOPLE HAVING TO APPLY FOR IT. TRUST ME ON THAT ONE. BUT THE OTHER--THE OTHER SIDE IS THIS USABILITY QUESTION. I SUBMIT THAT WE MAY HAVE BEAT THE BEZEBBERS OUT OF THIS PARTICULAR TOPIC SO LET'S--IF IT'S ALL RIGHT UNLESS THERE ARE OPINIONS TO THE CONTRARY-- >> I HAVE A QUESTION: WHO SAYS NO TO THE PEOPLE WHO WANT DATA? HOW DOES THAT WORK? >> IT DOESN'T WORK BUT GO AHEAD. THERE'S A LOT OF PRECEDENT WHERE THERE'S DATA ACCESS COMMITTEES SET UP AND PEOPLE MAKE AN APPLICATION FOR THE DATA AND THEY HAVE TO--THIS IS MOST ROBUST IN THE GENOMIC ARENA BUT IT'S ALSO BEING APPLIED FOR OTHER CATEGORIES OF DATA, I ACTUALLY SIT ON ONE OF THESE COMMITTEES AND YOU SAY NO BECAUSE PERSON EITHER DOESN'T HAVE THE FACILITIES TO DO THE KINDS OF ANALYSIS OR THE ANALYSIS ARE GOING TO IN ONE WAY OR ANOTHER NOT ADDRESS THE QUESTION THAT'S BEING POST, SO IT'S ESSENTIALLY A BLEND OF COMPETENCE AND CREDENTIALS THAT DETERMINES WHETHER SOMEONE GETS ACCESS AT WHICH LEVEL OF THE DATA. >> AND WHO SAYS NO TO THE CONTENT OF THE QUESTIONNAIRE? I MEAN WE TALKED ABOUT YOU CAN HAVE TOO MANY QUESTIONS, YOU CAN TRY TO COVER TOO MANY TOPICS AND I THINK PEOPLE WHO ARE EXPERIENCED FEEL THAT THE SIMPLER IT IS, THE MORE ELEGANT IT IS NOT TO TRY TO GO FOR TOO MUCH WHO SAYS NO IN THOSE CASES. HOW IS THAT DECISION MADE? >> SO IN THE NCS, THAT'S GOING TO BE DATA DRIB. WE ARE GOING TO TRY OUT DIFFERENT VARIATIONSTON DIFFERENT SEGMENTS OF THE VANGUARD STUDY AND WE WILL GET SOME SENSE OF COMPREHENSIBILITY, ITEM COMPLETION RATE, APPROPRIATENESS OF RESPONSES, WE HAVE A STRUCTURE WAY WHERE WE WORK WITH THE DATA COLLECTORS AND SAY, YOU KNOW, DID YOU UNDERSTAND THE QUESTION COULD YOU CONVEY THE QUESTION TO THE PARTICIPANTS? DO THE PARTICIPANTS UNDERSTAND IT? WE WILL ASK THE PARTICIPANTS SEPARATELY, YOU KNOW THE LAST 20 YEES YOU JUST 20 THROUGH, THEY DID IN SOME WAY OR ANOTHER OBSTRUCT YOUR ABILITY OR OBLIGATIONS SURE YOUR ABILITY TO RESPOND BECAUSE OF THE WAY THE QUESTIONS WERE, SO WE HAVE THESE DIFFERENT WAYS TO GET DATA ON IT, BUT THAT'S--THAT'S THE GENERAL PRINCIPLE. >> AND IF I CAN JUST ADD TO THAT, YOU KNOW CONCRETELY, IT'S UP TO THE PROGRAM OFFICE AT THE END TO MAKE THOSE DECISIONS, INFORM BOTH CLEARLY BY THE EXPERIENCE FROM THE VANGUARD BUT AT THE ADVICE OF A MORE 30,000-FOOT LEVEL, AND ADVICE FROM YOU ALL AS TO THE KINDS OF HOW TO WAIVE VARIOUS PRIORITIES KIND OF THING BUT IN THE END IT'S UP TO THE PROGRAM OFFICE TO MAKE THOSE DECISIONS. >> JENNIFER? >> I WOULD AGREE WITH THAT AT SOME POINT, YOU GUYS ARE GOING TO HAVE TO DECIDE AND EVEN AFTER GUTMACHER THROUGH ALL THE EVALUATION, YOU WILL HAVE MORE QUESTIONS THAT MEET YOUR CRITERIA THAN HAVE YOU ROOM FOR SO--YOU KNOW AND THEY'RE GOING TO MAKE SOMEBODY REALLY UNHAPPY. THAT'S JUST THE NATURE OF THE-- >> THE VANGUARD IS NO USE FOR IMPROVING THE-- >> WELL YOU GET A THICK SKIN IN THIS BUSINESS BUT IN TERMS OF DATA ACCESS, I THINK MOVING FORWARD IT WILL BE MUCH HARDER TO DENY ACCESS GIVEN THE PRESIDENTIAL AND OMB MEMOS ABOUT MAKING DATA ACCESSIBLE AND THE GREATER USE OF FOIA IN THIS AREA. I MEAN CERTAINLY THERE'S NO--YOU DO NOT HAVE TO EVER MAKE DATA AVAILABLE THAT WILL AFFECT CONFIDENTIALITY OR IDENTIFY ABILITY, BUT IN TERMS OF YOUR REASONABLE RESEARCHER AND YOU HAVE NOT, I THINK IT'S GOING TO BE MUCH, MUCH, HARDER. >> AND LET ME CLARIFY, I THINK YOU'RE RIGHT JENNIFER AND WITHOUT THOSE THERE ARE BASIC UNDERLYING PRINCIPLES FOR KNOWN EXHAUST ABLE RESOURCES THAT IS DATA OPPOSE TO SAMPLES, ENVIRONMENTAL BIOSAMPLES. BUT FOR DATA AS LONG AS CONFIDENTIALITY IS PROTECTED, THE DATA IS AVAILABLE TO ANYONE. AND YOU DON'T HAVE TO QUALIFY, YOU DON'T HAVE TO PROVE, YOU DON'T HAVE TO HAVE PASS EXPECTATIONS, WHATEVER BECAUSE PART OF IT IS PART OF THE REAL VALUE OF THIS IS THAT IT'S NOT SOME INNER CIRCLE HAVING, YOU KNOW SOLE OR EVEN PRIOR ACCESS TO THE DAILY BASIS AT. TELL BE RELEASED TO ANYBODY WHO WANTS TO LOOK AT IT AND LOOK AT IT IN WAYS THAT OTHERS OF US WOULDN'T IMAGINE AND SOME OF THOSE QUALIFIER AND WAYS WILL TURN OUT TO BE PRODUCTIVE SO FOR THE REASONS OF SORT OF FAIRNESS BUT ALSO ADVANCE IN SCIENCE OPTIMALLY, AS LONG AS THE CONTIARAS DENTIALITY CAN BE GUARANTEED AND OMB AGREES AND ALL THOSE OTHER THINGS, ET CETERA, TELL BE MADE PUBLICLY AVAILABLE. >> RIGHT AND WE DON'T WANT TO BE EXPLAINING OUR RESOURCES ON REVIEWING REQUESTS FOR DATA BUT ON THE OTHER HAND WE WANT TO ASSURE THERE ARE PEOPLE PROTECTIONS IN PLACE AND SO THAT'S THE BALANCE AND THAT'S WHY OUR PLAN IS TO PARTNER WITH DIFFERENT PEOPLE WHO ARE DOING THIS RIGHT NOW AND DO SOME PILOTS AND KIND OF SEE HOW THAT WORKS OUT, SO CMS HAS PILOTS, NCHS--CMS HAS RESOURCES, NCHS HAS RESOURCES, SAMSA, HAS RESOURCES, NATIONAL LIBRARY OF MEDICINE HAS RESOURCES AND WE ARE IN DISCUSSION WITH EVERYBODY, WHERE ARE WE GOING TO PARK OUR DATA IN THE SHORT-TERM AND THEN WE'LL SEE WHAT WE--SEEMS TO WORK FOR US. >> IN THE AGENDA FOR THE FUTURE, I DON'T THINK WE HAVE TO SOLVE THIS TODAY, THE CONFIDENTIALITY WILL NEVER BE GUARANTEED. I MEAN THE FUNDAMENTALLY THIS, IS WHAT I SPEND TIME TEACHING ABOUT, FUNDAMENTALLY EVEN IF THERE IS NOTHING WE CAN CONCEIVABLY DO FROM THE IDENTIFY OF THE RECORDS YOU RELEASE, SOMEBODY WHO HAS A BUNCH OF IDENTIFIEDED DATA WILL BE ABLE TO--IDENTIFIED DATAL BE ABLE TO JOIN THAT ABOUT SOMETHING THEY KNOW THE IDENTITY OF AND IF YOU--IF YOU RELEASE THINGS THAT ARE INDIVIDUAL LEVEL BUT EVEN IF YOU RELEASE CERTAIN THINGS THERE'S STATISTICAL AGGREGATE, I'M NOT SAYING THAT'S A BAD THING OR YOU SHOULDN'T RELEASE THE DAT THE GOVERNMENT IS MIXING UP CONFIDENTIALITY AND OPEN RELEASE OF SUBJECTS AND OPEN RELEASE OF DATA. >> WELL PEOPLE FROM BOOZE HAMILTON PLEASE STEP FORWARD. [LAUGHTER] >> YEAH, I THINK THAT'S SOMETHING WE SHOULD PUT ON THE AGENDA SOONER RATHER THAN LATER TO FIGURE OUT WHAT'S THAT STANDARD OF WE DIDN'T DO TOO MUCH HARM THAT THE COMMUNITY CAN LIVE WITH, RECOGNIZING THERE ARE A BUNCH OF PEOPLE WHOSE PERSONALLY ARKS DENTIFIED DATA WAS REIDENTIFIED WITH THEM THROUGH PEOPLE WHO HAD VARIOUS PARTS OF THAT DATA BUT NOT ALL OF IT. >> YEAH, I WOULD LIKE TO MOVE O IF I COULD TO THE DESIGN PROCESS BECAUSE WE'RE GOING TO BREAK FOR EXPLUNCH WE'RE GOING TO COME BACK AND GOING TO TALK ABOUT OUR RETENTION IS ONE OF OUR ITEMS OF BUSINESS. I GUESS THE FIRST CAUTIONARY POINT IS, THE HOPE THAT THE CHOICE OF WHAT APPEARS IN THE QUESTIONNAIRE IS DATA DRIVEN? I THINK THAT OPENS YOU UP TO OWN FORTUNATE--UNFORTUNATE THINGS HAPPENING IF FOR NO OTHER REASON THAT THE LATENCY BETWEEN THE PROCESS OF GETTING QUESTIONS INTO THE VANGUARD SITES GETTING THAT ALL CLEARED AND VETTED, GETTING THE DATA OUT, ANALYZE IT, COMING TO A CONCLUSION AND THEN AT THAT POINT SAYING, OKAY, I THINK IT'S READY FOR PRIME TIME, I THINK IT'S GOING TO JUST BE A WE BIT OF TOO LATE. SO, AT SOME POINT IT'S GOING TO COME DOWN TO TO THE LEADERSHIP OF THE PROJECT MAKING AN INFORMFORM ININFORMED DECISION AND GOD GOD HELP YOU JUST GET GOOD, GOOD, HELP. >> SO I WANT TO EMPHASIZE SOMETHING THAT DR. GUTTMACHER NOTED, I'LL TELL YOU WHAT WE DON'T DO AND THEN I'LL TELL WHAT YOU WE DO DO. WE DON'T DON'T WALK IN AND SEE WHAT'S ON THE TABLE AND SEE WHAT BUBBLES UP WITH OUR FIELD EXPERIENCE. WE WALK IN WITH SPECIFIC APPROACHES AND DOMAINS AND AREAS THATUSSING ON AND HOW WELL DO THESE WORK AND HOW DO WE NEED TO IMPROVE IT SO THAT'S I THINK THE BASELINE THAT WE WANT TO EMPHASIZE. >> SO ON THE DESIGN PROCESS, I THINK WHAT WOULD BE VERY HELPFUL AND WE'VE HAD KIND OF THESE BROAD BRUSH DESCRIPTIONS IS TO START TOO FLESH OUT MAYBE NOT AT THE ITEM IN THE QUESTION LEVEL, OKAY, BUT TO START ASKING WHAT ARE THE KEY DOMAINS THAT WE THINK OF AS BEING CORE AND AT WHAT AGES ARE THEY CORE, WHAT REGULARITY ARE THEY CORE BECAUSE SOME ITEMS, YES, YOU WANT TO COLLECT THEM FIRST, YOU MIGHT SAY, GREAT, I'LL--I'LL COLLECT IT A COUPLE OF TIMES AND ANYTHING BEYOND THAT, YOU KNOW I'M NOT SHOW SURE I'M WILLING TO MAKE THAT--THAT COMMITMENT OF TIME AND EFFORT. BUT ALSO, I THINK WE WANT TO--WELL THE PROBLEM IS THAT YOU HAVE TO GET THE NEXT FEW YEARS OF INSTRUMENTATION READY IN ORDER TO REALLY UNDERSTAND WHETHER THAT PLAN IS THE RIGHT PLAN TO BE USING YOU KIND OF HAVE TO KNOW WHAT THE STRATEGY IS IN THE OUTYEARS AND WHILE YOU DON'T NEED TO KNOW WHAT ALL THE ITEMS ARE, YOU DO NEED TO KNOW FOR EXAMPLE, AS CAME UP TODAY, WELL, THERE'S SO MUCH TIME WE WANT TO PUT IN WITH THE MOTHER AND FINDING OUTS WHAT'S GOING ON FOR THE KID AND HOW SHE'S INTERACTING WITH THIS CHILD AND THINGS THAT THE CHILD IS DOING OR NOT DOING THAT WE SIMPLY CAN'T SPARE THE TIME WHEN THE CHILD IS YOUNG TO DO THAT BUT WHEN THE CHILD IS EIGHT, NINE, OR 10 YEARS OLD, WELL MAYBE THEN THAT'S THE TIME TO START TO GO AFTER THE MATERNAL ABILITIES AND OTHER COGNITIVE ISSUES AND PSYCHOMETRICS, WHEN THOSE THINGS ARE LESS OF AN ISSUE O SO I THINK PROBABLY WE'LL BREAK FOR LUNCH BUT I THINK THAT'S WHERE I THINK WE WANT TO TART GOING IS TO START TO LOOK AT THE OVERALL ARCHITECTURE OF WHAT THIS DATA COLLECTION PROJECT IS GOING TO LOOK LIKE AND IT'S GOING TO BE HORRIBLY PREMATURE AND YOU KNOW, PROBABLY ONE WAY OR ANOTHER PEOPLE WILL MAKE SOME HORRIFYING MISTAKES, BUT AS MARTIN LUTHER SAID, SIN BOLDLY, BUT LET'S SET OUT A DESIGN WHERE THIS IS GOING AND THEN START TO SAY, ALL RIGHT, GIVEN THAT'S WHAT THE PATTERN WAS LOOKING LIKE FOR THE NEXT 21 YEARS WHERE DO WE START KNOWING THAT LATER DOWN THE ROAD, WE'RE GOING TO BE CAPTURING THIS DOMAIN WHEN IT'S--WHEN IT MAKES MORE SENSE. >> OKAY. >> SO ANY OTHER ADDITIONS. >> YEAH, JUST TELL US WHEN YOU WANT US BACK HERE. >> 45 MINUTES FROM NOW. I WANT EVERYBODY BACK. >> I WILL TAKE A FEW MINUTES TO REFLECT ON WHERE I THINK WE ARE. WHAT THIS ADVISORY COMMITTEE IS DOING AND WHAT WE MIGHT DO TO ENHANCE THE LIKELIHOOD THAT THIS WILL BE A SUCCESSFUL ENDEAVOR. SOME OF THIS IS JUST GOING TO BE KIND OF MY -- WE WILL NOT DARE SPEAK FOR ALL MEMBERS OF THE ADVISORY COMMITTEE INDIVIDUALLY, CERTAINLY ENCOURAGE THEM TO CHIME IN AND SPEAK WHAT'S ON THEIR MIND. BURR BEFORE I DO SO, BUT BEFORE I DO SO, TINA, INTRODUCE YOURSELF AND TELL US WHO YOU ARE AND WHY YOU'RE HERE. >> (INAUDIBLE). >> NOBODY EVER ASKS ME TO SPEAK LOUD. I'M TINA (INAUDIBLE) HERE REPRESENTING THE U.S. ENVIRONMENTAL PROTECTION AGENCY, I'M ON THE FEDERAL CONSORTIUM COMMITTEE AND LOW YEAHSON AS EXOFFICIO MEMBER OF THE ADVISORY COMMITTEE. >> (OFF MIC) >> SHE WANTED ME TO PUT THIS ON TAPE. THIS COULD BE A BIG MISTAKE. I BELIEVE WE'RE AT THE POINT WHERE WE REALLY NEED TO ARTICULATE JUST EXACTLY MAYBE NOT EXACTLY BUT MORE OR LESS WHAT THE NATIONAL CHILD STUDY IS GOING TO BE IN THE SENSE THAT AS WE CLOSE THE MORNING SESSION, WE NEED TO HAVE A ROADMAP THAT EXPLAINS WHAT THE KEY DOMAINS OF DATA COLLECTION ARE. WHAT'S CORE. WHAT'S DISCRETIONARY AND MIGHT BE RANDOMLY ASSIGNED. WHAT SORTS OF THINGS NEED TO BE ASKED AT PARTICULAR BENCHMARK POINTS OF DEVELOPMENT. WHAT THE CRUCIAL PHYSIOLOGICAL AND CLINICAL MEASURES ARE THAT NEED -- THAT STUDY IS GOING TO COLLECT AND AT WHAT FREQUENCY. AND THAT THIS LIST OF THINGS NEEDS TO BE NOT A LIST OF POTENTIAL ITEMS WHICH THE UNION OF WHICH WOULD COMPLETELY EXHAUST NOT ONLY THIS BUDGET CONSTRAINT BUT ANY IMAGINABLE BUDGET CONSTRAINT, BUT IT HAS TO BE A LIST OF ITEMS THAT ARE AT LEAST APPROXIMATELY FINANCIALLY FEASIBLE. SO THIS IS ALL ABOUT MAKING CHOICES AND THE PEOPLE HERE ON EITHER SIDE OF ME ARE ONES WHO NEED TO MAKE THOSE DECISIONS. SO LET'S GET STARTEDDED. SO I THINK WHAT THIS COMMITTEE CAN DO THAT WOULD BE CONSTRUCTIVE, IS TO START A PROCESS BY WHICH THERE ARE PERHAPS SMALLER WORKING GROUPS MADE UP OF STUDY STAFF AND MAYBE SELECTED MEMBERS OF THE ADVISORY COMMITTEE, REGULAR AND EXOFFICIO, THAT REVIEW OVER THE PHONE OR WITH IRREGULAR MEETINGS, THE CONTENT IN KEY DOMAINS GOING FORWARD WHAT THIS STUDY IS LOOKING LIKE. IOM IS DO GOING TO DO THEIR REVIEW. JOE IS GOING TO BE PUTTING TOGETHER HIS RENDITION OF WHAT HAS TRANSPIRED HERE AND WHAT -- THE OPINIONS AND THE GUIDANCE THAT THE ADVISORY COMMITTEE HAS. WE NEED CONSIDER FORWARDING THAT TO IOM SO THEY UNDERSTAND KIND OF WHAT'S -- WHAT OUR THINKING IS SO THAT -- I MEAN THIS SEEMS TO BE ASTUTE WITH LOTS OF CROOKS BUT IT WOULD BE HELPFUL IF THE VARIOUS CROOKS WOULD COOPERATE WITH THE SAME SPOON, METAPHOR HAS GONE BAD BUT ANYWAY, WE WANT TO COMMUNICATE WHERE WE ARE TO THE IOM TO AT LEAST GET THEM SO THEY KNOW WHAT WE'RE THINKING. BUT THE MORE IMPORTANT ISSUE IS WHERE THE STUDY IS GOING, HOW WE CAN HELP, NOT ONLY TERMS OF INSTRUMENTATION BUT ALSO RETENTION, THE DATA DECEMBER SIMILAR NATION, THERE ARE A LOT OF HUMAN SUBJECTS, THERE ARE A LOT OF HUMAN ISSUES THAT WE HAVE SOX EXPERTISE -- SOME EXPERTISE ON THE ADVISORY COMMITTEE THAT MIGHT BE USEFUL AND PRESUMABLY WE'RE HERE FOR A REASON, TO PROVIDE ADVICE, CONSTRUCTIVE ADVICE, THAT HELPS THE STUDY PRODUCE A FIRST RATE PRODUCT. SO I'LL STOP THERE AND AT THIS POINT SOLICIT VIEWS FROM OTHER MEMBERS OF THE ADVISORY COMMITTEE IN TERMS OF HOW THEY SEE THIS PROCESS AND WHAT THEY WOULD LIKE TO BUY INTO IN TERMS OF PARTICIPATION. >> ONE PROCESS QUESTION BEFORE WE COTHAT, I WANT TO CHECK WITH THE OFFICIAL FEDERAL OFFICIALSES FOR THE COMMITTEE SINCE THIS IS A FACA COMMITTEE, THERE ARE RULES ON THE SUGGESTION TO HAVE WORKING GROUPED JOINED BY STAFF, ET CETERA. THAT MIGHT -- I JUST WANT THE CHECK AND SEE IF IT'S POSSIBLE TO CONSTRUCT SUCH A THING IF IT DOES NOT RUN AFOUL OF FEDERAL FACA RULES. >> AND THE TERM WORKING GROUP WAS CERTAINLY NOT MEANT TO BE TERM OF ART. NOT SO MUCH THE WORKING GROUP NAME SAY IS THERE ARE VERY SPECIFIC -- THE ONLY WAY TO GIVE ADVICE TO FEDERAL OFFICIALS AS PART OF THIS IS TO HAVE OPEN MEETINGS, ET CETERA, ET CETERA. THE WORKING GROUPS SUPPLYING ADVICE TO PROGRAM STAFF I THINK WE HAVE TO BE QUITE CAREFUL HOW THOSE NEEDS ARE CONSTRUCTED. >> INDEED. I WOULD SEE THAT PARTICIPATION AS DISCOVERY, INFORMATION ACQUISITION BY THE ADVISORY COMMITTEE, THE SCOPE OF THE STUDY IS SO, SO LARGE, NO ONE CAN DO IT ALL. SO WE NEED MULTIPLE CONDUITS FROM THE STUDY BACK TO THE ADVISORY COMMITTEE. I THINK YOUR POINT IS ABSOLUTELY WELL TAKEN, WE ARE NOT IN THE BUSINESS OF DESIGNING THE STUDY, WE WANT TO PROVIDE ADVICE AND GUIDANCE BUT THE PROVISION OF ADVICE AND GUIDANCE IS PREDICATED ON GOOD INFORMATION SO THAT WE KNOW WHAT WE'RE GIVING A VICE AND GUIDANCE ON. LET'S HEAR MORE FROM THE GROUP THE DEVIL IS IN THE DETAILS. >> SERGEANT OF ARMS. >> SUGGESTING IN THE FORM OF WORKING GROUPS THERE ARE TWO THINGS, WORKING GROUPS AN SUBCOMITTEES. SUBCOMITTEES I THINK ARE WHAT DR. GUTTMACHER WAS REFERRING TO WHICH REQUIRE OPEN MEETINGS, MEETINGS NOTED IN THE FEDERAL REGISTER, SUBCOMITTEES MEANT TO BE ONGOING LONG TERM GROUPS. WORKING GROUPS HAVE A MORE FINITE TASK, THEY'RE SHORTER TERM AND THEY DON'T HAVE TO HAVE OPEN MEETINGS. THEY CAN COMMUNICATE WITHOUT PLANNING WEEKS IN ADVANCE AND PUTTING IT IN THE FEDERAL REGISTER. SO I THINK IF WHAT YOU'RE TALKING ABOUT SHORT TERM TASKS, TO GROUPS OF PEOPLE, WORKING GROUPS WOULD LIKELY BE THE WAY YOU WANT TO GO. >> PARTICULARLY, RANDY, YOUR POINT THIS WOULD BE NOT TO SUPPLY IN WORKING GROUPS ADVICE TO FOLKS BUT TO INFORM MEMBERS OF THE COMMITTEE ON THE WORKING GROUP SO THEY'RE MORE INFORMED WHEN THE OPPORTUNITY TO GIVE ADVICE DID OCCUR, IF I HEAR YOU CORRECTLY. >> IF I MAY, I'LL AMPLIFY THAT POINT. WE HAVE DONE THAT BEFORE IN THIS COMMITTEE AND THE CHARGE SAID THE DISCUSSIONS ARE FOR PLANNING AND INFORMATION GATHERING. AND THEN THE PRODUCTS OF THOSE WORKING GROUPS COME BACK TO THE LARGER COMMITTEE FOR DISCUSSION AND THEN POTENTIAL ADVICE. SO WE HAVE PRESS -- PRECEDENTS, WE HAVE AN ADMINISTRATIVE MECHANISM AND I'LL EXPRESS MY INTEREST AND I DON'T KNOW WHAT THE PROPER TERM IS, I DON'T KNOW TO SAY ENDORSEMENT, ACCEPTANCE OF THAT PROPOSAL. >> I HAVE TO ASK A RELATED QUESTION, SINCE I MISSED PART OF THE MORNING I MAYBE COMPLETELY OFF BASE HERE. BUT I UNDERSTAND, STEVEN, THERE IS ANOTHER IOM REVIEW THAT'S BEING LAUNCHED AND THE PUBLIC MEETING OF THAT IS IMMINENT. WHAT IS THE PURVIEW OF THAT REVIEW? I LOOKED AT IT ON THE WEBSITE BUT I WANT TO UNDERSTAND RELATIVE TO THIS COMMITTEE WHAT IS THAT IOM REVIEW GOING TO BE LOOKING AT? >> I WILL DEFER TO DR. GUTTMACHER WHO DISCUSSED AND PRESENTED THAT VERY TOPIC EARLIER TODAY. >> WE DID TALK ABOUT -- (OFF MIC) >> WE MAY NOT HAVE TALKED SPECIFICALLY ABOUT THE RELATIONSHIP THIS COMMITTEE, IT'S INDEPENDENT OF THIS COMMITTEE AND DOES NOT HAVE OVERLAPPING MEMBERSHIP OR WHATEVER. IT IS BEING ASKED TO LOOK AT ONLY ONE ASPECT, A LARGE ASPECT BUT ONE ASPECT OF NATIONAL CHILDREN'S STUDY WHICH IS THE FORMULATION FOR THE MAIN STUDY WHERE THIS ADVISORY COMMITTEE OBVIOUSLY IS A MUCH LONGER WHILE MEMBERS ROTATE OBVIOUSLY THIS ADVISORY COMMITTEE EXISTS FOR ENTIRE LIFE OF THE STUDY IS ASKED TO GIVE ADVICE THROUGHOUT THAT LIFE ALL ASPECTS INCLUDING ADVICE OVER THE PAST YEARS ABOUT THE VANGUARD STUDY, ET CETERA. SO THIS IS SORT OF A DIFFERENT FOCUS. BUT MANY OF THE QUESTIONS CONSIDERED OVERLAP CERTAINLY. (OFF MIC) >> IN TERMS OF THE QUESTIONS BEING ASKED THERE ARE A SUBSET OF THE QUESTION THAT OVER TIME THIS GROUP HAS CONSIDERED. (OFF MIC) >> PACKET IS BEING DEVELOPED AS WE SPEAK. AT LEAST BY TONIGHT. IT IS BEING DEVELOPED. IT WILL -- YES, ABSOLUTELY BE AVAILABLE TO MEMBERS OF THE GROUP. IN SOME WAYS WILL RESEMBLE DOCUMENTS THAT YOU HAVE ALREADY SEEN. >> IF YOU WANT TO GET A REALLY GOOD PREVIEW, USE THE BRIEFING DOCUMENT FOR THIS MEETING. >> THAT'S WHAT I WAS GOING TO ASK THE BRIEFING DOCUMENT (INAUDIBLE). >> YES. WHAT THE IOM GROUP GETS WILL SIMPLY BE SOMEWHAT AN ELABORATION, SEE THIS IS A DRAFT OF WHAT THE IOM GROUP WILL GET. ARE THERE ANY OTHER COMMENTS, JOE, PLEASE. >> AS SOMEBODY WHO COMES INTO THIS AS NOT THE SORT OF PERSON WHO RUNS STUDIES WITH THOUSANDS OF PEOPLE, ONE OF THE THINGS THAT I WOULD FIND VERY USEFUL AS WITH WE GO THROUGH THIS PROCESS IS CASE EXAMPLES. I REFERRED TO THIS BEFORE AS POTENTIAL HYPOTHESES, THAT MIGHT BE TOO NARROW A PHRASE BUT THE IDEA TAKING BOUNDING RESEARCH QUESTIONS THAT ONE OF THEM INVOLVES SOME PHYSICAL ENVIRONMENTAL FACTOR AND WHETHER IT LEADS TO SOME PARTICULAR DISEASE CONDITION. ONE MIGHT INVOLVE A DEVELOPMENTAL FACTOR RELATED TO SOMETHING, THAT'S BEHAVIORAL, SOME MIGHT BE MORE COMPLICATED THAN THAT, TO GIVE A CONCRETE EXAMPLE, SOMEBODY WANTED TO STUDY THE QUESTION OF WHETHER CHILDREN WHO HAVE OUT OF HOME DAY CARE END UP WITH BETTER PHYSICAL INDICATORS AT AGE 7 THAN CHILDREN WHO ARE IN THE HOME. I WOULD LIKE TO SEE A FEW OF THOSE TRACED THROUGH. WHAT WOULD THE DATA BE THAT COULD BE DELIVERED TO A RESEARCHER ASKING THAT QUESTION AGAINST THIS BASED ON THE CURRENT PLANS FOR THE QUESTIONS BEING ASKED. WHAT WOULD BE THE SAMPLING FRAME AGAINST WHICH THOSE DATA WOULD BE DRAWN? SOMETHING THAT WOULD BE EVENTUALLY ONE MIGHT IMAGINE IN A HANDBOOK FOR RESEARCHERS WHO WERE EXPLOITING THE BENEFITS OF THE STUDY, BUT REDUCE IT FROM A VERY ABSTRACT DESIGN LEVEL TO SOMEBODY YOU COULD CONCRETELY REVIEW TO LOOK AT HEY CAN I CONDUCT A PIECE OF RESEARCH AND FEEL CONFIDENT IN THE RESULTS. I DON'T KNOW THE TIME FRAME WHICH THOSE COULD BE THERE. THESE COULD BE QUESTIONS ALREADY USED INTERNALLY TO HELP EVALUATE THE RESEARCH AS IT'S GOING BUT AS WE THINK ABOUT WAYS OF DOING THIS, I REALIZE SOME PEOPLE GET IF YOU HAND THEM WE'RE GOING TO USE 12 SCALES. I'M NOT THE KIND OF PERSON WHO WILL COME BACK AND BE ABLE TO GIVE YOU A LOT OF FEEDBACK ON YOU USE THOSE SCALES UNHAPPY BUT IF YOU COULD SHOW ME A COUPLE OF EXAMPLES WORKED THROUGH HERE IS WHETHER YOU WILL GET. MAKES IT MUCH CLEARER TO WHAT YOU WILL ACTUALLY DELIVER. >> SO I'LL QUICKLY RESPOND. WE HAVE DONE THAT EXERCISE MANY TIMES INTERNALLY LOOKING AT ALL KINDS OF HYPOTHESES, DIFFERENT THRESHOLDS OF OUTCOMES DIFFERENT PREVALENCE OF EXPOSURES. FORMULATING THE FRAMEWORK FOR THIS STUDY. WE HAVEN'T PUT THOSE OUT PUBLICLY BECAUSE THEY ARE FAIRLY TECHNICAL THERE NEEDED A LOT OF CONTEXT AND WE DIDN'T WANT TO HAVE SOMEBODY SAY THAT'S WHAT THE NCS IS GOING TO DO. WE USE THESE AS THOUGHT EMPERIMENTS FOR MODELING. WE CAN CERTAINLY LOOK AT THAT GOING FORWARD AS A WAY TO EMPHASIZE CERTAIN PRINCIPLES. AN EXAMPLE IS OFTEN THE BEST WAY TO LEARN THE PRINCIPLE. (OFF MIC) >> I'M NOT SURE I WOULD EXCLUDE POWER IN THAT. I THINK FOR SOME PEOPLE GOING THROUGH THE EXERCISE THEY WOULD WANT TO UNDERSTAND IS, WILL THIS BE ABLE TO TELL ME WHAT I WANT TO KNOW EMPOWERS A FUNDAMENTAL PART OF THAT QUESTION, KNOWING THAT THE VARIABLES ARE THERE, NOT KNOWING IF YOU HAVE THE POWER TO DRAW A CONCLUSION WOULD BE THE CHALLENGE. >> THINGS LIKE POWER WHEN YOU'RE DOING A REGRESSION, THERE'S A DIFFERENT CRITTER BECAUSE IT DEPENDS ON EXACTLY WHAT THE VARIANTS, CO-VARIANTS MATRIX LOOKS LIKE. THE TRADITIONAL POWER VARIABLE THING IS KIND OF DUCK SOUP. ARE THERE OTHER POINTS OF VIEW WHO ARE -- OR OPINIONS OR ENDORSEMENTS FROM ADVISORY COMMITTEE? JEFF. >> I DIDN'T HAVE ANY ADDITIONAL COMMENTS. >> ALL RIGHT. IN THAT CASE, WE'LL -- WE WILL WE HAVE HANDLED THAT PARTICULAR ISSUE, NOW LET'S RETURN TO THE AGENDA WITH A DISCUSSION RETENTION NATIONAL CHILDREN'S STUDY WITH THE DISCUSSION CHAMPION BEING ALMA KUBY. SO LET'S START THAT. IT LOOKS LIKE DR. HUDAK, YOU'RE FIRST ON THE LIST. IF WE COULD, WE ARE GOING TO STEP OUT BEFORE WE GOT INTO OUR AFTERNOON SESSION. >> WE'RE GOING TO TAKE PICTURES. >> CORRECT. SO BEFORE WE -- SO WE DON'T INTERRUPT THE PRESENTATION PROCESS, WHAT WE ARE TRYING TO DO IS START TO DOCUMENT THESE MEETINGS A LITTLE BIT BETTER WITH SOME ATTENDANCE OF THE INDIVIDUALS AND NOTHING TELLS A STORY LIKE PICTURE OR PROCESS SO LIKE TO TAKE FIVE QUICK MINUTES IF WE CAN WALK OUTSIDE RIGHT OUT HERE TO RIGHT AND GATHER REAL QUICK AND WHEN YOU COME BACK THERE WILL BE A CONSENT FORM FOR YOU TO THE FILL OUT IN THAT PROCESS. IT I ALLOWS US TO PUBLISH A PICTURE IN ANY JOURNAL -- NO. IT'S A MANDATORY PROCESS FOR THE FEDERAL GOVERNMENT THAT AS WE DO THAT BUT WE LIKE TO DOCUMENT THE ADVISORY COMMITTEES PRESENCE AT THE MEETING AND DOCUMENT THE MEETING AS WE PUT OUT THE NOTES IN THAT PROCESS SO REAL QUICKLY RIGHT OUT HERE TO THE RIGHT. BRADLEY WILL KIND OF COORDINATE THIS ISSUE FOR US. THANK YOU. >> ALL RIGHT. THANK YOU FOR THE BRIEF RESPITE. SO AS WE MENTIONED, TALKED TO US THE SECOND HALF OF OUR DAY WHICH IS GOING TO BE FOCUSEDDED ON RETENTION. RETENTION OBVIOUSLY IS SOMETHING WE'RE HIGHLY FOCUSED ON WITHIN THE SAPP GUARD AS WE ENTER THAT VASE -- VANGUARD AS WE ENTER THAT PHASE AND TRANSITION. IT IS CRITICALLY IMPORTANT AS IT IS A SOURCE OF THE DATA THAT WE WILL BE USING. SO AS WE LOOK AT RETENTION, THE (INAUDIBLE) YOU'LL HEAR ABOUT IS A CRITICAL L PIECE OF METHODOLOGY TO INCORPORATE BEST PRACTICES THAT ARE OUT THEREl DEVELOP OUR OWN BEST PRACTICES AND IDEAS AND PROCESSES FROM VARIANTS BACKGROUNDS AND AREAS. AND FOCUS ON WHAT WE BELIEVE CAN BE THETIS FILE WE APPLY THIS TYPE OF -- THE FIRST TIME WE APPLY THIS TYPE OF METHODOLOGY TO RESEARCH TO THE QUALITY ASSURANCE LOOK IN THIS PROCESS SO — YOU'LL HEAR MORE ABOUT THAT IN THE PRESENTATION. BUT WHAT I WOULD LIKE TO DO IS ALLOW OUR COMMITTEE MEMBER ALMA KUBY TO LEAD THIS AND CHAMPION THIS DISCUSSION AND LET HER VARIED EXPERIENCE SPEAK TO US A LITTLE BIT ABOUT THE WISDOM AND PROCESSES THAT WE WILL ALSO UTILIZE IN OUR LOOK AT RETENTION FOR THE NATIONAL CHILDREN'S STUDY. >> WELL, RETENTION IS ONE WAY OF GUARDING AGAINST ATTRITION AND WE CAN THINK ABOUT THREE WAYS THAT WE HAVE ATTRITION, ONE BECAUSE THE PARTICIPANT NO LONGER IS REPRESENTATIVE OF THE POPULATION TWO, THERE ARE REFUSALS AN THREE, FAILURE TO TRACK AND LOCATE THE STUDY PARTICIPANTS. SO THESE ATTRIBUTES ARE IMPORTANT TO THINK ABOUT AND WE HAVEN'T DONE A LOT OF TALKING ABOUT WHO WOULD STAY IN THE STUDY. WILL WE FOLLOW PARTICIPANTS AND HOW FAR WILL WE FOLLOW THEM. THAT IS AN IMPORTANT ISSUE IN TERMS OF PROTOCOLS AND ALSO IN TERMS OF COST. AND IT RELATES TO THE WHOLE ISSUE OF WHAT WOULD BE THE RETENTION RATE. REFUSALS IN THE FIELD. WE TALK ABOUT DIFFERENT KINDS OF REFUSALS. THERE'S HARD REFUSALS AND SOFT REFUSE REFUSALS. HERE WE GET INTO THE ART OF DATA COLLECTION. WE NEED FLEXIBILITY AND WE NEED TO HAVE WELL-TRAINED INTERVIEWERS WHO ARE ALSO PERHAPS GIVEN MORE LEEWAY THAN WE WOULD LIKE TO GIVE IN OTHER ASPECTS OF THE STUDY SO THEY CAN CONVINCE PEOPLE TO THE STAY IN THE STUDY. THE THIRD PART THE FAILURE TO TRACK OR LOCATE STUDY PARTICIPANTS IS CRITICAL IN TERMS O IT CALLS FOR GOOD DATA SYSTEMS AND GOOD NOTES ON THE PART OF THE INTERVIEWERS PICKING UP THINGS THAT MIGHT BE RELEVANT IN THE FUTURE WHEN WE TRY TO LOCATE SOMEBODY. ALSO THE ISSUE OF FOLLOWING PARTICIPANTS RAISES ISSUES OF CONSENT. IF A CHILD IS PUT IN A FOSTER CARE SYSTEM, IF THEY'RE PUT -- IF IN JAIL, WHO WOULD BE THE PERSON WHO CONSENT? IT PERHAPS MAYBE THESE PROTOCOLS ARE GOING TO BE DEVELOPED IN THE FUTURE BUT THEY DO DESERVE SOME ATTENTION, IMPORTANT ATTENTION I THINK. THE OTHER ISSUE REGARDING LESSENING ATTRITION IS USE OF INCENTIVES AND MANY PEOPLE IN THE FIELD FOUND THAT USING DIFFERENCIAL INCENTIVES REALLY WORKS BEST, BOTH IN TERMS OF BUDGET FOR THE PROJECT AND ALSO FOR PUTTING MONEY WHERE YOU NEED IT, THERE MAY NOT BE A REASON TO GIVE AN INCENTIVE TO SOMEBODY, A COOPERATIVE SUBJECT. ON THE OTHER HAND YOU MIGHT USE YOUR MONEY FOR THOSE PEOPLE WHO ARE LESS INTERESTED IN BECOMING A MEMBER OF THE STUDY, RETAINING THEIR MEMBERSHIP IN THE STUDY. I DON'T KNOW WHETHER THIS WILL GIVE OMB -- US PROBLEMS WITH OMB, IF THERE ARE DIFFERENCIAL INCENTIVES BUT ONE CAN IMAGINE THAT AT SOME POINT WE SUGGEST THE DEVELOPMENT OF EXPERIMENTS SO THAT WE LOOK AT THE ISSUE OF INCENTIVE TO SEE WHAT MIGHT WORK BEST FOR US. AGAIN, A LOT OF THIS -- THESE ISSUES REGARDING LESSENING ATTRITION ARE SORT OF AN ART RATHER THAN A SCIENCE. SO WE HAVE TO TRY TO FOLLOW SOME OF OUR INSTINCTS AND THE INSTINCTS OF PEOPLE WHO ARE CLOSE TO THE SUBJECTS IN THE FIELD. THE OTHER ISSUE THAT HAS BEEN RAISED REGARDING RETENSION IS ARE WE TALKING ABOUT ONLY RETAIN ING RETAINING THE SUBJECTS OR ARE WE TALKING ABOUT RETAINING THE DATA? FOR EXAMPLE, SAY SOMEBODY IS NOT INTERVIEWED IN A WAY, WOULD WE GO BACK TO THAT PERSON AND WILL WE TRY TO CAPTURE EVENTS FROM PRIOR TIMES WHEN -- WHEN THEY WERE SKIPPED. COMPLICATING AN ISSUE BECAUSE OF RECRUITMENT AND SAMPLING PROCESSES BUT SOMETHING VERY IMPORTANT IF WE WANT A COMPLETE DATA SET. SO WE HAVE BEEN ASKED TO THINK ABOUT SOME QUESTIONS A WE LISTEN TO OUR PRESENTERS FROM THE VANGUARD STUDY. THESE QUESTIONS HAVE TO DO WITH BARRIERS WITH WE MIGHT ANTICIPATE AND WHAT SPECIFIC ACTIVITIES WE MIGHT RECOMMEND FOR DIFFICULT TO REACH POPULATIONS, NON-SPEAKING POPULATIONS AN HARD TO REACH. ALSO THE ISSUE OF KEEPING YOUR RETENTION HIGH, REDUCING ATTRITION IN EARLY YEARS AND PERHAPS OUR VANGUARD STUDIES WILL BE ABLE TO ANSWER THOSE QUESTIONS TODAY. SO WE'RE GOING TO MOVE AHEAD WITH OUR PRESENTERS. MARK, ARE YOU GOING FIRST? >> YES. GOOD AFTERNOON. MR. CHAIRMAN, THANK YOU FOR KEEPING EVERYBODY AWAKE. I LOOK AROUND THE ROOM, ALL EYES ARE OPEN, THAT'S A GOOD THING AFTER LUNCH. JOSLYN AND I WILL TAG TEAM A BIT. I'LL TART START WITH COMMENTS, JOSLYN WILL PICK UP WITH SOME SLIDES THAT I'LL FINISH OFF AT THE END BUT I DO WANT TO TAKE THE OPPORTUNITY TO SAY THAT I'M PRESENTING ON BEHALF OF MY SEVERAL CO-INVESTIGATORS AND STUDY COORDINATORS, IN THE COURSE OF THE STUDY MANY HAVE COME TO HAVE VERY STRONG RELATIONSHIPS, WE SPAN MULTIPLE DISCIPLINES. I'M A NEONATOLOGIST BUT I CONVERSE WITH EPIDEMIOLOGISTS AND STATISTICIANS AN OBGYNs AND THE WHOLE GAMUT OF PROFESSIONALS THAT STEPPED TO THE FRONT TO SHEPHERD THE STUDY TO ITS CURRENT POINT. I WOULD LIKE TO SAY, I NEED THE SLIDE SET IF SOMEBODY CAN GET THAT UP FOR ME. WILL THAT DO IT? OKAY. LISTENING TO DISCUSSION THIS MORNING THE TOPIC OF RETENTION IS CLEARLY IMPORTANT AND INTERESTING BUT LESS COMPLICATED THAN TALKING POPULATIONS STUDY, SAMPLING METHODOLOGY, PROSPECTIVE AND RETROSPECTIVE OPERATIONS, CONSENT PROCESSES, IMS SYSTEM, DATA COLLECTION INVENTORIES ET CETERA SO I'M HAPPEN P PI TO TALK ABOUT RETENTION. SO WITH THAT, WE WILL GO TO THE FIRST SLIDE HERE. THIS IS SOME THOUGHTS I REPRESENT THE INVESTIGATORS ON, I THINK TO A PERSON. IN TERMS HOW WE CONSIDER AN THINK ABOUT RETENTION.B?%– I'M NOT SURE CRITICALITY IS A GOOD CHOICE OF WORDS, I WAS THINKING ABOUT IT THIS MORNING. SOME THAT HAVE KNOWLEDGE OF PHYSICS OR EXPERIENCE WITH SUMMER BLOCKBUSTERS CRITICALITY IS ALSO DEFINED AS POINT IN THE NUCLEAR REACTION WHEN THE REACTION w BECOMES SELF-SUSTAINING SO I DON'T WANT TO MAKE ANY ANALOGIES TO THIS PRESENTATION. SO IN TERMS OF RETENTION, I THINK THE BRIEFING DOCUMENT CLEARLY POINTS OUT THAT IF YOU HAVE SMALL ATTRITION IN SAMPLE OVER TIME LONGITUDINAL STUDY HAVE A PROFOUND EFFECT UPON ULTIMATE NUMBER OF PARTICIPANTS AT X YEARS SO I THINK THE NUMBER IS IN THE BOOK IT SAID 99% RETENTION EVERY YEAR FOR 21 YEARS, 21 YEARS YOU STILL LOSE 20% OF YOUR P POPULATION. IF YOU HAVE A 97% RETENTION EVERY YEAR 21 YEARS, YOU'RE DOWN TO 54%. FACTORING OTHER INFORMATION IN TERMS OF WHAT IS YOUR COMPLIANCE WITH OR ADHERENCE TO DATA COLLECTION PROTOCOLS THAT NUMBER IS LESS THAN 100% YOU HAVE OBVIOUSLY GOT LESS ROBUST DATA FROM YOUR TARGET OF 100,000. SO IT IS VERY IMPORTANT GOING FORWARD TO RETAIN AS MANY PARTICIPANTS, THEY COLLECT AS MUCH DATA, GOOD QUALITY DATA OVER TIME AS POSSIBLE. SO THAT IMPACTS THE NUMBERS THAT YOU NEED TO RECRUIT, YOU NEED TO DECIDE OBVIOUSLY AT YOUR END POINT WHAT YOU LOOK TO ASSESS, SOMETHING TO MAKE SURE THAT AT THE BEGINNING YOU HAVE ENOUGH NUMBERS TO SUPPORT THAT. AND RETENTION IS IMPORTANT BECAUSE DIFFERENCIAL RETENTION ATTRITION LEADS TO PROBLEMS WITH FIDELITY, REPRESENTATIVES OF THE POPULATION YOU STUDY AT THE END POINT. CHEERILY IN A LENGTHY LONGITUDINAL REPLACING THE PARTICIPANTS IS NOT AN EASY OPTION AS OPPOSED TO OTHER STUDIES THAT OCCUR IN HOSPITAL FOR SHORT AMOUNTS OF TIME. AND I THINK ALL THE INVESTIGATORS UNDERSTOOD FROM THE BEGINNING THAT WE NEEDED TO REALLY WORK VERY HARD FROM THE VERY BEGINNING OF THE STUDY DURING THE RECRUITMENT PHASE TO WORK ON RETENTION AND WE EXPERIENCE THE FACT THAT TO DO THIS WE WERE LOOKING CERTAIN RESOURCES THAT HAD TO DO WITH OUR ABILITY TO ACCESS GOODS, CASE MANAGEMENT INFORMATION AND REAL TIME AND ANALYZE EASILY WHAT ARE SERIAL EXPERIENCES WITH RETENTION GIVEN VISITS AND OVER TIME WAS. FINALLY WE DISCOVERED THAT WE HAVE A VERY MOBILE POPULATION A LOT OF THESE PARTICIPANTS MOVED OUT OF THE AREA, THEY HAVE EXCEEDED THE 50-MILE RULE FOR THE STUDY. SO THERE ARE WITHOUT SYSTEMS IN PLACE TO ADEQUATELY FOLLOW THESE PATIENTS THERE HAVE BEEN SOME LOST. SO I WILL TURN IT TO JOSLYN NOW. >> I'M JOS LYNN LEVY, I BEGAN WORKING WITH UCLA ON THE COINS IN 2010. AND ALSO I WANT TO THANK DR. HERSHFELD AND THE COMMITTEE AN OPPORTUNITY TO SHARE WHAT WE HAVE BEEN DOING. DR. LAURA CAULIFIELD IS HERE TODAY, PRINCIPAL INVESTIGATORS FOR THE COIN AND WE HAVE MISCHA CORRELL AND (INDISCERNIBLE) ON THE TELEPHONE WHO WILL BE AVAILABLE TO ANSWER QUESTIONS AFTER THE DISCUSSION. I KNOW DR. NEAL HOUSEN FROM UCLA WHO INTRODUCED THE TECHNOLOGIES INTO THE STUDY PRESENTED BEFORE SO I WON'T GO INTO A LOT OF DETAIL ABOUT WHAT A COIN IS, AND MODELS AND METHODS THAT WE USE BUT TO GIVE ENOUGH THAT PEOPLE WHO MAY NOT BE FAMILIAR WILL HAVE THE BACKGROUND TO GET THE RICHNESS OF WHAT WE'RE DISCUSSING TODAY. THE NCS COIN PROVIDES A STRUCTURE AND TECHNICAL SUPPORTS TO HARNESS THE POWER OF COLLABORATION IN THIS BROAD STUDY TO ADDRESS SHARED PROBLEMS IN A COLLABORATIVE WAY AND WORK ADAPTING SOLUTIONS TO THE MANY DIFFERENT SETTINGS WHICH THOSE SOLUTIONS CAN BE APPLIED TESTIMONY COIN BILLS ON MODELS AND METHODS USED IN TECHNOLOGY AND HEALTHCARE SETTINGS AND OTHER INDUSTRIAL SETTINGS TO WORK ON IMPROVEMENT, FIRST THE STRUCTURE COMES FROM THE BREAK THANK YOU SERIES COLLABORATIVE DEVELOPED BY THE INSTITUTE FOR HEALTHCARE IMPROVEMENT IS USED BROADLY NATIONALLY AND INTERNATIONALLY TO ACHIEVE IMPROVEMENTS LARGELY ON IMPLEMENTING EVIDENCE-BASED PRACTICES BUT ALSO FOR INNOVATION. THEN THE FOUNDATION OF OUR WORK IS THE MODEL FOR IMPROVEMENT THAT WAS DEVELOPED BY ASSOCIATES AND PROCESS IMPROVEMENT AND P THE STUDY CYCLE USED IN INDUSTRIAL IMPROVEMENT FOR DECADES. THE MODEL FOR IMPROVEMENT THREE FUNDAMENTAL QUESTIONS. WHAT ARE WE TRYING TO ACCOMPLISH, WHAT IS OUR AIM? HOW WILL WE KNOW A CHANGE IS AN ACCOMPLISHMENT, WHAT Mj j CAN WE DEVELOP TO TRACK OUR PROGRESS TOWARDS OUR AIM. AND WHAT CHANGES CAN WE MAKE THAT WILL RESULT IN IMPROVEMENT, WHAT ARE THE IDEAS OR STRATEGIES THAT WE WANT TO TEST IN THIS INSTANCE, WHAT ARE THE IDEAS THAT WE HAVE FOR IMPROVING RETENTION, MITIGATING THE RISK OF LOSS FROM THE STUDY. ONCE WE HAVE THOSE [y DRIVE THEM FORWARD, WE TEST THEM USING PDSA CYCLES WHICH WE HAVE A VERY SPECIFIC PLAN ABOUT WHAT WE WANT TO TEST INCLUDING A PLAN FOR DATA COLLECTION, HOW WILL WE KNOW IF WE ACHIEVED OUR AIM FOR THAT TEST. WE CONDUCT THE TEST AND OBSERVE WHAT WENT ON, WE STUDY THE RESULTS OF THE DATA THAT E WE COLLECTED. THEN BASED ON THAT WE MAKE A DETERMINATION OF WHAT DO WE WANT TO DO NEXT. IN GENERAL WE TRY TO KEEP THESE PDSAs VERY SMALL, THE GOAL IS TO DO LOTS OF ITERATIVE CYCLES SO WE CAN ACCELERATE THE PACE OF LEARNING AND IMPROVEMENT. SO THE COIN BEGAN, DR. HELFMAN INTRODUCED THE COIN INTO THE NATIONAL CHILDREN'S STUDY IN 2010 AS A FORMATIVE RESEARCH PROJECT TO UNDERSTAND THE UTILITY AND FEASIBILITY OF APPLY ING IMPROVEMENT APPROACHES IN A RESEARCH CONTEXT. THIS WORK WAS FIRST DONE WITH THE DIRECT OUTREACH OR OTHERWISE KNOWN AS HIGH LOW GROUP OF ALTERNATIVE RECRUITMENT STRATEGY TO SEE HOW METHODS ARE USED, OPTIMIZE RECRUITMENT STRATEGY AND ACCELERATE AND MAXIMIZE ENROLLMENT INTO THE STUDY THAT. KICKED OFF IN 2010 AND CONTINUED THROUGH 2011. AT THE SAME TIME THE HOUSEHOLD BASED RECRUISEMENT SITES OBSERVED WHAT WAS GOING ON WITH HIGH LEVEL COIN AND WANTED TO AVAIL THEMSELVES OF THOSE SAME SUPPORTS AND METHODS SO WE ESTABLISH AD PARALLEL EHBR COIN. IT ONLY LAST AD FEW MONTHS BECAUSE SHORTLY THEREAFTER THE FOCUS OF THE VANGUARD STUDY SWITCHED FROM RECRUITMENT TO RETENTION AT WHICH TIME THE PROGRAM OFFICE ASKED US TO ESTABLISH TWO NEW COIN, ONE FOCUSED ON THE IMS SYSTEMS AND THE DATA QUALITY AND UTILITY USABILITY OF THE DATA SYSTEMS, AND THE OTHER TO FOCUS ON RETENTION. SO QUICKLY AGAIN, RETENTION INITIATED IN 2011 WITH WITH THE PURPOSE OF ACCELERATING INNOVATION AND OPTIMIZING IMPLEMENTATION OF NCS RETENTION STRATEGIES AND ULTIMATELY TO IMPROVE RETENTION OF PARTICIPANTS IN THE STUDY, THE KEY OUTCOME THAT WE WERE LOOKING FOR ENROLLED WOMEN, CHILDREN AND FAMILIES WILL PARTICIPATE IN ALL SCHEDULED DATA COLLECTION EVENTS. WE WE COULDN'T LOOK AT RETENTION LONG TERM SO WE WERE LOOKING AT WAY TO DEFINE HOW TO LOOK AT RETENTION IN VERY IMMEDIATE TERMS. THE PARTICIPANTS IN THE COLLABORATIVE HAVE BEEN -- WE HAD ABOUT 17 STUDY CENTERS AND THOSE REPRESENTED SOME OF THE ORIGINAL VANGUARD CENTERS ALL THREE ALTERNATIVE RECRUITMENT STRATEGIES, STUDY CENTERS FROM PROVIDER BASED SAMPLING, AT THIS POINT REGIONAL OPERATING CENTERS WERE PARTICIPATING AND WE HAD EXCELLENT PARTICIPATION FROM CONTRACTORS SUCH AS NORR SUPPORT TEAM, THE SPECIAL SCIENTIFIC SYSTEMS WORK BENCH TEAM AND MEMBERS OF PROGRAM OFFICE, SOME HERE TODAY, AND IT'S BEEN A GREAT VENUE FOR CRAB RATION ACROSS MULTIPLE PARTIES IN THE STUDY. OUR CO-CHAIRS ARE MENTIONED FROM HOPKINS AND PREVIOUSLY DR. HUDAK CHAIRD THE MEASURES WORK GROUP AND A TEAM FROM UCLA, MYSELF AND LLOYD PROVOST FROM THE PROCESS IMPROVEMENT SO THE GROUP BEGAP BY IDENTIFYING WHAT STRATEGIES WERE WITH POTENTIAL TO MACKMIZE RETENTION THAT WE WANTED TO FOCUS OUR WORK ON. OBVIOUSLY RETENTION IS A VERY BROAD TOPIC. AGAIN, WE ALWAYS FOCUS ON WITH IMPROVEMENT WHAT WE'RE TRYING TO ACCOMPLISH, IT'S IMPORTANT TO NARROW TO SOME EXTENT AND ZERO IN ON WHAT WE'RE WORKING ON. SO THESE WERE THE EIGHT AREAS THAT WERE IDENTIFIED SUCH AS MAINTAINING FREQUENT CONTACT WITH PARTICIPANTS, ENSURING THAT THEY -- THE PARTICIPANTS HAVE POSITIVE EXPERIENCE WITH DATA COLLECTION EVENTS, AND DISCUSSING IDENTIFYING MITIGATING LIKELY RISKS FOR PARTICIPANT DROP OUT AND OTHERS AND WITH THE ROCK COMING ON, AN IMPORTANT ONE ADDED WAS THAT ENSURING PARTICIPANTS RECKON SENT THROW THE TRANSITION PROCESS. SO I WANT TO GO THROUGH A COUPLE OF EXAMPLES OF THESE PLAN STUDY CYCLES CONDUCTED BY MULTIPLE STUDY CENTERS ON THE ROCKS TO GIVE YOU A FEEL FOR THE KIND OF WORK THAT WE HAVE BEEN DOING TO UNDERSTAND THE RETENTION CHALLENGES AND SOME OF THE POTENTIAL OPPORTUNITIES FOR MITIGATING THOSE CHALLENGES. THE FIRST EXAMPLE WAS DONE EARLY ON BY A GROUP OF STUDY CENTERS TO TEST A VISUAL CONFIRMATION OF CONTACT INFORMATION. A NUMBER OF STUDY CENTERS HAD NOTED WHEN ASKING -- THE PROTOCOL FOR CERTAIN VISITS WAS TO SIMPLY ASK PARTICIPANTS IF THEIR CONTACT INFORMATION CHANGED RATHER THAN CAPTURING THAT INFORMATION. OFTEN PARTICIPANTS SAID NO OR THEY LOOK LIKE THEY WEREN'T CERTAIN. THE DATA COLLECTION STAFF ASSESS THE PARTICIPANTS MAY NOT HAVE REMEMBERED WHAT THEY HAD ORIGINALLY GIVEN. SO ONE STUDY MANAGER HAD THE IDEA TO GO AHEAD AND TEST, TO TRY PRINTING EXISTING CONTACT INFORMATION AND WHEN ASKING THE QUESTION HAS YOUR CONTACT CHANGED TO SHOW PRINT OUT AND GET VISUAL CONFIRMATION OF THE CON CONTACT INFORMATION AND IN FACT, THAT DID LEAD TO PARTICIPANTS OFFERING A LOT MORE INFORMATION ABOUT UPDATING OF THEIR CONTACTS. BASED ON THAT A NUMBER OF OTHER STUDY CENTERS TESTED THAT SAME PROCESS. OVER JUST A SMALL NUMBER OF VISITS, MAYBE 20 VISITS, DIFFERENT STUDY CENTERS TO SAY OKAY, DID THAT ALSO HELP IN OUR SITUATION? AND IT DID AND BASED ON THAT SMALL AMOUNT OF DATA CAPTURED IN A QUICK AMOUNT QUICK BRIEF TIME, ALL THE STUDY CENTERS PARTICIPATING IN THE COIN ADDED THAT TO THEIR WORK. THE GOAL WAS TO MAKE RECOMMENDATIONS TO THE PROGRAM OFFICE TO ADD TO THE PROTOCOL AND A TRACING MODULE HAS BEEN ADDED TO THE MDES GOING FORWARD. THE SECOND PDSA FOCUSES ON IDENTIFICATION OF RISK VARIABLES AND THIS ACTUALLY A LONG SERIES OF PDSAs AND THAT ITERATIVE TESTING IS CRITICAL TO THE IMPROVEMENT WORK WE DO, WHAT'S LEARNED IN ONE CYCLE LEADS TO LEARNING IN THE NEXT. WE HAD A WORK GROUP FORMED TO FOCUS ON IDENTIFYING AND MITIGATING RISK OF PARTICIPANTS DROPPING OUT. BEGAN BY CHECKING IMPRESSION THROUGHOUT POTENTIAL RISK VARIABLES FROM MULTIPLE DATA CATEGORIES BASED ON EXPERIENCE. LET ME JUST SAY THAT THE GOAL OF THIS WORK IN DEVELOPING THIS KIND OF IDENTIFICATION OF RISK VARIABLES WAS REALLY TO FIGURE OUT A PROCESS FOR HOW THIS COULD BE DONE. AS WE HAVE DISCUSSED, OBVIOUSLY EVERYTHING ABOUT THE STUDY WILL CHANGE OVER THE YEARS AND WHAT ARE RISK VARIABLES NOW FOR MOTHERS AN INFANTS MAYBE DIFFERENT FROM WHAT THE RISK VARIABLES WILL BE IN BUT THIS WAS REALLY TO LOOK AT A PROCESS IDENTIFYING RISK VARIABLES, DIFFERENT WAYS WE CAN DO THAT. AND THEN HOW CAN WE USE THAT INFORMATION. SO A GROUP OF STUDY CENTERS PUT TOGETHER THIS LIST AND THEY SELECTED EACH OF THOSE STUDY CENTERS SELECTED A FEW PARTICIPANTS THEY CONSIDER HIGH RISK BUZZ THEY HAD DROPPED OUT OF THE STUDY, THAT EAR DIFFICULT TO REACH OR IN THE CASE OF THE HIGH, LOW RECRUITMENT STRATEGIES , CENTERS, THOSE PARTICIPANTS WERE NOT EAGER TO MOVE FROM LOW TO HIGH INTENSITY PROTOCOL. SO THEY LOOKED AT THOSE PARTICIPANTS AGAINST A LIST OF VARIABLES OF INTEREST. AND THEY DID SEE IN MANY CASES THEIR HYPOTHESES BORE OUT THESE VARIABLES WERE CHARACTERISTIC OF THESE PARTICIPANTS. HOWEVER THEY ALSO IDENTIFY UNEXPECTED CHARACTERISTICS AND ALSO SAW WHAT WAS A RISK VARIABLE FOR PARTICIPANTS IN ONE STUDY CENTER WAS QUITE DIFFERENT FROM WHAT RISK VARIABLES WERE FOR PARTICIPANTS IN ANOTHER STUDY CENTER. SO THAT WAS AN IMPORTANT PIECE OF LEARNING THAT IT MAY NOT BE WE CAN CREATE A BLANKET SET OF VARIABLES BUT IT NEEDS TO BE RESPONSIVE TO THE PARTICULAR SETTING OR CATEGORIES, CATEGORIES OF PARTICIPANTS. SO AFTER THAT, THE GROUP DECIDED THAT -- TO GO IN TWO DIRECTIONS. ONE TO CONTINUE THIS LINE OF INVESTIGATION AND LOOK AT OTHER FEATURES. ANOTHER WAS TO TAKE IT MORE INTO RESEARCH REALM BECAUSE AGAIN, THIS IS IMPROVEMENT WORK WE USE AS SMALL, AS LITTLE DATA AS WE CAN, SMALL TEST TO MOVE QUICKLY TO LEARN BUT THIS IS A SCIENTIFIC STUDY AND THERE ARE VERY IMPORTANT QUESTIONS AROUND WHAT ARE FACTORS THAT INFLUENCE RETENTION THEREFORE, A GROUP OF INVESTIGATORS FROM THE COIN HAVE FORM AD WRITING GROUP TO CONTINUE TO LOOK AT THIS IN A MUCH MORE EXPANSIVE WAY TO BE ABLE TO MAKE SOME DEFINITIVE STATEMENTS ULTIMATELY ABOUT RETENSION. SO FOLLOWING THAT FIRST LOOK, BASED ON SOME OF THE LEARNING THE GROUP OF STUDY CENTERS DECIDED THEY WANTED TO LOOK AT ADDITIONAL VARIABLES TO TRY TO EXPAND UNDERSTANDING, THERE ARE OTHER VARIABLES THAT MIGHT BE IMPORTANT AND ALSO THEY WANTED TO LOOK AT NOT JUST THOSE PARTICIPANTS CONSIDERED TO BE HIGH RISK BUT ALSO TO THE EXTENT POSSIBLE LOOK AT THEIR FULL COHORT AND LOOK AT SOME PATTERNS OF DIFFERENCES IN THOSE VARIABLES. ONE OF THE THINGS IN PARTICULAR THAT THE GROUP DECIDED TO LOOK AT WERE PARTICIPANT RESPONSES TO SELF-ADMINISTERED QUESTIONNAIRES AROUND MOTIVATION FOR PARTICIPATION IN THE STU DI. AND THE GROUP AGAIN FOUND SOME SONSES THAT WERE THINGS THAT SURPRISED THEM. THEY ALSO WORKED ON DEVELOPING A -- TESTING A DYNAMIC RISK INDEX OR RISKCAL L QUEUE LAYTOR THAT WAS BASED ON NUMBER OF CONTACTS ATTEMPTS THAT IT TOOK TO COMPLETE DATA COLLECTION. IN SOME CASES BECAUSE OF ALL THE TRANSITION CERTAIN STUDY CENTERS HAD A HARDER TIME ACCESSING THEIR DATA AT THIS POINT, SO COULDN'T NECESSARILY LOOK AT THOSE RISK VARIABLES BUT THEY WERE STILL INTERESTED IN CATEGORIZING PARTICIPANTS BY LEVELS OF RISK SO THEY CAN TEST RISK MITIGATION STRATEGIES. SO USING NUMBER OF CONTACTS ATTEMPT PROVIDE AD MORE ACCESSIBLE WAY DIVIDING PARTICIPANTS INTO THESE DIFFERENT RISK CATEGORIES. SO GIVEN THE PATTERNS THAT WERE SEEN AND THE CONFIDENCE THAT THE PARTICIPANTS, COIN MEMBERS HAD DOING THIS WORK, THEY FELT THAT THE RICHNESS OF THE DATA ALLOWED THEM TO IDENTIFY DIFFERENCES, THEY MADE A RECOMMENDATION THAT ALL CHILDREN IN THE STUDY GOING FORWARD DO I BE ASSESSED FOR RISK USING BDR DATA. THAT KIND OF RISK ASSESSMENT AND RISK MITIGATION IS IN FACT BUILT INTO THE GUIDANCE AT THE TIME, AT LEAST PRELIMINARY WORK. SO HERE ARE SOME OF THE RISK VARIABLES THAT THEY IDENTIFIED, THEY WERE BROKEN INTO FOUR CATEGORIES, SOCIO DEMOGRAPHICS, LIFE STRESSORS, DIFFICULT TO REACH AND MOTIVATION FOR THE STUDY. AS I MENTION, THE SEGMENT GEOGRAPHY MATTERS, WHAT WAS THE RISK FACTOR IN ONE SEGMENT, MIGHT NOT HAVE BEEN IN ANOTHER. THEY WERE ALSO SOME INTERESTING RESPONSES FOR INSTANCE EMPLOYMENT STATUS WHERE ONE MIGHT ASSUME WHETHER THE RISK IS IF YOU WERE EMPLOYED OR NOT, IN FACT YOU FOUND THE HIGHER RISK ASSOCIATED WITH NO RESPONSE TO THAT QUESTION OR RELATED IN PART TO NUMBER OF HOURS THAT SOMEONE WAS WORKING. UNDER LIFE STRESSORS, MARK WILL SPEAK MORE ABOUT THIS. BUT WE CERTAINLY SAW SOME SUGGESTIVE DIFFERENCES IN RETENTION RELATED TO THE VERY EARLY POSTNATAL PERIOD AS OPPOSED TO CHILDREN THAT GOT A LITTLE BIT OLDER. IN TERMS OF RELOCATION POTENTIAL, IT POINTED TO THE DIFFERENCES IN TYPES OF RELOCATION ISSUES SO MIGRANT WORKERS VERSUS WE SAW THAT FOR MILITARY FAMILIES, MILITARY INSURANCE, THAT WAS A PROXY FOR MOBILITY, THAT WAS INDICATOR OF RISK. AND UNDER MOTIVATION FOR THE STUDY, IS THERE MOTIVE IN CERTAIN AREAS, MOTIVATION FOR STUDY AREAS WAS THERE INCENTIVE, THAT AN INDICATOR OF RISK. SO REASON FOR ESTABLISHING RISK ASSESSMENTS CATEGORIES WAS REALLY TO BE ABLE TO CREATE SOME KIND OF WAY OF PUTTING PEOPLE INTO DIFFERENT CATEGORIES OF RISK SO THEN THERE COULD BE A STRATIFIED RISK MITIGATION APPROACH. SO THIS IS A SIMPLE, SPLIT INTO TWO CATEGORIES ORIGINALLY THE GROUP WAS WORKING WITH A MORE COMPLEX STRATIFICATION SCHEME. BUT IN DISCUSSIONS WITH THE ROK WE'RE NOW DOWN TO TWO CATEGORIES. AND FOR THE HARD TO REACH STRATEGY THEY'RE TESTING USING MORE PERSONALIZED MORE INTENSIVE CONTACT, HAND DELIVERED INCENTIVES, FEDEX ENVELOPES TO BRING MORE ATTENTION, INVITATIONS TO COMMUNITY EVENTS FOR FACE TO FACE TIME FOR THESE PARTICIPANTS. WORKING WITH PARTICIPANTS PREFERRED MODE OF COMMUNICATION, BE IT TEXTING OR EMAIL VERSUS TELEPHONE. ON TOP OF THE UNIVERSEAL STRATEGIES USED TO MAINTAIN FREQUENT CONTACT, PERSONALIZED CONTACT. >> WHEN YOU TALK COMMUNITY EVENTS YOU TALK COMMUNITIES OF PARTICIPANTS? >> SO I KNOW THERE HAVE BEEN AT LEAST ONE OF THE STUDY CENTERS THAT IN FACT HAS DONE THAT. I THINK FOR THE MOST PART IT'S MORE GENERALIZED COMMUNITY EVENTS BUT I WOULD LEAVE THAT TO SOMEONE IN THE STUDY. >> I'M WONDERING, APART FROM CONFIDENTIALITY ISSUES ASSOCIATED, I WORRY ABOUT ANYTHING IN THE DESIGN OF A 21 YEAR STUDY THAT PUTS THE PEOPLE IN THE STUDY TOGETHER WITH EACH OTHER IN WAYS THAT MIGHT LEAVE THEM TO ALTER BEHAVIOR IN A WAY THAT WOULD BE DIFFERENCE FROM THE POPULATION THEY'RE SUPPOSEDLY AS CLOSE TO RANDOM DRAWN FROM AS POSSIBLE. SO I MEAN RETENTION IS EXTREMELY IMPORTANT. IT'S NOT IMPORTANT ENOUGH TO TO SOMETHING THAT WOULD UNDERMINE INTEGRITY OF THE DATA OF THE STUDY IN NAME OF RETENTION. IF THAT WERE THE CASE, WE WOULD PICK PEOPLE COMPLIANT UP FRONT AND NOT WORRY ABOUT RECRUITING PEOPLE WHO ARE NOT LIKELY TO DROP OUT. THAT WORRIES ME BECAUSE WE SAW NOTHING IN THE PROCESS HERE WHERE EXPERTS EVALUATE WHAT ARE THE NEGATIVE SIDE EFFECTS OF ONE OF THESE RETENTION STRATEGIES. >> I ONLY KNOW OF ONE GROUP THAT DID THAT. I THINK IN LARGE PART OTHER COMMUNITY EVENTS STRONG PART OF THE FEELING OF STUDY CENTERS THAT WAS AN IMPORTANT THING TO BE TESTING. IN TERMS OF LOOKING AT NEGATIVE SIDE EFFECTS WHENEVER WE TEST ANYTHING IN THE COIN PART OF WHAT WE'RE LOOKING FOR IS WHAT HAS POSITIVE IMPACT, HAS NEGATIVE AGAIN NOT A RESEARCH STUDY, TO ANSWER A QUESTION OF THE COMPLEXITY YOU'RE ASKING THAT COULD NOT PROBABLY BE ANSWERED THROUGH THE COIN PROCESS. THE COIN PROCESS IS USED IF WE'RE GOING TO IMPLEMENT SUCH AN EVENT WHAT'S THE BEST WAY TO DO IT. WHAT CAN WE LEARN ABOUT HOW WE DID IT. BUT RESEARCH GOES BEYOND THE COIN. >> I THINK THE ONE JOSLYN REFERS TO IS THAT IN THAT DIFFICULT TO REACH GROUP THERE'S OPPORTUNITY GIVEN TO PARTICIPANTS TO HAVE DATA COLLECTION OPPORTUNITIES FOR INTERVIEWS AT PLACES OF THEIR CHOICE. MCDONALDS IN IN BAKER COUNTY WAS A GREAT PLACE TO CATCH UP WITH PARTICIPANTS AND DOING AN INTERVIEW AND BUILD RELATION L SHIP SO THERE'S MORE OF THAT IN THAT LINE ITEM. >> I'M TRYING TO UNDERSTAND WHAT WE'RE SUPPOSED TO BE GETTING FROM THIS. MY IMPRESSION OF LOOKING AT THE PROCESS, IT MAYBE A TUNE PROCESS IT RATING IDEAS TO SEE IF THEY HAVE THE DESIRED EFFECT BUT WITHOUT SOME SORT OF ADVISORY ROLE OR CENTRAL MANAGEMENT ROLE IN THE PROGRAM OFFICE, IF NOBODY IS LOOKING AT THE DOWNSTREAM EFFECTS THAT ARE NOT RELATED TO THE GOALS YOU'RE TESTING, IT'S NOT NECESSARILY A USEFUL PROCESS FOR US TO LOOK AT GOING FORWARD. IF THE POINT OF THIS IS TO GIVE US A SET OF EXAMPLES OF STRATEGIES WE KNOW WORK, THAT'S GREAT BUT PERHAPS WE CAN GO THROUGH THEM QUICKLY SO WE CAN MOVE TO THE NEXT ELEMENTS OF THE DISCUSSION. >> THE QUESTION IS WHAT EXTENT THESE ARE STRATEGIES WE HAVE ACTUALLY KNOW WORK FOR MANY YEARS. IF YOU TOOK A FIELD MANAGER OR STUDY DIRECTOR FOR NORTH OR WESAT THAT TIME AND TOLD THEM THESE THINGS THEY WOULD SAY YEAH. THE QUESTION TO WHAT EXTENT IS THIS NEW. THE OTHER QUESTION, HOW MUCH DOES THIS COST BEYOND THE THINGS THAT THE FIELD ORGANIZATIONS DO IN A NORMAL COURSE OF BUSINESS? HOW MUCH DOES THAT COST? IF YOU PUT THOSE DOLLARS INTO AS ALMA SUGGESTED EARLIER INTO RESPONDENT FEES TO BRING PEOPLE OVER. ARE YOU AHEAD OR ARE YOU BEHIND? >> I ONLY RUN A STUDY WITH 6,000 PEOPLE BUT WE HAVE BEEN FOLLOWING THEM FOR EIGHT YEARS NOW SO IF YOU COULD DO A MONDAY MORNING QUARTERBACK ON STUDIES YOU'RE DOING NOW, WHAT WOULD BE IF YOU WERE TO RANK THE MOST IMPORTANT POSITIVE RETENTION ASPECT VERSUS THE LEAST -- THE WORST NEGATIVE ASPECT. BECAUSE CERTAINLY IN APPENDIX THREE PEOPLE LOOKED AT RETENTION OF ALL THESE OTHER STUDIES. SO TO ANSWER JOE'S QUESTION ABOUT WHETHER OR NOT A QUESTIONNAIRE OR CARDS OR NEWS LETTERS AFFECTED HOW THE SUBJECTS WERE WHETHER OR NOT IT BIASED FINDINGS SOMEONE SHOULD HAVE BEEN ABLE TO ANSWER THAT AND ALL THESE LISTED IN APPENDIX 3, DOES ANYONE KNOW? THE NURSES HEALTH STUDY, FRAMINGHAM HART STUDY, PEOPLE GOT NEWS LETTERS ALL THE TIME. >> I THINK THE BEST PRACTICES THAT ARE LISTED THAT COME FROM THESE STUDIES, THEY WORK IN SOME SITUATIONS AND NOT OTHERS. IT REALLY IS AN ART AND THE ART IS CALLED TAILORING. YOU NEED SOMEBODY WHO DECIDES WHAT NEEDS TO BE DONE PER THIS PARTICULAR RESPONDENT. I MEAN, THAT'S MY BOTTOM LINE FROM DOING A LOT OF THESE. >> I AGREE. CONNECTIONS MADE LONGITUDINALLY WITH THE SAME STUFF PERSONAL AND I HAVEN'T SAID IT ALL, I DON'T KNOW ON NATIONAL LEVEL ANYONE ACTUALLY ASKED THIS, I APOLOGIZE, BUT IN TERMS OF INCENTIVES RAFFLES, THINGS THAT KEEP PEOPLE IN STUDY, I AGREE, PARTICIPATION IS ONLY -- THERE'S SO MANY VARIABLES TO THAT. BUT CONNECTIONS WITH THE IMMUNITY. DISCUSSING AMONG DIFFERENT HEALTH PROFESSIONALS. THERE'S MANY LEVELS PROVIDING EDUCATION THAT ALSO HELPS RETENTION. WONDERING IF THAT ALSO CAME INTO PLAY IN THE NATIONAL CHILDREN'S STUDY. >> ALL THOSE FACTORS WE WORKED ON, GIVING A IF EXAMPLES. JUST WONDERING IF ANYONE ELSE FROM THE STUDY WANTS TO ADDRESS THIS. I'M A QUALITY IMPROVEMENT COACH SO I'M NOT GOING TO SPEAK ON RETENTION STRATEGIES THEMSELVES. >> I'LL TAKE A STAB AT IT. THERE'S A LARGE LITERATURE ABOUT WHAT ARE EFFECTIVE STRATEGIES FOR RETENTION IN DIFFERENT CIRCUMSTANCES AND TO SOME EXTENT THEY VARY FROM POPULATION TO POPULATION. CLEARLY A POPULATION OF PREGNANT WOMEN AND SHORTLY POST PREGNANT WOMEN VERY UNIQUE POPULATION THAT WE ALL TRY TO BE SENSITIVE TO. I THINK WHAT WE DID IN THIS STUDY IS THAT WE WERE GIVEN PROTOCOLS ON DATA COLLECTION BUT THERE WASN'T A WHOLE LOT OF RAP AROUND THAT WENT WITH THAT. WE DID HAVE THE BENEFIT 30 ARS SITES FROM HAVING THE BEST SEVEN ORIGINAL VANGUARD CENTERS PILOTED INITIALLY. I THINK WHAT WE DID, YOU ADOPT FOR INSTANCE BUNDLES OF CARE SO YOU WANT TO DECREASE INCIDENCE OF ASSOCIATED PNEUMONIAS, YOU DO THINGS THAT PEOPLE SAID ARE BEST PRACTICES, EMPLOY THEM AND I GUARANTEE YOU IF YOU GO INTO A HOSPITAL THAT HAS A HIGH RATE OF ASSOCIATED PNEUMONIA, YOU EMPLOY THIS BUNDLE YOU WILL BRING DOWN YOUR PNEUMONIA RATE. CAN YOU IDENTIFY WHICH OF THOSE INTERVENTIONS SPECIFICALLY WERE EFFECTIVE? NO. BUT THE -- I THINK THAT'S WHAT HAPPENED HERE. WE ADOPT AD LOT OF DIFFERENT PRACTICES, WE HAD SOME VERY ORIGINAL IDEAS I THINK, HOW TO APPROACH WOMEN FROM THE STUDY COORDINATORS OF THE STUDY CENTERS AN THROUGH THE COIN PROCESS. PLACES EMBRACE THE PACKAGE AND P THERE'S DATA THAT'S BENEFICIAL EFFECT. MAYBE THAT DOESN'T ANSWER YOUR QUESTION COMPLETELY. >> ANOTHER PIECE OF IT WAS HOW DO YOU IMPLEMENT THESE BEST PRACTICES LOOKING AT HEALTHCARE WE KNOW WHAT BEST PRACTICES ARE. THE CUSTOMIZATION, OR ART OR SCIENCE OF PUTTING THEM IN PLACE IS OFTEN CHALLENGING. (OFF MIC) >> I THINK YOU HAVE TO UNDERSTAND AS WELL WITHIN THE HISTORICAL CONTEXT WE'RE OPERATING, AS MARK SAID ORIGINAL RETENTION FIGURES CAME FROM THE VANGUARD SITE THAT THEN IN THINKING ABOUT THE IMPLICATIONS OF WHAT THAT ORIGINAL RETENTION RATE WAS FROM THE TIME THEY T BABY IS BORN TO SIX MONTHS, IF THAT RETENTION RAY CARRIED THROUGH WOULD BE A SIGNIFICANT ALSO OF SAMPLE SO WE WERE TASKED WITH AS I SEE IT, PART OF THE CHALLENGE IS HOW TO DRIVE THAT DOWN MEANING THAT WE HAVE MUCH LESS LOSS TO FOLLOW-UP OVER THAT TIME PERIOD AS WE GO FORWARD. AS WE REFLECTED A PROCESS THROUGH WHICH DIFFERENT STUDY CENTERS IN EXPERIENCE BECAUSE THERE'S A WEALTH OF COHORTS PARTICULARLY MATERNAL AND CHILD HEALTH HOW TO KEEP PEOPLE THROUGH PREGNANCY BIRTH AND PEDIATRIC PERIOD AND BRING TOGETHER THOSED IDEAS AND DECIDE ACROSS THE UNITED STATES WHICH ARE THE STRATEGIES RESONATE AND SHOULD BE CARRIED FORTH FOR THE MAIN STUDY. SO IT WAS WITHIN THAT SPIRIT THIS PROCESS IS INVOLVED FOR THE LAST YEAR OR SO. AND WE HAVE MADE RECOMMENDATIONS FOR THINGS IN TERMS OF THE INFORMATION CASE MANAGEMENT SYSTEM AS WELL AS SPECIFIC STRATEGIES THAT RESONATE ACROSS DIVERSE SETTINGS THAT ARE CURRENTLY PART OF NATIONAL CHILDREN'S STUDY. SO I THINK IT'S IMPORTANT TO CONSIDER THE CONTEXT WHICH WE WERE MOVING FORWARD. >> IMPORTANCE OF GOOD CASE MANAGEMENT IS SOMETHING WE CAN APPRECIATE. IT'S CRITICAL. TO FOLLOWING PEOPLE TO GOING ON WITH THE STUDY. ARE THERE DATA THAT INDICATE WHERE YOUR GREATEST LOSSES WERE? FROM YOUR PREGNANCY GROUP AND BIRTH, IS THERE ANYTHING YOU CAN EITHER SHOW US OR TELL US ABOUT? >> IF I MAY, THE BIGGEST DECREASE WE SAW WAS BETWEEN A WOMAN ENROLLING, IN THIS CASE EVERYONE ENROLLED DURING PREGNANCY. WITH FEW EXCEPTIONS OUT OF THE 5,000 CHILDREN THERE MAYBE 50 OR SO ENROLLED PER NATALLY. THE VAST MAJORITY DURING PREGNANCY. THE BIGGEST DROP WAS ENROLLMENT AND BIRTH OF CHILD. SUBSEQUENTLY WE HAVE BEENNB[œ LOOKING AT DIFFERENT -- WE HAVE VARIABLE DATA BUT NOTHING ALARMING. BUT TO US WHAT'S MOSTLY ALARMING WAS A GAP BETWEEN WHEN A PREGNANT WOMAN ENROLLS AND THE BIRTH OF THE CHILD. THAT NUMBER WAS ORDER OF MAGNITUDE VARIED FROM LOCATION TO LOCATION BUT IT WAS HIGH AS 20%, LOW AS 10%. 10% WAS OKAY, BUT NORTH OF THAT WE GOT WORRIED SO WE SAID WE HAVE TO START EXAMINING THIS. >> SO WE'RE THE REFUSALS WERE THEY REINTERVIEWS TO FIND OUT WHY THEY REFUSED AS MUCH >> THAT IS IN PROCESS RIGHT NOW. WE'LL REPORT BECOME BUT WE GOT SOME DATA FROM OUR EXPERIENCE BUT WE WANT TO GET MORE COMPREHENSIVE DATA. >> THERE IS A DIFFERENCE BETWEEN PEOPLE IF YOU CAN'T FIND THEM AGAIN AT SOME LEVEL POPULATION OR THEY REFUSE. AND IT SOUND LIKE SOME OF THESE -- LIKE YOUR TRIGGERS FOR WANT OF BETTER WORD WOULD BE MORE LETD TO LOSING PEOPLE LIKE IF SOMEONE IS UP EMPLOYED LIKELY TO MOVE. SO FORT. AND OTHERS THAT MIGHT BE MORE RELATED TO REFUSAL. THEIR ATTITUDE ABOUT THE SURVEY. I TONE KNOW IF YOU ANALYZED THE DATA BY THOSE TWO THINGS BECAUSE IN OUR EXPERIENCE VERY DIFFERENT. VERY DIFFERENT STRATEGIES FOR THIS TWO GROUPS. >> WHICH AREN'T ANALYZING PER SE BUT THERE ARE STRATEGIES WHEN WOMEN MAKE REFUSALS REFER THEM OVER TO SPECIALISTS WHO THEN ADDRESS THAT, WHO HAVE TRAININGENING THOSE TECHNIQUES. WHAT I HARD YOU TOO TO THE DO IS NOT WAIT UNTIL THE EVENT HAPPENS ABOUT YOU HAVE TO UNDO IT. BUT ARE THERE THINGS THAT SAY THIS IS A LIKELY CASE TO MOVE, I HAVE TO DO SOMETHING MORE ON MY TRACING. SO THERE'S A LOT OF LITERATURE, LIKELY TO REFUSE NEXT TIME BECAUSE THEY HAVE MORE CALL ATTEMPTS OR THEY MADE APPOINTMENT, BROKE AN APPOINTMENT, THAT'S A GOOD ONE. I'M NOT EXACTLY SURE WHAT YOU DO, BUT AT LEAST YOU'RE TRYING TO MAKE THAT REFUSAL NOT HAPPEN AND YOU DON'T WANT TO LOSE THAT PERSON. FINDING THEM AFTER THE FACT. >> SO THAT'S THE WORK THAT'S BEING DONE ON IDENTIFYING WHAT THE RISK VARIABLES ARE AND P DEVELOPING RETENTION STRATEGIES TO GO WITH THEM, THAT WOULD BE SPECIFIC TO WHAT RISK VARIABLES WERE. SO IDENTIFICATION OF RISK VARIABLES. MORE WORK IS DONE ON THAT, HAVEN'T BEEN ABLE TO GET TO RETENTION STRATEGIES YET. PART OF THAT HAD TO TO WITH TRANSITION FROM STUDY CENTERS TO THE ROCK, SOME OF THAT WORK GOT PUT ASIDE FOR NOW. WILL BE RESUMED WITH THE ORIGINAL OPERATING CENTERS. >> FOLLOWING UP ON THAT, I HAVE A COLLEAGUE WHO HAS DONE SPECIFICALLY THE WORK IN THE AREA OF CHURN PREDICTION AND ONLINE GAMES. VERY DIFFERENT DOMAIN, IT'S ABOUT WHEN PEOPLE WILL STOP PLAYING AND EVENTUALLY STOP PAYING. BUT THE DATA MINING TECHNIQUES ARE GENERALIZABLE. IT'S WELL POLISHED -- PUBLISHED WORK. FOR PEOPLE WHO WANT TO TAKE LOTS OF DATA CAN WE FIGURE OUT WHO IS AT RISK. THERE'S A LOT OF WORK DONE IN ALL SORTS OF MARKETING SPACE NOT JUST SURVEY SPACE. THERE A RECURRING MESSAGE HERE THAT AT A CERTAIN POINT IT BECOMES A SKILLED INDIVIDUAL DEALING WITH INDIVIDUAL PARTICIPANTS. FIGURING OUT WHAT THIS PERSON NEEDS TO MAKE THINGS HAPPEN AND ONE OF THE THINGS THAT THEREFORE MUST COME CENTRALLY IS COMBINATION OF GUIDELINES OF HOW FAR CAN YOU GO. BUT ALSO THE TRAINING, IN THIS CASE I WOULD ALSO SAY EDUCATION TO SAY WHEN IS WHAT YOU'RE THINKING ABOUT DOING A GREATER JEOPARDY TO THE PROJECT. HAS THAT STUFF BEEN DEVELOPED YET? IS THIS A PLAN FOR WHERE THE TRAINING MATERIAL COMES FROM? THERE ARE A LOT OF PEOPLE MT. FIELD TRYING TO RETAIN 100,000 PARTICIPANTS. PLUS. THAT SEEMS LIKE STUFF THAT'S NEEDED IN THE NEXT COUPLE OF YEARS. >> WE AGREE WHOLEHEARTEDLY WITH YOU. I BELIEVE THAT PART OF WHAT WE'RE HOPINGING FROM THE LITERATURE REVIEW WHICH IS DONE BEFORE AND THE ACTIVITY IS GOING TO LOOK AT HELPING USkd„ DEFINE HOW THAT TRAIN AND THAT PROCESS, THE INDIVIDUAL RELATIONSHIPS ARE THE KEY TO SUCCESS IN THE PROCESS, THERE IS NO MAGIC WAND ACROSS THE BOARD EVEN IF YOU CAN PUT PEOPLE IN GROUPS WHY THEY LEFT, THERE'S REASONS THEY FELL WITHIN THOSE INDIVIDUAL GROUPS AND PROCESSES SO THE INDIVIDUALIZED TRAINING IS CRITICAL FACTOR IN THAT PROCESS AND IS A COMPONENT WHY WE'RE TRYING TO GET THE DATA SPECIFICALLY FOR OUR VANGUARD AND STUDY CENTERS IN THOSE PROCESSES ALONG WITH WITH WHAT'S OUT THERE IN THE FIELD. WHAT'S OF INTEREST IS DIVERSITY OF DOMAIN YOU'RE TALKING ABOUT AND YOUR YOU RUNG A BELL WITH ME, THAT PROCESS OF RETENTION IN DIFFERENT ARENAS IS STILL BASED ON HUMAN BEHAVIORS AND WHAT'S THE HOOK. IN THOSE PROCESSES. SO AGREED. >> >> WHAT IS THE PROCEDURE NCS WILL HAVE ABOUT METRICKING THIS AS STUDY IS ONGOING SOMETHING ONE THING I HAVE FOUND HELPFUL GETTING THE 6,000 WE HAVE RETAINED IS CONSTANT MONITORING EVERY SIX MONTHS MAKING SURE WE'RE HITTING OUR MILESTONES AND IF WE'RE NOT WE GET FEEDBACK FROM THE PEOPLE IN THE FIELD TELL US WHAT'S WRONG, WHAT'S RIGHT AND WE MACK SURE WE CHANGE IT ACCORDINGLY SO WE CAN SPEAK TO THE NEEDS OF THE PEOPLE WE'RE TRYING TO TAKE CARE OF. >> I WANT TO EMPHASIZE THE IMPORTANCE OF RELATION SHIMS. IN THE ENTERRUE QUESTIONNAIRES, THERE IS A QUESTION AT SEVERAL PLACES PRE-NATAL AND MAYBE POSTNATAL ABOUT WHAT IS THE WOMAN SUPPORT STRUCTURE. IN TERMS OF SPECIFICALLY THE NCS. IS IT SUPPORTED BY PARENTS, GRANDPARENTS, FRIENDS, COWORKERS, SO FORTH. WHAT WE FOUND IN OUR COUNTY WHERE I HAVE GOT -- VERY LUCKY TO HAVE THREE EXCELLENT STUDY NURSES THAT LIVE IN BAKER COUNTY AND KNOW THE POPULATION AND ARE THE CULTURAL SENSITIVITY, IS THAT THAT QUESTION SHOULD ALSO ASK ABOUT SUPPORT STRUCTURE IS YOUR STUDY NURSE. I THINK THAT IS WHAT HAPPENS. OUR STUDY NURSES ARE NOT -- THEY CAN'T PROVIDE ANY CARE BUT WE HAVE ALWAYS SAID THEY CAN SAY THEY'RE A RESOURCE TO THE WOMEN, IF THEY HAVE QUESTIONS WHERE TO ACCESS HEALTHCARE OR PROVIDERS THAT THAT IS SOMETHING THAT THEY CAN DO AND WE EMPOWER THEM TO DO THAT. STUDY NURSES HELP GET THEM TO MEDICAID BY GETTING THEM TO DOOR. SO THOSE ARE THINGS THAT I THINK HELP BUILD RELATIONSHIPS AND ENGENDER LOYALTY TO THE STUDY. >> I DON'T WANT TO DIMINISH THE IMPORTANCE OF THESE PERSONAL THINGS. YOU'RE TALKING 100,000 PEOPLE, IT'S BIG STUDIEN APPROXIMATE TOUGH HAVE CONSISTENCY ACROSS THAT COLLECTION AND I HOPE YOU -- PERHAPS YOU'RE GOOD RECRUITING STUDY PERSONNEL AND PICK THOSE VERY, VERY SPECIAL PEOPLE. BUT IF THERE'S NOT SOMETHING CENTRAL AND PART OF THE DATA COLLECTION PROCESS, PART OF THE PROTOCOL, I CAN GIVE TO EVERYONE AND SAY YOU DO THIS ALL THE TIME, YOU MAY DO MORE, YOU MAY HAVE TO MODIFY BUT LEAVING IT TO THE PERSONAL PART OF IT WILL INTRODUCE A KIND OF VARIABILITY THAT WE WILL SEE IN THE RATES WHEN YOU'RE DONE, IT WILL BE RELATED TO PLACE AND ANYTHING RELATED TO PLACE IS GOING TO BIAS YOUR SAMPLE, SO IT HAS TO BE DUAL, I MEAN IT HAS TO BE ACROSS THE BOARD BUT THINGS THAT YOU CAN INSTITUTIONALIZE ARE EASIER TO DO, AND I THINK WILL PROBABLY BE PAID -- >> WE ABSOLUTELY AGREE, PART OF THE PROCESS HERE IS TAKING INVENTORY AND ANALYSIS ACROSS OUR CENTERS AND TAKING THOSE ELEMENTS PUTTING THEM INTO A STANDARD PROTOCOL ABSOLUTELY. >> SO WITH REGARD TO QUESTION OBJECT TRAINING ONE TEST THE REGIONAL OPERATING CENTERS ARE ENROLLED IN RIGHT NOW IS USING, ONE TIME THE STUDY CENTER, THERE WASN'T THE FAQ DATA FOR ALL VISITS. SO THEY DECIDED TO TRY WHERE THE DATA COLLECTORS WOULD SHARE PERCEPTIONS ABOUT PARTICIPANTS ENGAGEMENT IN THE DATA COLLECTION VISIT, IN ORDER THE USE THAT TO THINK ABOUT RISK ASSIGNMENT. SIN THAT TIME THE ROCKS HAVE DECIDED TO USE THAT SAME DATA COLLECTION PERCEPTIONS OF THE VISIT THEIR SINCE OF WHAT WENT’md ON, AS A TRAINING TOOL SO AFTER THE VISITS, THE DATA COLLECTORS ARE COMPLETING QUESTIONS ABOUT THE EXPERIENCE AND STAFF WILL REVIEW THAT WITH DATA COLLECTORS TO UNDERSTAND WHAT ADDITIONAL TRAINING THEY MAY NEED TO WORK ON MITIGATING RISK. THEN ROCKS HAVE BEGUN TO DO TESTING AROUND RECKON SENT PROCESS. AND REALLY LOOKING AT BEST WAYS TO MAINTAIN CONTINUITY ACROSS THE STUDY CENTER TO ROCK TRANSITION, TRYING A NUMBER OF DIFFERENT APPROACHES, SOME OF THAT DEPENDING ON GUIDELINES FOR STUDY SENORS AROUND THEIR IRB OR SOME OTHER PROCESSES BUT IN SOME INSTANCES THE STUDY THENTERS ARE WRITING PERSONAL NOTES TO INTRODUCE THE ROCK STAFF AND OTHER INSTANCES DOING JOINT VISITS FOR STUDY STAFF AND ROCK STAFF ARE GOING OUT TOGETHER. SO FAR RESPONSES TO THOSE PAIRED SRI HAVE BEEN POSITIVE AND PERSONAL INTRODUCTIONS. BUT THERE WILL HAVE TO BE MUCH MORE EXAMINATION OF IMPACT OF THOSE EFFORTS ON RECKON SENT RATES ONCE THE DATA IS AVAILABLE TO THE MOST EFFECTIVE STRATEGIES SO WE HAVE TOUCHED ON ISSUE OF OF MEASUREMENT AND MEASUREMENT IS EXTREMELY IMPORTANT FOR DOING THIS KIND OF IMPROVEMENT WORK AS YOU KNOW WE NEED THAT KIND OF REAL TIME FEED BACK. OBVIOUSLY THERE ARE CHALLENGES TO DOING THAT IN A FEDERAL STUDY OF THIS TYPE BUT CERTAINLY THE DATA WE HAVE HAD HAS BEEN EXTREMELY USEFUL. AND I'M GOING TO HAND IT OVER TO MARK, CHAIR OF MEASURES ECONOMY TO SPEAK MORE ABOUT THAT. >> SO I HOPE I TOUCH ON QUESTIONS YOU'RE ALLUDING TO HERE. ONE INTERESTING THING ABOUT THIS PROJECT IS THAT STUDY SENORS HAD A DIFFERENCIAL SPECIFICATION IN TERMS OF ABILITY TO DO CASE MANAGEMENT AND DO IN A FACILE WAY IN REAL TIME LOOK AT RETENTION RATES BY VISIT OR CUMULATIVE RETENTION RATES. WE IDENTIFY THAT AS A REAL OPPORTUNITY FOR IMPROVEMENT IN COIN. SENORS WERE FOCUSED ON RECRUITMENT, ACTIVE RECRUITMENT UNTIL NOVEMBER 2011, THEN WE HAD PASSIVE RECRUITMENT THAT CONTINUES THROUGH 2012 SO IT WAS AT THIS POINT IF CENTERS BEGAN TO TRANSITION FROM -- TO RECRUITMENT IN MANY CASES. THAT'S WHEN THE FIRST SESSION WAS DONE IN FEBRUARY OF 2012. I'LL SHOW YOU SOME DATA ABOUT THAT. BUT THE METRICS FOR RETENTION, THIS IS COMPLICATED BECAUSE THERE'S MANY CATEGORIES YOU CAN PUT MOTHERS AND BABIES INTO. WE FELT IN TERMS OF COIN WE NEEDED POPULATION THAT WAS ACTIONABLE. IF A STUDY CENTER AN MOTHER HAS LOST A BABY BEFORE BIRTH, THERE'S NOTHING STUDY CENTER CAN DO ABOUT THAT. IF A STUDY CENTER ENCOUNTER A MOTHER WHO MOVES FROM FLORIDA TO LOUISIANA THERE'S NOTHING WE CAN DO ABOUT THAT SO WE CAN'T IMPACT RETENTION IN THAT CASE. IF YOU HAVE A BABY WHO IS ADOPTED AND YOU ARE NOT ALLOWED TO KNOW WHO THE ADOPTIVE FAMILY IS, YOU HAVE LOST THAT BABY AND NOTHING TO DO ABOUT THAT. SO WE FOCUS TON POPULATION THROUGH PDSA PROCESS HAS SOME IMPACT ON IMPROVING RETENTION O. SO THE DENOMINATOR WAS DIFFERENT COMPARED TO THE ENTIRE POPULATION FETUSES BABIES BUT GRANTED THE LATTER IS ALSO EXTREMELY IMPORTANT IN TERMS HOW THE MCS IS DESIGNED, WHAT THE NATURAL HISTORY OF THE TREE WILL BE OVER TIME. SO -- WE ALSO WANTED MEASUREMENTS THAT LENT THEMSELVES TO RAPID ASSESSMENT, ITERATIVE ASSESSMENT FOR THE PDSA PROSECRETARIES -- PROJECTS DONE. SO WE LOOKED AT MEASURES SPECIFIC TO VISITS AND WE CAME UP WITH A DEFINITION ONE OF MANY POSSIBLES LOOKING AT OVERALL RETENTION KNOWING A MOTHER COULD MISS A VISIT BUT PICKED UP AGAIN ON THE NEXT VISIT. WE DIDN'T WANT THE SAY SHE WAS NOT RETAINED BECAUSE SHE MISSED ONE VISIT. I NEED TO USE THE RIGHT THING SO THE METRICS ON THIS WERE BASICALLY A CONSENSUS BASED APPROACH, A CONSULTATION ACROSS STUDY CENTERS. THIS IS CLEAR TO UNIFORM IMPLEMENTATION, WE WILL HAVE STATIC MEASURES AT EACH VISIT TO SEW WHAT THE RETENTION TRENDS WERE BY VISIT OVER TIME. AND DYNAMIC OVER SERIAL VISITS TO LOOK AT THE REAL WORLD. ISSUED HAD TO BE DYNAMIC TO USE IN PDSA PROJECTS AND ASSESS WHAT THE INTERVENTION WAS. BE APPLIED TO A REASONABLE POPULATION OF PEOPLE. WHAT WE WERE TASKD TO COIS IDENTIFY AND FOLLOW EVERY CONFIRMED FETUS IN LIVE BORN INFANTS. INTERESTINGLY, WHAT WE FOUND OUT IN THE STUDY IS BECAUSE A WOMAN SAYS SHE'S PREGNANT DOESN'T NECESSARILY MEAN SHE'S PREGNANT. WE KNOW THAT FOR A FACT OR MAYBE CONSENTED AS PREGNANT, THEN WE LOSE HER AT FOLLOW-UP BEFORE SHE DELIVERS -- SHE HAS ANY IDEA WHETHER SHE'S PREGNANT OR NOT. THIS ALSO HAPPENED. WE FOUND OUT SERVING -- IF A BABY WAS IN BETWEEN WINDOWS, AND WINDOWS INITIALLY FOR THE NCS FOR THEY WEREN'T CONSECUTIVE SO YOU HAVE A GAP BETWEEN THREE AND SIX MONTHS VISIT, THAT DATA COLLECTORS WEREN'T COLLECTING DATA. IF NCI THAT WINDOW YOU CAME ACROSS THE MOTHER SO YOU SAID WELL, IT'S NOT IN THE PROTOCOL BUT AN OPPORTUNITY TO CONTACT MOTHER, SIT DOWN WITH HER, HAVE A DATA COLLECTION REINFORCE THE IMPORTANCE AND VALIDITY OF THE STUDY AND HOPEFULLY THE NEXT STUDY VISIT OCCURS ON CYCLE L SO THAT'S SOMETHING THAT RAISED EYE BROWS. BOTH NCS DENOMINATOR, AND THE COIN DENOMINATOR, WE FOUND OUT VERY FEW CENTERS HAVE THE CAPABILITY TO LOOK WHAT YOU WERE ASKING FOR WITH WITH -- WHAT IS RETENTION TODAY AS I WALK INTO THE OFFICE. WHICH MOTHERS I NEED TO OUTREACH AS PRIORITY BECAUSE THEY'RE NEARING THE END OF WINDOW. OR THIS MOTHER HAS BEEN DIFFICULT TO CONTACT, SHE MISSED A VISIT, SHE -- WE HAVEN'T BEEN ABLE TO GET IN TOUCH WITH HER. HOW DO I ORGANIZE DAY AND WEEK IN OTHER WORDS. SO THE VISIT WINDOWS, WE CAME UP WITH A GENERAL PRINCIPLE DATA COLLECTION OPPORTUNITIES ASSIST RELATION L SHIPS FACILITATE RETENTION GOING FORWARD. ALL DATA I THINK ARE VALUABLE. AND COLLECT THE DATA AND DECIDE WHETHER OR NOT IT'S USEFUL BUT CERTAINLY FOR INSTANCE IF YOU COLLECTED A THREE MONTH VISIT FOUR AND A HALF MONTHS WHERE THE WINDOW IS 2 TO 4 AND 6 MONTH IS 5 TO 7, THEY ARE NOW CONSECUTIVE. BUT IF YOU GET A WEIGH OF FOUR AND A HALF MONTHS THAT PROVIDE VALUABLE INFORMATION FOR GROWTH. YOU KNOW WHAT PERCENTILE THE CHILD IS AT THAT POINT IN TIME AND YOU CAN PIECE TOGETHER A STORY WITH MEASUREMENT OVER TIME BUT THAT'S OOH IMPORTANT INFORMATION TO GET. THE BIRTH IN THE WIPE TOE WAS TECHNICALLY TEN DAYS. THERE ARE LOTS OF MOTHERS WE MISSED IN HOSPITAL OR COULDN'T GET BEFORE TEN DAYS BUT FORM WAS THE ONLY ONE WE COLLECTED BIRTH WEIGHT. SO IF YOU DIDN'T DO A BIRTH VISIT YOU DON'T HAVE BIRTH WEIGHTS IN THE DATABASE WHICH IS A REAL PROBLEM. SO WE DID BIRTH VISITS AFTER TEN DAYS TO GET THE INFORMATION WE THOUGHT IMPORTANT ABOUT THE BIRTH. WE PROPOSED TO EXPAND WINDOW TO ALLOW DATA COLLECTION AT ALL TIMES. THAT WAS DONE CLEARLY. SO EVERYTHING WAS CONSECUTIVE, ANY POINT IN TIME THERE IS A WINDOW WHICH A BABY FITS FOR DATA COLLECTION. THAT WAS A VERY IMPORTANT DEVELOPMENT. THESE ARE BABYS WE EXCLUDED FROM VARIABLE POPULATION SO MISCARRIAGES OR LATE FETAL DEMISE, INFANT DEATH, INFANTS GIVEN UP FOR ADOPTION, SO I CAN GIVE A FLAVOR OF THIS BY GIVING NUMBERS FROM BAKER COUNTY TO SHOW YOU WHAT HAPPENED IF PERMISSIBLE. >> I THINK JUST IF YOU COULD TOUCH ON THE CONCLUSIONS BECAUSE THE NUMBERS ARE RATHER SOFT AND THEY'RE NOT RIGOROUS. >> WE DEVELOPED RETENTION TRACKING TOOL WHICH ASSISTS CENTERS TO KNOW ANY POINT IN TIME DESIGN ELEMENTS REQUIRE THE CENTERS TO PUT IN A MINIMAL NUMBER OF DATA ELEMENTS TO LOWER THE BURDEN, MAINTAIN EASILY, PROVIDE VISUAL GRAPHIC FEEDBACK AND BE ABLE TO SHOW SOME CHANGE OVER TIME YOU CAN LOOK AT VERY STRAIGHT FORWARD WAY. THIS IS A SLIDE THAT SHOWS YOU THE IMPORTANCE OF THIS SLIDE IS THIS IS A GLOBAL ASSESSMENT OF THE PROJECT. 17 CENTERS THAT CAME TOGETHER TO SHARE AGGREGATE DATA ABOUT RETENTION. THIS WAS OUR BASED ON OUR DEFINITION, THIS IS THE RETENTION RATE OF THREE MONTHS 6 MONTHS 9 MONTHS 12 MONTHS. WE TRACK THAT AND A MONTH LATER GO BACK IN AND ASK FOR REPEAT DATA AND CENTERS PROVIDE THAT AND WE LOOK TO SEE WHAT HAPPENED OVER TIME. YOU CAN SEE ON THE SLIDE BETWEEN THE FIRST DATA SETS WHERE CENTERS SEND DATA AND LAST DATA SET, THE RETENTION RATES EVERY VISIT TENDED TO GO UP. WHICH DOESN'T MEAN ANY SPECIFIC COIN PDSA HAD AN EFFECT BUT THERE IS ASSOCIATION BETWEEN DOING COIN PROJECT AS MUCH FOCUSING ON RETENTION AND MAKING PEOPLE AWARE OF THIS, AS SPECIFIC INTERVENTION WE DON'T KNOW BUT IT WAS FAVORABLE IMPROVEMENT >> WHAT WE ALSO NOTICED HERE IS THREE MONTHS VISIT IN EVERYBODY'S HANDS WAS A LITTLE BIT OF A PROBLEM, I'M A KNEE NATAL GIST, BABY DOCTOR, THIS IS NOT SURPRISING. THREE MONTHS AT THE TIME OF LIFE OF MOTHER AND CHILD, DIFFICULT TO THROW SOMETHING ELSE ON TOP OF WHAT'S GOING ON. BY SIX MONTHS PEOPLE SORTED THINGS OUT BETTER AND IT'S EASIER TO DO THAT. THIS SHOWS YOU AMONG TREE DIFFERENT ARS GROUPS TRENDS IN RETENTION, OVERALL RETENTION RATE, WHICH MEANS AT LEAST ONE OF TWO ELIGIBLE VISITS THERE HAVE DATA COLLECTION ON THE BOTTOM SHOWS YOU EHER, RECRUITED DOOR BY DOOR. RECRUITED DOOR TO DOOR. THAT'S AT THE LOWEST LEVEL. IN THE MIDDLE ARE PROVIDERS SO THESE ARE WOMEN IN THE PROVIDER OFFICE, OB OFFICE CONSENTING TO THE STUDY. AT THE TOP IS DIRECT OUTREACH POPULATION AND ONE CAN SPECULATE THAT IF A MOTHER RESPONDS TO A NON-FACE TO FACE DIRECT OUTREACH INITIATIVE SHE'S MORE LIKELY MOTIVATED AND RETAINED SOMEONE WHO YOU SORT OF FIND BY KNOCKING DOOR TO DOOR AND FOR SOME REASON THEY GIVE CONSENT SO THIS MADE GOOD SENSE TO US. THIS IS HOW WE TRACK IN OUR BAKER COUNTY, THIS IS HOW -- WHAT WE WOUND UP WITH JUNE 30th, WHEN WE ENDED THE ACTIVITY. YOU CAN SEE THIS IS SOMETHING PRINTED OUT FROM THE EXCEL SPREADSHEET THAT CHANGES DYNAMICALLY AS YOU PUT MANY THE ADDITIONAL DATA ELEMENT. WE HAD PRETTY GOOD SUCCESS THROUGH 12 MONTHS RETAINING MOTHERS. THIS IS RETENTION RATE OVER TIME. THIS IS AVAILABLE TO EVERY CENTER AND HELPS CENTER UNDERSTAND WHERE THEY WERE IN THE PROCESS. I SHOW FOR THAT REASON JUST KNOWING YOUR DATA, INDIVIDUAL CENTER SORT OF OPENINGS YOUR EYES WHAT'S GOING ON IN A WAY YOU CAN IDENTIFY BARRIERS AND IN FACT, SOLUTIONS. THIS WAS OUR -- THIS IS COMPLETELY SPURIOUS DATA BUT EXAMPLE IN THE MATRIX, THIS IS WHAT THE PEOPLE CAME IN TO WORK EVERY DAY AND LOOKED AT. THEY PULLED THIS UP, THEY WOULD SEE FOR EVERY LINE ITEM HERE FOR THE PARTICIPANTS WHAT WAS GOING ON. SO IT'S COLOR CODED SO THAT MEANS THAT YOUR GREEN SHADE VISITS COMPLETEDDED WITHIN THE WINDOW, YOUR BLACK SHADED BOX WITHIN THE DATE ARE MISSED WINDOWS, BLACK SHADED BOX WITHOUT A DATE ARE BEYOND THE WINDOW. THEN YOU GOT YOUR ORANGE AND RED COLORED THINGS SO YOU HAVE A VISIT IN ORANGE PENDING, THE WINDOWS OPEN AND THEN YOU HAVE THE RED WHICH IS RED FLAG, THIS IS WINDOWS GOING TO SHUT WITHIN 15 DAYS SO THIS IS WHERE YOU NEED TO PUT YOUR FOCUS ON TODAY OR THIS WEEK. THAT WAS HELPFUL FOR PEOPLE TOLL MANAGE THEIR THINGS. SO ONE THING FOR INSTANCE THE LA CENTER FOUND WHEN THEY LOOKED AT THIS AND WERE ABLE TO ACCESS THE INFORMATION IN REAL TIME, THEY NOTED VERY LOW THREE MONTH RETENTION RATE AND COULDN'T UPS WHY THIS WAS HAPPENING THAT I MEAN TO WENT BACK TO A COUPLE OF THINGS. COMPUTER ASSISTED TELEPHONE INTERVIEW FORMS WEREN'T UP LOADED IN THE TRACK GROUP SO THEY FIXED THAT HOLE. AND THEN THEY FOUND OUT THAT THEIR EARLY AWARENESS SYSTEM WASN'T FUNCTIONING CORRECTLY AND THEY WERE GETTING NOTICEED THAT THE WINDOW WAS OPEN UNTIL ONE MONTH INTO THE WINDOW. THREE MONTHS INSTEAD OF TWO MONTHS. SO THEY FIXED THAT PROBLEM, THEY PAID MORE VIGILANCE TO THIS, OVER THREE MONTHS THEY DOUBLED THEIR THREE MONTHS RETENTION RATE OR RETENTION VISIT RATE FROM 32% TO 60%. THIS IS A BY-PRODUCT OF KNOWING WHERE YOU WERE. AND LOOKING AT ROOT CAUSES. SOUNDS SIMPLE BUT THIS CAPABILITY IS SOMETHING THAT WAS NOT OBTAINED AT EACH CENTER. IN TERMS OF RECKON SENT PROCESS THIS IS ALSO VERY IMPORTANT. I DON'T KNOW THE COMMITTEE IS FAMILIAR WITH ALL THE DIFFERENT VARIABILITY HOW THE TRANSITION FROM STUDY CENTER TO ROCK IS OCCURRING BUT THERE ARE A LOT OF SPECIFIC DIFFERENCES IN TIMING AND MECHANICS THAT HAVE TO DO WITH STUDY CENTER ROCK RELATIONSHIP AND HOW THAT CENTER IS ADJUDICATED WITHIN THE GOVERNING IRB. SO THERE ARE A NUMBER OF CENTERS OUT THERE NOT UNDER THE NIH IRB BUT UNDER THEIR OWN AREA SPECIFIC IRB. AND THAT HAS CHANGED DYNAMICS. A BIT. SO WHAT WE LIKE TO LOOK AT IS WHAT ARE THE INFLUENCES OF THESE FACTORS AND MECHANICS OF TIMING ON THE TRANSITION ON THE RECKON SENT RATE. SO THE OPERATIONAL CHARACTERISTICS WE CAN QUANTITATE ARE WHETHER OR NOT THESE MOTHERS WERE PREPARED FOR THE TRANSITION AHEAD OF TIME WHETHER OR NOT RECKON SENT WAS OPPORTUNITY -- RECKON SENT WAS DONE -- RE-CONSENT FROM SOMEBODY WITH A ROCK OR WHETHER DONE SEPARATELY.’&‡ FOR INSTANCE WE HAVE TO HAVE -- OUR IRB HAS TO SAY OUR STUDY CENTER FOLK GETS AUTHORIZATION TO TRANSMIT THE PERSONAL INFORMATION TO THE ROCK AND THEN THE ROCK HAS TO DO THAT CONSENT OR RE-CONSENT. THAT'S NOT THE SAME AT ALL SITES. THE DURATION OF TIME ELAPSED BETWEEN THE LAST CONTACT BY THE STUDY CENTER AND FIRST ATTEMPT BY ROCK TO RE-CONSENT, DURATION BETWEEN IN FIRST CONTACT BY THE ROCK FOR RE-CONSENT. THIS IDENTIFIES A FEW MORE IN TERMS OF STUDY VISIT VARIABLES IN TERMS OF WHETHER OR NOT THIS IS A PHONE RE-CONSENT OR FACE TO FACE, WHETHER EXTRA CONTACT OR TAKES PLACE DURING A REGULAR DATA COLLECTION VISIT. THE NUMBER OF CONTACTS REQUIRED TO ACHIEVE THAT PHONE OR FACE TO FACE INTERVIEW, I THINK THESE THINGS WILL GO FORWARD AND EVOLVE ANY FUTURE RETENTION ANALYSIS PROJECTS. >> CAN YOU SEPARATE OUT, THERE'S TWO DIFFERENT ISSUES. THERE'S THE CONSENT ISSUE, WHICH ONE WOULD HOPE WE CAN FIND A WAY AROUND. BUT IN THE WORST CASE WHAT WE'RE TALK ABOUT IS SOMEBODY WHO MOVES AND RE-CONSENT MAYBE PART OF THE PROCESS BUT I WOULD LIKE TO KNOW IF WE CAN GET AWAY FROM THAT. THEN THERE'S HAND OFF ISSUE. WHETHER THERE'S A NEED FOR RE-CONSENT OR NOT, THE PERSON WHO YOU DEVELOPD THE WONDERFUL RELATIONSHIP WITH THE PAST EIGHT YEARS SAYS I CAN'T FOLLOW YOU WHEN YOU MOVE FROM ST. LOUIS TO LA. AND HAS NO HAND-OFF TO THE PERSON THAT'S GOING TO BE CONTACTING VISITING AND RETAINING THEM IN LA. THAT SEEMS LIKE A POINT WHERE THERE'S SIGNIFICANT RISK OF LOSS. CURIOUS HOW MUCH ISSUE IS REALLY CONSENT. AND HOW MUCH OF THE ISSUE IS REALLY JUST HAND-OFF. THAT YOU'RE GOING TO DEAL WITH NEW PEOPLE NOW, THAT'S AS GOOD A TIME AS ANY TO POTENTIALLY DECIDE, I DIDN'T REALLY WANT TO DO THIS ANYWAY. >> STATISTICALLY MOST OF IT IS -- IF I UNDERSTAND YOUR QUESTION CORRECTLY, MOST PARTICIPANTS ARE TABLE IN THEIR COMMUNITIES. SO IT IS A HAND-OFF THAT OCCURS. BUT CONSENT IS OBTAINED BY DEPENDING ON IRB BY STUDY CENTER PERSONNEL AND HANDED OFF TO PERSON ON THE ROCK WHO IS RECEIVING THE CARE OR THE ROCK PERSON MAY NEED TO DO THAT EITHER PRIMARILY OR IN CONJUNCTION WITH THE STUDY CENTER PERSONNEL. >> MAYBE I SHOULD BE MORE CLEAR. IS SIZE OF EFFECT OF INCREASED ATTRITION OF DIFFERENT VARIATIONS HOW THE CONSENT IS DONE ANYWHERE NEAR AS LARGE AS JUST THE OVERALL EFFECT OF SIMPLY HANDING SOMEBODY OFF FROM ONE PERSON IN STUDY CENTER TO ANOTHER PERSON EMPLOYED BY THE ROCK. OR IS THIS PLAYING AROUND THE EDGES, EDGES ARE IMPORTANT IN WHAT WE'RE DOING AND THE BIG THING IS TO FIGURE OUT HOW WE MINIMIZE THE NUMBER OF HAND OFFS EVEN IF THAT MEANS FINDING WAYS AND FUNDING TO MAINTAIN A GOOD RELATIONSHIP AT A DISTANCE AND THINK ABOUT LONG HAND-OFFS WHERE THE HAND-OFF PERIOD MIGHT BE A YEAR OR LONGER THAT YOU MAINTAIN CONTACT WITH SOMEBODY AFTER THE HAND OFF OCCURS. >> I CAN'T ADDRESS MOST CONTRACTUAL ISSUES BUT I WILL SAY THAT THE POINT OF THIS IS TO BASICALLY TRY TO STUDY, UNDERSTAND THOSE ISSUES. SO BY LOOKING AT THE DIFFERENCES IN STEP FUNCTION, IFAL WITH RETENTION RATES ASSOCIATED WITH THE TRAP SITION LOOKING AT OPERATIONAL AND TIMING FACTORS AN SEEING WHICH REALLY ADD, THAT'S AS MUCH AS I CAN SAY, DR. HERSCHFELD MAY WANT TO ADDRESS OTHER QUESTIONS YOU BROUGHT UP. >> I THINK IN THE INTEREST OF TIME, WE'LL KEEP MOVING BUT THE SHORT RESPONSE IS THAT WE HAVE CENTRALIZED POLICIES AND PROCEDURES TO TRY TO GUIDE THE HAND OFF PART AND RE-CONSENT PART. WE ARE PUTTING TOGETHER DATA PACKAGES TO ADDRESS EXACTLY POINTS YOU'RE RAISING ABOUT EFFECT SIZE AND METHODS. (OFF MIC) >> NO. WE NEED RE-CONSENT AT THIS POINT ONCE OR TWICE, DECADE WHENEVER NEW CONTRACTORS COME INTO PLACE. >> OTHER EXAMPLES THE CONSENSUS TO THE STUDY HAS NOTHING TO DO WITH DATA COLLECTORS. >> SO DOES THIS MEAN BECAUSE IT WAS RE-CONSENT, THAT PERHAPS WE ARE NOT GOING TO LEARN TOO MUCH ABOUT THIS FROM THE EARLY PARTS OF THE VANGUARD STUDY? >> I CERTAINLY HOPE NOT. I THINK THE KEY IS TO LOOK AT RETENTION, RE-CONSENT WITHOUT GOING INTO FURTHER DETAIL ABOUT IT IS WE INITIALLY HAD A LOT OF LOCAL IRBs OVERSEEING THE HUMAN SUBJECT PROTECTION. NOW WE CONSOLIDATED THAT. AT THIS -- WHEN WE FINISH WITH OUR TRANSITION PROCESS, THERE WILL BE ONE IRB. THE NICHD INTRAMURAL IRB WHICH WILL HAVE HUMAN PROTECTION RESPONSIBILITY ACROSS THE ENTIRE SYSTEM. >> SO I THINK I HAVE ONE LAST SLIDE. I THINK JUST IN SUMMARY IN TERMS OF WHAT THE VALUE I THINK THE NCS COIN ACTIVITIES HAVE BEEN, CLEARLY I THINK THESE ACTIVITIES SUPPORTED AN ENVIRONMENT OF SHARED LEARNING. AND IMPROVES THE GLOBAL KNOWLEDGE BASE ACROSS THE STUDY CENTERS. WE HAD PRODUCTIVE COLLABORATIONS THAT RESULTED. WE HAD WHAT I CALL ROBUST INPUT WITH A LOT OF DIFFERENT PEOPLE WITH DIFFERENT GRAB GROUNDS, EXPERIENCE CONTRIBUTING TO -- BACKGROUNDS, CONTRIBUTING TO INNOVATIONS AN SOLUTIONS. THE COIN ITSELF PROVIDED TRAINING FOR STUDY SITE STAFF AND NOW FOR THE ROCKS. AND PROVIDES ADDITIONAL DOCUMENTS AND TOOLS PRODUCES FEEDBACK USING CONTINUOUS MEASUREMENTS FOR THE PDSAs, MEANT TO SHORTEN THE TIME WHEN WE INTRODUCE CHANGES IN A FEW SITES AND ADOPT THEM IN MANY SITES, I THINK FACILITATES THE SPREAD OF SUCCESSFUL STRATEGIES ACROSS SITES. SO I THINK IT'S BEEN A VERY WORTHWHILE ENDEAVOR. SO I WILL END THERE. >> THANK YOU VERY MUCH. I WONDER IF ANY OF THE QUESTIONS POSED TO US COULD BE ANSWERED YOUR EXPERIENCE FOR EXAMPLE VARIOUS SUBPOPULATIONS THAT YOU HAD PROBLEMS WITH OR WHAT YOU FELT THE GREATEST BARRIERS WERE. >> I THINK ONE OF THE THINGS WE[ LEARNED IS THAT WE HAVE TO BE CAUTIOUS ABOUT DEFINING SUBPOPULATIONS, THE DEEPER WE LOOKED WE FOUND A LOT OF GEOGRAPHIC VARIATION AND WHAT WAS THE RISK FACTOR IN ONE AREA MIGHT NOT BE IN ANOTHER SO I THINK THE IMPORTANCE OF REALLY USING WHATEVER DATA WE HAVE TO CONSTANTLY MONITOR THAT AND SEE WHAT THOSE -- LOOK AT WAIVER RETENTION RATES OR MISSED VISITS OR NUMBER OF CONTACTS IT'S TAKING TO UNDERSTAND THE RELATIONSHIP BETWEEN DIFFICULT TO SCHEDULE STUDY VISIT AND VARIABLES. AGAIN, TO SEE HOW THAT CHANGES OVER TIME. BECAUSE IT WILL CHANGE OVER TIME AS CHILDREN GET OLDER, PARENTS GET OLDER. AND THE ENVIRONMENT CHANGES. >> SO LIKE TO OPEN UP THESE QUESTIONS TO OTHER PEOPLE SITTING AROUND THE TABLE, OTHERS IN THE ROOM, MANY PEOPLE HAVE HAD EXPERIENCE WITH LONGITUDINAL STUDIES AND I WONDER IF WE CAN FOCUS A WHILE ON THE FIRST QUESTION WHICH IS WHAT MAJOR BARRIERS DO YOU THINK NCS WILL FACE WITH REGARD TO RETENTION. I LIKE TO SPEAK FIRST. THAT IS, IF WE DON'T HAVE THE GOOD DATABASE WE'RE GOING TO BE IN VERY BAD TROUBLE. THE TRACKING THE DATABASE IS REALLY KEY TO WHAT HAPPENED. >> THE DATABASE FITS THE REQUISITE CALLS, AS YOU, PREACHING TO THE CHOIR YOU KNOW MORE THAN ANYONE, THE VARIABILITY WITH WHICH INTERVIEWERS RECORD THAT RECORD OF CALLS, WITH THAT KIND OF GETS OUT OF SORTS. >> SO THAT REQUIRES TRAINING FOR HOW TO FILL OUT RECORD OF CALLS THAT THEY NEED TO NEED TO BE ATTUNED TO WHAT THEY WERE SAYING OR TALKING ABOUT THEY MAY HAVE TO VISIT THEIR GRANDMOTHER, DO SOMETHING. THOSE ARE NOTES THAT MUST BE RECORDED IN RECORD OF CALL AND GIVE HINTS WHAT SHOULD HAPPEN IN THE FUTURE. >> ONE STRATEGY WORTH EMPLOYING THERE IS TO HAVE DUMMY SUBJECTS THAT ARE FEEDING DATA IN AND THE DATA CAN BE CENTRALLY MONITORED TO HAVE THE QUALITY OF THE FIELD WORK TO SEE WHEN DUMMY SUBJECT THINKS I'M VISITING MY COUSINS IN SPAINK DOES THAT START A DATABASE? WE'RE GETTING RID OF CELL PHONES DID THAT SHOW UP. THIS IS A COMMON STRATEGY IN ANY CUSTOMER SERVICE MARKETING AREA. YOU EMPLOY PEOPLE WHOSE JOB IS TO BE SMART CUSTOMERS WHO COME BACK FOR THE SCALE WE'RE TALKING ABOUT, THAT MIGHT BE A WORTHWHILE STRATEGY HERE. >> ONE EXPENCESIVE LESSONS IS HETEROGENEITY RULE IT IS DAY COMES TO RESPOND ATTENTION. THERE ARE SOME RESPONSE FOR WHOM THE ONLY HINT IS DIAL NUMBER CORRECTLY. WE'LL DO THE INTERVIEW. THERE ARE OTHERS THAT NO MATTER WHAT YOU DO, THEY'RE NOT GOING TO DO IT. YOU MIGHT GET A FIRST INTERVIEW BUT AFTER THAT, THEY'RE OUT OF HERE. THE PROBLEM HETEROGENEITY DRIVES THE REALITY THE THAT VARIATION AND COST PER CASE FOR EASY VERSUS HARD ONES CAN BE QUITE ALARMING. A FACTOR OF TEN OR MORE. SO YOU HAVE TO MAKE THIS HARD DECISION OF -- IF I GET THIS RESPOND SENT THIS IS COSTING ME TEN TIMES AS MUCH AS -- HOW DO I SPEND MY MONEY. UNLESS YOU HAVE THIS SOCIETY HEART DEAL WITH CONGRESS THAT I KEEP HEARING ABOUT AT SOME POINT YOU HAVE TO DECIDE WHERE YOU SPEND YOUR MONEY BECAUSE THEY'RE NASTY PEOPLE. >> I THINK THAT'S ONE OF THE HINGES THAT JENNIFER WAS ALLUDING TO, THERE HAS TO BE SOME NATIONAL STANDARDS. MAX CALLS ARE ONE THING THAT USUALLY YOU USE. SO WHILE YOU HAVE VARIABILITY OF RESPONDENTS AND FIELD STAFF, THERE NEED TO BE SOME STANDARDS IMPOSED IN THE FIELD. >> LET ME PROVIDE A LITTLE INSURANCE, WE HAVE A MAX CALL POLICY NUMBER ONE. NUMBER TWO IS WE ARE ENGAGING A -- WE HAVEN'T GONE INTO THE DUMMY DATA MODEL. I'M FAMILIAR WITH THE MODEL, USED IT IN OTHER SETTINGS BUT IN THIS CASE WE HAVEN'T. WHAT WE HAVE DONE IS WE HAVE A INDEPENDENT QUALITY CONTROL CONTRACTOR THAT CHECKS AND SELECTS RECORDS FOR VERACITY AND COMPLETENESS SO IT'S A DIFFERENT APPROACH BUT WE'LL KEEP -- BE OPEN TO ALL OPTIONS. QUALITY ON PROCESS AND COMPLETENESS OF THE DATA COLLECTION. >> ARE WE GOING TO FOLLOW PEOPLE BEYOND 50-MILES? >> WE'RE FOLLOWING PEOPLE, WHEREVER THEY MOVE IN THE CONTINENTAL U.S. WHICH IS ONE OF THE -- FOR US, WAS A CONSIDERATION HAVING REGIONAL RATHER THAN P LOCAL ORGANIZATION WE HAVE HAD PEOPLE WHO CAN GO FROM ONE LOCAL TO ANOTHER LOCAL. SO WHEN THEY -- EXAMPLE GIVEN EARLIER, SOME WENT TO LOUISIANA, WE HAVE LOCATION IN NEW ORLEANS BUT WITH REGIONAL APPROACH, WE HAVE FLEXIBILITY AND CERTAINLY THE CONTRACTORS ARE COMMITTED AS PART OF TASK ORDERS TO FOLLOW PEOPLE WHEREVER THEY END UP. WE EXPECT BASED ON SOME PROJECTIONS MADE THAT WITHIN 20 YEARS, ROUGHLY 20% OF PEOPLE WILL BE WHERE THEY START AND THE OTHER 80% WILL BE ELSEWHERE. WE EVEN HAVE SHOULD IT OCCUR WE HAVE A PARTNER STUDY WE MIGHT BE ABLE TO GET SOME CONTINUITY THERE. BURR THAT WE HAVEN'T TESTED. WE HAVE DISCUSSED IT THOUGH. >> EXCELLENT. WHAT ABOUT THE IDEA OF IN ADDITION TO FOLLOWING PEOPLE, IF A PERSON CHANGES STATUS, CHANGES PARENTS, ARE THERE PROTOCOLS FOR THAT IF THEY ARE ADOPTED, IF THEY BECOME A WARD, IF THEY GO TO JAIL IN LATER YEARS OR TOO FAR I WAY TO THINK ABOUT NOW? >> NOT TOO FAR AWAY TO THINK ABOUT. WE HAVE SOME PROTOCOLS BUT I WILL SAY THERE'S A LOT OF VARIATIONS ON WHAT CONSTITUTES FAMILY STRUCTURE IN THE U.S. AND MAYBE WE HAVEN'T COVERED ALL OF THEM BUT WE CANNOT COVER ALL OF THEM THEM. >> GREAT, JENNIFER. >> QUICKLY, WHAT YOU'RE TALKING ABOUT SOUND SIMILAR TO THE PARADATA RESPONSIVE DECIDE, DATA QUESTION A LITTLE DIFFERENT BECAUSE IT'S LONG NUDE TALL. -- LONGITUDINAL. THAT PUSHD TO AUTOMATE THIS BASIC INFORMATION ACTING ACTIVITY AROUND COLLECTING ITSELF KAYLY WEEKLY MONTHLY REPORTS, TARGET HOW YOU'RE GOING TO DO YOUR NEXT CONTACT WITH AT ONE TIME, WHAT KIND OF FOLLOW UP, WE'LL GET THEM AND COMET8h– BACK, USING WHATEVER INFORMATION YOU HAVE AND IT IS A FLUID FIELD BUT IT'S CERTAINLY TAKING OFF. YOU MIGHT BE ABLE TO USE SOME OF THE METHODOLOGY AND SOFTWARE BEING DEVELOPEDDED TO TAKE ADVANTAGE OF THAT. AND AUTOMATE DATA COLLECTION THEMSELVES. ENTERED AUTOMATICALLY GET AS MUCH AS YOU CAN THAT WAY RATHER THAN DIRECT ENTRY. >> IN TERMS OF OUR TIME BUDGET THE LAST ITEM ON THE AGENDA WE DID RIGHT AFTER LUNCH SO WE HAVE GOT THAT OUT OF THE WAY. SO I HAVE HEARD EXPRESENTATIONS OF INTEREST IN A BREAK BUT BEING ONE OF THE OLDER GUYS IN THE ROOM, IF I CAN MAKE IT YOU CAN MAKE IT. >> ANYONE WHO NEEDS TO LEAVE WE CAN TAKE A ROLLING BREAK OR MAYBE A SHORT BREAK. BUT I WOULD PROPOSE MANY OF OUR GUESTS HAVE BEEN HERE FOR THE GREATER PART OF THE DAY. SO HOW ABOUT A QUICK FIVE MINUTE BREAK, WE'LL COME BACK, WE'LL TAKE COMMENTS FROM OTHER OBSERVERS AND THEN THAT WILL GIVE JOE SOME TIME TO COLLECT HIS THOUGHTS. AND WE'LL HAVE JOE GIVE HIS REPORT, COMMENTS ON THAT AND THEN WE CAN WRAP UP BECAUSE I KNOW THERE ARE PEOPLE WHO HAVE 4 O'CLOCK CABS. DOES THAT SOUND LIKE A PLAN? OKAY. FIVE MINUTES. AT THIS POINT I WOULD LIKE TO OPEN THE MEETING UP TO COMMENTS OR QUESTIONS, CLARIFICATIONS FROM THE FLOOR. SOMETHING ASKED A QUESTION, GIVE JOE A BREAK HERE. HE'S TRYING TO GATHER HIS THOUGHTS TOGETHER. HE'S READY? GO AHEAD, ASK YOUR -- USE THE MICROPHONE, I WAS JUST PLAYING WITH YOU. (OFF MIC) >> IF ONE HAS ANYTHING THAT'S AN ACCEPTABLE NON-RESPONSE. REFUSAL IS A REJIT MITT DISTRIBUTION CODE. >> ALL RIGHT. THANK YOU VERY MUCH. JOE, PLEASE. >> OKAY. THE RESEARCH CHALLENGES WE FACE IS VERY BOLD, STUDY FROM BIRTH UNTIL 21 YEARS OLD. GATHER DATA ON ENVIRONMENT WELL BEING AN HEALTH WITHOUT OVERLY TAPPING OUR NATIONAL WEALTH. THOUSANDS OF RESEARCHERS AWAIT OUR DATA, PLANNING TO PUMP IT INTO TOOLS LIKE (INAUDIBLE) BUT EACH WANT SOMETHING DIFFERENT FROM AUTISM TO X-RAY EXPOSURE TO MODERATE AMONG REQUESTS WILL REQUIRE LEADERSHIP AND COME POW INSURE. PUTTING THE THE -- COME POE INSURE. PUTTING THE REST THE HIGHLIGHTS IN BRIEF NOT IN FULL. [APPLAUSE] >> IF YOU STILL FEEL THAT WAY LATER, I ORGANIZED TO EIGHT ISSUES NOT EQUAL SIZE, MANY -- THESE ARE ROUGHLY BUT NOT EXACTLY CHRONOLOGICAL BECAUSE I TRIED THE PULL THINGS BACK TOGETHER. RIGHT UP FRONT THE ISSUE CAME OUT OF PUBLIC ACCESS TO DATA. YOU HEARD THE NOTION THE PLAN FACILITY WITH OPEN ACCESS NON-PERSONALLY IDENTIFYING OR IDENTIFIABLE DATA, LINE ITEM DATA BUT ONLY IMPROVED ACCESS FOR SPECIFIC LINE UP ITEM DATA, THERE WAS A LOT OF ADVOCACY FOR AS MUCH OPEN ACCESS AS POSSIBLE FOR MANY REASONS INCLUDING JUST GETTING THE SUPPORT IN THE RESEARCH COMMUNITY, SOME MIGHT BE MOTIVATED TO DEFEND A STUDY IF THERE ARE TIMES WHEN BUDGETS ARE THREATENED. BUT FIGURING WHAT THIS MEANS IS A CHALLENGE. NOTHING IS PERFECTLY NON-IDENTIFIENING PRESENCE OF OTHER COLLECTIONSES OF DATA. THIS ISSUE NEEDS TO BE PUT ON THE AGENDA GOING FORWARD. ISSUE TWO WAS NICE AND SMALL. HOW WELL DID WE LEARN FROM PILOT EFFORTS, TIMING LOGISTICS AND DATA AND STUDY QUESTIONS. WE CAME AWAY ASSUREDDED THERE ARE HUNDREDS OF SUCH ELEMENTS OF OPERATIONAL DATA AND WE DIDN'T GO INTO TREMENDOUS DETAIL. THE THIRD THAT CAME UP WAS THIS CHALLENGE OF ADDRESSING RACIAL ETHNIC ECONOMIC AND OTHER DISPARITIES. PART OF THIS HELPS BY GOING TO HOSPITALS AND BIRTHING CENTERS BECAUSE OF THE OVERALL LARGE PERCENTAGE OF BIRTHS THAT OCCUR IN THESE, THERE ARE COMMUNITIES DEALT WITH SEPARATELY. THERE IS TRADE OFF BETWEEN EARLY AND LATE PREGNANCY RECRUITMENT INCLUDING BIRTH RECRUITMENT AND DIVERSITY. THE EARLIER YOU GO THE MORE YOU HIT PEOPLE WITH BETTER HEALTH COVERAGE. THE WHOLE SUPPLEMENTAL RECRUITMENT COHORTS. THE POINT WAS MADE THIS IS NOT JUST A ISSUE IN RECRUITMENT, IT'S ALSO A HUGE RETENTION ISSUE. THAT FIGURING OUT HOW TO RETAIN THE GROUPS IN ROUGH PROPORTION TO THEIR REPRESENTATIVENESS IN THE POPULATION IS NONTRIVIAL. FOUR, GAMING DATA ON PRE-NATAL EXPOSURE WHICH IS AN ISSUE THAT THE ADVISORY COMMITTEE HAS RAISED AS AN AREA OF CONCERN, EVERYBODY IS CONCERNED. THERE'S A BUNCH OF STRATEGIES HERE. P YOU HEARD THE MAGIC NUMBERS THE 45,000 RECRUITS DURING PREGNANCY, THE 15 TO 20,000 SIBLING BITS THAT GET DATA EARLY BUT NOT NECESSARILY. YOUTH THAT WILL BE STUDIED THROUGH AGE 21. PERHAPS 5,000 OR SO IN A PRECONCEPTION SUPPLEMENTAL COHORT. THERE ARE CHALLENGES AND CONCERNS PARTICULARLY WITH ANY OF THESE HOW YOU ISOLATE BEHAVIOR CHANGE AS A RESULT OF PARENTHOOD VERSUS PARTICIPATION IN THE STUDY, IF YOU'RE TALKING ABOUT LATER SIBLINGS AND HOW YOU DEAL WITH THINGS THAT ARE CONFIRMTY VERSUS REPORTING ERROR WHEN TALKING ABOUT THINGS THAT ARE SURVEYS OF. SO HOW MUCH TOBACCO DID YOU HAVE IN THE HOUSE A MONTH BEFORE WE CONTACTED YOU. THERE'S A LOT OF QUESTIONS LIKE THAT. FIVE, THE ISSUE OF IOM REVIEW AND QUESTIONS FOR THE PANEL, ONE QUESTION WAS TO SPECIFICALLY ASK THE COMMITTEE TO LOOK AT RECRUITMENT GROUPS AN WHO THEY DO OR DON'T WORK TOGETHER TO ADDRESS VARIOUS WEAKNESSES OF THE COHORTS BEING RECRUITED. AT THIS POINT WE WERE GIVEN REQUEST FOR FEEDBACK ON ISSUES OF FRONT LOADING. CORE QUESTIONNAIRE AND TOPIC AND ROBUST PHENOTYPE, THE CO-QUESTIONNAIRE PIECE HAVE A LOT OF FEEDBACK. THE POINT WAS BROUGHT THAT, YEAH, DEVELOPMENT MOSTLY HAPPENS EARLY AND FROM THAT PERSPECTIVE THE IDEA OF KEEP THEM CLOSE TOGETHER MAKES SENSE. THERE WAS STILL A QUESTION WHAT ARE WE MISSING LATER, THE ONE THAT KEPT COMING UP IS PUBERTY, HAPPENS AFTER AGE 5 MOST KIDS,t IT MAY CHANGE BY THE TIME THE STUDY STARTS BUT NOW THAT SEEMS TO BE THE CASE. IF WE WERE DEALING WITH, I DIDN'T WRITE THAT DOWN, THAT'S AD LIBBED. IF WE'RE DEALING WITH HAVING TO SHIFT WITH OTHER SOURCES OF INFORMATION, WHAT DO WE LOSE? THERE ARE CERTAIN THINGS PEOPLE WERE NOT COMFORTABLE WITH THE IDEA IT REMEMBERED ACCURATELY 23, 24 MONTHS LATER. THERE'S IMPACT ON ABILITY TO MEASURE CERTAIN KINDS OF BEHAVIORS AND CERTAIN ISSUES. BULLYING, SEXUAL HEALTH, CYBER ANYTHING RIGHT NOW. AND QUESTION WHETHER SUPPLEMENTAL ONLINE STUDIES MIGHT BE THERE AND WE HEARD THE ANSWER THAT SES WOULD LIKE TO BE A PLATFORM THAT DOES THIS BUT I HEARD LATER THE QUESTION, THE PLATFORM OTHER PEOPLE CAN ADMINISTER STUDY, WHAT'S THE RISK TO THE CORE MISSION, IF YOU HAVE THAT, PEOPLE PERCEIVE THAT AS INCREASED IN BURDEN, THEY PERCEIVE THAT AS ANOTHER VENUE THAT'S LINKED TO THE STUDY ITSELF AND IF THEY HAVE A BAD EXPERIENCE OF SUPPLEMENTAL QUESTIONNAIRE DO THEY USE THAT AS AN EXCUSE ABANDON THE MAIN STUDY. THERE'S CHALLENGES EVEN AT THE PLATFORM IDEA SEEMS LIKE A NICE IDEA. ARE WE TAKING ADEQUATE NOTE OF FATHERS, MEASUREMENTS OF PARENTS IN GENERAL, INTELLECTUAL CAPACITY, PHYSICAL, VARIOUS THINGS. WE GOT BACK THE MESSAGE THAT WE'RE ADDRESSING FATHERS AND MOTHERS BUT ONLY WITHIN THE CONTEXT OF THE CHILD. THE PRIMARY FOCUS IS CHILD. YOU GET MORE FOCUS ON MOTHERS DURING PREGNANCY BECAUSE THAT'S SORT OF DIRECT ROOT TO THE. BUT REALLY THE FOCUS IS ON PARENTS IN AREAS THEY MOST CREATE THE ENVIRONMENT AND SUPPORT DEVELOPMENT OF THE CHILD, OR SHOULD SAY HELP THEM DEVELOP IT. ALSO THE REMINDER CAME OUT AT THIS POINT VANGUARD STUDIES ENCOUNTERS ISSUE AS WE FIGURE OUT SOMETHING IS THERE, SOMETHING IS NOT THERE, THAT APPLIES NOT JUST TO PARENTS BUT TO ALL ISSUES. THE HOPE IS WE'RE SWIFT REACTING BECAUSE GOVERNMENT IF ONE THING IS NOT SWIFT. WE ALL KNOW THIS. WHAT IS THE CORE PHENOTYPE DATA. THIS WAS AN INTERESTING CONVERSATION. THERE WAS EMPHASIS ON THE FACT WE DON'T HAVE A COMMON TERMINOLOGY, THERE WAS A SEPARATE VALUE FOR INVENTORY SYMPTOMS APART FROM THE VALUE OF THE ANALYST CODING OF THEM BECAUSE WHAT PEOPLE CALL SOMETHING CHANGES. THERE IS -- THE POINT MADE WHEN WE COLLECT DATA ON OUTCOMES THAT ARE OBSERVED WE GET IN TROUBLE IF WE DON'T FACTOR IN COMMUNITY FAMILIES SCHOOL, SIBLINGS AND OTHER INPUTS. THERE WAS DISCUSSION OF TRIGGERS WENT OFF IN MANY DIRECTION. A CERTAIN PIECE OF INFORMATION, A CERTAIN CONDITIONING THERES SUPPLEMENTAL INVESTIGATION AND THE HOLE QUESTION, WHEN ARE TRIGGERS CONFUSING CHOICES AN PREFERENCES WITH ENVIRONMENTAL AND GENETIC CAUSES AND MIGHT WE BY MISTAKENLY GOING TO WRONG PATH DOWN A TRIGGER MOVE FROM SOMETHING APPROXIMATING RANDOM SAMPLING TO CHOICE BASED SAMPLING. SOME OF THAT TOO WAS VOCABULARY. WHEN IS A TRIGGER REALLY JUST LOOKING AT QUESTIONS THAT DON'T MAKE SENSE, BRANCH LOGIC SURVEY AND WHEN IS A TRIGGER DEEPLY INVESTIGATING SOMETHING THAT COULD BE INVESTIGATING -- INVESTIGATED IN EVERYONE. TO THE EXTENT WHAT WE MEANT HERE IS HERE IS SOMETHING MORE DEPTH FOR PEOPLE MORE RELEVANT. THE SUGGESTION WAS MADE EVEN IF YOU WANT TO DO THAT YOU HAVE TO RANDOMLY ADMINISTER THAT UNIT, THAT MODULE TO SOME SET OF PEOPLE SO YOU CAN NORMALIZE AND UNDERSTAND EVEN IF YOU OVERSAMPLE PEOPLE WHO FIT THE TRIGGER. THAT LENGTHENED THE DISCUSSION THAT BOY, THIS IS COMPLEX AND THE KIND OF COMPLEXITY AND POTENTIAL CONFUSION OF TOO MUCH VARIATION SKIP LOGIC IS TRIGGER ING, THIS COULD GET REALLY HARD. IT COULD GET HARD FOR PEOPLE TO UNDERSTAND EVEN SOME SIMPLE THINGS LIKE WHO WAS ASKED THIS QUESTION WHICH SHOULDN'T BE HARD. PART OF WHAT NEEDS SAMPLING MATH TO COVER THE POSSIBILITIES AND MAYBE EVEN A SMALL SET OF OPTIONS FOR SAMPLINGS THAT WILL BE DONE TO AVOID THINGS THAT ARE INCOMPREHENSIBLE. TIDE TO THAT THE PRACTICE OF STORING RAW DATA FIGURE OUT WHAT ELSE GOLD STANDARD HERE. THERE ARE CERTAINLY CLEAR LOGIC THAT SAYS SKIP PATTERNS MAKE SENSE. YOU HAVE A MOLE, HOW BIG WAS IT? IF THEY SAY NO, WE DON'T ASK HOW BIG IT WAS. NOBODY WAS QUESTIONING THAT, BUT TO BE MORE CAREFUL ABOUT THE OTHER TRIGGERS. THERE MAYBE CERTAIN CASE IT IS TRIGGER IS GEOGRAPHIC. HURRICAN OCCURRED IN PLACE AND WE WANT TO DEAL WITH THE FOLLOW-UPMENT THERE MAYBE CENTRALLY MANAGED QUERIES „– LIKE THAT. BUT TOO MUCH OF THIS COULD BE A PROBLEM ET CETERA SPECIALLY WHEN YOU -- ESPECIALLY GETTING TO THE WADING NEEDEDDED WITH SUBSAMPLES AND REALLY GETS DOWN TO THE QUESTION CAN ANYBODY USE THE DATA WHEN IT'S DONE. THAT BROUGHT US TO DISCUSSION OF DRIVERS VERSUS OUTCOMES WHICH IS YOU DON'T WANT TO RANDOMIZE OUT DRIVERS BECAUSE THEY HURT YOUR MODEL. BUT IT IS NOT ALWAYS CLEAR WHAT IS WHAT. IF WE HAD THE PATIENTS AND TIME AND MONEY ANSWER IS COLLECT EVERYTHING FROM EVERYONE. THAT'S NOT AN OPTION. IT IS A REAL CHALLENGE. OUTPUTS ARE OUTPUTS AND ENVIRONMENT AND BEHAVIOR ARE INTERTWINED. ALL THIS STUFF IS TRICKY. THERE ARE THINGS THAT CAN BE DONE TO SIMPLIFY INCLUDING LOOKING AT WHEN YOU HAVE GOTTEN SOMETHING TO A SAMPLE SIZE IN A PARTICULAR RESPONSE CATEGORY SO SMALL IT'S NOT USEFUL ANYWAY. AT THAT POINT YOU DECIDE YOU WON'T GO ANYWHERE. THEN GOT INTO POINT THAT THE TIMING SCHEDULE, AGE 10, 11, 12, 13, CRITICAL FOR PUBERTY, DOING 11, 13, IS THAT ENOUGH, DO WE NEED SOMETHING MORE AT THIS POINT. I FORGET WHO BUT WE'RE NOT ATTRIBUTING POINTERS TO PHENOTYPIC ONTOLOGIES FOR INDICATOR QUESTIONS THAT MIGHT BE HELPFUL AND POINT TO DETAILED MODULES. THERE'S VALIDATED STANDARDIZED QUESTIONNAIRES OUT THERE. MORE REALLY ADDRESSING MORE DIRECTLY IMPACT OF MEDICARE, MEDICAID, THE HEALTHCARE REFORMS INCENTIVES ON ELECTRONIC HEALTH RECORDS. GOING TO BE A LOT MORE DATA OUT THERE USED EITHER DIRECTLY OR MORE LIKELY FOR CROSS VALIDATION OF HEALTH EVENTS. AND THE TIME FRAME, HOW DO WE KNOW AND WHAT WILL WE KNOW ABOUT DIFFERENCIAL ACCESS TO THOSE RECORDS BY POPULATION. AFTER LUNCH WE MOVED INTO A ROLE OF ADVISORY COMMITTEE ITSELF, REALLY WHAT CAME OUT OF THIS IS THE COMMITTEE NEEDS TO WORK PROACTIVELY TO GET ON TOP OF WHERE THE STUDY REALLY IS TODAY AT CERTAIN LEVEL DETAIL SCALE SAMPLING, HUMAN SUBJECTS AND WE NEED TO WORK TO SUPPORT THE PROGRAM AND HAVING THIS AS SUCCESSFUL AS POSSIBLE. IMMEDIATELY RAISED, UNDER FACA WHICH CAME BACK WITH WORKING GROUPS WITH A THICK SHORT DURATION TASK CAN BE DEALT WITH. WORKING GROUPS CAN EXIST. PUBLIC MEETINGS OF FULL COMMITTEE HAVE RULE AROUND THEM. THE QUESTION WAS RAISED WHAT'S PROVIDED TO THE IOM. IMAGINE THE SUCCESSOR TO THIS PACKET LOOKING SIMILAR BUT UPDATED AND THAT WILL BE MADE AVAILABLE AT LATER DATE WHEN ACTUALLY DONE. THE REQUEST FOR MORE EXAMPLES. THEN MOVED TO THE LOVELY TOPIC OF RETENTION. WE HAVE OVERVIEW STRATEGIES TO REDUCE ATTRITION, REALLY GOT INTO THE SUBTLETIES OF DIFFERENT REFUSALS AND TRACKING AND LOCATING AND SUPPORT FROM THE FEELS EXPERIENCED ABOUT DIFFERENCIAL INCENTIVES. I WILL NEVER AGAIN ANSWER ON THE FIRST REQUEST WHEN ANY OF YOU CONDUCT A STUDY OF ME BECAUSE I'LL HOLD OUT FOR A BETTER INCENTIVE NOW THAT I KNOW THIS. YOU LEARN SOMETHING EVERY DAY. WE GOT AN INVESTIGATORS PERSPECTIVE ON RETENTION WHICH STARTED WITH THE VERY SOBERING POINT THAT EVEN REALLY SMALL RATES OF NON-RETENTION PILE UP OVER THE COURSE OF 21 YEARS. AND YOU START SAYING WE'RE DOING 95%, AT THE END OF 95% YOU'RE AT ZERO. SO WE'RE TALKING ABOUT 99% WHICH IS A SCARY NUMBER. WE GOT THE BACKGROUND ON THE COLLABORATIVE INNOVATION AND IMPROVEMENT NETWORK FOCUSED ON RETENTION AND SOME OF THE STRATEGIES THEY FOUND TO BE SUCCESSFUL. VISUAL CONFIRMATION, STRATEGIES TO BEST RATE ATTRITION RISK. WE GOT INTO THE ISSUE HOW DO YOU JUDGE THE NEGATIVE OUTCOMES OF RETENTION STRATEGIES NOT RELATED TO RETENTION. WHAT HAPPENS IF THROUGH RETENTION YOU SAY I KNOW WHAT WE'LL DO. I WILL TAKE YOU OUT AND WE'LL DO ALL OF OUR SESSIONS AT MCDONALD'S AND THEN YOU DISCOVER THE PEOPLE IN THAT GROUP HAVE HIGHER CHOLESTEROL AND WEIGHTS THAN ONES YOU DIDN'T TAKE OUT TO MCDONALDS. HOW DO YOU DEAL WITH THINGS THAT CHANGE BEHAVIOR. PROBABLY NOT AN ISSUE AGE 7 TO 21 ONCE EVERY TWO YEARS BUT EVERY THREE MONTHS. WE DON'T KNOW. PARTICULARLY A SIX MONTH OLD. MCDONA 69 BE QUITE COMPELLING. THERE IS ALSO PERHAPS, IF ANYTHING THE OVERALL MESSAGE IS TAILORING, THE ABILITY OF INDIVIDUAL CONTACT TO TAKE THE STEPS NEEDED AT THE INDIVIDUAL LEVEL. NOTHING IS GOING TO WORK IF WE DON'T HAVE A GOOD CASE MANAGEMENT SYSTEM. THAT BRINGS BACK TRAINING AND QUALITY, HAVING THE MATERIALS THAT ALLOW PEOPLE TO UNDERSTAND WHAT'S ALLOWABLE BUT ALSO EDUCATION THAT ALLOWS THEM TO UNDERSTAND WHAT'S GOOD AND WHEN IT -- I'LL COME BACK TO THOSE, WE CAME BACK TO THEM. I'LL REMIND EVERYONE THE GREATEST ATTRITION IN THE VAN GUARD IN BIRTH AT LEAST SO FAR. FOR 45% OF THE POPULATION WE WILL SOLVE THAT PROBLEM. 55% WE HAVEN'T. REFUSING LOSING TRACK OF PEOPLE AND DIFFERENT STRATEGIES ADDRESS THESE DIFFERENTLY, IN BOTH CASES, ESPECIALLY WHAT IS USEFUL IT IS EASIER TO AVOID THAN TO CORRECT. IF YOU CAN TAKE ADVANTAGE OF TOOLS THAT IDENTIFY PEOPLE AT RISK TO DO SOMETHING TO MAKE PEOPLE MORE WELCOME, MORE VALUED, BETTER COMPENSATED, WHATEVER IT IS. THAT'S EASIER THAN GOING BACK AND E CLAIMING AFTER THEY MADE THE DECISION AND ARTICULATED IT. ED FROM QUESTIONS RAISED ABOUT CENTRAL MANAGEMENT METRICS, WE WE HEARD ABOUT PROJECT LOOKING AT DIFFERENT VERSIONS OF HAND OFF AND RE-CONSENT. THE RE-CONSENT WON'T BE THE SAME ISSUE WITH THE SAME IRB BUT HAND-OFF IS POTENTIAL ISSUE. THERE WERE QUESTIONS ASKED ABOUT SUBPOPULATIONS AND SPECIFIC BARRIERS TO RETENTION AND IT WASN'T THAT SIMPLE. PROBLEMS IN ONE AREA WEREN'T IN OTHERS. WE DON'T UNDERSTAND HOW MUCH AN AREA IS AN AREA. GEOGRAPHICALLY, HOW MUCH IT'S A STUDY CENTER, HOW MUCH IT'S A POP LACE, A LOT O FIGURE OUT. -- POPULATION, A LOT TO FIGURE OUT. THERE'S DIFFERENT STRATEGYS FROM THE INDEPENDENT QUALITY CONTROL TO DUMMY PARTICIPANTS TO WHIPS AND CHAINS TO GET GOOD DATA IN THE DATABASE. SO FAR, PROGRAM OFF THE AT THIS IS NOT ORDERED WHIPS BUT I UNDERSTAND THERE IS FUNDING AVAILABLE IF NEEDED. COULD BE USEFUL TO AUTOMATE METADATA COLLECTION, DATA ABOUT DATA COLLECTION AND AS MUCH DATA COLLECTION AS POSSIBLE. WE OPENED IT UP FOR PUBLIC COMMENTS. THERE WERE NONE. I WILL CONCLUDE WITH THE COMMITTEE WORKED HARD, THE COMMITTEE'S BRAIN HURTS. SOME OF US SHOULD SPEAK A WEEK IN (INAUDIBLE). SO MANY DISCUSSIONS AND COLLECTIONS PERHAPS WE'LL RETAIN THEM WITH HIGHEST CONFECTIONS. SOME WANT EVERY SURVEY TO LOOK THE SAME, IF WE ENTER THE MATRIX COMPLEXITY WE MUST TAME, BUT ALL OUR EFFORTS WILL BE NULL AND VOID IF PARTICIPANTS LEAVE AND ARE RESEARCH AVOID. I LACK MUSICAL TALENTS THAT'S WHY VERSE AND NOT TUNE, I WISH ALL OF YOU A MOST GOOD AFTERNOON. [APPLAUSE] >> THANK YOU VERY MUCH, JOE. WELL DONE.