>> GOOD MORNING, EVERYONE. I THINK IN JUST MOMENTS WE WILL BE LIVE TO WELL OVER 200 WEBINAR PARTICIPANTS. SO CONGRATULATIONS TO EVERYONE, I THINK THIS REALLY SHOWS YOU HOW WHAT YOU'RE DOING IS SO PHENOMENALLY IMPORTANT. I WANTED TO SAY GOOD MORNING, MY NAME2’ IS WENDY NILSEN FROM THE OFF FOIFS BEHAVIORAL AND SOCIAL SCIENCES RESEARCH HERE AT NIH. I CAN'T TELL YOU HOW GLAD I AM TO HAVE Y'ALL HERE TODAY AND FROM THE PLANNING COMMITTEE WE'RE KIND OF BUSTING OUT EXCITED. THIS IS GOING TO BE SUCH AN EXCITING STIMULATING INTERESTING DAY THAT WE HOPE REALLY MAKES SOME -- HELPS US CHANGE THE SCIENCE IN A WAY WE ALL THINK IT NEEDS TO BE CHANGED. COUPLE OF THINGS BEFORE WE START. THE RESTROOMS ARE OUTSIDE, FOR THOSE IN THE WEBINAR WE WILL BE TAKING BRIEF BREAKS, WE ARE GOING TO BE DRA CONE KENYAN ABOUT TIME SO THOSE ON THE WEBINAR CAN PLAN ON US BEING THERE WHEN WE SAY WE WILL BE THERE. THE PLANNING COMMITTEE WILL COME UP AND TELL YOU WHY THEY'RE HERE. BUT ONE THING I WANTED TO TELL YOU ABOUT IS THAT THE BOTTOM HERE IS TO WEIGH IN ON mHEALTH AT NIH, THE COMMON FUND WHICH IS A FUND THAT -- OUT OF OUR DIRECTOR'S OFFICE WHICH IS VERY IMPORTANT, IT'S TRANS-NIH, CUTTING-EDGE HIGH INNOVATION RESEARCH. RECENT HI THEY ASKED FOR PUBLIC COMMENTS. ONE OF THE CONCEPT AREAS THEY'RE LOOKING FORE FEEDBACK ON IS INNOVATIVE MOBILE AND WIRELESS TECHNOLOGY SO mHEALTH. WHAT'S REALLY IMPORTANT ABOUT THIS, I WANT THE PEOPLE IN THE WEBINAR TO THINK OF OTHER PEOPLE IN THE ROOM, PLEASE MAKE COMMENTS, PLEASE GIVE FEEDBACK. THERE ARE 26 AREAS, PLEASE GIVE ANY THAT YOU THINK ARE INTERESTING OR IMPORTANT OR NOT. GIVEN EVERYBODY'S INTEREST IN THE ROOM THIS IS AN IMPORTANT AREA FOR PEOPLE TO COMMENT ON. SO WE CAN GIVE YOU THE LINK, I'LL SEND IT OUT AGAIN BUT PLEASE COMMENT. SO WITH THAT, I REALLY YOU KNOW YOU'RE GETTING OLD WHEN THOSE NUMBERS GET TINY AT THE BOTTOM. WITHOUT FURTHER ADIEU I WOULD LIKE TO INTRODUCE THE AGENCIES IN THE PLANNING COMMITTEE. I WANTED TO START FIRST WITH MY BOSS. DR. ROBERT CAP PLAN, DIRECTOR OF THE OFFICE OF BEHAVIORAL AND SOCIAL SIGNS RESEARCH AT NIH. I CAN TELL YOU WONDERFUL THINGS ABOUT ALL THE PEOPLE TODAY BUT WHAT I WILL TELL YOU THAT WILL EXCITE YOU MOST ABOUT BOB HE'S TOTALLY PSYCHED ABOUT mHEALTH. WITH THAT I'LL LET HIM TELL YOU. [APPLAUSE] >> I WANT TO WELCOME YOU FROM THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCE RESEARCH. WE ARE PART OF THE NIH OFFICE OF DIRECTOR, ONE OF FOUR SUBSTANTIVE AREAS AT THE NIH OFFICE OF DISEASE PREVENTION, OFFICE OF WEM'S HEALTH RESEARCH AND OFFICE OF AIDS RESEARCH. TOGETHER WE TRY TO COORDINATE ACTIVITIES ACROSS THE NIH AND TRY TO REACH OUT TO COORDINATE ACTIVITIES WITH OTHER PUBLIC AND PRIVATE PARTNERS. SO WE'RE THRILLED TO DO THIS IN COLLABORATION WITH ROBERT WOOD JOHNSON FOUNDATION, WITH MCKESSON AND WITH NSF WHO IS TURNING OUT TO BE A VERY IMPORTANT PARTNER FOR US IN A VARIETY OF DIFFERENT ACTIVITIESCH ADS WENDY MENTIONED, I'M EXCITED ABOUT MHEALTH, IT'S SOMETHING THAT I'M JUST BEGINNING TO LEARN ABOUT. IT'S A WONDER ALL THESE GREAT TECHNOLOGIES, THINGS ARE EVOLVING SO QUICKLY. WE ALSO FEEL mHEALTH IS AN IMPORTANT PART OF NIH'S FUTURE. SO WITHIN THE NEXT MONTH OR SO THE DIRECTOR IS HOLDING A SPECIAL MEETING FOR THE INSTITUTE DIRECTORS, 27 INSTITUTES AND CENTERS AT THE NIH, WE'RE GOING TO FOCUS A BIG PART OF THAT PROGRAM ON MHEALTH AND WE'LL BE TALKING ABOUT THE THINGS I'LL WILL LEARNING ABOUT TODAY LISTENING TO YOU FOLKS. SO AGAIN, IT'S A GREAT PLEASURE TO BE HERE AS WENDY MENTIONED, THIS IS AN EXCITING GROUP AND AN EXCITING OPPORTUNITY FOR US. SO THANKS FOR COMING. >> NOW THE PLANNING COMMITTEE AS BOB SAID IS REALLY COMING TOGETHER BRINGING PRIET AND PUBLIC PARTNERSHIPS TOGETHER IN A WAY THAT MADE SENSE. SO MY NEXT -- OUR OTHER PARTNER IN THIS, A VERY IMPORTANT PARTNER IS THE ROBERT WOOD JOHNSON FOUNDATION, IT'S A PIONEER GROUP SO I WANTED TO INTRODUCE AL SHAR, AL IS AN AMAZING PERSON AND HE IS VISIONARY AND HE'S THE GUY WHO MADE THE FIRST PHONE CALL THAT SAID LET'S DO THIS. SO WITH THAT I'LL INTRODUCE AL. >> THANK YOU, WENDY. [APPLAUSE] >> WELCOME. THE PIONEER TEAM AT THE ROBERT WOOD JOHNSON FOUNDATION FOCUSES ON THE FUTURE SEEKING BREAK-THROUGHS WITH THE POTENTIAL TO GENERATE SIGNIFICANT IMPACT. WHEN WE BEGAN TO LOOK AT MOBILE HEALTH IT WAS CLEAR THIS WAS A FIELD WE WANTED TO BE IN. WHERE AND WHEN AND HOW IS THE QUESTION. WE BEGAN TO EXPLORE WHAT MIGHT MAKE A DIFFERENCE. LAST YEAR AT THE mHEALTH SUMMIT, I NOTICED A HUGE INCREASE IN INTEREST IN PARTICIPATION WITH REMARKABLY LITTLE EVIDENCE THAT ANYTHING ACTUALLY WORKED. I THOUGHT THAT MIGHT BE A PRODUCTIVE DIRECTION FOR US TO GO IN. AS I BEGAN TO LOOK FURTHER, WHILE SOME PEOPLE WERE TRYING TO DEMONSTRATE EVIDENCE IT SEEMED TO ME THAT MANY WERE INAPPROPRIATELY TRYING TO SHOE HORN THEIR RESEARCH INTO A RANDOMIZED CONTROL STUDY. THEY TENDED TO THINK THIS WAS THE GOLD STANDARD AN TO HAVE CREDIBILITY THEY HAD TO WIND UP USING IT. I DON'T KNOW TO DENIGRATE THE APPROPRIATE USE OF AN RCT BUT IT'S NOT THE END ALL OR BE ALL OF ALL RESEARCH. I USED TO SAY THE U.S. WENT OFF THE MONETARY GOLD STANDARD IN 1933 WITH GREAT RESULTS. I'M NOT SURE I SHOULD SAY THAT NOW. [LAUGHTER] >> OUR TASK TODAY IS TO LOOK AT LO LEGITIMATE VALID WAYS OF DEMONSTRATING EVIDENCE AND HELP ADVANCE WHAT IS AND WILL BE A VERY IMPORTANT CONTRIBUTION TO HEALTH AND HEALTHCARE. MY HOPE IS THAT WE CAN FIND AN ACTIONABLE AGENDA THAT CAN HELP US MOVE FORWARD. ONCE AGAIN, THANK YOU AND WELCOME. >> [APPLAUSE] >> NEXT I WANT TO INTRODUCE OUR PARTNER FROM MCKESSON. KERRY (INAUDIBLE) IS HERE, SHE'S PRESIDENT OF MCKESSON FOUNDATION. I BUMP INTO HER EVERYWHERE. EVERYBODY IS SAYING EVERY TIME I GO TO A MEETING SOMEBODY SAYS DO YOU KNOW KERRY AT MCKESSON? SHE'S VISIONARY SO IT'S IN THE A SURPRISE WE'RE SEEING HER HERE THIS MORNING. [APPLAUSE] HI, EVERYBODY. I'M A SURFER TO I FEEL COMFORTABLE SAYING I'M TOTALLY PSYCHED ABOUT mHEALTH. THE MCKESSON FOUNDATION IS DEDICATED TO SERVING LOW INCOME POPULATIONS IN PARTICULAR THOSE WITH CHRONIC DISEASES. AND ABOUT A YEAR AND A HALF AGO WE LUNCHED OUR MOBILIZING FOR HEALTH GRANT INITIATIVE WHICH INVESTS ONE 1/2 MILLION DOLLARS A YEAR IN BUILDING THE EVIDENCE BASE FOR MOBILE HEALTH IN LOW INCOME POPULATIONS. SOME OF YOU ARE GRANTEES. WE CONTINUE THAT WORK TODAY AND WE'RE REALLY BELIEVERS IN THE POWER AND POTENTIAL OF MOBILE HEALTH. SO WE'RE VERY, VERY GRATEFUL THAT ALL OF YOU ARE HERE TODAY TO SHARE YOUR KNOWLEDGE AND TIME SO THAT WE CAN REALLY ADVANCE MOBILE HEALTH. IN PARTICULAR, BUILD THE EVIDENCE BASE. SO THANK YOU, EVERYBODY. THANK YOU, WENDY FOR BEING THE PERSON THAT IS LEADING THE CHARGE HERE. THANK YOU. [APPLAUSE] >> AND LAST BUT DEFINITELY NOT LEAST, NOTICE OFTEN AT NIH WE'RE DOING ONE THING, NSF IS DOING A PARALLEL THING. AND WE'RE DETERMINED TO MAKE THAT A DIFFERENT WAY. SO WITH THAT, I WOULD LIKE TO INTRODUCE MISHA PAVEL, DIRECTOR OF THE SMART HEALTH AND WELL BEING PROGRAM AT NSF. FOR THOSE OF YOU WHO THINK WE DON'T WORK TOGETHER, MISHA AND I TALKED, WE PROBABLY TALK ABOUT FIVE DAYS A WEEK AT THIS POINT. mHEALTH HAS BROUGHT THINGS TOGETHER IN A WAY THAT YOU MIGHT NOT HAVE THOUGHT. MISHA. [APPLAUSE] >> GOOD MORNING. FIRST I WOULD LIKE TO WELCOME YOU ON BEHALF OF NSF TO THIS WORKSHOP AND I WOULD ALSO LIKE TO THANK WENDY AND ALL THE OTHER ORGANIZERS FOR THIS HARD WORK THAT PRECEDED, YOU WOULDN'T BELIEVE THE MEETINGS WE HAD EVERY WEEK. THE DEDICATION OF ALL THE PEOPLE INVOLVED. THIS MADE IT POSSIBLE. AS WENDY TOLD YOU I'M DIRECTOR OF A NEW PROGRAM AT NSF CALLED SMART HEALTH AND WELL BEING, IT'S ONE OF THE FIRST VENTURE INTO HEALTH BY NSF. THE REASON I CAME ACTUALLY WAS MY PERSONAL GOAL AND THE GOAL OF MY BOSS IS TO ACTUALLY DOVE TAIL OUR PROGRAM WITH NIH AND MY DREAM IS TO HAVE SOLICITATION TO BE JOINT SOLICITATION. I DON'T KNOW IF YOU CAN ACCOMPLISH THAT BUT THAT WOULD BE MAJOR GOAL. THE FOCUS OF SMART HEALTH AND WELL BEING IS TRANSFORMING ON THE ISSUES THAT ARE NEEDED TO SOLVE IN ORDER TO TRANSFORM HEALTHCARE IN A WAY THAT HAS TO BE TRANSFORMED TO BE MORE PROACTIVE, PREVENTIVE AND PATIENT-CENTERED. THAT REQUIRES TECHNOLOGY WE DONE KNOW YET HOW TO DO. MOBILE HEALTH IS ONE OF THE MAIN GAME CHANGERS IN THIS DIRECTION. WE HOPE TOGETHER WITH mHEALTH AND OTHER INFRASTRUCTURE WE WILL MAKE A HUGE DIFFERENCE IN THE NEXT FIVE YEARS. I'M LOOKING=y(JUU8‡q O THE WORKSHOP. I WOULD LIKE TO MAKE ONE MORE -- I HAVE A COUPLE OF DIFFERENTIAL EQUATIONS I WAS GOING TO PUT UP. [LAUGHTER] >> BUT I WANT TO MAKE ONE TECHNICAL COMMENT. I WOULD LIKE TO EMPHASIZE THAT mHEALTH, MOBILE HEALTH, IS JUST ONE OF THE TECHNOLOGIES THAT HAS THAT PROBLEM. WHERE THE TECHNOLOGY ADVANCES FASTER THAN OUR RANDALL CONTROL TRIALS CAN CATCH UP WITH. SOLVING THAT PROBLEM IS GOING TO GENERALIZE TO MANY OTHER TECHNOLOGIES AND MANY OTHER AREAS IN HEALTHCARE AND HEALTHCARE DEVELOPMENT. THIS IS WHY MOBILE HEALTH TO ME IS EXTREMELY EXCITING. THANK YOU. [APPLAUSE] >> NOW ON BEHALF OF THE PLANNING COMMITTEE I WOULD LIKE TO INTRODUCE THE CHAIR OF THIS MEETING SO YOU WANT TO TALK ABOUT THE WORK MIND THIS. SANTOSH KUMAR IS HERE FROM THE UNIVERSITY OF MEMPHIS, HE'S AN AMAZING RESEARCHER AND WONDERFUL PERSON. HE'S BEEN A GREAT CHAIR AND HAD LED US THROUGH WHAT WAS AN AMAZING KIND OF PROCESS TO GET TO THIS. I THINK AS MISHA SAID, mHEALTH IS THE PERFECT TEST CASE TO START TO REALLY TALK ABOUT HOW DO WE GENERATE EVIDENCE. BUT THIS IS IMPORTANT TO ALMOST ALL OF THE AREAS OF HEALTH. SO SANTOSH HELPED LEAD US THROUGH THIS, KIND OF MAKE THIS COME TOGETHER AND IS REALLY THE REASON WE'RE ALL HERE TODAY. WITH THAT I'LL LET HIM TELL US WHY WE'RE REALLY HERE. [APPLAUSE] >> GOOD MORNING, EVERYONE. ON BEHALF OF THE ORGANIZING COMMITTEE I WOULD LIKE TO WELCOME ALL OF YOU TO THIS WORKSHOP. WE WOULD LIKE TO THANK EVERYONE WHO SUBMITTED A WHITE PAPER, WE HAD OVER 70 SUBMISSION, ALL GREAT IDEAS. I WOULD ALSO LIKE TO THANK EVERYONE ON THE WEBINAR FOR YOUR INTEREST IN THIS IMPORTANT ISSUE. WHAT I WOULD LIKE TO DO IN THIS SHORT TALK IS SET AGENDA AND TO SET A GOAL FOR OURSELVES SO WE CAN IDENTIFY THE PROMISING DIRECTIONS. IN THIS WORKSHOP WE WOULD LIKE TO EXPLORE INNOVATIVE EFFORTS TO EXPLORE THE EFFICACY AN SAFETY OF MOBILE HEALTH. I'LL BEGIN WITH A BRIEF EXAMPLE OF WHAT MOST OF US KNOW ALREADY, MOBILE HEALTH IS HERE. AND PRESENTS US WITH AN INTERESTING OPPORTUNITY. I FEEL FORTUNATE THAT WE WE ARE LIVING IN THESE VERY INTERESTING TIMES. HERE I HAVE A FEW EXAMPLES OF MOBILE HEALTH SYSTEMS. ON THE LEFT IS AN ARTISAN SYSTEM WE DEVELOPED UNDER THE GI PROGRAM AND SUPPORT FROM NSF AS WELL. THIS SYSTEM CAN MEASURE VARIOUS PHYSIOLOGICAL PARAMETERS, FOR ALCOHOL AND VARIOUS OTHER ACTIVITIES BEHAVIORS AND TRANSMIT THE DATA DIRECTLY ON THE MOBILE PHONE IN THE NATIONAL ENVIRONMENT OF INDIVIDUALS SO YOU CAN ACCESS THEIR STRESS LEVEL, ADDICTIVE BEHAVIOR, THEIR ENVIRONMENT, THEIR SURROUNDINGS, THEIR SYSTEM STATUS, BEHAVIORAL STATUS, MENTAL STATUS AND ALL THAT DATA SYNCHRONIZED WITH EACH OTHER CAN BE COLLECTED IN REAL TIME IN THE NATIONAL ENVIRONMENT OF INDIVIDUAL. AS WE SPEAK, SEVERAL PARTICIPANTS HAVE BEEN ABLE TO (INAUDIBLE) FOR FEW DAYS UP TO A WEEK IN THAT NATURAL ENVIRONMENT WITHOUT ANY ISSUES, IT LED TO MANY ADVANCES. THIS IS NOT THE ONLY SYSTEM. THERE IS ANOTHER SYSTEM CALLED (INAUDIBLE) THERE IS A THIRD AND A FOURTH. SO WE HAVE SEEN AN EXPLOSION IN THIS MOBILE HEALTH DEVICES THAT ARE BEGINNING TO EMERGE WHICH CAN IF USED APPROPRIATELY CAN TRANSFORM THE WAY WE DO HEALTHCARE. SO WE COULD ENVISION VARIOUS INTERVENTIONS AND TREATMENTS THAT COULD BE DELIVERED IN REAL TIME ON THE MOBILE DEVICE IN THE MOBILE ENVIRONMENT OF INDIVIDUALS TO INTERVENE WHEN AND WHERE NEEDED. ALL OF THAT COULD BECOME A REALITY. BUT WE DON'T WANT IT TO BE THERE IS 100 APPS OUT THERE THAT CAN DO IT. THEY ALL NEED TO BE SIGN SCIENTIFICALLY BASED. HOW DO WE EVALUATE MOBILE HEALTH BASED INTERVENTIONS. OF COURSE RCD IS THE GOLD STANDARD. THE ISSUE IS A HUGE TIME LAG. IF IT'S GOING TAKE US 5.5 YEARS INCLUDING PREPARATION TIME TO START THE STUDY TO 6 YEARS, THAT'S TOO LONG FOR MOBILE HEALTH. THE BASE OF DEVELOPMENT IN THE MOBILE HEALTH ARENA IS THAT TECHNOLOGY CHANGES EVERY SIX MONTHS, EVERY YEAR. SO IF WE ARE REALLY TO MOVE FORWARD WE WOULD LIKE TO SET A GOAL TODAY FOR OUR COMMITTEE. IN FIVE YEARS FROM TODAY COULD WE SHOOT FOR DECREASING THIS TIME TO EVALUATE MOBILE HEALTH-BASED TREATMENT INTERVENTIONS BY TWO YEARS TO 3.5 YEARS FROM DESIGN TO THE WORK PUBLISHED CAN WE GET TO 3.5 YEARS THAT'S A CHALLENGE THAT WE WOULD LIKE TO POSE TO OUR COMMITTEE TODAY. BUT WE DON'T WANT TO GET CARRIED AWAY. YES, WE DO WANT TO REDUCE THE TIME AND THE EFFORT AND EXPENSES BUT WITHOUT COMPROMISING OTHER SAFETY. SO WE WOULD LIKE TO BALANCE THE TIMELINESS AND HAVE CONFIDENCE IN THE RESULTS THAT IS PRODUCED OR THE EVIDENCE PRODUCED. WE NEED TO CONTROL FOR CONFOUNDING FACTORS, WE NEED TO BALANCE FOR PROGNOSTIC FACTORS. BUT THE QUESTION IS, IF WE ARE NOT RANDOMIZING, WHAT IS THE ACCEPTED LEVEL? WHAT EVIDENCE WILL SUFFICE? AND THAT'S A QUESTION THAT WE WOULD LIKE TO OPEN UP TO THIS COMMITTEE. AS REVIEWERS OF OF GRANTS, AS REVIEWSER OFF RESULTS PUBLISHED. WHAT WOULD LIKE TO SEE AS ACCEPTABLE EVIDENCE IF YOU LERRING TO LET GO OF SAYING (INAUDIBLE). SO I WOULD LIKE TO SEE SOME APPROACHES SOME ANSWERS TO THIS QUESTION TODAY AND GOING FORWARD. SO WE KNOW THAT MOBILE HEALTH BRINGS MANY INNOVATIONS. PERHAPS MOBILE HEALTH TECHNOLOGY BY ITSELF COULD REDUCE, GET US CLOSER TO THIS GOAL, IT COULD REDUCE IF WE CAN REDUCE OR ELIMINATE FACE TO FACE INTERACTIONS. CUT DOWN ON TRANSPORTATION COST, THE STUDY COST, REDUCE THE BURDEN ON THE PARTICIPANTS, AND BROADEN TO MORE PARTICIPANTS, MANY PARTICIPANTS DON'T HAVE THE TIME OR ENERGY TO GO THROUGH ALL THE BURDEN WE PUT THEM THROUGH. WE BRING THEM EVERY WEEK TO THE LAB. OR EVERY DAY. USING MOBILE HEALTH COULD ALSO IMPROVE THE QUALITY OF MEASURES AND PERHAPS THAT COULD HELP US CUT DOWN ON THE NUMBER OF PARTICIPANTS WE NEED TO RECRUIT. IT COULD HELP US CLOSELY MONITOR COMPLIANCE. SO AGAIN, WE GET GOOD DATA FROM THE PARTICIPANTS WE NEED TO INCLUDE. IT PROVIDES US WITH LONGITUDINAL DATA WITH ECOLOGICAL VALIDITY SO WE DONE NEED TO WAIT SIX MONTHS TO START ANALYZING THE DATA. MAYBE WE CAN DO TREND ANALYSIS EARLY ON. AND IT ALSO PROVIDES US WITH A PLATFORM TO TEST ADAPTIVE DESIGN. SO THERE ARE VARIOUS WAYS MOBILE HEALTH ITSELF CAN HELP BUT IS IT SUFFICIENT TO CUT IT DOWN BY TWO YEARS? CUT THE TIME DOWN BY TWO YEARS? NOT SURE. SO WHAT -- WE WOULD LIKE TO INITIATE OR CALL FOR TRANSFORMATIVE INNOVATIONS IN THIS AREA. INNOVATIONS IN INFRASTRUCTURE, FOR INSTANCE WHAT INFRASTRUCTURE DO WE NEED TO SUPPORT RECRUITMENT AND COLLECTION. CAN WE UPDATE MOBILE MEASURES THAT'S NOT BEEN POSSIBLE TODAY WHICH COULD POTENTIALLY IMPROVE THE QUALITY OF EVALUATION, QUALITY OF CARE AND AS WELL AS REDUCE THE TIME AND FOR EVALUATION. SHORT ON DEPLOYMENT TIME WITH OPEN SOURCE AN MODELER DESIGN. ON THE MOBILE STUDY DESIGN THERE'S SEVERAL STUDY DESIGNS OUT THERE TODAY. GIVEN THAT MOBILE HEALTH OFFERS THESE OPPORTUNITIES OR THESE NEW ADVANTAGES COULD WE REVISIT THE STUDY DESIGNS AND HAVE MORE ACCEPTABILITY FOR THEM? IS IT TIME TO START INVESTIGATING THEM MORE BROADLY? THERE ARE SEVERAL MATTERS EMERGING, REALITY MINING, MACHINE LEARNING, MODELING, TREND ANALYSIS. IS IT WORTH INVESTIGATING THESE ISSUES MORE DEEPLY OR THESE MATTERS MORE DEEPLY SO THEY CAN CUT DOWN ON THE TIME IT TAKES TO SUBMIT THE FIRST PAPER. TODAY THE MEDIAN TIME TO SUBMIT THE FIRST PAPER AFTER THE STUDY IS COMPLETE IS ONE 1/2 YEARS. MAYBE BY USING SOME MATTERS, INTENTION LONGITUDINAL DATA WE CAN CUT THAT TIME TO SIX MONTHS OR EVEN SHORTER. SO THOSE ARE JUST SOME IDEAS. BUT TO -- IN TODAY'S MEETING WE WE WOULD LIKE TO HEAR ARE WHAT ARE SOME APPROACHES THAT WE AS A COMMITTEE BELIEVE THAT WE COULD DO TO ACHIEVE OUR GOAL OF REDUCING THE EVALUATION TIME FOR THE INTERVENTIONS TO 3.5 YEARS OR LESS BY 2016. WITH THAT I WOULD LAKE TO OPEN UP THE MEETING. THANK YOU. [APPLAUSE] >> OKAY. WITH THAT, YOU HAVE GOT YOUR CHARGE NOW. I'M GOING TO INVITE THE FIRST PANEL UP. I'M ALSO GOING TO INVITE TISHA WILEY FROM THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCE RESEARCH, MY GREAG COLLEAGUE WHO WILL START US ON THE FIRST SESSION. SESSION ONE PANEL MEMBERS PLEASE COME UP. DON HEDEKER IS A PROFESSOR OF BIOSTATISTICS T THE UNIVERSITY OF ILLINOIS AT CHICAGO IN THE DEPARTMENT OF SCHOOL OF PUBLIC HEALTH IN EPIDEMIOLOGY AND BIOSTATISTICS. DON IS AN EXPERT IN ANALYZING LONGITUDINAL AND CLUSTER DATA, PARTICULARLY IN THE MIX FACTS SORT OF FRAMEWORK SO HE'LL START US TALKING ABOUT HIS WORK AND GIVE AN OVERVIEW HOW THE REST OF THE PAPERS FROM THE SESSION FIT INTO THAT. SO IT'S MY PLEASURE TO INTRODUCE DON. [APPLAUSE] >> THANKS SO MUCH. I WAS SO EXCITED THIS MORNING I CUT MYSELF SHAVING. THIS IS AN EFFORT OF SWEAT AND BLOOD. [LAUGHTER] >> WHAT I'M GOING THE TALK A LITTLE BIT IN THE FIVE MINUTES I HAVE IS THE MODELING APPROACHES I HAVE TAKEN TO THE SORT OF -- THESE SORT OF DATA AND TALK ABOUT BETWEEN AN WITHIN SUBJECT VARIATION OF mHEALTH DATA. AND THIS IS JOINT WORK I HAVE DONE AT SCHOOL OF PUBLIC HEALTH WITH ROBIN MERMALSTEEN. SO WE HAVE BEEN LOOKING AT ECOLOGICAL ASSESSMENT DATA FOR A WHILE, ALSO CALLED EXPERIENCE SAMPLING DIARY METHODS. NECESSARY DESIGNS SUBJECTS PROVIDE FREQUENT REPORTS ON EVENTS AN EXPERIENCE OF THEIR DAILY LIVES. UP TO 30, 40 RESPONSES PER SUBJECT COLLECTED OVER A COURSE OF A WEEK OR SO. WHEN I FIRST GOT THESE DATA I THOUGHT GREAT, NOW WHAT DO I DO WITH THE DATA? THE CHALLENGE IS TO FIGURE OUT INTERESTING WAYS TO ANALYZE ITCH THESE ARE GOTTEN BY A VARIETY OF WAYS, ELECTRONIC DIARY, PALM PILOTS, PDAs, VOICE RESPONSE SYSTEMS CELL PHONE, WEB-BASED, A VARIETY OF METHOD TOTS COLLECT THESE DATA. THE GOAL OF THESE STUDIES ARE TO CAPTURE PARTICULARS OF AN EXPERIENCE WITH MORE TRADITIONAL DESIGNS LIKE RCTs. INSTEAD WE CAN EXAMINE HOW SOMETHING DEVELOPS ACROSS TIME WITHIN AN INDIVIDUAL. THE REPORTS WE GET FROM SUBJECTS COULD BE TIME-BASED, COULD BE RANDOMLY BASED, EVENT PROMPTED AND WE LOOKED AT DIFFERENT SORTS OF PROMPTS LIKE THIS. BASICALLY THE BOTTOM LINE HERE IS THE DATA ARE RICH AND FROM THE STATISTICAL POINT OF VIEW, THEY OFFER REALLY A LOT OF INTERESTING MODELING POSSIBILITIES. I THINK IT'S A GOLD MINE FOR METHODOLOGICAL DEVELOPMENT. SO HERE SHOWS ONE PAPER I HAVE WORKED ON PUBLISHED IN BIOMETRICS RECENTLY. THIS IS A PLOT OF WHAT THE MODEL CAN DO. THE SOLID LINE IS BASICALLY USUAL MEAN RESPONSE, HOW HIGH OR LOW IS THE PERSON. WHAT WE LOOKED AT IS MOTION, BASICALLY HAPPY, SAD YOU MIGHT THINK OF IT THIS WAY. CO-VARIANTS COULD AFFECT HOW HAPPY OR SAD A PERSON IS. COULD BE SUBJECT CO-VARIANTS OR TIME CO-VARY VARIANTS. ON THE TOP IS A PERSON HAPPIER THAN OTHER ON BOTTOM. THOSE DISPERSION OF THE DOTTED LINES REPRESENT IT IS BETWEEN SUBJECT VARIATION, HOW HETEROGENEOUS ARE THE SUBJECTS. IN THIS MODEL WE DEVELOPED WE ALLOWED CO-VARIANTS TO INFLUENCE THAT AS WELL. OFTEN PEOPLE THINK OF HETEROGENEITY IN TERMS OF DIFFERENTIAL MEAN RESPONSE BY GROUPS OF SUBJECTS. I WANT TO OPEN THAT UP TO HETEROGENEITY OF VARIANTS, THAT IS THE HETEROGENEITY BETWEEN SUBJECT VARIANTS CAN BE DIFFERENT FOR MEN AND WOMEN LET'S SAY FOR CONTROL AND TREATED SUBJECTS. THE DISPERSION OF THE DOTTED LINESSER THAT'S PRECISELY WHAT THAT GETS AT, THE BETWEEN SUBJECT VARIATION, WE CAN TALK HETEROGENEITY OF THAT. NOW, ADDITIONALLY YOU SEE INDIVIDUAL DOTS, THAT REPRESENT THE OBSERVED MEASUREMENTS, THE 30 PROMPTS OR SO. NOW, THERE'S A PERCENT VERY ERRATIC RELATIVE TO THE PERSON ON THE BOTTOM. THAT'S ADDITIONALLY A COMPONENT IN OUR MODEL WITHIN SUBJECT VARIANTS. AGAIN WE CAN HAVE HETEROGENEITY OF THE IN SUBJECT VARIANTS. SOME ARE MORE CONSISTENT, SOME ARE MORER RATTIC. MAYBE YOUR INTERVENTIONS ARE MEANT TO MAKE PEOPLE BE IN BETTER CONTROL OF THEIR SYMPTOMTOLOGY. A CO-VARIANT WOULD BE THEREFORE INTERESTED IN MODELING THE WITHIN SUBJECT VARIANTS. CAN A CO-VARIANT BE RELATED TO THE DISPERSION OF DOTS AROUND THEIR LINE. SO IN THIS MODEL WE HAVE DEVELOPED WE ALLOW THE CO-VARIANTS TO INFLUENCE THE MEAN LEVEL WE ALL KNOW ABOUT, HOW HIGH OR LOW AS A FUNCTION OF CO-VARIANT, THE BETWEEN SUBJECT VARIANTS, HOW DISPARATE ARE THOSE DOTTED LINES AND ALSO WITHIN SUBJECT VARIANTS T DISPERSION OF THE POINTS. AS A TECHNICAL POINT FOR THE STATISTICIANS OVER THERE, I'LL MENTION WE ALSO HAVE RANDOM EFFECTS ON BOTH THE MEAN AND WITHIN SUBJECT VARIANT STRUCTURE. SO WE USE THIS KIND OF MODELING IN EMA DATA, IT HAS MORE BROAD APPLICATION WHERE YOU'RE INTERESTED NOT JUST IN HOW HIGH OR LOW ARE INDIVIDUALS BUT HOW SPREAD OUT ARE THEY, HOW SPREAD OUT IS THE DATA WITHIN SUBJECTS. WHAT ARE THE DETERMINANTS OF THOSE SOURCES OF HETEROGENEITY. HERE ARE SOME OF THE RESULTS WE GOT. WE GOT CONSISTENT RESULTS FOR POSITIVE AND NEGATIVE MOOD. IN TERMS OF MEAN RESPONSE THEY WENT IN OPPOSITE DIRECTIONS BUT THE VARIANTS' RESPONSES WERE SIMILAR FOR THINGS THAT WE LOOKED AT LIKE BEING ALONE, BEING A MALE, BEING A NEGATIVE MOOD REGULATOR, ET CETERA, ET CETERA. SO WON'T FOCUS ON THOSE, IF INTERESTED YOU CAN LOOK AT THE PAPER BUT THE IDEA IS TO MODEL THIS HETEROGENEITY. HERE SHOWS ANOTHER PAPER WE WORKED ON IN A DICTION THAT ALSO LOOK AT HETEROGENEITY BUT ACROSS TWO DIFFERENT SETS OF EVENTS. ON THE LEFT-HAND SIDE I HAVE RANDOMLY PROMPTED EVENTS. INDIVIDUALS ARE RANDOMLY PROMPTED AND THEY GAVE THEIR MOOD RESPONSE. ON THE RIGHT HAND SIDE THEY WERE ALSO GIVING SMOKING EVENTS. HERE WHAT I HAVE PUT IS YOU CAN THINK OF THE SLOPE, THAT'S THE MEAN RESPONSE. YOU HAVE INDIVIDUALS GIVING RANDOM PROMPTS, INDIVIDUALS GIVING SMOKING RESPONSE. YOU MIGHT ALSO THINK OF THIS IN A CONTROL AND TREATMENT SORT OF SITUATION. YOU HAVE GOT A CONTROL, YOU HAVE TO TREAT THE RESPONSE. THERE CAN BE A DIFFERENTIAL TREND BY INDIVIDUALS BECAUSE YOU COLLECT SO MUCH DATA IN EACH OF THESE TWO SCENARIOS. WHAT WE'RE INTERESTED IN, THIS IS THE DISPERSION OF THE LINE, HOW DISPARATE ARE THE LINES, HOW MUCH HETEROGENEITY IS THERE ACROSS SUBJECTS. THIS SHOWS FOR EXAMPLE ON THE TOP I HAVE GOT DISPARTICIPANT LINES THAT GET MORE DISPARATE. SO THE HETEROGENEITY OF THE SLOPES IS INCREASE FROM THE TOP THE BOTTOM. ALTERNATIVELY YOU MIGHT THINK OF THE HETEROGENEITY BEING DIMINISHEDCH THIS IS WHAT WE SAW IN THIS PARTICULAR STUDY LOOKING AT THE VARIATION IN THE SMOKING-RELATED MOOD RESPONSE WHEN THESE INDIVIDUALS BECAME MORE EXPERIENCED SMOKERS THEIR MOOD RESPONSE HETEROGENEITY WAS DIMINISHED SO THIS MODELING OF NOT JUST THE MEAN STRUCTURE BUT THE VARIANT STRUCTURE VERY INTERESTING AND VERY DOABLE IN THESE SCENARIOS WITH LOTS OF DATA WITHIN SUBJECTS. FINALLY WE ALSO LOOKED AT EVENT TRIGGERED EMA. WHEN DO SUBJECTS GIVE US THESE SMOKING REPORTS. SO SUBJECTS HAVE A PDA THEY DAIRY AROUND, A SMOKING REPORT. WHEN DO THESE EVENTS OCCUR? DO THEY VARY ACROSS DAYS, TIMES OF THE DAY? SO WHAT WE DID WAS THIS. WE CLASSIFIED EACH DAY AND WEEK OF THE EMA DATA COLLECTION AT DIFFERENT TIME INTERVALS TO WHETHER AN EVENT WAS OBSERVED OR NOT. SO THINK ABOUT WE HAVE SEVEN DAYS WITHIN EACH DAY WE CREATED FIVE TIME BINS AND WE OBSERVED WHETHER OR NOT THERE WAS AN EVENT IN ONE OF THOSE TIME BINS. WE USE THOSE AS 35 TEST ITEMS JUST LIKE IN AN EDUCATIONAL TESTING SCENARIO USED AN EDUCATIONAL TESTING MODEL TO FIND OUT WHICH TIME BINS WERE TIME BINS WHEN PEOPLE WERE SMOKING AND NOT ONLY THAT, WHICH TIME BINS WERE MORE DISCRIMINATING IN TERMS OF RELATING WHO IS A LOW-LEVEL SMOKE TORE A HIGH-LEVEL SMOKER. HERE ARE THE RESULTS OF THE DIFFICULTY ESTIMATES, THESE SHOWS WHICH DAYS OF THE WEEK WERE DAYS THAT WERE EASY, THOSE ARE ON THE BOTTOM AN WHICH ARE HARD. MONDAY AND SUNDAY ARE TIMES OF INFREQUENT REPORTS. FRIDAY IS FREQUENT REPORTS. THERE'S DAY OF THE WEEK. THREE MINUTES. I KNOW I'M GOING TOO LONG. SO WHAT WE HAVE ON THE BOTTOM IS THAT EARLY MORNING SMOKING YOU CAN SEE VERY INFREQUENT TIME OF RESPONSE TOWARDS THE EVENING MORE FREQUENT. THAT'S ALL STUFF THAT'S EASILY GAINED. WHAT'S MORE INTERESTING ARE THE DISCRIMINATION PARAMETERS. THE TIME BINS THAT SEPARATE FROM MORE EXPERIENCED SMOKING. MIDDLE SMOKING IS A MORE TELLING TIME FOR DISCRIMINATING SMOKING LEVEL AND ON THE BOTTOM EARLY MORNING SMOKING A MORE TELLING TIME BIN TO DISCRIMINATE SMOKING LEVEL. SO IN THE ONLY DO WE FINE OUT ABOUT WHEN THESE OCCUR BUT WHICH ONES ARE MORE DISCRIMINATING IN TERMS OF SUBJECTS. OKAY. SO THAT'S MY BIT. LET ME COMMENT A LITTLE BIT ABOUT THE GREAT SESSIONS WE'RE GOING TO HAVE RIGHT NOW. MARGARET HANDLEY WILL SHOW US ABOUT STEP WEDGE DESIGN, VERY USEFUL APPROACH IN ANALYSIS OF mHEALTH DATA, SHE HAS SUBJECTS FROM PATIENT REGISTRY, RANDOMLY ASSIGNED TO TREATMENT AND WEIGHT LIST, AUTOMATED TELEPHONE SELF-MANAGEMENT SUPPORT SYSTEM MANAGEMENT OF DIABETES. THERE THEY MIGHT BE INTERESTED IN LOOK AT STABILITY OF RESPONSES ACROSS TIME. WHAT'S VERY USEFUL HERE IS THEIR ABILITY TO ADAPT TO NON-ENGLISH SPEAKERS SO RANDOMIZATION OF SUBJECTS TO TIME OF TREATMENT RECEIVED IS VERY IMPORTANT. USING RANDOMIZATION TO STRATIFY SUBJECTS IN TERMS OF WHEN THEY GET THE TREATMENT. SO THE TREATMENT EFFECT HERE AS I THINK IN MANY OF THESE STUDIES INVOLVES THE COMPARISON OF WITHIN SUBJECTS DATA AS WELL AS BETWEEN SUBJECTS DATA. AT ANY GIVEN TIME POINT SOME SUBJECTS ARE RECEIVING THE TREATMENT, SOME ARE NOT. ACROSS TIME YOU HAVE INDIVIDUALS GETTING THE TREATMENT AND NOT GETTING THE TREATMENT. SO STATISTICALLY WE HAVE TO DEAL WITH THIS. SO I ONLY HAVE ONE MORE MINUTE. LET ME GO THROW A LITTLE BIT MORE. BETHANY HANDLEY WILL SHOW US ABOUT PRE-POST TRIALS, WE'RE NOT GOING ANALYZE AT THE INDIVIDUAL LEVEL, WE RANDOMIZE AT THE PROVIDER LEVEL, VERY EFFICIENT DESIGN FOR mHEALTH STUDIES. SORRY TO BE SHORT BUT THEY'LL GUF YOU MORE DETAILS. I'M RUNNING OUT OF TIME SO I'LL CHASE TO THE CUT. DR. (INAUDIBLE) WILL TALK ABOUT CONTINUOUS MEASUREMENTS OF PERSONAL EXPOSURES TO ADDICTIVE SUBSTANCES IN PSYCHOSOCIAL STRESS, A MOBILE PLACED PLATFORM WHICH IMIENS RECRUITMENT OF SUBJECTS AND COLLECTION OF DATA FROM PHONES AN ON BODY SENSORS, THEY HAVE THE PACKAGE, THE COLLECTION OF DATA, THE DATABASE ENTRY, TREND ANALYSIS, REGRESSION DISCONTINUITY. IN HIS WHITE PAPER WAS A NICE FOCUS ON BURDEN PARTICIPATION INCENTIVES PRIVACY ISSUES. MISSING DATA. IN THESE STUDIES WE GET SO MUCH DATA WE FORGET ABOUT THERE ARE TIMES WHEN THEY ARE MISSED. A CHALLENGE BASICALLY FOR US TO DEAL WITH THAT. DR. KRAVITZ WILL TALK ABOUT HETEROGENEITY OF TREATMENT EFFECTS. THAT'S A VITAL ASPECT AND HE CAN LOOK IN THE TRIALS WHERE WE HAS INDIVIDUAL WHOSE ARE EXPERIENCING THESE TRIALS, YOU CAN SEE THE HETEROGENEITY RESPONSE ACROSS INDIVIDUALS, I WOULD URGE NOT JUST HETEROGENEITY MEAN RESPONSE BUT MANY TERMS OF VARIANT. ANOTHER INTERESTING THING TO LOOK AT. I'LL STOP SHORT AND APOLOGIZE TO THE SPEAKERS FOR (INAUDIBLE). (OFF MIC) [APPLAUSE] >> PLEASE HOLD YOUR QUESTIONS UNTIL THE END AND WE'LL TAKE THE QUESTIONS AT THE END AND HAVE TIME FOR DISCUSSION THEN. SO NOW WE'LL TURN TO MARGARET HANDLEY'S PRESENTATION. IF YOU WANT TO COME ON UP. >> THANK YOU VERY MUCH. AS DON SAID, I'M GOING TO TALK ABOUT STEP WEDGE DESIGN, OUR AUTOMATED TELEPHONE SUPPORT PROGRAM, MANAGEMENT SUPPORT PATIENTS TAILORED TO PATIENTS WHO ARE NON-ENGLISH SPEAKERS. THE IT OF WHAT WE'RE DOING HERE, WE HAVE AN AUTOMATED TELEPHONE SUPPORT PROGRAM. WE HAVE USED THIS IN A RANDOMIZED CONTROL TRIALS CAN DIABETES. QUERIES ON THE PHONE THAT ASK THEM SOME QUESTIONS ABOUT MANAGEMENT BEHAVIORS AND THOSE ANSWERS GO BACK TO A CALL SYSTEM THAT GENERATES A REPORT FOR A CARE MANAGER TO CALL PEOPLE BACK. SO INTERVENTION INCLUDES THINGS THAT HAPPEN ON THE PHONE IN AN AUTOMATED WAY AS WELL AS INTENTIONSIVE COUNSELING. THE INTERVENTION BENEFIT WITH WHAT WE HAVE DEVELOPED IS THAT FOR PATIENT WHOSE ARE NON-ENGLISH SPEAKERS GET COUNSELING IN LANGUAGE SO THERE'S CONCORDANT CARE FOR DIABETES HEALTH MANAGEMENT. OFTEN IT'S BETWEEN THE REPORTS AN PROVIDERS AND CLINICS AS APPROPRIATE FOR THIS. WE DEVELOP THE RANDOMIZED CONTROL TRIAL AND BECAUSE A REGIONAL MANAGED CARE PLAN APPROACHED US AND WANTED TO DEVELOP THIS AS A MEMBER BENEFIT, A COVERED BENEFIT. WE DIDN'T WANT TO DO A RANDOMIZED CONTROL TRIAL BECAUSE THEY SAID WE'RE A HEALTH PLAP AND DON'T WANT TO DO THAT WE WANT TO STUDY EFFECTIVENESS IN OUR SYSTEM BUT WE WAN YOU GUYS TO COME UP WITH AN EVALUATION DESIGN BUT NOT AN RCT. SO WE PURR PSEUDOD DIFFERENT -- PURSUED DIFFERENT OPTIONSCH WE WANTED TO WORK WITH THE PATIENT REGISTRY, THEY HAVE A WONDERFUL DIABETES REGISTRY WE HAVEN'T USED FOR INTERVENTIONS, THEY USED IT FOR SENDING REPORTS AND THINGS LIKE COME GET YOUR IMMUNIZATION BUT WE HAVEN'T USED IT FOR ANYTHING INTERACTIVE SO WE WANTED TO WORK WITH A SYSTEM THEY HAD. THIS IS A MAP OF THE QUASI EXPERIMENTAL DESIGN AN AT THE TOP YOU SEE IN THE BLUE ARE WHAT WE CONSIDER GOLD STANDARDS IN TERMS OF INDIVIDUAL PATIENT RANDOMIZED CONTROL TRIALS AN CLUSTER RANDOMIZED TRIALS. BELOW THAT YOU SEE THINGS IN DECREASING ORDER OF RIGOR AND OPPORTUNITY, SOME OF THE QUASI EXPERIMENTAL DESIGN SO THE STEPPED WEDGE AND WEIGHT VARIANT OF THAT IS WHAT I'M DESCRIBING BUT THERE'S OTHERS PEOPLE ARE DISCUSSING SUCH AS TIME SERIES AN BEFORE AN AFTER CONTROL DESIGNS. SO STEP WEDGE DESIGN, HOW MANY HAVE HEARD OR SEEN IT OR FAMILIAR WITH THEM? SOME RF YOU ARE. IN THE BASIC DESIGN IT'S DEVELOPED MORE FOR CLINIC OR CLUSTER BUT WE ADAPTED IT FOR PATIENT REGISTRIES BUT IF YOU LOOK IN THE WHITE IS THE CONTROL TIME AND THE PURPLE WOULD BE THE INTERVENTION TIME. SO ON THE BOTTOM IF YOU LOOK AT THE FIRST COLUMN WOULD BE TIME PERIOD ONE, EVERYBODY IS UNEXPOSEDCH NOBODY GETS THE INTERVENTION. YOU COLLECT DATA AT THE BOX TIME SO IT INVOLVES MULTIPLE DATA COLLECTION POINTS F YOU'RE WORKING WITH A HEALTH PLAN AND DATA REGISTRY THERE'S DATA COMING INTO THAT. THEY HAVE A LABORATORY, THEY HAVE CLAIMS DATA, THEY HAVE A LOT OF DIFFERENT SOURCES FOR WHICH WE DON'T HAVE TO G AND COLLECT ADDITIONAL DATA AT THESE DIFFERENT TIME POINTS. IN A LOT OF STUDIES YOU DO. WE ADDED EVALUATIONS TO COLLECT ADDITIONAL DATA AT THESE DIFFERENT TIME POINTS. SO AS PEOPLE MOVE THROUGH TIME YOU RANDOMLY ASSIGN WHICH CLINIC OR CLUSTER GETS THE INTERVENTION SO IN A SECOND COLUMN YOU SEE THE PURPLE BOX AT THE BOTTOM UNDER THE TWO IS WHERE THE FIRST GROUP STARTS GETTING THE INTERVENTION AN EVERYONE ELSE IS STILL CONTRIBUTING CONTROL TIME. IF YOU CAN ALLOCATE THIS IN A RANDOM WAY THAT'S THE BEST OPTION BUT A LOT OF TIMES YOU HAVE TO STRATIFY SO THERE'S EXAMPLES WHERE CLINIC SIZE INFLUENCES HOW YOU WANT TO ROLL OUT THIS DESIGN. IT'S A STEP WEDGE BECAUSE IN THEORY THAT PURPLE SHAPE LOOKS LIKE A WEDGE SO THAT'S WHY IT'S CALLED THAT. SO OUR VARIANT OF THIS IN OUR SMART STEPS PROGRAM, THIS IS AN ADAPTATION OF OUR AUTOMATED TELEPHONE SUPPORT PROGRAM WE CALL SMART STEPS. IN SMART STEPS WE WANTED 130 PATIENTS IN THE INTERVENTION, 130 PATIENTS IN WHAT WE CALL THE WEIGHTLESS CONTROL PERIOD. OVER TIME IN EACH OF THESE WAVES WE CALL THEM, THEY'RE NOT CLUSTERS THAT REPRESENT ANYTHING GEOGRAPHIC OR PHYSICAL LICK A ACTUAL CLINIC SITE BUT THE PAGE REGISTRY IS HEALTH PLAN WOULD HAVE RECRUITMENT LISTS SO FOR EXAMPLE IF THE FALL CALL MEMBERS AND SAY WE'RE ENROLLING IN OUR PROGRAM CALLED SMART STEPS. YOU'RE GOING GET IT NOW OR IN SIX MONTHS SO AT THE TIME OF ENROLLMENT THE RANDOMIZATION PROTOCOL WAS ACTUALIZED SO THEY TOLD THE PATIENT THEY WERE GOING TO GET THE MEMBER BENEFIT NOW OR LATER. THIS IS NOT JUST BECAUSE IT WAS A GOOD DESIGN FOR CONTROL DATA, THEY DIDN'T HAVE A STAFF TO MANAGE THE CARE COUNSELING BACK AT THE HOME BASE WHERE PEOPLE WERE TRIGGERING BASED ON THEIR CALL BACK DIABETES SELF-MANAGEMENT NEEDS SO THEY WERE WANTING TO UNROLE IT IN A STAGGERED FASHION SO THEY COULD HAVE ONE PART TIME PERSON DO THE CALL BACKS. IN THE FIRST WAVE YOU SEE AT THE TOP CERTAIN PATIENTS WERE IMMEDIATELY GETTING THE INTERVENTION AND OTHER PATIENTS WERE RANDOMIZED TO WEIGHTLESS. AFTER SIX MONTHS THAT GROUP OF PATIENTS WOULD CROSS OVER FROM THE WEIGHT LIST AND THAT'S THE BLUE LINE GOING UP INTO THE WEIGH LIST ONE INTERVENTION ARM. THEY'RE NOW IN THE INTERVENTION PERIOD FOR SIX MONTHS. SO IN EACH OF THESE WAVES, IF YOU LOOK AT THEM LIKE THREE WAVES OVER TIME YOU SEE THERE'S COMPARISON DATA THAT'S CHECKED AT EACH TIME INTERVAL. WE HAVE RANDOMLY ALLOCATED THE WAY IN WHICH THOSE PATIENTS WERE GETTING THIS. TO SOME EXTEN, IT'S A SEMIRANDOMMIZATION OFF THE WEIGHT LIST. SO THAT'S HOW WE DID IT USING A PATIENT REGISTRY, THERE'S -- THE PROS OF THIS ARE THAT YOU'RE ABLE TO INTEGRATE SOMETHING LICK THIS AN STAGGER IT OVER TIME AS WELL AS HAVE SOME LEVELS OF CONTROL OVER THE RANDOMIZATION. AND DOWN SIDE IS SPEAKING TO THE ISSUES THAT WERE BROUGHT UP EARLIER THAT IT DOES EXTEND THE PROJECT TIME FOR SOMETHING LIKE THE DIABETES OUTCOME THAT WE WERE LOOKING AT BECAUSE YOU HAVE TO WEIGHT FOR THE WEIGH LIST POPULATIONS TO CROSS OVER. FOR SOME OUTCOMES IT DOES EXTEND THE TIME PERIOD SO THAT'S A LIMITATION. IF YOU'RE CHECKING YOUR OWN DATA YOU HAVE TO REINTERVIEW PEOPLE SO IT ADD AS STEP OF DATA COLLECTION THAT'S CUMBERSOME BUT IF YOU'RE WORKING WITH AUTOMATED SYSTEMS AN ELECTRONIC MEDICAL RECORDS AN LABORATORY DATA THAT'S NOT AN ISSUE JUST SLICE THE DATA AT DIFFERENT TIME PERIODS AND LOOK AT IT AS CONTROL DATA BEFORE YOU PROCESS PEOPLE OVER TO THE INTERVENTION. THANKS. [APPLAUSE] >> NOW WE'LL HEAR FROM BETHANY HEDT. >> THANK YOU FOR EVERYONE FOR ALLOWING ME TO PARTICIPATE IN THE WORKSHOP TO TALK ABOUT SOME OF OUR WORK. I APOLOGIZE A BIT BECAUSE I RECOGNIZE THERE'S SOME OVERLAP WITH WHAT MARGARET WAS TALKING ABOUT BUT I DO TALK ABOUT BEFORE AN AFTER IN DESIGN MUCH BIGGER PICTURE PERSPECTIVE, NO NECESSARILY TALKING ABOUT THE BENEFITS AND THE DRAW BACKS OF EACH OF THESE DESIGNS. MY INTEREST IN mHEALTH IS REALLY TWOFOLD. ONE IS MY PRIMARY RESEARCH IS DEVELOPING EFFICIENT MONITORING AND EVALUATION SYSTEM. THE DAWN OF mHEALTH TECHNOLOGY CAN IMPROVE THE EFFICIENCIES OF THESE SYSTEMS. MY SOUTHERN DRAW AMPLIFIES OVER THE MIC SO I TRY NOT TO GET TOO CLOSE. THE SECOND IS I'M THE PRIMARY STATISTICIAN FOR D TREE INTERNATIONAL. D TREE TRIES TO DEVELOP PROTOCOLS TO IMPROVE DELIVERY OF CLINICAL CARE BUT ALSO USES MOBILE HEALTH AND MOBILE TECHNOLOGY AND ELECTRONIC TECHNOLOGY TO FACILITATE THE USE OF THESE PROTOCOLS. AND IT'S THAT WORK THAT I'LL TALK ABOUT OVER THE NEXT FEW MINUTES. SO THE FIRST QUESTION IS WHY CLUSTERS AN WHY DOES IT MATTER? WHAT I'M SHOWING YOU HERE ARE PRELIMINARY RESULTS FROM A STUDY I'M DOING WITH D TREE INTERNATIONAL AT CLINICS IN TANZANIA. THIS IS A STUDY AROUND THE USE OF INTEGRATED MANAGEMENT CHILDHOOD ILLNESSES PROTOCOL. WE KNOW THIS PROTOCOL IS USED WELL, WE CAN DECREASE UNDER FIVE MORTALITY IN SETTINGS THAT ARE -- HAVE HIGH -- UNDER FIVE MORTALITY RATES. THE CHALLENGE IS THAT THESE SYSTEMS ARE NOT BEING USED WELL. WHEN COUNTRIES ADOPT IMCI PROTOCOLS AN TRAIN THEIR STAFF, THE ADHERENCE TO THE PROTOCOL DECREASES OVER TIME. SO WHAT YOU'LL SEE IN THE PINK IS THE ADHERENCE TO DIFFERENT COMPONENTS OF THE IMCI PROTOCOL UNDER THE CURRENT PAPER-BASED SYSTEM AND THEN IN THE BLUE YOU'LL SEE THE ADHERENCE WHEN WE GIVE THEM THE SAME PROTOCOL PROGRAMMED ON A MOBILE PHONE T ADHERENCE UNDER THAT SYSTEM. ONE OF THE THINGS THAT I WANT TO POINT OUT IS YOU CAN SEE THAT THE ADDED BENEFIT OF THE MOBILE TECHNOLOGY REALLY DEPENDS ON THE CLINIC THAT YOU'RE WORKING IN. SO I WOULD ARGUE THAT IT'S IMPORTANT TO STUDY MOBILE HEALTH TECHNOLOGY IN MULTIPLE CLUSTERS BECAUSE THE ADDED BENEFIT CAN VARY BY THE INDIVIDUALS WHO YOU'RE WORKING WITH. SO WE'RE ALL FAMILIAR WITH CLUSTER RANDOMIZED TRIALS AND HOW THEY WORK. I'M GOING TO SKIP THIS FOR TIME. THE QUESTION IS WHY NOT CLUSTER RANDOMIZED TRIAL? IN OUR EXPERIENCE D TREE INTERNATIONAL IS PRIMARILY AN IMPLEMENTING ORGANIZATION SO WE'RE BROUGHT IN BY DIFFERENT ORGANIZATIONS TO HELP FACILITATE CARE. WHAT YOU'LL FIND IS IMPLEMENTING ORGANIZATIONS AND RESEARCH BODIES ARE OFTEN AT ODDS WITH EACH OTHER BECAUSE OF THE TIME AN RESOURCE INVOLVED. SO THE QUESTION IS, IS THERE A WAY TO INTEGRATE THE CLUSTER RANDOMIZED TRIALS INTO IMPLEMENTATION WHILE MAINTAINING THE RIGOR OF THE STUDY. SO THE BEFORE AND AFTER TRIAL, THIS IS ESSENTIALLY WHAT WE DID FOR THE STUDY RESULTS THAT I SHOWED YOU A FEW SLIDES AGO. YOU ESSENTIALLY RANDOMLY CHOOSE SOME CLINICS WHO WILL BE INVOLVED IN YOUR STUDY OUT OF ALL CLINICS THAT ARE ELIGIBLE TO PARTICIPATE. YOU TAKE YOUR BASELINE MEASUREMENT AND IMPLEMENT YOUR mHEALTH TECHNOLOGY IN THIS CASE AND YOU TAKE FOLLOW-UP MEASUREMENTS. WHAT'S NICE ABOUT THIS IS THAT THE CLINICS SERVE AS THEIR OWN CONTROLS WHICH IS OPTIMAL FOR CONTROLLING CONFOUNDING. THE DOWN SIDE IS YOU'RE ACTUALLY INTRODUCING THE TEMPORAL CON FOUNDER SO A BENEFIT BETWEEN YOUR INTERVENTION, A BENEFIT OF YOUR INTERVENTION, IT'S HARD TO TEASE OUT WHETHER OR NOT THERE WERE OTHER FACTORS GOING ON. I WILL SAY THE REASON WHY WE FELT THAT THE BEFORE AND AFTER CLUSTER TRIAL WAS FINE IN OUR PARTICULAR STUDY WAS THAT FROM THE TIME THAT THE FIRST OBSERVATION UNDER THE BASELINE MEASUREMENT THE TIME OF THE LAST OBSERVATION ON THE FOLLOW-UP MEASUREMENT WAS A TOTAL OF (INAUDIBLE) FOR A PARTICULAR CLINIC SO WE FELT ANY TEMPORAL CONFOUNDING WOULD BE NEGLIGIBLE OR ALMOST OBSOLETE BECAUSE OF THE LIMITED TIME SCALE THAT WE WERE STUDYING. THE STEP WEDGE CLUSTER DESIGN JUST MENTIONED, I WANT TO BRING IT UP AGAIN BECAUSE IN A LOT OF WAYS THIS IS A WAY WE GET THE SAME BENEFIT TO IMPLEMENTERS BUT CONTROL FOR THE TEMPORAL CONFOUNDING. SO THE BENEFIT TO THE IMPLEMENTERS IS BY THE END OF THE STUDY EVERYONE HAS THE INTERVENTION WHICH IS WHAT OUR IMPLEMENTING PARTNERS WANT. THE BENEFIT IN TERMS OF STUDY DESIGN IS THAT WE'RE ABLE TO GET AROUND THE TEMPORAL CONFOUNDING BECAUSE THE COMPARISON THAT WE HAVE IN A STEP WEDGE DESIGN IS NOT THE PRE-POST CLINIC, THE COMPARISON IS ACTUALLY AT ANY GIVEN TIME POINT THOSE SITES THAT HAVE THE INTERVENTION TO THOSE SITES THAT DON'T HAVE THE INTERVENTION. FOR THE NEXT PHASE OF OUR WORK WITH THE IMCI STUDY, THERE'S ENOUGH EVIDENCE TO SHOW IT'S EFFECTIVE IN TERMS OF IMPROVING CLINICAL ADHERENCE TO PROTOCOLS. WE'RE WORKING WITH PARTNERS INTERESTED IN ADOPTING THIS TECHNOLOGY AND USING A STEP WEDGE DESIGN TO SCALE UP TECHNOLOGY TO MAKE SURE WE CAN SEE THE LONG TERM BENEFIT OF HAVING THE mHEALTH IMCI PROTOCOL. THANK YOU. [APPLAUSE] >> OKAY. NEXT WE'RE GOING THE HEAR FROM DEE DEEPAK GANESAN. >> OKAY. THIS IS JOINED WITH WHITE PAPER WITH FOLKS AT UNIVERSITY OF MEMPHIS AND (INAUDIBLE) FROM UCLA. ALL RIGHT. SO WHAT I'M GOING TO DO IS QUAKILY TALK ABOUT THE TOOLS THAT WE HAVE BEEN DEVELOPING, THE COMPARING TO mHEALTH TECHNOLOGY TOOLS AND HOW THAT INFLUENCES METHODOLOGY, BOTH POSITIVES BENEFITS IT CAN BRING AND CHALLENGES AS WELL. SO THE TEAM IS WORKING ON SEVERAL MOBILE HEALTH TOOLS AND THE EXCITING PART IS THAT WE USE THESE TOOLS WE CAN SCALE UP THE SIZE OF EITHER STUDY TO BE RUN. IT'S ESTIMATED ONLY 2% OF THE U.S. POPULATION ACTUALLY PARTICIPATES IN ANY CLINICAL OR BEHAVIORAL TRIAL, OBVIOUSLY ROOM TO IMPROVE ON THAT. PART IS THE CHALLENGE OF THE PERCEIVED BURDEN ON THE PATIENT. COUPLE OF WAYS TO CHANGE THAT. ONE BROADWAY IS BY REMOTE RECRUITMENT AND MANAGEMENT TOOLS SO LIMITING THE AMOUNT OF FACE TO FACE INTERACTION PERHAPS ELIMINATING IT ALL TOGETHER, RECRUITING PEOPLE USING MOBILE TOOLS WHERE STUDIES POP UP ON THE PHONE, YOU CAN CHOOSE THESE STUDIES THAT CONNECT TO VARIOUS SENSORS THAT YOU CAN USE PUSH OFF THE SHELF OR HIP TO PARTICIPANTS. AND THEN THEY'RE OFTEN READY TO GO. YOU HAVE REMOTE MANAGEMENT TOOLS THAT YOU CAN RUN THESE STUDIES WITHOUT GAINING INTERACTION WITH STUDY DESIGNERS. THE OTHER BROADWAY IN WHICH YOU CAN SCALE UP USER STUDIES IS GIVING MORE CONTROL OVER WHEN USERS PROVIDE THE DATA, PROVIDING PRIVACY KNOBS, WHOM TO SHARE THE DATA WITH, THAT IS THEIR DATA SO THERE ARE WAYS TO SHARE DATA BEYOND THE SCOPE OF A PARTICULAR STUDY. THAT HOPEFULLY WILL MAKE THEM MORE COMFORTABLE WITH T WAY THE STUDIES ARE RUN. THESE ARE EXCITING AND HOPEFULLY THESE CAN ENABLE STUDIES BUT THERE ARE CHALLENGES -- THERE ARE METHODOLOGICAL ADVANTAGES AN CHALLENGES. METHODOLOGICAL ADVANTAGES ARE FOR EXAMPLE ADAPTIVE DESIGNS HAVE BEEN AROUND A LONG TIME INCREASINGLY. LAR BUT USING THE TOOLS YOU CAN HAVE TIGHTER CONTROL OF FAST ERA PID CHANGING RAPID DESIGNS. THE SIMPLE EXAMPLE, IF YOU DO SAMPLE SIZE REESTIMATION IN THE MID COURSE OF THE TRIAL AND YOU WANT TO CHANGE THE NUMBER OF PARTICIPANTS YOU USE THE REMOTE MANAGEMENT AND REMOTE RECRUITING TOOLS, YOU CAN CHANGE IT SO INSTEAD OF HAVING TO RECRUIT PEOPLE FOR OTHER PHASE FOR MANY DAYS MAYBE IN A FEW HOURS YOU'LL HAVE MORE PARTICIPANTS. SO IT CHANGE IT IS CONTROL LOOP. MAKES THE LATENCIES MUCH LOWER. THERE ARE OBVIOUS BENEFITS WITH RESPECT TO THE GRANULARITY OF DATA THAT YOU GET, NOT JUST PRE AN POST DATA, IT'S ALSO DATA DURING THE PROCESS, YOU CAN FIGURE OUT IF THE INTERVENTION TAKES PLACE QUICKLY, SLOWLY, THE LATER PART OF THE STUDY AND THE EARLY PART OF THE GENDER DIFFERENCES, SO FORTH. THOSE ARE GOOD BUT THERE ARE CHALLENGES THAT WE HAVEN'T QUITE ADDRESSED THAT NATURALLY EMERGE WITH THESE mHEALTH TOOLS. ONE CHALLENGE IS MISSING AND NOISY DATA. NOISY DATA IS NOT MENTIONED IN THE SLIDE. BUT THE MOMENT YOU USE MOBILE TOOLS ON BODY SENSORS YOU HAVE TO DEAL WITH MISSING DATA COULD BE JUST EITHER BECAUSE THE USER DOESN'T WANT TO PROVIDE IT OR CONNECTIVITY PROBLEMS, ARE DROPPED, THINGS BECAUSE THEY WERE BUSY. BUT WITH SENSORS ON THE BODY, SMALL CHANGES IN ORIENTATION, MAKE HUGE CHANGES IN THE QUALITY OF DATA. THAT NOT ONLY EFFECT IT IS RAW SEN SENSOR OF THE DATA, THERE MIGHT BE GAPS IN THE QUALITY. ANY BASIS AL GOR G RHYTHM YOU RUN ON TOP OF IT THE QUALITY IS CHANGING BECAUSE OF THAT. AND ERRORS CAN CASCADE OVER TIME SO THERE'S CHALLENGES IN TERMS OF WHAT ERRORS WE OBSERVE. THE GIEWD NEWS THAT'S A WEALTH OF TOOLS TO DEAL WITH NOISY DATA THAT HAVE -- BEING DEVELOPED TO DEAL WITH SORT OF VARIOUS MISSING DATA ESTIMATING MODELS OF THESE MISSING DATA, CLASSIFICATION, WHEN THERE IS NOISY AND MISSING DATA SO THAT YOU KNOW IF YOU IDENTIFY THAT SOMEBODY IS STRESSED YOU KNOW THEY'RE STRESSED WITH A CERTAIN PROBABILITY AND SO ON. SO THERE'S SOME TOOLS THAT WE CAN BRING TO BEAR FROM THE COMPUTING COMMUNITY BUT IT'S CLEARLY A BIG CHALLENGE WHEN YOU'RE RUNNING THESE mHEALTH STUDIES WITH CONTINUOUS SENSOR DATA STREAMS. THERE ARE OTHER QUESTIONS AS WELL. FOR EXAMPLE, WE RELY ON SELF-REPORTING QUITE A BIT. CAN USE mHEALTH TECHNIQUES TO INCREASE COMPLIANCE IN GENERAL. FOR EXAMPLE SELF-REPORTING REQUIRES SOME IS A QUESTION ABOUT SOCIALLY STIGMATIZED BEHAVIOR, OR IF IT IS -- USING A POPULATION LIKE A DRUG ABUSE POPULATION WHERE YOU DON'T KNOW THE QUALITY OF THE SELF-REPORTED INFORMATION THAT'S COMING THROUGH. YOU CAN USE MOBILE HEALTH VALIDATION. THERE'S MORE INFERENCES THAT WE CAN MAKE AND SO MANY PEOPLE WORKING ON WHAT KINDS OF THINGS YOU CAN INFERENCE FROM ON-BODY SENSOR, MORE YOU CAN SAY WHETHER OR NOT THE SELF-REPORTED INFORMATION IS TRUE BY CROSS VALIDATING WITH THE SENSORS AND INFERENCES. FOR EXAMPLE, SOMEBODY MIGHT CLAIM SOMETHING MORE SOCIAL INTERACTION OR THE AMOUNT OF SOCIAL INTERACTION THEY HAVE BUT USING RESPIRATION SENSORS YOU CAN VALIDATE THAT AND ACTUALLY FIGURE OUT HOW TRUTHFUL THEY'RE BEING OR HOW VALID IS THE DATA. OTHER THINGS WE CAN DO TO IMPROVE COMPLIANCE. FOR EXAMPLE, THE FACT THAT YOU HAVE THESE INCENTIVE MECHANISMS TO CHANGE FINE GRANULARITY MEANS THAT YOU CAN CHANGE -- YOU CAN FIGURE OUT PEOPLE HOW INDIVIDUALS RESPOND, CHANGE IN SENSORS OVER TIME SO THAT YOU INCREASE COMPLIANCE IN GENERAL AS WELL. HOPEFULLY THIS IS GIVEN A FLAVOR OF THE TYPES OF THINGS, THERE ARE NEW TOOLS AND CHALLENGES AN BENEFITS THAT THEY DOCUMENT WITH. [APPLAUSE] >> OUR LAST SPEAKER FOR THIS SESSION IS RICHARD KRAVITZ. >> HI. IT STRIKES ME AS IRONIC IN A MEETING THAT'S SUPPOSED TO BE ABOUT EVIDENCE I'M NOT GOING TO PRESENT MUCH EVIDENCE BUT ONLY MAKE A SERIES OF ASSERTIONS BECAUSE THAT'S ALL THE TIME THERE IS. SO FIRST SOP FIRST -- SOME FIRST PRINCIPLES. AS DON INDICATED, MOST CLINICAL RESEARCH PARTICULARLY RANDOMIZED CONTROL TRIALS FOCUS ON COMPARISONS BETWEEN GROUPS AND REPORT AVERAGE EFFECTS. BUT IT'S AT LEAST HYPOTHETICALLY POSSIBLE AND DEMONSTRABLY TRUE IN SOME SWIGS THAT INDIVIDUALS -- SITUATIONS THAT INDIVIDUALS VARY IN TREATMENT. , THEY MAY NOT VARY AS SHOWN ON THIS DIAGRAM BUT FOR ANY TREATMENT THE AVERAGE HERE IS REPRESENTED BY THE VERTICAL LINE. THERE MAYBE INDIVIDUAL WHOSE DERIVE GREATER THAN AVERAGE BENEFIT SHOWN ON THE RIGHT SIDE AND THOSE WHO DEVELOP LESS. SO WHAT WE'RE INTERESTED IN IS NOT THE AVERAGE EFFECT FOR GROUPS OF PATIENT BUS THE INDIVIDUAL TREATMENT EFFECT FOR A PERSON. THERE ARE REALLY ONLY THREE WAYS TO ESTIMATE INDIVIDUAL TREATMENT EFFECTS. ONE IS ONE APPROACH THROUGH INTERACTION ANALYSIS OR TREATMENT BY SUBGROUP ANALYSIS AND PARALLEL GROUP RCTs, THIS WOULD INCLUDE SUBGROUP ANALYSIS. SECOND IS A STATISTICAL TECHNIQUE KNOWN AS MATCH PAIR ACENALSIS, THIRD IS THE FOCUS OF MY REMARKS TODAY, N OF ONE CLINICAL TRIALS WHICH IS REALLY IN SWIGS SITUATIONS WHERE APPLICABLE IT'S ESTIMATING TREATMENT EFFECTS. N-1 TRIALS HAVE THREE KEY ELEMENTS BUT FIRST WHAT IS AN N OF ONE TRIAL? IT'S REALLY A CROSS-OVER TRIAL IN AN INDIVIDUAL PATIENT. YOU CAN SEE IN THE VERY TOP, SEE IF I CAN BRING EVERYTHING IN HERE. AT THE TOP RIGHT THE DIAGRAM SIMPLY SHOWS, DURING A PARTICULAR EPISODE FOR AN INDIVIDUAL THEY MIGHT BE ASSIGNED TO TREATMENT A, AND THEN TO TREATMENT B AND MAYBE TREATMENT A AGAIN, B, A, ET CETERA. AND THEN OUTCOMES ARE MEASURED AT THE END. THREE KEY ELEMENTS OF A FEW BLOWN N OF ONE TRIAL ARE RANDOMIZATION, THE PURPOSE OF WHICH IS TO CONTROL FOR TEMPORAL AND ORDERING EFFECTS. SO AS I SORT OF SUGGESTED BY ALTERNATING A AN B IN A MORE OR LESS RANDOM FASHION ONE CAN BEGIN TO CONTROL FOR THAT. THE GREATER THE NUMBER OF EPISODES THE BETTER THE CONTROL FOR THOSE FACTORS IT BECOMES. THE SECOND ELEMENT IS BLINDING. SO OBVIOUSLY THIS IS MOST RELEVANT FOR MEDICATION OR A PILL IN WHICH TREATMENTS ARC AND B CAN BE -- A AN B CAN BE PLACED INTO IDENTICAL LOOKING DELIVERY PACKETS. AND THEN THIRD AND ARGUABLY MOST IMPORTANT IS SYSTEMATIC MEASUREMENT AND mHEALTH OF COURSE OFFERS THE OPPORTUNITY TO IMPLEMENT SYSTEMATIC MEASUREMENT IN A MORE REFINED AND MORE FREQUENT WAY. A FULL-BLOWN N OF ONE TRIAL HAS ALL THREE ELEMENTS BUT ARGUABLY ONE COULD ELIMINATE ONE OR MORE OF THESE PARTICULARLY RANDOMIZATION AND BLINDING AND STILL RETAIN ADVANTAGES OF THE N OF ONE TRIAL. SO IN TERMS OF SYSTEMATIC DATA COLLECTION, IN THE OLD DAYS WE HAD SYMPTOM DIARIES WHERE PATIENTS WRITE DOWN THEIR SYMPTOMS OR FILL OUT QUESTIONNAIRES ON A RECURRING BASIS. IN THE YEAR OF mHEALTH WE HAVE ELECTRONIC DIARIES. SOME ARE ALREADY ON THE MARKET. FEW OF THEM HAVE BEEN VALIDATED. ON THE FAR RIGHT HAND SIDE YOU SEE ONE ITEM FROM THE ITEM DATA BANK ORGANIZED BY PROMISE.ORG, MY KIDS OFTEN ACCUSE ME OF TAKING GROUCH PILLS SO I HAVE STARTED TO COMPLETE THIS QUESTIONNAIRE ON A REGULAR BASIS. I FIND UNFORTUNATELY MY IN-PERSON VARIANCE IS RATHER LOW. SO THE VALUE PROPOSITION FOR N OF ONE HEALTH, IN THE EARLY PHASE WE USE N OF ONE TRIALS IN ISOLATION USING mHEALTH AS A TOOL TO GUIDE THERAPY FOR INTERVENTION FOR INDIVIDUALS, SORRY. BUT LATER ON AS WE BEGIN TO ACCUMULATE DATA, N OF ONE DATA ON A SERIES OF INDIVIDUALS YOU COULD USE BAYESIAN APPROACHES TO COMBINE DATA ON INDIVIDUALS WHO HAD PREVIOUSLY BEEN ENTERED INTO N OF ONE TRIALS FOR A PARTICULAR TREATMENT WITH THE DATA IN COMING FROM A NEW INDIVIDUAL TO BORROW FROM STRENGTH AND IDENTIFY THE MOST APPROPRIATE TREATMENT FOR THAT PERSON AS WELL AS DEVELOPING A BETTER NOTION OF WHAT THE IDEAL TREATMENT IS FOR BOTH INDIVIDUALS AND POPULATIONS. THAT'S IT. THANKS. [APPLAUSE] >> SO WE'RE GOING TO OPEN THIS UP TO QUESTIONS. I INVITE YOU TO ASK ANY OF OUR PANELISTS QUESTIONS. ALSO THERE WERE SOME WHITE PAPERS IN THIS SESSION THAT WERE IN YOUR PACKET THAT DIDN'T PRESENT SO IF YOU HAVE QUESTIONS OF THOSE TWO YOU CAN ASK THOSE HERE. SO I OPEN UP TO QUESTIONS NOW. >> PLEASE GO TO THE MICROPHONE. >> OR WENDY WILL BRING THEM TO YOU. >> THIS IS FOR RICH. SO ABOUT THREE OR FOUR WEEKS AGO I WAS WITH A PAIN GROUP WORKING WITH PEOPLE WITH FIBROMYALGIA AND THEY WANT TO DO AN N-OF-1 TRIAL. THEY HAVE ABOUT 200 PATIENTS THEY WANT TO DO THIS WITH. THEY HAVE I THINK FOUR OR FIVE DIFFERENT TREATMENTS SOME OF WHICH ARE MEDICATION, ONE OF WHICH IS A BEHAVIORAL THERAPY. ONE OF THE CONCERNS I HAD WAS IF SOMEONE IS DOING mHEALTH SUPPOSE YOU GIVE JOE A TREATMENT A AND HE DOES REALLY WELL. NOW YOU'RE GOING TO TAKE HIM OFF THAT TREATMENT? YOU HAVE TO YOU HAVE TO FIGURE OUT WHICH IS BEST. YOU MAY DO BETTER BUT YOU DON'T KNOW. SO THERE'S THIS WHOLE ISSUE OF DO YOU THEN AUGMENT, IT'S NOT CLEAR WHAT DO THE SUBSEQUENT -- WHICH OF THE SUBSEQUENT TREATMENTS BE IF A PATIENT DOES WELL ON A TREATMENT. THEN THERE'S CARRY-OVER EFFECT. MAYBE YOU SHOULD COMMENT ON THE FIRST BECAUSE THAT WAS THE MOST CONCERNING TO THE CROWD. THEY DO WELL ON A TRIAL AND THEN YOU TAKE THE TREATMENT AWAY. OF COURSE (INAUDIBLE). >> RIGHT. THE QUESTION IS HOW TO GET PATIENTS TO PARTICIPATE IN N-OF-1 TRIAL, PATIENTS AND CLINICS FOR THAT PEART WHEN THEY'RE CONVINCED A PARTICULAR TREATMENT IS SPEAR WROR TO THE OTHERS OR THEY COME TO THAT CONCLUSION AS A RESULT OF THE EARLY EXPERIENCE. N-1 TRIALS HAVE CONSTRAINT, ONE THE PATIENT AND CLINICIAN HAVE TO HAVE SOME DEGREE OF EQUIPOISE AT THE BEGINNING OF THE TRIAL AND AS PART OF THE ONGOING PARTICIPATION. IF EQUIPOISE APPEARS BECAUSE THE TREATMENT IS DRAMATIC, LIKELY THAT PATIENT IS GOING TO DROP OUT OF THE TRIAL AN MAYBE THE STATISTICIANS CAN OFFER UP WAYS OF HANDLING THAT. THAT'S GOING TO HAPPEN. YOU MENTION CARRY-OVER EFFECT. I'LL MENTION OTHER LIMITATIONS. I MEAN N-OF-1 TRIALS DON'T WORK FOR ONE TIME EVENTS. THEY DON'T WORK WHEN THERE'S PROLONGED CARRY OVER EFFECTS OF TREATMENT, THEY DON'T WORK WHEN TREATMENT TAKES A LONG TIME TO TAKE EFFECT SO THERE'S A NUMBER OF LIMITATIONS BUT FOR PAIN IS ONE OF THE CONDITIONS IN WHICH N OF OF 1 TRIALS MOST FREQUENTLY APPLIED. WE DID A SYSTEMATIC REVIEW OF N OF ONE TRIALS IN LITERATURE, AROUND A THOUSAND HAVE BEEN PUBLISH, A THOUSAND CASES PUBLISHED. MAYBE A THIRD HAVE TO TO WITH PAIN RELATED CONDITIONS. COUPLE OF QUESTIONS ABOUT THE STEP DESIGN AND THE CHALLENGES THERE. (INAUDIBLE) SECOND OF THE STEP TALKS ABOUT HOW SOME CLINICS OR INDIVIDUALS BENEFIT AND SOME MIGHT NOT. SO AS YOU GO THROUGH THE STEPS IT SEEPS THERE MAYBE AN OFFSETTING FOR DIMINISHING TO SEE BENEFIT SO THE SITES ARE NOT -- DONE HAVE THE PERSONNEL OR THE INFRASTRUCTURE TO MAKE THE IMPACT WORK WHERE OTHER ONES DO. THEY MAY BLEND. AND DIMINISH YOUR BENEFITS SO HOW MANY STEPS DO YOU NEED, HOW MANY SITES DO YOU NEED BEFORE YOU SHOW YOUR IMPACT. >> I'LL GIVE MY ANSWER, IT DEPENDS HA YOU'RE STUDYING AND THE EFFECT. IT'S -- THE ACTUAL COMPARISON IN A STEP WEDGE DESIGN ARE SITES THAT HAVE AT A GIVEN TIME POINT THE SITES THAT HAVE INTERVENTIONS COMPARED TO THOSE THAT DON'T. TO YOUR POINT WHAT YOU'LL SEE IS SITES THAT COME ON IN EARLY PHASE. WHEN THEY'RE FOUR YEARS OUT MAYBE IT TAKES FOUR YEARS TO PHASE IN YOUR INTERVENTION, IF IT'S SOMETHING THAT TAKES A LONG TIME TO SCALE TO A SITE AND GET A SITE ONLINE AND RUNNING. THE EFFECT AT FOUR YEARS MAYBE SUSTAIN -- THE SUSTAINED EFFECT AT ONE YEAR OR MAYBE A DIMINISHED EFFECT. IT'S HARD TO PREDICT WHAT'S GOING TO HAPPEN. IT ALLOWS YOU TO SEE THE LONGER TERM EFFECTS. SO WHAT YOU MAY END UP HAVING AT THAT LATER TIME POINT ARE SITES THAT DON'T HAVE THE INTERVENTION, SITES THAT HAVE HAD THE INTERVENTION FOR A YEAR, SITES WITH INTERVENTION FOR TWO YEARS. SO YOU MIGHT HAVE MULTIPLE ARMS, NOT JUST INTERVENTION ARM AN NON-INTERVENTION ARMS. IN TERMS OF THE NUMBER OF CLINICS YOU NEED ONLINE, IN ORDER TO SEE AN EFFECT IT DEPENDS ON THE AMOUNT OF CLUSTERING YOU SEE IN YOUR DATA. SO WHAT'S GOING TO DRIVE THAT TO MAKE SURE YOU HAVE SUFFICIENT POWER IS THE INTRACLUSTER COORDINATION THAT DRIVES NUMBER OF CLUSTERS YOU NOOD FOR TRADITIONAL RANDOMIZED TRIAL. WE HAD 20 CLINICS WE HAD BEFORE AN AFTER MEASUREMENTS. THAT'S SMALL FOR CLUSTERED RANDOMIZED TRIALS BUT OUR EFFECT WAS SO BIG WE WERE SUFFICIENTLY POWERED TO SEE A DIFFERENCE. I DON'T KNOW IF THAT'S A SATISFYING ANSWER. IT REALLY DEPENDS ON WHAT YOUR CLUSTERING EFFECT IS. >> IT MAY REQUIRE MULTIPLE -- MULTI-CENTER EFFORT WITH CERTAIN INTERVENTIONS TO PULL THAT OFF. MY OTHER CONCERN TOO mHEALTH IS SINCE IT'S EVOLVING SO QUICKLY, IF YOU'RE A YEAR OUT BY THE TIME YOU GET NEXT GROUP ON, THE INTERVENTION CHANGED OR THERE'S A LEARNING CURVE FROM THE SITE THAT STARTED EARLY THAT MAY CHANGE ITS BEHAVIOR HOW THEY'RE WORKING SO YOU HAVE MULTIPLE EFFECTS GOING ON, MULTIPLE EXPERIMENTS GOING ON AT THE SAME TIME. >> JUST TO COMMENT ON A SPECIFIC APPLICATION WE'RE WORKING ON. ONE OF THE PARTNERS THAT HAVE SEEN OUR PRELIMINARY RESULTS SHOW THAT WE CAN REALLY IMPROVE HEALTHCARE WORKERS ADHERENCE TO A PROTOCOL THEY HAVE ADOPTED. ONE OF OUR PARTNERS HAS COME TO US AND SAID WE HAVE 20 SITES WE WANT YOU TO DEVELOP THIS TOOL FOR. SO WE ALREADY KNOW WHAT OUR MAXIMUM CLUSTER SIZE IS GOING TO BE, IT'S 20, IT'S AN IMPLEMENTATION PROJECT. WE WANT TO VALUE THE EFFECTIVENESS WITHIN THAT. SO IT'S A DISCUSSION BETWEEN THE IMPLEMENTATION TEAM AND THE RESEARCH TEAM. WE KNOW TO GET A SITE ONLINE WILL TAKE ABOUT THREE MONTHS. SO OUR TIME BETWEEN PHASES WILL BE THREE MONTHS, IT WILL MATCH THE IMPLEMENTATION SCALE-UP PLAN. WHICH SITES COME ONLINE AT WHICH PERIODS, WE'RE GOING TO DO IN A RANDOM PROCESS, WE DO ACTUALLY HAVE THE CONTROLS TO DO IT IN A RANDOMIZED WAY, THAT'S NOT ALWAYS POSSIBLE WITH IMPLEMENTING PARTNERS. SO IT'S A DIALOGUE BETWEEN THE IMPLEMENTING PARTNERS AND THE RESEARCH TEAM TO COME UP WITH WHAT THAT WILL LOOK LIKE. >> COUPLE OF OTHER COMMENTS TOO, I'M WORKING WITH GROUPS NOW, IT'S THE SAME THING WHERE THEY HAVE A FIXED NUMBER OF CLUSTER SITES BUT WHAT THEY'RE TRYING TO DO IS LOOK AT ARE THERE SHORTER-TERM OUTCOMES THEY CAN MEASURE, THEY HAVE SOME OUTCOMES THEY KNOW TAKE AT LEAST 6 TO 8 MONTHS TO OBSERVE. THEY'RE TRYING TO LOOK TO SEE CAN THEY HAVE MORE INTERVALS OF TIME, ANOTHER WAY OF DEALING WITH SOME OF THE CONFOUNDING AND OTHER DESIGN ISSUES. SO IF YOU CAN LOOK AT MORE INDICATORS OVER PERIODS OF TIME THAT'S ANOTHER WAY OF ADDRESSING THE DESIGN IF YOU HAVE A NUMBER OF FIXED CLUSTERS. >> I WAS GOING TO NOT ASK MY QUESTION BECAUSE YOU ASKED IT BUT THEY DIDN'T ANSWER IT. I THINK THE GOAL OF HAVING -- WHAT WAS YOUR GOAL TO REDUCE IT TO HOW MANY YEARS? 3.5 YEARS STARTING IN 2/16 SO WE'RE PLANNING NOW STUDIES GOING ON IN 2017, 18 AND TALKING ABOUT SMS AND THINGS LIKE THAT. THAT'S A DISCONNECT BETWEEN THE TECHNOLOGY. IS THERE EXPERIENCE FROM PAST HEALTH INTERVENTIONS WHERE WHAT YOUR INTRODUCING NINE MONTHS LATER IS DIFFERENT FROM WHAT YOU INTRODUCED THREE GROUPS BEFORE? BUT WITHIN THE SAME STUDY. HOW DO YOU DEAL WITH THAT SINCE OUR -- THE NATURE OF THE INTERVENTION AT SOME LEVEL WILL BE SIMILAR FROM A CLINICAL PERSPECTIVE BUT THE WAY IT'S DELIVERED WILL BE CHANGING. >> THE EXPERIENCE -- >> THE EXPERIENCE I HAVE HAD IN THAT IS LOOKING AT IT FROM A FIDELITY FRAMEWORK WHEREAS IF MANY THE IMPLEMENTATION PROJECT WE HAVE CURRENTLY IT'S NOT THAT THE INTERVENTION IS CHANGED, IT'S A COUNSELORS THAT HAVE BEEN PROVIDED BY THE HEALTH PLAN CHANGE ALL THE TIME SO WE'RE GOING THE LOOK AT THE FIDELITY EFFECTS BUT IT'S MORE OBSERVATIONAL. AFTER WE COLLECT THE DATA WE MAY HAVE INSIGHTS HOW THAT CAN BE BETTER ADDRESSED BUT AT THIS POINT WE'RE GOING TO TAKE A LOOK AT THAT CHANGE OVER TIME. IT IS THE ISSUE, WITH ANY INTERVENTION THAT OVER TIME THERE'S CHANGES. >> HI. I HAVE BEEN PART OF THIS PANEL TOO WE'RE PRESENTING BUT I HAVE BEEN -- MY NAME IS MARC FROM COLUMBIA UNIVERSITY WHERE I WAS DOING SIX YEARS OF RANDOMIZED CONTROL AND RANDOMIZED CONTROL TRIALS. I THINK THE QUESTION IN MY EXPERIENCE IS NOT SO MUCH RANDOMIZED, WE CAN DO 3.5 YEARS OR 5.5 YEARS BUT THERE IS REALLY ANOTHER FIELD OF EXPERIENCE THAT SHOWS LIKE VACCINE SAFETY, AS AN EXAMPLE. THEY CAN GET THE DATA IN A WEEK AN ANALYZE IT SO WHY WAIT 3.5 YEARS IF YOU CAN GET IT IN ONE WEEK. THE THING IS HOW DO YOU GET THIS DATA FROM MOBILE HEALTH BECAUSE THE VACCINE DATA THEY HAVE A LOT OF ELECTRONIC HEALTH RECORDS THAT FIT THIS DATA SO THEY CAN GET THIS DATA ON ONGOING BASIS. WE CANNOT. WE DID THE STUDY SIX MONTHS, WHAT HAPPENS AFTER SIX MONTHS? SO THE QUESTION FOR ME WOULD BE NOT ONLY HOW TO REDUCE THE TIME BECAUSE THE STATISTICAL METHODS THEY USE AN (INAUDIBLE) ORGANIZATIONS USE ARE MUCH MORE SOPHISTICATED, THAT THE METHODS WE'RE USING IN MOBILE HEALTH BUT HOW CAN WE GET A CONTINUOUS DATA STREAM OF PATIENT DATA BEYOND SIX MONTHS, BEYOND THE STUDY THAT STAYS IN THERE. SO SHOULD WE FUN FOR EXAMPLE, IF (INAUDIBLE) IS GOING TO GIVE A GRANT TO AN TORSION DO SOMETHING TO IMPROVE MATERNITY CARE IN AFRICA, SHOULD THEY GIVE IT TO RESEARCH TEAM AT COLUMBIA OR TO (INAUDIBLE) THAT THERE START UP A THING GATHERING DATA FOR TEN YEARS? WHAT HAPPENS AFTER THE GRANT IS GONE? THERE IS NO MORE DATA. THEN WE'RE NO DIFFERENT THAN ANYBODY ELSE. I THINK WE'RE IN THE BUSINESS NOT OF RANDOMIZED OR NON-RANDOMIZED. WE'RE IN THE BUSINESS OF BIG DATA. HOW CAN WE PRODUCE BIG DATA THAT IS NOT CAPTURING THE ELECTRONIC MEDICAL RECORDS, IT SETS US APART FROM THE REST OF THE (INAUDIBLE) SO I THINK WE SHOULD ALSO FOCUS TO METHODS HOW TO DRIVE PATIENT ENGAGEMENT BECAUSE IT'S THE PATIENT THAT WILL PROVIDE WAS THE DATA. HOW CAN WE DERIVE PATIENT ENGAGEMENT AND WHAT SENSE CAN WE FIND TO GET THIS DATA ON ONGOING BASIS IS THE KEY. BOTH RANDOMIZED AND NON-RANDOMIZED. MY EXPERIENCE YOU FINISH THAT YOU DEMONSTRATE THE EVIDENCE IS THERE. HOW MANY PEOPLE IN THIS ROOM KNOW ABOUT (INAUDIBLE)? PROBABLY NOBODY. I HAVE (INAUDIBLE) 50%, THINGS THAT COULD BE DONE NATIONWIDE SO THE EVIDENCE IS NOT ENOUGH. TO MAKE THIS EVIDENCE GO INTO THE REAL WORLD. SO YOU HAVE TO ESTABLISH PRIVATE/PUBLIC PARTNERSHIPS TO BRING THIS THING IN. SO THE METHODS ARE INTERESTING BUT IT'S ONLY ONCE YOU HAVE THE CONTINUOUS DATA STREAM YOU CAN BEGIN TO REFINE THE METHODOLOGIES. >> SO ON THAT QUESTION, ON THAT COMMENT I NOTE THAT THERE'S ALREADY SOMEWHAT OF A MOVE TOWARDS PEOPLE COLLECTING AND THEIR OWN DATA CONTINUOUSLY FOR REASONS THAT ARE PERSONAL, THAT I HAVE MY DATA AND I CAN SHARE IT WITH SIMILAR COMMUNITIES, PATIENTS LIKE ME AT THE WEBSITE WHERE PEOPLE SHARE THE DATA AND THEY HAVE DONE ANALYSIS, THAT KIND OF DATA. AND IT GETS TO I MEAN I THINK AT SOME LEVEL THERE WILL BE FINITE STUDIES, THE QUESTION IS, WHETHER INDIVIDUALS ARE EQUIPPED TO COLLECT THE DATA CON WASLY. ONE CAN BUY AWFUL THE SHE WOULDS AN AND PERHAPS FIT THOSE ON THE BODY. CERTAIN KINDS OF INTERVENTIONS YOU CAN ACTUALLY DO THAT. BUT THERE ARE REASONS WHY INDIVIDUALS KEEP COLLECTING THE DATA AND PERHAPS THERE WILL BE A MARKETPLACE FOR THAT WHERE YOU CAN BUY DATA FROM INDIVIDUALS AND USE FOR FUTURE STUDIES. >> GOOD QUESTION. >> I HAVE ONE INDIRECT RESPONSE TO AN ISSUE YOU RAISED. PATIENT ENGANGMENT IS KEY. ONE WAY SOME N OF ONE ENGAGERS HAVE BEEN ALLOWED TO SELECT THE OUTCOME MOST IMPORTANT TO THEM. ANNULS OF INTERNAL MEDICINE IN THE MID 80s, HE ASKED HIS PARABLES TO SELECT OUTCOME, MANY SELECTED THINGS SUCH AS BEING ABLE TO CLIMB UP TO THE SECOND FLOOR OF THEIR HOUSE WITHOUT STOPPING BECAUSE OF SHORTNESS OF BREATH. THAT IN ITSELF IS MOTIVATING AND ENGAGING, IT RAISES SOME QUESTIONS IN TERMS OF COMPARISON BETWEEN INDIVIDUALS AND MAYBE EVEN IN TERMS OF MEASUREMENT VALIDITY BUT IN TERMS OF ENGAGEMENT, IT BROUGHT THE PATIENTS IN. >> INITIATIVE FROM THE FATD REACH OVER 100 MILLION PEOPLE IF SOMEBODY FINDS SOMETHING -- IF I WAS A FUNDER IF YOU DONE TELL ME YOU'RE ABLE TO REACH AT LEAST 10% OF THE TARGET POPULATION ON ONGOING BASIS, I WILL NOT FUND YOU. BECAUSE THAT SAME STUDY IMPROVE IMMUNIZATION RATES BY 50% BUT IF SO -- IT'S SO INTENSIVE AFTER THE FUNDING IS GONE THIS CANNOT BE SUSTAINED SO I'M NOT INTERESTED IN THAT. I'M INTERESTED IN STUFF THAT IS SUSTAIN BIBLE AFTER THE FUNDING IS GONE. AND THAT IS ABLE TO REACH 100 MILLION PEOPLE OR TEN MILLION, IT'S A BENCHMARK, 10% OF THE PATIENT POPULATION NATIONWIDE. IT CAN PUT THE SENSORS IN THE MARKET BUT WHO PAYS FOR THE SENSORS, ARE THE SENSORS OF COMMERCIAL VALUE, ARE THE PATIENTS WHEN THEY HAVE TO DO THE MAINTENANCE OF THE SENSORS P LOOING TO DO THESE MENTORS? SENSORS CAN WORK IN AN EFFICACY ENVIRONMENT, IN AN EFFECTIVENESS ENVIRONMENT IT DOESN'T WORK. TRYING TO RAISE THE FUND EFFECTIVENESS NOT FUND ACTIVITY. >> IN THE EUROPEAN COUNTRIES THEY HAVE MORE SOCIALIZED MEDICINE SO THEY HAVE VAST MEDICAL RECORDS FOR ALL THEIR PATIENTS. I'M NOT SURE HOW THAT WORKS HERE BUT I'M WONDERING IF IN THAT SET UP PEOPLE ARE ACCUMULATING THESE LARGE DATA BASIS AND ARE ABLE TO MAKE PROGRESS YOU'RE TALKING ABOUT. LET'S GO TO THE VA THAT'S THE ANALOG IN THE UNITED STATES. YOU HAVE MASSIVE DEA SETS, HUNDREDS OF THOUSANDS OF PATIENTS ELECTRONIC MEDICAL RECORDS, PERHAPS THAT'S THE WAY TO PARTICIPATE YOU HAVE TO -- YOU HAVE DIABETES, YOU WANT TO WEAR CERTAIN SENSORS OR WHATEVER, MAYBE THAT'S THE WAY THESE BIG, BIG HEALTH SYSTEMS ONE HAS SOME HOPE OF COLLECTING THAT AMOUNT OF DATA PEOPLE WHO VOLUNTEER ON THE WEB THAT'S A SPECIAL GROUP OF PEOPLE BUT (INAUDIBLE). >> SUSAN MURPHY. UNIVERSITY OF (INAUDIBLE). >> IN THE U.S. THERE'S BEEN AN EXAMPLE CALLED (INAUDIBLE) THAT THE COMPANY WAS THE GOVERNMENT, A NUMBER OF ORGANIZERS ARE DOING. IT'S PRE-NATAL CARE FOR WOMEN AND IT'S OPEN TO THE WHOLE PEOPLE WHO ARE ABOUT TO HAVE A BABY. AND THERE'S HUNDREDS OF PEOPLE GETTING THE INTERVENTION SO THAT CONVINCES ME YOU CAN DO SOMETHING LIKE THAT. THERE IS NO TIME, ANYONE CAN SIGN UP, THERE'S SOME THAT YOU STILL HAVE TO FIGURE OUT THE BUSINESS MODEL AROUND THAT. BUT THIS IS MUCH MORE INTERESTING TO ME THAT SOMETHING IS ONLY AFFECTING 500 PATIENTS. IN EUROPE YOU COULD DO THIS. BUT IN THE -- WHO PAYS FOR THE OUTCOMES, WHO IS INTERESTED IN THE OUTCOMES IF THE GOVERNMENT HAVING TO DO THAT OR NOT. >> (INAUDIBLE) OF MARYLAND. MY QUESTION THAT I HOPE WE TALK ABOUT TODAY IS A LITTLE BIT MORE THE PRACTICALITIES OF OUR RESEARCH AS WELL. SO FOR EXAMPLE, YOU CAN HAVE THESE INCREDIBLE DATABASES, SENSOR DATA, THE ECOLOGICAL MOMENTS. HOW ARE WE GOING -- HOW DO WE WORK THIS THROUGH OUR IRBs AND THAT PATIENTS KNOW IN ADVANCE THEIR DATA IS GOING TO BE USED OTHER THAN IT JUST BEING DEIDENTIFIED? BECAUSE OUR IRBs ARE REALLY STILL IN THE MOLD OF RANDOMIZED CLINICAL TRIALS. SO I'D BE INTERESTED IF ANY PANELISTS CAN TALK ABOUT THEIR EXPERIENCE. >> IN THE PROJECT THAT WE HAD BECAUSE WE WERE WORKING WITH OUTSIDE ORGANIZATION AND THIS THEY WERE IMPLEMENTERS DEFINED AS A QUALITY IMPROVEMENT PROJECT WE HAD A LIMITED REQUIREMENT WITHIN OUR IRB. WE HAD TO DESCRIBE IN GREAT DETAIL TO THEM WHY WE CARE TO EVALUATE SOMETHING THAT WE WEREN'T OURSELVES IMPLEMENTING. THAT WAS PROBABLY THE BIGGEST CHANGE IN TERMS OF THEIR THINKING THAT THAT MIGHT BE -- WE MIGHT HAVE THAT KIND OF A ROLE BECAUSE USUALLY YOU DO THE WHOLE STUDY, YOU DEVELOP IT AND YOU EVALUATE FT IT SO ONCE WE CROSS THAT HURDLE IT WAS SIMPLE, QUALITY IMPROVEMENT ORIENTED IRB PROCESS. >> I THINK THE SHORT ANSWER IS WE DON'T UNDERSTAND. AND THE IRBs FOR MOBILE HEALTH ARE -- IT'S A COMPLICATED QUESTION. WE'RE STILL TRYING TO UNDERSTAND IT. OTHERS MAY HAVE MORE EXPERIENCE WITH DOING SOME OF THESE STUDIES. BUT THE FIRST QUESTION IS IT WAS HARD ENOUGH TO UNDERSTAND PAPER-BASED IRBs, THERE ARE IRBs ON A COMPUTER SCREEN, WAY HARDER TO UNDERSTAND AN IRB THAT POPS UP, LET'S SAY YOUR PHONE CONNECTS WITH SENSOR ON THE BODY. IT'S WAY HARDER TO UNDERSTAND WHAT -- NOBODY READS THESE ANYWAY, NOBODY READS, FOR APPS WE DOWNLOAD, WHY WOULD WE READ AN IRB IN SMALL FONT? SO THAT'S ONE QUESTION. THEN OF COURSE THE OTHER BIGGER QUESTION ALSO IS THERE'S SO MANY THINGS WE DONE UNDERSTAND ABOUT WHAT'S INFERED FROM THE DATA. IT'S ONE THING TO SAY HERE ARE THINGS WE'RE MEASURING BUT OFTEN TIMES WE GO BACK TO THE DATA AND THERE'S CORRELATION BETWEEN THE PHYSIOLOGICAL, THE GPS, THE LOCATION INFORMATION AND SO MANY OTHER THINGS YOU CAN MEASURE. WE DONE KNOW BEFOREHAND WHAT WE CAN INFER FROM IT. AN EXAMPLE, YOU PUT THESE CHEST BANDS AND DRUG ABUSE POPULATION YOU CAN FIGURE OUT WHERE THEIR (INAUDIBLE). YOU CAN FIGURE OUT THINGS ABOUT THEIR PATTERNS AND SO MANY THINGS THAT THEY MAY NOT NECESSARILY KNOW OR THINK ABOUT BEFOREHAND. SO I THINK THIS IS A QUESTION, THIS IS AN ONGOING QUESTION. I DON'T THINK THERE'S A QUICK ANSWER TO THE QUESTION. BUT FROM BOTH FROM THE PERSPECTIVE HOW DO YOU REDUCE IRBs TO BE MEANINGFUL ALL THE WAY TO WHAT ARE WE GOING -- HOW DO WE ACTUALLY CONVEY WHAT WE'RE GOING TO DO WITH THE DATA AS WE GO THROUGH IT. SO IT'S A HARD QUESTION TO ANSWER BUT WE'RE IN THE PROCESS OF FIGURING IT OUT AND WE HAVE TO DO IT. >> BARBARA MIDDLEMAN FROM THE NIH. I WANT TO FOLLOW-UP ON THAT QUESTION. I THINK THIS IS AN ISSUE THAT HAS NOT ONLY TO DO WITH IRBs BUT TYPICALLY LOOKING AT CLINICAL RESEARCH. EVALUATION ISN'T RESEARCH AT LEAST IN THE LEXICON OF THE IRB. SO THESE ARE PROGRAMS THAT DON'T NECESSARILY NEED TO GO THERE. AND AS YOU MENTIONED, IF YOU CAN INFER WHERE SOMEBODY IS, WHAT THEY'RE DOING AND WHETHER THEY'RE DOING THINGS ILLEGAL OR NOT LEGAL OR THAT MAY REALLY COMPROMISE THEIR PRIVACY IN IMPORTANT WAYS, I THINK WE ARE NOT COMMUNICATING TO PATIENTS WHAT THE REAL RISK IS. NOT OUT OF MALICE ON THE PART OF THE INVESTIGATOR BUT BECAUSE I THINK WE DON'T APPRECIATE WHAT THE REAL RISK IS. SO THE PRIVACY RULES IS BEING RE-EVALUATED AND REDRAFTED NOW, WE HAVE IRBs THAT WORK UNDER A SET OF RULES AN DIRECTIONS WHICH ARE NOT REALLY SUITED TO THESE KINDS OF TECHNOLOGICAL STUDIES. WE'RE GENERATING DATABASES WHICH ALTHOUGH THEY'RE WITHIN THE HIPAA CONSTRAINTS AND THE DATA IS DEIDENTIFIED IN FACT PROVIDE VERY IDENTIFYING INFORMATION AND I THINK WE HAVE A REAL DISCONNECT HERE. THIS IS VERY MUCH LIKE THE DISJUNCTURE TALKED ABOUT EARLIER ABOUT TECHNOLOGY AND RCTs, THIS IS THE TECHNOLOGY BETWEEN HOW WE DEAL WITH PATIENT, SUBJECTS, HOW WE DEAL WITH THE PUBLIC AND HOW WE CONDUCT OUR WORK AS RESEARCHERS AND OUR ATTEMPT TO UNDERSTAND BETTER THE RELATIONSHIP BETWEEN HEALTH AND A WHOLE BUNCH OF OTHER FACTORS. SO THIS IS NOT EXACTLY A QUESTION, IT'S MORE A REQUEST. BECAUSE AT SOME POINT IF WE IN THE RESEARCH IMMUNITY DON'T PROVIDE INSIGHT AND PROVIDE RECOMMENDATIONS AS TO HOW TO REFI GUR THE RULES, HOW TO REFI GUR THE EVALUATION CRITERIA FOR WHAT IS AND ISN'T RESPONSIBLE RESEARCH, WHAT IS AND ISN'T PROTECT THING PATIENT'S PRIVACY AND CONFIDENTIALITY THE RULES WILL GET IMPOSED ON US ARE NOT NECESSARILY GOING TO BE USEFUL FROM A RESEARCH STANDPOINT. OR USEFUL FROM A PUBLIC HEALTH STAND POINT. IF YOU GUYS HAVE COMMENTS I WOULD LOVE TO HEAR THEM. BUT I THINK IT'S A HUGE PROBLEM AND A LOOMING ONE THAT REALLY DOESN'T -- IT'S MORE A BLACK CLOUD IN THE BACKGROUND, IT DOESN'T HAVE A DEFINED SHAPE, BECAUSE IT DOESN'T HAVE A DEFINED SHAPE IT'S HARD TO DEAL WITH. I'M INTERESTED IN YOUR OPINION. >> CAN I JUST EXPAND ON BAR BREAS COMMENT? I THINK THERE'S A REALLY IMPORTANT THING IN WHAT BARBARA SAYING. HHS IS RIGHT NOW IN PUBLIC COMMENT ON THE COMMON RULE. THE COMMON RULE IS THE RULES THAT DRIVE ALL YOUR IRBs. SO RIGHT NOW IS THE TIME TO BE THINKING ABOUT THIS AN COMMENTING ON THIS. FOR THOSE OF YOU THAT DO IRB STRUCTURED RESEARCH. THEY'RE FUNDAMENTALLY RETHINKING HOW WE DEFINE RISK IN IRB RESEARCH. THIS IS AN IMPORTANT PART TO THINK ABOUT. I'LL LET THE PANEL ANSWER BUT FOR THOSE OF YOU WE'LL SEND OUT A LINK TO THE COMMON RULE SITE THAT REALLY THEY'RE LOOKING FOR YOUR COMMENTS BUT I THINK THIS IS A SPECIAL PART OF THE RESEARCH mHEALTH THAT MIGHT REQUIRE ITS OWN SET OF COMMENTS. >> DONNA (INAUDIBLE) UNIVERSITY OF CALIFORNIA. I JUST A SIDE BAR, NOT SO MUCH A SIDE BAR, PRIME PMR IS THE YOUNG IRB CONFERENCE LAST YEAR WE HAD A SESSION ON MOBILE HEALTH. THERE IS A GREAT NEED FOR PUBLIC COMMENTARY AND PUTTING IDEAS DOWN ON PAPER. I'M BEGINNING THAT PROCESS AS PART OF MY FUNCTION AS DIRECTOR OF RESPONSIBLE RESEARCH. THOUGH I'LL MOBILE RESEARCH I HAVE A BACKGROUND IN MEDICAL ETHICS, THIS IS AN EXTREMELY IMPORTANT AREA. SO I HAVE -- MY EMAIL IS IN THE BOOK, PLEASE ALL THE STUFF SEND TO ME BECAUSE I'M TRYING TO PUT SOMETHING TO THAT'S COHEREIN. LET ME REITERATE HOW IMPORTANT IT IS FOR Y'ALL TO COMMENT RIGHT NOW ON LINKS THAT WENDY WILL SEND OUT. THE COMMENTS ARE MORE EFFECTIVE IF YOU CAN DO IT AS A GROUP WITHIN YOUR UNIVERSITY. WHAT THEY'RE PROPOSING, HAVE A LOOK YOURSELF BUT WE NEED PUBLIC COMMENT BADLY. >> IN THE BACK, AMY. >> DUKE UNIVERSITY. I WOULD LIKE TO FOLLOW-UP ON THIS BECAUSE IT IS SO FUNDAMENTALLY CRITICAL. IT'S NOT THAT IRBs ARE IMPOSING ON US BUT WE HAVE A RESPONSIBILITY AS WE TAKE CARE OF PEOPLE'S HEALTH INFORMATION TO USE IT AND MAKE USE OF IT IN THE MOST PRACTICAL IN PATIENT SENSITIVE MANNER. THIS IS A BIG CONFERENCE GOING ON AT -- CONVERSATION GOING ON AT THE IOM RIGHT NOW AND OTHER PLACES HOW TO ENSURE PUBLIC TRUST IN THE OVERALL SYSTEM OF HEALTH INFORMATION FOR OUR FUTURE. SO IT'S NOT ONLY HOW DO WE HELP ENSURE THE IRBs, HAVE THE RIGHT KIND OF INFORMATION SO THEY CAN DO THEIR DECISION MAKING, BUT IT'S ALSO HOW TO COMMUNICATE WITH THE PUBLIC TO MAKE SURE THEY UNDERSTAND WE CARE ABOUT USING THE INFORMATION THE RIGHT WAY AND WE'LL WORK TOGETHER TO DO THAT. SO I WOULD LIKE TO CONTINUE THIS CONVERSATION AN IMPORTANT THING TO MAKE SURE IS ON OUR AGENDA AND PUT IT SOME OW ON OUR EVIDENCE DEVELOPMENT AGENDA WE NEED TO DEVELOP BETTER EVIDENCE ABOUT HOW TO DEAL WITH THIS IMPORTANT ETHICAL AS WELL AS PUBLIC TRUST QUESTION. >> ONE THING WE HAVE DONE IN A CHRONIC PAIN STUDY, CANCER AN NON-CANCER CHRONIC PAIN IS CLINICS INVOLVED IN THAT STUDY HAVE ADDED IN STANDARDIZED DATA ELEMENTS AND ALSO MADE THE STANDARD OF CARE PATIENTS ANSWERING QUESTIONS ABOUT WHAT IS CONTROLLING OR MAKING THEIR PAIN WORSE IN BETWEEN THEIR VISITS TO THE DOCTOR'S OFFICE. SO WHAT WE HAVE DONE IS COMBINED THE DATA THAT'S PHYSICIAN AND OTHER CLINICS FELT THEY NEED FORD THE ACTUAL CARE PROCESS WITH GETTING ALL THESE EXTRA PIECES OF INFORMATION. BY MAKING IT THE STANDARD OF CARE THERE WAS NOT AN ISSUE WITH REGARD TO IRB IN THAT REGARD. IF YOU ANALYZE THE DATA SUBSEQUENTLY YOU GO THROUGH AN IRB. AS MARGARET WAS SAYING IT WAS EASY AT THAT POINT BECAUSE IT'S HIPAA COMPLIANT DATA AND IT'S BEEN ABLE TO BE PUT TOGETHER IN THIS WAY AND RESPONSIBLY USED AS YOU HAVE BEEN TALKING ABOUT. ONE OF THE THINGS I FEEL STRONGLY ABOUT ALL OF THESE DATA MECHANISMS IS THAT IF WE CAN COMBINE THE DATA WE COLLECT AUTOMATICALLY TO DO THE BEST JOB WE CAN WITH CARE FOR PATIENTS AS WELL AS HAVING IT BECOME STANDARD OF CARE SO TO SPEAK, WE WON'T HAVE THIS SEPARATION OF THIS IS RESEARCH, THIS IS ACTUAL CARE. IT WILL ALLOW FOR THE CONTINUITY YOU'RE ALSO TALKING ABOUT. EPI >> SUSAN (INAUDIBLE) FROM THE CENTER FOR CLINICAL RESEARCH AND OUTCOMES ACTIVITY. >> PLEASE REMEMBER TO STATE YOUR QUESTION SO THE WEBINAR VIEWERS KNOW WHO IS ASKING WHAT QUESTIONS. WILL IS NEXT. >> OVER HERE. I'M WILL VARGAS. NATIONAL SCIENCE FOUNDATION. MY QUESTION IS GETTING TO TISSUE HOW TO SHARE DATA. WHAT ARE THE BIGGEST BARRIERS, A NEED FOR MORE OPEN mHEALTH STANDARDS OR IS IT AN INCENTIVE QUESTION FOR RESEARCHERS OR FOR PATIENTS TO SHARE THEIR DATA? IS IT IRB STUFF? WHAT ARE THE ACTUAL -- THESE ARE CHALLENGES, BUT WHAT'S THE BIGGEST HURDLE? >> ANYONE FROM OUR PANEL HAVE COMMENTS ON THAT? >> I DO. IT TOUCHES ON SOME CHALLENGES, OTHERS HAVE HIGHLIGHTED, IN ADDITION TO WORK WITH D TREE I HAVE WORKED ON ELECTRONIC SYSTEMS WITH THE MINISTRY OF HEALTH. SOME OF THE SETTINGS THE PARTNERING ORGANIZES ARE CLEAR THAT THOUGH WE HAVE SOME RESEARCH QUESTIONS, THE DATA BELONGS TO THE HEALTHCARE PROVIDERS. THAT IT'S -- WE MAY APPLY TO DO RESEARCH AROUND THAT DATA THAT WE FACILE FATE AT A TIMED AND COLLECTING IN THIS CASE IN A MOBILE PLATFORM AND OTHER CASES IN COMPUTER PLATFORMS. IT MAKES ITTR TRICKY TO DO THE RESEARCH. BUT IN A LOT OF WAYS IT HEPS BE MORE CLEAR WHAT IS STANDARD OF CARE, WHAT IS HEALTHCARE SETTING AND WHAT IS THE RESEARCH. I ALSO REALLY I THINK IT WAS BARBARA YOU WERE TALKING BEFORE, I WORK IN MONITORING EVALUATION. I FIND THE BLUR BETWEEN MONITORING AN RESEARCH MAKES IT MORE COMPLICATED. BECAUSE I COULD SAY I'M DOING MONITORING AND EVALUATION, THIS IS NON-HUMAN SUBJECTS DETERMINATION, IT'S -- IF I DEIDENTIFY DATA BUT IT'S PLURRED AND DOESN'T MAKE IT EASIER, IT MAKES IT MORE COMPLICATED. ONGOING CHALLENGES FOR RESEARCH NOT SIMPLIFYING ANYWAY. >> I GUESS WHEN YOU TALK ABOUT SHARING OF DATA YOU'RE TALKING ABOUT INDIVIDUALS SHARING THEIR DATA WITH RESEARCHERS. THAT QUESTION WHAT IS THE STUDY AN INCENTIVES. INCENTIVES ARE KNOWN TO WORK IN CERTAIN TYPES OF STUDIES SO THERE WE HAVE DONE WORK ON TRYING TO UNDERSTAND WHAT KIND OF INCENTIVES MIGHT WORK, PERSONALIZABLE INCENTIVES, WORKS FOR ONE PERSON WORKS FOR THE NEXT PERSON, WAYS IN WHICH WE CAN PROVIDE A LIBRARY OF INCENTIVES THAT A STUDY ORGANIZER CAN PICK FROM. PRIVACY ISSUES ARE A CONCERN FOR SOME SEGMENT OF THE POPULATION. YOU HAVE TO CONVINCE THEY'RE SHARING THE DATA THEY'RE DOING IT FOR THE RIGHT CAUSE OR AND SO ON. SO IT'S A HARD ANSWER TO GENERALIZE SHARING ACROSS IS MUCH HARDER QUESTION. THAT CERTAINLY IN THE INTERNET CONTEXT PEOPLE SHARE THE DATA BUT THERE'S CHRONIC DISEASE SITUATION. SO THOSE PEOPLE WITH CHRONIC DISEASES IF YOU HAVE THE REGULAR PARTICIPATING STUDY Z MAYBE THEY DO IT AS WELL. BUT IT REMAINS TO BE SEEN. >> WE'RE TALKING ISSUES THAT CUT ACROSS AREAS. CAN THE PANEL COMMENT ON HOW METHODOLOGIES FIT TOGETHER WITH THE TECHNOLOGY, THE DIFFERENT ALTERNATIVE DESIGNS WITH THE STATISTICS AND THOSE METHODOLOGIES TO MAKE THE MOST OUT OF THE DATA YOU'RE GETTING. HOW DO YOU MAKE THE MOST OF THESE DATA THAT THIS TECHNOLOGY GIVES YOU THE OPPORTUNITY GET AND HOW DO THE TECHNIQUES FIT TOGETHER. WONDERING IF ANYONE HAS COMMENTARY HOW TO SEE THESE TOOLS FITTING TOGETHER. >> ONE THING, THERE'S A DANG NER AGGREGATING THE DATA. ONCE YOU AGGREGATE YOU LOSE WITH A FINE DETAIL AND SO THESE METHODOLOGIES AFFORD US A LOT OF INTERESTING DATA, A LOT OF INTERESTING QUESTIONS THAT CAN BE ADDRESSED BUT ONCE YOU AGGREGATE YOU'LL LOSE THE FINE DETAIL. SO I WOULD URGE AGAINST THAT. WE'RE BIG PICTURE QUESTIONS, SURE AGGREGATION IS FINE BUT THAT'S SORT OF THE ADVANTAGE OF ALL THESE APPROACHES, TO FINE OUT WHAT GOES ON AT MORE AN INDIVIDUAL AND TIME BASIS. THAT'S ONE THING THAT TIES TOGETHER. THE OTHER THING THAT TIES TOGETHER IS ALL THESE METHODS ARE KIND OF VERY MUCH INTERESTED WITHIN SUBJECT CHANGES AS WELL AS BETWEEN SUBJECT CHANGES. THE STEP DESIGN, THE N OF ONE TRIAL, IT'S ABOUT WHAT'S HAPPENING TO AN INDIVIDUAL WHEN THEY GET THE TREATMENT VERSUS WHEN THEY'RE NOT. IT ADDRESS IT IS COUNTER FACTUAL ISSUE THAT RCTs CAN'T GET AT. RCTs INFER WHAT'S GOING TO HAPPEN TO A SUBJECT IF THEY HAVE GOT THE NEIGHBOR TREATMENT B THEY NEVER GET ONE OR THE OTHER SO IT'S LEAP OF FAITH ONE TAKES, RANDOMIZATION HELPS. BUT NONETHELESS IN THESE STUDYS WE'RE GETTING INTERESTING DATA USED IN LOTS OF NICE WAYS AND WITHIN SUBJECT COMPARISON BUYS US POWER TO EXAMINE HOW INDIVIDUALS REACT TO THESE DIFFERENT SITUATIONS AN TREATMENTS. >> I'LL COMMENT TOO, I ECHO THAT BUT BEING ABLE TO LOOK AT THE GRANULARITY IN TERMS OF WHO ENDS UP IN THE STUDY DESIGNS THAT WE'RE TALKING ABOUT SO THERE'S A MUCH WIDER REACH IN TERMS OF OPPORTUNITIES OF LOOKING AT DIFFERENT PATIENT POPULATIONS. THAN MORE TRADITIONAL DESIGNS OFFER, NOT LOSE SIGHT OF THAT, TO GENERALIZE MORE TO POPULATIONS FOR WHOM THE DATA WAS GENERATED FROM THAN TO GO AHEAD APPROXIMATE AGGREGATE. SO IT'S AN IMPORTANT OPPORTUNITY TO LOOK AT MORE DIVERSE POPULATIONS AND AT THE SAME TIME BE AWARE THAT THAT GENERALIZATION PROCESS MAYBE DIFFERENT THAN WHAT WE HAVE DONE WITH RCTs IN THE PAST. >> ONE INTERESTING CONNECTION ALSO RELATES TO THE DATA PROCESSING STEPS AND HOW YOU PROPAGATE QUALITY INFORMATION. FOR EXAMPLE, IN MY SLIDES I MENTION THE FACT THAT'S MISSING DATA ALL THE WAY AT THE BOTTOM. AS YOU REACH UP THE LEVEL YOU FIGURE HIGHER INFORMATION LEVEL CONTEXT AND YOU USE THAT TO RUN BETWEEN PERSON OR WITHIN ANALYSIS WHAT DO YOU PROPAGATE, HOW DO WE UTILIZE KNOWLEDGE ABOUT THE LOW LEVEL AND THINGS FOR ABOUT THE QUALITY OF DATA WHEN WE MAKE THAT ANALYSIS. SO THAT'S THE DATA WE HAVE TO CONTEND WITH. >> I HAD A COMMENT HERE FIRST. >> KEVIN PATRICK, UC SAN DIEGO. THRES TWO JOURNAL EDITORS IN THE ROOM, I SEE BONNIE SPRING HERE, I WANT TO ASK A QUESTION, BECAUSE WE'RE GOING TO GET O THE 3.5 YEARS FROM IDEA TO DELIVERY OF THE IDEA TO THE MASSES. CURRENTLY THESE GO THROUGH JOURNALS SO ANY INSIGHTS WHAT WE MIGHT DO FOR OUR PARTICULAR PART OF THAT PIPELINE TO FACILITATE THIS. ALL THESE THINGS YOU'RE TALKING ABOUT CHOKE REVIEWERS IN MY EXPERIENCE, THESE ARE DIFFICULT THINGS FOR PEOPLE TO DEAL WITH. SO WHAT TIPS CAN YOU GIVE US AS JOURNAL EDITORS REVIEWSERS AND DESIGNERS OF STUDIES THAT DO OUR WORK TO THINK HOW TO FRAME AN FACILITATE GETTING THIS INFORMATION OUT >> I JUST FINISHED A ROUND AS ETOR OF ANNULS STACKS WHICH IS MATT JOURNAL THAT TAKES TWO YEARS BEFORE PATIENTS GET OUT, IT WAS AWFUL. AND SO WHAT WE DID, WE ADDRESSED THIS VERY ISSUE. IN COMPUTER SCIENCE THE TURN AROUND TIME IS VERY FAST SO WE WENT TO COMPLETE ELECTRONIC SYSTEM, THEN WE GOT BUY-IN FROM EVERY SINGLE ASSOCIATE EDITOR THAT JOINED THAT JOURNAL THAT THEY WOULD TURN AROUND THE PAPER WITHIN THREE MONTHS. THAT'S A BIG DEAL ON A MATHEMATICAL FIELD, THREE MONTHS, I KNOW IT'S NOT A BIG DEAL LIKE SCIENCE. THE JOURNAL SCIENCE. THEY'RE MUCH FASTER BUT I THINK YOU HAVE TO GET BUY-IN FROM YOUR ASSOCIATE EDITORS THE PEOPLE WHO HANDLE THE JOURNALS AND THEN ALL THE REVIEWERS YOU HAVE TO GET BUY-IN FROM. WITHIN ONE AND A HALF YEARS I THINK WE HAD OVER 75% OF THE PAPERS WERE BEING GIVEN THEIR FIRST DECISION WITHIN THREE MONTHS. THAT WAS GOING FROM OVER A YEAR AND A HALF TO THREE MONTHS. SO YOU CAN DO IT BUT YOU MOVE TO COMPLETELY ELECTRONIC SYSTEM WHICH IS WHAT WE DID. I'M SURE YOU HAVE DUB THAT BUT BUY-IN FROM ASSOCIATE EDITORS, THAT WAS ABSOLUTELY CRUCIAL. THEY WEREN'T ALLOWED TO JOIN THE BOARD UNLESS THEY AGREED TO DO THIS. >> OTHER COMMENTS YOU GUYS HAVE? (OFF MIC) >> IT WAS ONLY JUST A CATALOGING QUESTION LIKE WHEN TRYING TO FIND EXAMPLES OF THE DESIGNS THERE'S NO COHERENT WAY TO FIND THEM. I DON'T KNOW, THESE ARE NOT -- THESE ARE BROADER ISSUES ABOUT HOW YOU IDENTIFY DIFFERENT TYPES OF DESIGNS BUT I DON'T KNOW IF THERE'S SOMETHING THAT THE EDITORS CAN THINK OF THAT WOULD ALLOW THAT IN THE PROCESS OF REVIEWING PAPERS TO COME UP WITH THE MORE COHERENT METHODOLOGIES. >> THIS IS LINDA COLLINS PENN STATE. SUSAN MURPHY WAS TALKING HOW TO MOVE PAPERS THROUGH THE PIPELINE MORE QUICKLY. I WANT TO COMMENT ON SOMETHING KEVIN SAID A MINUTE AGO. ANOTHER PART WHICH IS REVIEWERS BEING A LITTLE BIT TAKEN ABACK BY THE CONTENT OF THESE PAPERS AND THAT IS CERTAINLY HAPPENED TO ME IN MY WORK WHERE I PUBLISHED -- COLLABORATED WITH PEOPLE AND A PAPER WILL HAVE AN UNUSUAL DESIGN OR ANALYSIS. I DON'T KNOW WHAT CAN BE DONE ABOUT THIS BUT I WOULD LIKE TO EXPRESS THE IDEA THAT I THINK EVERY SCIENTIST OUGHT TO HAVE CONTINUING EDUCATION FOR THEIR ENTIRE CAREER AB METHODOLOGY BECAUSE METHODOLOGY IS CHANGING ALL THE TIME. WHAT YOU LEARNED IN GRADUATE SCHOOL 20 YEARS AGO IS NOT NECESSARILY WHAT IS BEING DONE TODAY. I ALSO DON'T THINK IT HAS TO BE COMPLICATED. I DON'T THINK PEOPLE LEARN HOW TO DO DIFFERENTIAL EQUATIONS, I THINK THEY NEED TO UNDERSTAND THE CONCEPTUAL LINK BETWEEN THE RESEARCH QUESTIONS THEY HAVE AND THE TYPE OF DESIGN OR DATA ANALYSIS METHOD. IT'S NOT JUST JOURNAL REVIEWs, IT'S GRANT REVIEWERS WHERE THIS COMES UP A LOT. AND WHERE -- UNFORTUNATELY TO ME IT'S CREATING A HUGE CONSERVATIVE SHIFT IN SCIENCE, BEHAVIORAL SCIENCES OVERALL. I JUST SEE THIS AS ENORMOUS ISSUE ANION WHAT CAN BE DONE ABOUT IT BUT I WANTED TO VENT ABOUT THAT FOR A MINUTE. >> PART IS SECONDING WHAT LINDA SAYING. PART, I MEAN I LEARNED A LOT BY SERVING ON A STUDY SECTION, THE EPIPI METHODS, INNOVATIVE STUFF SO I LEARNED A LOT AS A JOURNAL EDITOR. PART OF THIS IS ALSO HOW YOU AS AUTHORS REFERENCE AND ANCHOR YOUR WORK IN THINGS THAT ARE ACCESSIBLE TO OTHERS SO PART IS THAT. SO DON'T MAKE IT DIFFICULT TO HAVE PEOPLE FIND THE SEMINOL ARTICLE OR THE SEMINOL REVIEW PAPER THAT RELATES TO THIS. BUT I WOULD LEAVE THIS AS A CHALLENGE FOR US TO ADDRESS TODAY, THIS IS A QUESTION I WOULD BRING UP TO EVERY PANEL, HOW CAN WE THINK ABOUT THIS? BECAUSE THIS IS ONE OF THE RATE-LIMITING STEPS. WHAT WE'RE TALKING ABOUT HERE. IF WE'RE GOING TO MAKE THE AD VAXES WE'RE TALK -- ADVANCES WE'RE TALKING ABOUT HERE WE NEED THE FIND BETTER WAYS TO COMMUNICATE. IT'S TRUST. PART OF THE ROLE, GEORGE LUND BERK EDITOR OF THE JAMA, SORT OF MY HERO AS FAR AS MEDICAL EDITOR, ONE OF THE ROLES IS TO SHED LIGHT, TAKE HEAT AND GIVE HEAT. YOU NEED TO BASICALLY YOUR ROLE IS TO DEAL WITH THE TRUST BECAUSE ACTUALLY YOU'RE THE FINAL PATHWAY BY WHICH THIS INFORMATION POTENTIALLY GETS OUT. YOU'RE A JOURNAL. SO WE TAKE THIS SERIOUSLY, I'S THE ISSUE OF TAKING SERIOUSLY THIS ABILITY TO FIND THE WAY IN WHICH WE CAN INFORM OUR EDITORS AND OUR REVIEWERS ON THIS PROCESS AND GET -- IT'S VERY OPEN QUESTION BUT I THINK I WOULD LIKE TO SEE THIS GROUP KEEP DISCUSSING. >> COULD I MAKE ONE COMMENT AS A PANELIST? JUST WHILE WE'RE ON TOPIC OF JOURNALS AN JOURNAL EDITING, SPEAKING AS CO-EDITOR AN CHIEF OF JOURNAL -- GENERAL MEDICINE, SOMEONE WITH A STAKE IN LONGEVITY AN SUCCESS OF JOURNALS, I WOULDN'T BE SURPRISED IF JOURNALS DON'T EXIST IN TEN YEARS. WE SHOULD THINK DISRUPTIVELY WHETHER THINGS WILL MOVE TOWARDS THE WAY THEY DO IN ASTRO PHYSICS WHERE SCIENTISTS POST THEIR WORK AND THERE'S ESSENTIALLY POST APPROXIMATELY DAITION PEER REVIEW. THAT MAYBE WHERE WE'RE GOING. >> I HAD MY HAND UP A LONG TIME AGO. SO I DON'T KNOW IF IT'S VALUABLE BUT I'LL TRY TO CONNECT IT TO THIS CONVERSATION. THE LAST WAVE WAS TALKING ABOUT DATA AND DATA AVAILABILITY. ONE THING WE'RE WORKING ON TRYING TO MAKE SOME PROGRESS ON IS DEVELOPMENT OF COMMON DATA ELEMENTS AND ELECTRONIC MEDICAL RECORDS. SO WE HAVE A GROUP WORKING ON THAT, WE'RE SPONSORING. I JUST WANT TO SAY WE LOOKED AT WHAT SOME OF THE REALLY BIG MEDICAL GROUPS ARE DOING, TURNS OUTS THEY CAN'T GIVE YOU BASIC INFORMATION LIKE DEMOGRAPHIC BREAKDOWN OF THE POPULATIONS THEY SERVE. SO WE ARE TRYING TO DEVELOP CONSENSUS AROUND WHAT SHOULD BE IN ELECTRONIC MEDICAL RECORDS, PARTICULARLY PSYCHOSOCIAL ELEMENTS. IT DOES CONNECT TO THIS CONVERSATION BECAUSE WE REALIZE THAT WE FOUND OUT WHO THE ENEMY IS. IT'S US. WHAT HAPPENS IS THAT EVERYBODY DEVELOPED THEIR OWN MEASURE FOR SOME SUB COMPONENT. AND WE HAVE BEEN REWARDED FOR THAT IN OUR STRUCTURE AND PUBLICATIONS AND REVIEWERS DONE LIKE TO HEAR THAT YOU'RE USING THE SAME MEASURE AS EVERYBODY ELSE. SO I THINK AS AN ACADEMIC COMMUNITY IT'S IMPORTANT TO FOR US TO PULLING TO AND RECOGNIZE THE IMPORTANCE OF COMING TOGETHER AS A COMMUNITY TO MEASURE THE SAME THINGS. >> I IDA SIM UCSF. LIKE THE DISCUSSION HOW TO GENERATE EVIDENCE AND DISSEMINATE IT. WHAT WE'RE ABOUT HERE IS NOT JUST GENERATING GOOD EVIDENCE BUT FUNDAMENTALLY GETTING IT USED AND GETTING IT USED IN A RAPID ENOUGH FASHION THAT WE CAN HAVE WHAT THE THE IOM CALLED A LEARNING (INAUDIBLE). SO I WANT TO GO BACK TO SANS TO'S CHARGE TO US TO REDUCE OUR TIME AN GENERATION AND PUBLICATION TO THREE AND A HALF YEARS AN RECONSIDER THAT IT. 'S NOT SO MUCH THAT WE PUBLISH OUR DATA BUT HOW DO WE GET OUR EVIDENCE OUT TO THE IMMUNITY IN A WAY THAT'S REVIEWED. SO TO ME MAYBE THE CHARGE MIGHT BE ONE YEAR TO GENERATE THE EVIDENCE AND HAVE IT USED SO WE CAN HAVE THAT LEARNING SYSTEM. I THINK JOURNALS ARE PART OF THE SOLUTION. I DON'T THINK THEY ARE THE ENTIRE SOLUTION AS RICH SAID. I THINK WE'RE THINKING DISRUPTIVELY ANYWAY, WE MIGHT AS WELL THINK DISRUPTIVELY ABOUT THE JOURNAL PROCESS AS WELL. HOW DO WE GET OUR INFORMATION AND WHAT WE LEARN OUT THERE IN A WAY THAT WE CAN REUSE IT IN A MUCH MORE RAPID FASHION? THAT TO ME, THAT IS THE CHARGE OF THE EXCITEMENT OF WHAT WE DO. >> I FEEL LIKE PHIL DONAHUE, I DON'T KNOW WHOW TO CALL ON. >> MY NAME IS TOM CIRCHNER IN (INAUDIBLE). I THINK IT'S IMPORTANT TO CONSIDER NOT ONLY THE METHODS WE'RE USING BUT THE BEHAVIORAL OUTCOMES THAT WE'RE TALKING ABOUT. AND I THINK THIS GROUP SHOULD DISCUSS WHAT DO WE THINK IS ACCEPTABLE AND I THINK THIS GOES BACK TO KEVIN'S QUESTION HOW TO GET THINGS PUBLISHED AS WELL. THERE ARE OUTCOMES THAT ARE EVENTS, IT'S POSSIBLE TO MEASURE QUICKLY OVER TIME. THERE'S VACCINATION EVENTS FOR INSTANCE, THERE ARE ALSO OUTCOMES THAT ARE CHRONIC DISEASE STATES. AS AN EXAMPLE IN THE WORLD OF SMOKING CESSATION YOU HAVE THE STANDARD POINT PREVALENCE AT SIX, NINE, 12 MONTHS BUT THEN YOU HAVE RESPONSE TO MEDICATION IN THE FIRST WEEK. RIGHT NOW I CAN TELL YOU, YOU'RE NOT GOING TO GET -- YOU'RE PROBABLY NOT GOING TO GET A PAPER PUBLISHED ON RESPONSE TO MEDICATION, ONLY ON THE FIRST WEEK BUT IF THAT'S WHERE THE SIGNAL IS, IF THAT'S WHAT'S ACTUALLY RELATED TO THE MHEALTH INTERVENTION, THAT'S WHAT IT IS. MAYBE THAT SHOULD BE PUBLISHABLE. SO THE QUESTION I HAVE IS WHAT IS ACCEPTABLE AS FAR AS OUTCOMES ARE CONCERNED, NOT JUST METHODS, THE METHODS HELP US GET THERE. >> SO I WANT TO RESPOND BECAUSE I THINK ACTUALLY THE STUDY THAT I WAS PRESENTING SOME OF THE PRELIMINARY RESULTS SPEAKS WELL TO THAT. WE'RE INTERESTED IN NOT IMPROVING ADHERENCE OF A PARTICULAR PROTOCOL. WE'RE INTERESTED IN IMPROVING THE HEALTH OF CHILDREN IN THE COMMUNITIES WE'RE WORKING. AS EVERYONE KNOWS, SHOWING A DIFFERENCE IN UNDER FIVE MORTALITY IS THOSE ARE THE FIVE, TEN, 15 YEARS STUDIES AND TALKING CHANGES IN MOBILE HEALTH OVER 15 YEARS, IT BECOMES OBSOLETE WHAT YOU'RE ANSWERING. WE HAVE COME UP WITH A LOGICAL MECHANISM OF HOW WE THINK THAT HAPPENS. WE HAVE A PROTOCOL THAT HAS BEEN SHOWN TO DECREASE UNDER FIVE MORTALITY. SO IF WE CAN SHOW THAT WE CAN IMPROVE ADHERENCE TO THAT PROTOCOL, THEN THE LOGICAL LINK IS THAT WE EXPECT TO HAVE AN IMPACT ON UNDER FIVE MORTALITY. SO WE HAVE ACCEPTED SHOWING THAT ADHERENCE IS ACCEPTABLE FOR THAT LOGICAL LINK. I THINK WE MAY RUN INTO CHALLENGES PEOPLE SAYING WELL, WHERE IS YOUR DECREASE, YOUR SIGNIFICANT DECREASE IN UNDER FIVE MORTALITY BUT WHICH FEEL CONFIDENT THAT THIS IS SUFFICIENT. IT GOES A BIT -- I HAVE MISSED YOUR NAME SOMEONE FROM COLUMBIA, GOES TO WHAT YOU WERE SAYING, THE CHALLENGE BETWEEN EFFICACY RESEARCH AND EFFECTIVENESS RESEARCH, WE NOW HOPE TO INTEGRATE THIS INTO A MORE LONG-TERM STUDY WITH A PARTNER WHO WILL BE IMPLEMENTING THIS AS THEY GO ALONG TO SHOW THAT ONE, IT'S SUSTAINABLE, THIS TECHNOLOGY DOESN'T ARE THE ADDED BENEFIT WEEKS OUT THAT HAD THE ADDED BENEFIT THREE YEARS OUT. AND THAT IF WE LOOK AT A THREE YEAR TIME FRAME WE CAN SHOW BIGGER LONG TERM OUTCOMES AS WELL. >> TO QUICKLY FOLLOW-UP. I'M WITH YOU. I THINK PART OF THE DANGER THOUGH IS THIS SLIPPERY SLOPE ONE WAY OR THE OTHER, SOMEBODY MENTIONED BEING AT THE mHEALTH SUMMIT. I HAD A SIMILAR EXPERIENCE TO A LOT OF APPLICATIONS AND AMAZING TECHNOLOGIES WITH VERY LITTLE LINK TO OUTCOME. SO YOURS SOUNDS GOOD TO ME BUT THERE'S A QUESTION WHERE DOES THE SLOPE STOP. SO YOU HAVE THIS APPLICATION INCREASES CLICK-THROUGH ON A WEBSITE OR PERHAPS WITHIN THE APPS. THAT IS A PROXIMAL OUTCOME, IS THAT TOO PROXIMAL? WHERE DO WE DRAW THE LINE AND WHAT DO WE BELIEVE IS ACCEPTABLE? >> I WAS GOING TO SAY THAT I THINK THAT'S A STRONG ANALOGY BETWEEN THAT ISSUE AND THE USE OF BIOMARKERS OR PROXY MEASURES IN CLINICAL RESEARCH, SO THIS SIGNALS IN mHEALTH APPLICATIONS ARE KIND OF LIKE BIOMARKERS. WE OUGHT TO PROBABLY HAVE SIMILAR STANDARDS TO USE OF BIOMARKERS IN CLINICAL RESEARCH. THERE OUGHT TO BE WORK THAT SHOWS THOSE BIOMARKERS ARE LINKED TO CLINICALICALLY MEANINGFUL OUTCOMES BEFORE WE USE THEM. WE'RE IN AN ENVIRONMENT WHERE OFTEN WE HAVE THE PROBLEM WE CAN'T GET ENOUGH DATA, WITH MHEALTH WITH I CAN SEE US HAVING TOO MUCH DATA. WE'RE GOING TO DROWN IN DATA. >> JUST AS A REMINDER WE ONLY HAVE TEN MINUTES LEFT. I JUST WANTED TO GIVE EVERYONE A HEADS UP. >> (INAUDIBLE) FROM GW AGAIN. PART OF OUR CHALLENGE HERE IN LOOKING AT STUDY DESIGNS, IS THAT mHEALTH LOOKING AT A WHOLE DIFFERENT RANGE OF INTERVENTIONS AN DISEASES, THERE'S SHORE TERM ONE, IMMUNIZATION MAYBE ONE SHOT INTERVENTION. VERSES CHRONIC CARE DIABETES, CONGESTIVE HEART FAILURE. MENTAL HEALTH ISSUES. AND SO I WOULD BE INTERESTED WHETHER OR NOT CERTAIN STUDY DESIGNS ARE MORE APPLICABLE TO THESE CHRONIC LONG TERM DISEASES WHICH I'M MORE INVOLVED IN AS CARDIOLOGIST VERSUS VERY IMPORTANT INFECTIOUS DISEASE ISSUES AS WELL. >> IN THE CHRONIC DISEASE WORK WE HAVE DONE WE HAD TO WORK HARD TO FIND OUTCOMES THAT WE COULD MEASURE IN REAL TIME WITH A DIABETES SELF-MANAGEMENT SUPPORT MODEL. SO THE STEP WEDGE DESIGN DOES WORK WITH US FOR THE INTERVENTION WE HAVE DEVELOPED. BUT WE WOULD LIKE TO HAVE LONGER TERM OUTCOMES BE MEASURED SO THEN YOU'RE LOOKING AT HOW DO YOU FUND PROJECTS WITH RESEARCH DOLLARS THAT WILL EXTEND OUT OVER 7, 8 YEARS TO LOOK AT A VARIETY OF OUTCOMES WE LIKE TO LOOK AT. >> AND THE CHALLENGE AS SAID EARLIER, WHAT ARE YOUR OUTCOMES? WHAT ARE END POINTS? YOU HAVE TO DEFINE THOSE. SO WE'LL USE HELIX CRITERIA, HOW MANY PATIENTS GET THEIR EYE EXAM, THEIR FOOT EXAM, HEMOGLOBIN A 1C OVER TIME AND WHETHER OR NOT THAT'S PICKING UP BECAUSE THEY'RE HAVING THE INTERVENTION OR NOT. RATHER THAN JUST LOOKING AT HEMOGLOBIN A-1C LEVELS YOU CAN LOOK AT -- >> PROCESSES OF CARE. >> PROCESS OF CARE, ALSO EVEN JUST SELF-MANAGEMENT SKILLS. >> YESSjj– AT A LOT OF THOSE AND WE ALSO TRY TO LOOK AT SOME OF THE MORE CLINICAL LABORATORY OUTCOMES BUT YOU RUN INTO THE ISSUES WHEN YOU'RE PUBLISHING IT THERE'S LESS INTEREST IN THE PROCESSES OF CARE OUTCOMES AT LEAST IN SOME OF THE EXPERIENCE THAT WE HAVE HAD WITH OUR EFFICACY TRIALS. SO IT'S TRYING TO DO BOTH USUALLY. BUT YOU'RE REALLY I THINK TARGETING MORE. WE'RE MORE INTERESTED IN GIVING PEOPLE THE SKILLS FOR THE LIFE-LONG PERSPECTIVE. RIGHT. EXACTLY. SO OUR WORK WITH THE HEALTH PLAN, THEY WANT OUTCOMES IN A VERY SHORT TIME FRAME BECAUSE THEIR HEALTH PLAN AND DECIDING IN NEXT YEAR'S BUDGETS WHETHER OR NOT TO CONTINUE WITH THIS PROGRAM. SO IT'S DEPENDS ON THE ENVIRONMENT YOU'RE WORKING IN. >> WE JUST HAVE TIME FOR A FEW MORE QUESTIONS. SO WE'RE GOING TO GO HERE, TWO QUESTIONS UP HERE. AND THEN BACK TO THE BACK AND WE'LL GIVE THE PANEL AN OPPORTUNITY TO SAY ANY WRAPPING UP COMMENTS. START HERE. >> DAVID MOORE FROM NORTHWESTERN UNIVERSITY. I WANT TO GO BACK TO SOMETHING KEVIN RAISED ABOUT EVALUATING OUR OWN RESEARCH. SEEMS LIKE HERE WE HAVE BEEN TALKING ABOUT TWO LEVELS OF EVALUATION, ONE IS EVALUATION OF THE EFFECT OF THE INTERVENTION ON POPULATIONS AND THEN THE OTHER IS HOW WE EVALUATE OUR OWN RESEARCH WITHIN OUR COMMUNITY BY GRANTS THROUGH REVIEWS OF PAPERS. AND SEEMS TO ME THERE ARE VERY DIFFERENT -- MAYBE WE SHOULD THINK ABOUT THESE IN DIFFERENT WAYS. IN -- WITH POPULATIONS OF PATIENTS WE'RE WANTING TO SEE SOME SORT OF SEPARATION BETWEEN TWO GROUPS WITH VARIABILITY BUT SEEMS TO ME WHEN WE TALK THAT WHEN WE EVALUATE OUR OWN WORK CERTAIN CERTAINLY WHAT I HAVE SEEN IN REVIEW COMMITTEES IS IT SEEMS LIKE THE MOST INNOVATIVE WORK IS WHERE YOU GET THE LARGEST SPREAD WHERE YOU GET THE MOST DISAGREEMENT, THAT VARIABILITY IN FACT WHEN WE JUDGE OUR OWN WORK MAYBE A MARKER OF SOMETHING THAT IS REALLY INNOVATIVE, NOT THAT EVERYTHING WHERE YOU GET A SPREAD IS INNOVATIVE BECAUSE I'M SURE -- AND CERTAINLY WHEN I THINK IT'S BAD IT SHOULD BE BAD. [LAUGHTER] >> BUT -- AND I WONDER IF THERE'S SOME WAY THAT TO BEGIN TO TAKE INTO ACCOUNT THAT VARIABILITY WHEN WE EVALUATE OUR WORK. I BELIEVE I HER AT ONE POINT FOR EXAMPLE THE GATES FOUNDATION WILL ALLOW GRANT REVIEWERS TO MAKE A DECISION TO FUND ONE GRANT. AND SO IF THEY'RE REVIEWING A LOT OF GRANTS F THEY THINK ONE THING IS REALLY INNOVATIVE, THEY HAVE THE OPPORTUNITY TO GO IN AND PULL THAT OUT. SO IEP NOT SUGGESTING THAT THE NIH IS GOING TO DO OR SHOULD DO SOMETHING LIKE THAT BUT IF WE CAN BEGIN TO THINK ABOUT HOW TO USE THAT VARIABILITY A A MARKER OF SOMETHING THAT'S POTENTIALLY INNOVATIVE, IT MAYBE SOMETHING THAT WILL BENEFIT OUR FIELD. >> KEEP THE QUESTION SHORT. WE ONLY HAVE A LITTLE BIT OF TIME LEFT. >> SO ON THE QUESTION OF WHEN YOU EVALUATE THE OUTCOME -- >> INTRODUCE YOURSELF. >> BOB EVANS BOOG L. WHEN -- GOOGLE. WHEN YOU EVALUATE THE EVIDENCE THE TIME FRAME SEE AN EFFECT I UNDERSTAND THAT IN BEHAVIOR CHANGE IT MAY TAKE MULTIPLE YEARS WHETHER THERE'S STILL A 47% LAPSE RATE AFTER FIVE YEARS. SO IN THE COMPUTER SCIENCE WORLD THERE'S DEBATE ABOUT THIS RECENTLY IN WORKSHOPS AND SOMEONE MADE THE DISTINCTION BETWEEN THE ACI ASPECT AND BEHAVIOR CHANGE ASPECT. SO IF YOU'RE BUILDING A TECHNOLOGY TO DO INTERVENTION, IN TERMS OF THE USE OF TECHNOLOGY, DOES IT CREATE SHORT TERM EFFECTIVE MEASURES, SOMETHING YOU CAN SEE LIKE IF U YOU'RE DOING SELF-MONITORING DO THEY ACTUALLY CHANGE THEIR IMMEDIATE LOCAL BEHAVIOR IN THE SELF-MONITORING BUT ONCE THEY QUIT THE SELF-MONITORING BEHAVIOR LAPSES BACK TO FORMER STATE SO MAYBE USEFUL IN TERMS OF EVALUATION TO HAVE STAGED EVALUATION OF THE mHEALTH TECHNOLOGY IN TERMS OF ITS TECH NO LJCAL USAGE AND IMPACT HAS AN IMMEDIATE EFFECT AND WE CAN TALK ABOUT THAT NOW, WHICH IS PROBABLYISTICLY AN INDICATOR THERE MIGHT BE AN EFFECTIVE BEHAVIOR CHANGE DOWN THE ROAD WHICH IS ANOTHER EVENT. THOUGHT ON THAT? >> I ACTUALLY JUST WANT TO RESPOND BECAUSE IT TIES IN WELL WITH WHAT TOM WAS SAYING. IT IS A SLIPPERY SLOPE IN TERMS OF DO YOU FALL TOO SHORT ON WHAT YOU'RE MEASURING AND NOT GOING TO HAVE BIGGER IMPACTS THAT EVERYONE WANTS TO BE MEASURING BUT ARE FEELING THEIR HANDS ARE TIED TO DO. SO MY HOPE FOR THAT, THIS IS WHAT WE TRIED TO DO IS COME UP WITH A LONG TERM RESEARCH PLAN THAT SHOWS THINGS IN STAGES. SO WE HAVE THESE SHORTER-TERM EFFECTS WE HOPE TO MEASURE THAT WE THINK LOGICALLY LINK TO THESE OTHER EFFECTS AND OUR LONG-TERM RESEARCH PLAN SO TO MEASURE AT EACH STAGE AS THEY ARE APPROPRIATE. SO IT GOES VERY MUCH TO WHAT YOU'RE SAYING WHICH IS TO HAVE OUTCOMES THAT MATCH WHERE YOU ARE IN TERMS OF IMPLEMENTATION BUT THEN TO CONTINUE MEASUREMENT NOT JUST STOP AT THE FIRST. >>d (INAUDIBLE). (OFF MIC) >> SO WE'RE ACTUALLY LOOKING AT THE CHANGE IN CLINICAL CARE. SO THE FIRST IS LOOKING AT WHAT WE SHOW HERE CAN WE CHANGE THE IMMEDIATE ACTION OF THE DOCTOR. AND THEN THE NEXT WILL BE DOES THE CHANGE OF THE DOCTOR BEHAVIOR EFFECT THE HEALTH OF THE CHILD. MEASURING THE CHANGE ON THE HEALTH OF THE CHILD IS A MUCH MORE DIFFICULT THING TO MEASURE. SO WE THOUGHT IF IT CAN SHOW THE FIRST THING FIRST THAT GIVES JUSTIFICATION FOR A BIGGER STUDY TO SHOW THE NEXT STEP. >> CAN YOU SAY H YOU MEAN BY HCI FOR THOSE THAT MIGHT NOT -- (OFF MIC) >> THANK YOU. >> IF I CAN ADD ONE OTHER THING, WE HAVE LOOKED AT THE QUALITATIVE MEASURES OF THIS BECAUSE ONE OF THE BIG PUSH BACKS WE HAVE IN OUR SETTING, WILL HEALTHCARE WORKERS ACCEPT IT, WILL THE PATIENTS ACCEPT IT. I THINK THAT'S A REALLY IMPORTANT PIECE NOT JUST THE HARD COWTIOMS OF ARE THEY DOING THE THINGS WE WAN THEM TO DO, HOW ACCEPTABLE IS IT SHORT AND LONG TERM. >> WE HAVE ONE LAST COMMENT IN THE BACK. >> CRAIG LEFEBVRE, THE LAST COMMENT IS A SEGUE. I DESIGN MY RESEARCH AND GET INVOLVED IN RESEARCH PROJECTS THINKING FROM THE POINT OF VIEW OF PEOPLE WHO WILL USE THE TECHNOLOGY. SO I WIEBD UP THINKING ABOUT STUDIES THAT AREN'T N OF ONE STUDIES AND AREN'T RANDOMIZED STUDIES BUT LOOKING AT TEENAGERS AND CONTROL OF ASTHMA, I HAVE N OF 3 AN N OF 4 STUDIES BECAUSE TEENAGERS USE A MOBILE PHONE FOR OTHER THINGS THAN CHECKING DATA AND SENDING DATA BACK AND FORTH. THEY HAVE ALSO GOT A PARENT. THEY HAVE MAYBE ONE OR TWO KIDS IN THE SCHOOL WHO ARE ALSO PART OF THIS NETWORK OF ASTHMA MANAGEMENT WHO THEY NATURALLY INCORPORATE INTO WHAT THEY'RE USING THEIR MOBILE PHONE OR FACEBOOK PAGE THEY HAPPEN TO DO. SO THIS SOCIAL ASPECT WHAT WE CAN DO WITH THE MOBILE PHONE, I HAVEN'T HEARD ADDRESSED BY DESIGN. AND I'M CURIOUS IF YOU GUYS HAVE BEEN THINKING ABOUT WHAT KINDS OF DESIGNS DO YOU USE WHEN IT'S NOW -- IT'S NOT A PARTICIPANT AND NOT A SUBJECT, IT'S A GROUP. IT MAYBE NATURALLY DEFINED IN MANY WAYS BUT SEEMS TO ME THAT JUST PRESENTS ALL KINDS OF A OPPORTUNITIES BUT THEN ALL KINDS OF ISSUES. I JUST LIKE TO HEAR HOW YOU'RE THINKING ABOUT THAT AT THIS POINT. >> DOES THE PANEL HAVE ANY COMMENTS? I THINK THAT'S -- SEEMS TO BE AN INTERESTING THOUGHT-PROVOKING QUESTION THAT YOU CAN DISCUSS OVER THE BREAK. I THINK WE HAVE ONE FINAL COMMENT. (OFF MIC) >> AND I THINK WE NEED TO LOOK AT THAT AS A COMPLIMENTARY SYSTEM AND THAT JOURNALS ACTUALLY NEED TO BEGIN TO ACKNOWLEDGE WHERE PEOPLE ARE PUBLISHING BITS AND PIECES OF THEIR RESEARCH AT AN EARLIER POINT ONLINE, IF JOURNALS CAN BEGIN TO ACKNOWLEDGE THOSE AND THERE WOULD BE SOME MOTIVATION FOR PEOPLE PUBLISHING THINGS EARLIER AS WELL AS ANONYMIZED RESULTS, ET CETERA. SO WE NEED TO LOOK AT OTHER AREAS OF SCIENCE THAT MIGHT HAVE GONE DOWN THIS TRACK. IN ADDITION TO IMPROVING THE SPEED. JUST TO MENTION, THIS IS NOT A QUESTION, THIS IS A COMMENT, BUT A COMMON DATA ELEMENT, COMMON METRIC, COMMON OUTCOMES, WE NEED SOME CONSENSUS. WE NEED SOME -- WE NEED PEOPLE TO PUT TOGETHER OR BEGIN TO POSITION DIFFERENT METRICS PEOPLE CAN CONGREGATE AROUND INPUT AND PROCESS. A LOT OF THIS CONSENSUS AROUND OUTCOME FOR THE MOST PART IN HEALTH BUT THERE'S LESS SO ON THE INPUT ON THE mHEALTH SIDE WHETHER IT'S AND THE PROCESS ON THE HEALTH SYSTEM IMPROVEMENTS AND AS WELL AS AT THE PROVIDER LEVEL AS WELL AS THE CLIENT LEVEL. I THINK THAT'S A SCENARIO WHERE AT LEAST WE'RE VERY EXCITED AT THE POTENTIAL FOR COMMON METRICS EMBEDDED SYSTEMS THAT IDA AND OTHERS ARE (INAUDIBLE). >> SO THIS HAS BEEN I GUESS THE PLANNING COMMITTEE IS HE CAN STATIC. WE KNEW THAT Y'ALL NEEDED TO TALK ABOUT THIS. WE KNEW AND YOU CAN SEE WE'RE BURSTING AT THE SEAMS WITH QUESTIONS AND COMMENTS. THAT'S WHY WE HAVE ALL DAY TO WORK THROUGH THIS AN LOTS OF TIME FOR DISCUSSION, WE HAVE HIT MANY TOPICS, NOT JUST THE METHOD BUT THE DISSEMINATION ISSUE, THE PRIVACY, ALL THESE OTHER ISSUES INCLUDING COMMON METRICS THAT WE'RE GOING TO DISCUSS THROUGHOUT TODAY. I TOLD YOU I WAS DRACONIAN ON TIME. I'M ALL ABOUT THAT SO YOU HAVE 15 MINUTES ON THE BREAK. THOSE ON THE WEBINAR FEEL FREE TO TAKE A BREAK TOO. DO SOMETHING FUN. THE REST OF YOU COME OUT HERE, HAVE A CUP OF COFFEE APPROXIMATE WE'LL START AGAIN AT QUARTER TO. THANK YOU. WELCOME BACK, EVERYBODY. FOR THOSE OF YOU ON THE WEBINAR, WELCOME BACK. FOR THOSE OF YOU ON THE WEBINAR, YOU SHOULD BE HAPPY, PEOPLE IN THE ROOM DO NOT HAVE INTERNET SO WE'RE SORRY ABOUT THAT. IT'S JUST A CHALLENGE AT NIH TO HAVE INTERNET SO WE DON'T. THOSE ON THE WEB YOU DO, BE GLAD. THERE'S AN ADVANTAGE. NEXT SESSION I'LL INTRODUCE WILLIAM RILEY FROM THE NATIONAL HEART LUNG AND BLOOD INSTITUTE. HE'S A HEART LUNG GUY. HE'LL MODERATE OUR NEXT SESSION. >> THANK YOU. IT'S A PRIVILEGE TO MODERATE THIS SESSION. I'M GOING TO DO INTRODUCTIONS SIMPLY BECAUSE YOU ALREADY HAVE THEM IN YOUR BOOK. WITH BIOS. SO CHAIR AND FIRST SPEAKER IS LINDA COLLINS FROM PENN STATE UNIVERSITY. [APPLAUSE] >> I HAVE BEEN GIVEN THE OPPORTUNITY TO INTRODUCE THIS SESSION. I WOULD LIKE TO SHARE SORT OF A LARGER PICTURE PERSPECTIVE ON ALL OF THIS. AND EXPRESS HOW I THINK THESE THREE TALKS KIND OF FIT TOGETHER. I WOULD LIKE TO SHARE WITH YOU A VISION FOR THE FUTURE OF BEHAVIORAL INTERVENTION SCIENCE. YOU HAVE BEEN TALKING A LOT ABOUT THE RCT AND ITS ROLE IN WHAT WE'RE TALKING ABOUT TODAY. THE RCT IS GOING TO CONTINUE THE PLAY AN IMPORTANT CONFIRMATORY ROLE IN ULTIMATELY CONFIRMING THE BOTTOM LINE OF WHETHER AN INTERVENTION IS EFFECTIVE OR NOT BUT I THINK THERE'S GOING -- THERE'S EMERGING A DIFFERENT PERSPECTIVE ON THE DEVELOPMENT OF INTERVENTIONS. I WOULD LIKE TO SEE US MOVE TOWARD A MORE ENGINEERING-BASED FRAMEWORK FOR THIS. SO THE IDEA HERE IS THAT A BEHAVIORAL INTERVENTION CAN BE CONSIDERED A DEVICE OR PROCESS THAT'S INTENDED TO ACCOMPLISH SOMETHING, MUCH THE SAME WAY THAT A VACUUM CLEANER IS INTENDED TO VACUUM UP DIRT OR CHEMICAL PROCESS IS INTENDED TO END UP WITH A PARTICULAR DRUG. AND IF YOU THINK OF A BEHAVIORAL INTERVENTION IN THIS WAY, IT'S A VERY COMPLEX THING BUT IF YOU THINK OF A BEHAVIORAL INTERVENTION IN THIS WAY YOU REALIZE THAT YOU CAN BUILD A BEHAVIORAL INTERVENTION, YOU CAN THINK ABOUT IMPROVING IT, YOU CAN THINK ABOUT OPTIMIZING IT. IN MUCH THE SAME WAY THAT OTHER DEVICES AND PROCESSES ARE OPTIMIZED. OF COURSE, THE FIELD OF ENGINEERING HAS BEEN DEVELOPED OVER THE YEARS TO DO PRECISELY THIS, TO BUILD THINGS, IMPROVE THEM AND ULTIMATELY TO OPTIMIZE THEM. PAGE DOWN ISN'T WORKING. I THINK I HAVE IT NOW. THIS THING IS NOT WORKING VERY WELL. OKAY. SO ONE THING I WANTED TO SUGGEST BEFORE WE GO ON WE GET SOME TERMINOLOGY CLEAR. I HAVE ALREADY SEEN A LITTLE BIT OF CONFUSION ABOUT THIS HERE AND THERE SO I WOULD LIKE TO SUGGEST THAT FOR THE REST OF TODAY WE'RE CAREFUL TO DRAW A DISTINCTION BETWEEN INTERVENTION DESIGN AND EXPERIMENTAL DESIGN. BECAUSE I HAVE SEEN THE WORD DESIGN USED JUST BY ITSELF. IT'S SOMETIMES NOT CLEAR WHICH WE MEAN. SO AN INTERVENTION DESIGN I WOULD LIKE TO SUGGEST REFERS TO THE CONTENT AND APPROACH OF BEHAVIORAL INTERVENTION, FOR EXAMPLE, AN ADAPTIVE INTERVENTION IS ONE KIND OF INTERVENTION DESIGN. THE EXPERIMENTAL DESIGN RATHER THAN INTERVENTION DESIGN, IS THE APPROACH TO COLLECTING INFORMATION THAT'S GOING TO INFORM THE CHOICE OF THE INTERVENTION DESIGN. SO FOR EXAMPLE YOU MIGHT CONDUCT A FACTORIAL EXPERIMENT, RCT, SUSAN WILL TALK ABOUT THE SMART TRIAL, THOSE ARE ALL EXPERIMENTAL DESIGNS. SO WE NEED TO BE CLEAR ABOUT WHICH ONE OF THOSE WE'RE TALKING ABOUT. SO IF YOU ADOPT ENGINEERING BASED PERSPECTIVE ON BEHAVIORALLER INT VENGSES IT SUGGESTIONS HOW WE SHOULD PROCEED. ONE THING IT SUGGESTIONS IS THAT WE NEED TO HUSBAND OUR RESEARCH RESOURCES CAREFULLY BY SELECTING THE MOST APPROPRIATE AND EFFICIENT EXPERIMENTAL DESIGN. A LOT OF TIMES THAT'S NOT NECESSARILY GOING TO BE AN RCT. THERE'S A WORLD OF DESIGN OUT THERE. WE SAW SOME OF THEM IN THE PREVIOUS SESSION. THE IDEA HERE IS TO ACHIEVE, WE WANT TO ACHIEVE THE MOST SCIENTIFIC GAIN GIVEN RESOURCES THAT WE HAVE AVAILABLE. SO YOU CAN LOOK AT THIS VERY DISPASSIONATELY, I HAVE THIS MUCH MONEY, I HAVE THIS MUCH TIME, I HAVE ACCESS TO THIS MANY SUBJECTS. THESE ARE THE SCIENTIFIC QUESTIONS THAT I NEED THE ADDRESS, -- NEED TO ADDRESS HOW CAN I CHOOSE A DESIGN THAT GETS ME THE MOST GIVEN RESOURCES I HAVE. THIS MEANS RESEARCH QUESTIONS HAVE TO BE STATE AND PRIORITIZED. YOU CAN'T DO EVERYTHING IN ONE EXPERIMENT. I WOULD LIKE TO SEE US START TO THINK IN TERMS OF OPTIMIZING BEHAVIORAL INTERVENTIONS. BONNIE'S PRESENTATION TALKS ABOUT GOING FROM GOOD TO GREAT. I THINK WE CAN TALK -- WE HAVE A LOT OF VERY GOOD BEHAVIORAL INTERVENTION, WE CAN HAVE GREAT BEHAVIORAL INTERVENTIONS WITH I THINK ABOUT OPTIMIZING THEM. THE DEFINITION OF OPTIMIZATION THAT I'M THINKING OF IS THE PROCESS OF FINDING THE BEST POSSIBLE SOLUTION TO A PROBLEM SUBJECT TO GIVEN CONSTRAINTS SO I'M NOT TALKING THE BEST SOLUTION IN AN ABSOLUTE SENSE OR AN IDEAL SENSE, I'M TALKING ABOUT THE BEST WE CAN GET GIVEN THE CONSTRAINTS THAT WE HAVE ON HAND. FOR EXAMPLE, IF WE'RE VERY CLEAR ABOUT THE OPTIMIZATION CRITERION, YOU MIGHT SHOOT FOR THE MOST EFFECTIVE INTERVENTION YOU CAN DELIVER FOR LESS THAN $200 A PERSON. OR THE MOST EFFECTIVE INTERVENTION YOU CAN DELIVER FOR IN LESS THAN A HALF HOUR. THERE'S A LOT OF DIFFERENT OPTIMIZATION CRITERIA THAT ARE POSSIBLE BUT THE POINT I WANT TO MAKE IS THAT IT IS POSSIBLE GIVEN TECHNOLOGY WE HAVE TODAY, TO ENGINEER BEHAVIORAL INTERVENTIONS TO MEET SPECIFIC OPT MY -- OPTIMIZATION CRITERIA. WE SHOULD CONSIDER OPTIMIZATION OF PROCESS, IT'S NOT AN END GOAL. WE SHOULD NEVER CONSIDER AN INTERVENTION FINISHED. WE SHALL CONSIDER IT SOMETHING BEING IMPROVED. EVALUATION AND OPTIMIZATION ARE IMPORTANT, NOT THE SAME THING, EVALUATION, POSES THE QUESTION IS THE INTERVENTIONS AFFECTS STATISTICALLY SIGNIFICANT. IT'S AN IMPORTANT QUESTION. BUT THERE'S ANOTHER QUESTION WHICH IS THE QUESTION OF OPTIMIZATION. IS INTERVENTION THE BEST POSSIBLE GIVEN THE AVAILABLE CONSTRAINTS. SO IT COULD BE AN EFFECT IS NOT STATISTICALLY SIGNIFICANT BUT THE INTERVENTION IS NOT OPTIMIZED SO YOU MIGHT WANT TO DO FURTHER OPTIMIZATION WORK. IT MIGHT BE TO BELIEVE THAT POSSIBLE THAT THE INTERVENTION IS NOT OPTIMIZED, STILL NOT STATISTICALLY SIGNIFICANT. THERE YOU WANT TO GO BACK TO THE DRAWING BOARD, PERHAPS. THIS IS WHERE MOST OF OUR BEHAVIORAL INTERVENTIONS ARE TODAY. STATISTICALLY SIGNIFICANT, NEVER BEEN OPTIMIZED. THIS IS WHAT I THINK WE SHOULD BE AIMING FOR. INTERVENTIONS THAT ARE BOTH HAVE A STATISTICALLY SIGNIFICANT EFFECT AND HAVE AN OPTIMIZEDCH THIS IS COMPLETELY DOABLE. SO BEING AN EEGTISCAL ACADEMIC I HAD TO MENTION MY WORK THOUGH THAT'S NOT WE'RE HERE FOR. I HAVE BEEN WORKING ON APPROACHING FOR BUILDING OPTIMIZING AND EVALUATING BEHAVIORAL INTERVENTIONS AN IDEAS FROM ENGINEERING CONTROLS TO OPTIMIZE VARYING ADAPTIVE BEHAVIORAL INTERVENTIONS. THE THREE PRESENTATIONS TODAY ARE SUPER INTERESTING. BONNIE WILL TALK ABOUT WHERE BEHAVIORAL INTERVENTION SCIENCE FIELD IS HEADING, SHE'LL TALK FROM GOOD TO GREAT IN OUR BEHAVIORAL INTERVENTIONS AND HOW IMPORTANT IT IS TO OPTIMIZE IN SEVERAL WAYS TO GET WHERE WE'RE GOING. SUSAN WILL TALK ABOUT A PARTICULAR EXPERIMENTAL DESIGN, THE SEQUENTIAL RANDOMIZED TRIAL. THIS IS AN EXPERIMENTAL DESIGN THAT INFORMS THE ADAPTIVE INTERVENTION DESIGN, REALLY GREAT FOR ANSWERING THOSE SPECIFIC QUESTIONS. I'M LUCKY TO WORK WITH BONNIE AND SUSAN. I NEVER MET DAVID BEFORE TODAY BUT HAD A CHAN TO SEE A SLIDE. I THINK YOU'RE GOING THE LIKE HIS PRESENTATION. HE'LL TALK ABOUT BEHAVIORAL INTERVENTIONS. AND BUILDING ON PRIOR RESULTS WHICH IS NOT SOMETHING WE DO ENOUGH IN BEHAVIORAL SCIENCES. WE TALKED THIS MORNING ABOUT SINGLE SUBJECT EXPERIMENTS AND I THINK HIS WORK ALSO OPENS UP THAT POSSIBILITY. SO I'LL CLOSE NOW AN YOU HAVE THREE GREAT PRESENTATIONS TO LOOK FORWARD TO. THANK YOU. [APPLAUSE] >> NEXT SPEAKER IS BONNIE SPRING FROM NORTHWESTERN UNIVERSITY. >> THANKS, BILL FOR PUTTING THIS MEETING TOGETHER. THIS IS MUCH NEEDED, VERY EXCITING. THANK YOU. WHAT I HAVE BECOME THROUGH MY WORK ON mHEALTH I HAVE BECOME INTERESTED IN SCIENCE. I'M GOING THE TAKE A CONTRARIAN VIEW POINT. AFTER LINDA SAID YOU'RE TALKING ABOUT INTERVENTION DESIGN, NOT NECESSARILY THE SYSTEMATIC USE OF DATA TO FIGURE OUT HOW TO MOD FIE A DESIGN. FROM A TEAM SCIENCE PERSPECTIVE THAT'S THE LAST THING I TELL MY TEAM SCIENCE COLLABORATORS. WHAT TO HIRE DOING THAT THEY CONSIDER RESEARCH IS NOT HOW THEY SHOULD BE CHECKING DATA TO DETERMINE THE DESIGN OF AN INTERVENTION. SO BECAUSE I HAVE LEARNED SOMETHING FROM THEM, WHAT I'M GOING TO DO IS JUST TALK ABOUT FOUR DIFFERENT WAYS THAT THE PEOPLE THAT I WORK WITH CONSIDER THAT THEY'VE GOTTEN ENOUGH INFORMATION TO FIGURE OUT WHAT TO DO NOW WITH THEIR DESIGN. SO I'M AN RCT PERSON FROM WAY BACK. I LOVE THESE DESIGNS. THEY ARE OUR GOLD STANDARD. THEY ARE FABULOUS IN TERMS OF PRESERVING INTERNAL VALIDITY, THAT'S THE MAIN THING THEY'RE FOR, TO PROTECT US AGAINST ASSUMING THAT WE HAVE AN INTERVENTION THAT WORKS WHEN IT DOESN'T. THE WAY THEY DETERMINE THAT IS IT HAS TO BE COMPARED TO SOMETHING BUT IT COMES DOWN TO A QUESTION OF IT'S BETTER THAN SOMETHING ELSE. IF WE'RE GOING PILL TRIALS WE CAN USE PLACEBOS, WE CAN CHECK THE TREATMENT INTEGRITY, WE'LL PROBABLY ASSAY THE PILLS BUT WHEN IT COMES TO BEHAVIORAL INTERVENTIONS, THE CORE CHALLENGES THAT WE FACE ARE THOSE OF TREATMENT FIDELITY. THIS ISSUE HAS COME UP ALREADY FOR TREATMENT FIDELITY DELIVERED AS INTENDED. TREATMENT INTEGRITY. THERE SHOULD BE DIFFERENTIATION FROM THE CONTROL CONDITION, NO BLEEDING. AND THE KEY THING. AND THE KEY ISSUE IS THE INTERVENTION SHOULD BE GIVEN EXACTLY THE SAME AT END OF THE TRIAL AS THE BEGINNING OF THE TRIAL. SO THIS IS WHERE THINGS GET HAIRY BECAUSE THERE'S NO ITERATION ALLOWED WITNESS YOU START THE RCT. NOW, RCTs DO HAVE -- THEY HAVE AN ADVANTAGE WITH TECHNOLOGY INTERVENTIONS BECAUSE TO THE EXTENT THE INTERVENTION IS ONLY THE TECHNOLOGY, THAT SHOULD BE THE CASE. YOU'RE NOT REDESIGNING THE TECHNOLOGY MIDSTREAM. INTERVENTION SHOULD BE THE SAME AT THE END. TO THE EXTENT YOU HAVE A HUMAN ELEMENT THAT MAY OR MAY NOT BE THE CASE. YOU MAY GET INVENTION ALONG THE WAY. MY ARGUMENT IS THAT WITH RCTs, THESE ARE THE BEST THING TO DO IF YOU'RE AT THE END OF AN INTERVENTION DEVELOPMENT PROCESS. YOU KNOW WHAT YOUR ACTIVE COMPONENTS ARE. YOU BUNDLE THEM. WHERE I THINK WE OFTEN SHOOT OURSELVES IN THE FOOT IS WE TRY TO DO THIS FIRST. WE DON'T SPEND ENOUGH TIME DEVELOPING AN OPTIMIZING THE INTERVENTION. HERE IS ABOUT AS RADICALLY DIFFERENT AN APPROACH AS YOU CAN POSSIBLY TAKECH THIS IS DONALD NORMAN. WHO IS ONE OF MY COLLABORATOR, HE STARTED HIS LIFE AS A COGNITIVE SCIENTIST AND THEN BECAME THE DESIGN GURU OF USER-CENTERED DESIGN. AND THE PREMISE OF THIS APPROACH IS THAT UNLIKE WHAT WE -- WHAT MANY OF US DO WITH INTERVENTIONS WHERE WE BUNDLE THEM UP, TIE THEM WITH A RED RIBBON AN EXPECT PEOPLE THE ADAPT TO THEM THE WAY WE DELIVER THEM, THESE PEOPLE SAY NO, THAT'S NOT WHAT WE WANT TO DO. WE WANT TO OPTIMIZE THE INTERVENTION AROUND USER NEEDS, BOTH THOSE THEY KNOW AN EXPRESS, AND THOSE THEY CAN'T EXPRESS. SO THE WAY THEY DO THIS IS THEY GET RIGHT AWAY INTO A CYCLE OF ANALYZING THE USER NEEDS, DESIGNING A ROUGH PROTOTYPE, HAVING PEOPLE CARRY AROUND THE PROTOTYPE, GET THEIR FEEDBACK ON IT, TAKE THE USER FEEDBACK REDESIGN AND GO THROUGH A CONSTANT CYCLE OF ITERATION. IT'S ONGOING ITERATION. ABOUT AS OPT SIT AS YOU CAN BE FROM AN RCT. HOW DO THEY KNOW WHEN THEY'RE DONE? WELL, THEY KNOW WHEN THEY'RE DONE, WHEN THE USERS SAY THEY'RE SATISFIED AND THEY GET SATURATION OF THEMES. I HAVE HAD THIS ISSUE, WITH WORKING WITH DESIGN FOLKS, YOU MAKE UP A BUDGET SAYING OKAY, YOU'RE GOING TO TEST 22 PEOPLE, THEY COME BACK AND THEY HAVE TESTED TEN AND I SAY I WANT HALF THE BUMENT BACK. THEY SAY NO, THIS IS -- WE DID IT WITH ENOUGH DEPTH TO REACH SATURATION. RADICALLY DIFFERENT VIEW. WHAT'S GOING ON WITH THIS GENTLEMAN HERE? THEY DISCOVERED UNEXPRESSED NEED. THE GUY WANTS TO USE A SMART PHONE IN THE BATHTUB SO THEY DEVELOPED A WAY TO DO THAT. MULTI-FADES OPTIMIZATION5wF,K JEAN THEM, FIND OUT WHICH ARE ACTIVE, WHICH ARE DEAD WOOD. AND WHICH MAXIMIZE YOUR INTERVENTION UTILITY ACCORDING TO A NUMBER OF CRITERIA, MONEY, TIME, SO THE PREMISE IS YOU KNOW WHAT THE POSSIBLE INTERVENTION INGREED WENTS ARE -- INGREDIENTS ARE AND ORGANIZE THEM STRATEGICALLY TO COME OUT WITH THE BEST COMPONENTS. THIS IS ONE FROM SOFTWARE ENGINEERING COLLEAGUESCH BLACK BOX ACCEPTING TESTINGCH WE SHOULD DO MORE OF THESE WITH OUR INTERVENTIONSCH THIS IS WHAT THEY DO WHEN THEY FINISHED A TECHNOLOGY AND NOW THEY WANT TO KNOW HOW IS IT GOING TO BE USED. AND THEY MAKE THE ASSUMPTION THAT THERE ARE ALWAYS BUGS IN THE TECHNOLOGY. WHEN THEY'RE TESTING THIS IN ONE WAY THEY CALL IT WHITE BOX TESTING, THEY TELL THE TESTERS WHAT THE SOFTWARE IS LIKE SO THEY CAN TEST WHETHER IT'S FULFILLING ITS MAJOR FUNCTIONS. BUT WHEN THEY ARE -- WHEN THEY'RE TESTING AT THIS OTHER WAY, THEY CALL THE BLACK BOX ACCEPTANCE TESTING THEY DONE KNOW, TESTER DOESN'T KNOW WHAT IS BUILT INSIDE THE BOX. THEY DON'T KNOW THE WIRING DIAGRAM. SO THEY USE IT. THEY JUST TEST IT, THEY KNOW THE INPUT, KNOW WHAT THE OUTPUT IS SUPPOSED TO BE BUT DISCOVER THESE VARIOUS UNUSUAL WAYS THAT PEOPLE WIND UP USING TECHNOLOGY. HERE WE HAVE THE UNUSUAL USE OF DECIDING TO CHEW ON THE ELECTRONIC TECHNOLOGY. SO VERY DIFFERENT VIEW AND AGAIN, A WAY THEY COLLECT DATA TO KNOW IF THEY'RE DONE. I THINK CORE QUESTIONS FOR US WHEN WE THINK ABOUT DESIGN IS WE NEED TO THINK ABOUT WHAT WE'RE TRYING TO LEARN FROM IT. WITH THE RRT WE'RE TRYING TO LEARN IS IT BETTER THAN THE COMPARISON CONDITION. IF WE'RE OPTIMIZING OR USING USER SENORRED DESIGN WE ASK IS IT AS GOOD AS IT CAN BE. BLACK BOX ACCEPTANCE TESTING WE'RE ASKING ARE WE DONE YET. I THINK WE'RE AT THE BEGINNING OF A BLUE SKY ERA, VERY EXCITING. WE HAVE A NUMBER OF CONSIDERATIONS WE NEED TO BALANCE AND FIGURING OUT RESEARCH DESIGN APPROACH AND THANKS FOR HAVING ME. [APPLAUSE] >> NEXT SPEAKER IS SUSAN MURPHY FROM UNIVERSITY OF MICHIGAN. >> LINDA MENTIONED ACTUALLY THE WORD ADAPTED PHRASE, ADAPTIVE INTERVENTION, I WANT TO MAKE SURE WE'RE ON SAME PAGE SO THOUGHT I WOULD PUT A DEFINITION. I WANT TO POINT OUT, THIS AN INTERVENTION DESIGN, THE DESIGN OF A SEQUENCE OF TREATMENTS. SO ADAPTIVE INTERVENTIONS ARE INDIVIDUALLY TAILORED SEQUENCES OF TREATMENT DECISIONS. WHAT DO YOU PROVIDE FIRST, IF THE PERSON DOES WELL, WHAT DO YOU PROVIDE SECOND IF THE PERSON DOESN'T DO WELL, WHAT DO YOU DO THEN? THE IDEA IS ADAPTIVE INTERVENTIONS TAYLOR THESE DECISIONS, THESE TREATMENT DECISIONS ACCORDING TO DYNAMICALLY INVOLVING INFORMATION WHETHER OR NOT THE PERSON RESPONDED TO THE INITIAL TREATMENT, WHETHER THEY'RE SHOWING POOR ADHERENCE TO THE INITIAL TREATMENT. THEN WHAT SHOULD YOU DO. ACROSS DIFFERENT FEELS THEY'RE KNOWN BY DIFFERENT NAMES. IN COMPUTER SCIENCE THESE ARE OFTEN CALLED TREATMENT POLICIES. THIS HAS TO DO WITH MULTI-STAGE DECISION MAKING. IN MENTAL HEALTH SOMETIMES THEY'RE CALLED ADAPTIVE TREATMENT STRATEGIES, IT DEPENDS ON THE AREA HERE WE'RE JUST CALLING THM ADAPTIVE INTERVENTIONS. A LITTLE BIT MORE ON INTERVENTION DESIGN, ONE MORE JUST LITTLE BIT MORE EXPLICATION. SOME EXAMPLES OF DYNAMIC INDIVIDUAL INFORMATION COULD BE THE PERSON'S LEVEL OF CRAVING AT THAT POINT IN TIME. COULD BE WHETHER OR NOT THEY HAVE JUST EXPERIENCED A SLIP, THEY JUST LIT A CIGARETTE. COULD BE THEIR ACTIVITY LEVEL, HOW MUCH THEY'RE WAP WALKING AT THIS MOMENT, THEIR PAIN LEVEL. COULD ALSO BE MORE GEOGRAPHICAL INFORMATION, WHETHER OR NOT THEY'RE CLOSE TO A BAR, IF THEY'RE SOMEONE IS AN ALCOHOLIC, SO ON. COULD BE SYMPTOMS. SOME EXAMPLE, DECISIONS COULD BE MESSAGES THAT ARE SUPPLIED TO THE INDIVIDUAL AT THAT POINT IN TIME, DIFFERENT TYPES OF BEHAVIORAL MESSAGES, A DOSE -- A SUGGESTION AS TO A DOSAGE LEVEL THEY MIGHT USE AT THAT POINT IN TIME. GOALS THAT THEY MIGHT USE SAY TO IMPRO-THEIR EXERCISE. EMAIL TO A BUDDY ABOUT A PROBLEM THEY'RE EXPERIENCING OR EMAIL OR ALARM TO A HEALTHCARE PROVIDER. THESE ARE TYPES OF DECISIONS THAT ONE COULD HAVE BASED ON1An THE DYNAMIC INDIVIDUAL INFORMATION. AGAIN WE'RE TALKING ABOUT INTERVENTION DESIGN. THE SMART DESIGN, THE ACRONYM IS SEQUENTIAL MULTIPLE ASSIGNMENT RANDOMIZED TRIAL DESIGN. WHAT DOES THIS MEAN? IT MEANS THAT EACH INDIVIDUAL IS MULTIPLY RANDOMIZED. SO AN INDIVIDUAL IS RANDOMIZED REPEATEDLY OVER TIME. THIS IS NOT A STANDARD RCT, NOT MEANT TO EVALUATE A COMPLETELY FORMED ADAPTIVE INTERVENTION. INSTEAD IT'S MEANT TO CONSTRUCT AND OPTIMIZE AN ADAPTIVE INTERVENTION. IT FITS PERFECTLY INTO THE -- LINDA'S FRAMEWORK, THIS IDEA OPTIMIZING INTERVENTIONS AS YOU GO THROUGH. IT'S VERY DIFFERENT FROM WHAT'S KNOWN IN THE CLINICAL TRIAL LITERATURE AS THE ADAPTIVE CLINICAL TRIAL THAT'S SOMETHING CLEAT COMPLETELY DIFFERENCE. ADAPTIVE CLINICAL TRIAL ADAPT IT IS TRIAL ON PAST INDIVIDUALSCH THIS IS THE TREATMENT IS ADAPTED TO THE PERSON TO EACH PERSON DURING THE TRIAL. ONE THINKS ABOUT WANTING TO OPTIMIZE AN ADAPTIVE OR BUILD ADAPTIVE INTERVENTION TREATMENT POLICY, SEQUENCE OF TREATMENTS FIRST THING WE WANT TO DO IS PINPOINT A -- IT COULD BE EXPLICATION. WHAT DO I DO WHEN SOMEONE HAS A SLIP. THEY JUST LIT A SEG CIGARETTE. WHETHER CAN WE DO? YOU THINK ABOUT WHAT ALTERNATIVE DECISIONS SHOULD YOU MAKE, SHOULD THERE BE AN ENCOURAGING MESSAGE, IT'S JUST A SLIP OR SOME OTHER MESSAGE OR SHOULD YOU PROVIDE A MESSAGE AT ALL. SO HERE IS SOME -- SOME EXAMPLES, A CRITICAL DECISION WOULD BE AN INDIVIDUAL IS IN A DANGEROUS TO THEM LOCATION, IS IT EFFECTIVE TO SEND A MESSAGE? WHICH OF SEVERAL TYPE OF MESSAGES IS MOST EFFECTIVE? ONE FOUNDATION OF THIS DESIGN IS IT HAS RANDOMIZATION AND YOU RANDOMIZE BETWEEN SENDING A MESSAGE, NOT SENDING A MESSAGE EVERY TIME THE PERSON IS IN ONE OF THESE DANGEROUS LOCATIONS. OR COULD BE THAT YOU RANDOMIZE BETWEEN TWO VERY DIFFERENT TYPES OF MESSAGES IF THEY HAVE A SLIP. THEY JUST LIT A CIGARETTE, WHAT TYPE OF MESSAGES YOU SHOULD GO. AGAIN WE'RE TALKING ABOUT EXPERIMENTAL DESIGN. WHY RANDOMIZATION, THE PURPOSE OF RANDOMIZATION IS IT ALLOWS US TO FIRST DECIDE WHAT TYPE OF -- WE OPEN THE BLACK BOX, WHAT TYPE OF TREATMENT DECISION TO MAKE AT THESE CRITICAL DECISIONS. AND DISENTANGLE THE FACT OF THE MESSAGE FROM WHY THE PERSON GOT THE MESSAGE. IF YOU HAVE ANY QUESTIONS LATER YOU CAN SEND ME AN EMAIL. [APPLAUSE] >> ALWAYS GOOD WHEN SPEAKERS ARE ON THEIR LAST SLIDE AS I PUT UP THAT ZERO. PERFECT. SO OUR NEXT SPEAKER IS DAVID MOORE, NORTHERN WESTERN UNIVERSITY. >> THANKS. I WANT TO ACKNOWLEDGE N AIRKSWA DWAN WHO HELPED PUTTING THIS TALK TOGETHER. HE WASN'T ABLE TO BE HERE TODAY. SO THE PROBLEM I THINK THAT WE HAVE -- OR ONE OF THE PROBLEMS THAT'S COME UP A NUMBER OF TIMES TODAY IS THAT WHEN WE'RE LOOKING AT mHEALTH WE'RE DEALING WITH INTERVENTIONS IN PERPETUAL DATA. THEY'RE SLIGHTLY CHANGING OR ADDING TO THE TREATMENT TARGET TO IMPROVE EFFICACY, BROADENING TO IMPROVE THE ABILITY INCORPORATING ADVANCEMENTS IN TECHNOLOGY, IMPROVING DESIGN AND APPEAL. SO I THINK WE HAVE ALL -- WE HEARD THIS OVER AN OVER AGAIN THE ONE TRIAL AT A TIME METHODOLOGY JUST DOESN'T CUT IT HERE. THEY REQUIRE CONSIDERABLE TIME AN RESOURCES AND THEN BY THE TIME YOU'RE AT THE END OF THE TRIAL, WHATEVER YOU HAVE IS A HORSE AND BUT GI, IT'S OBSOLETE. AND THEN AGAIN, THE ISSUE OF SUBSEQUENT CHANGES TO THE INTERVENTION MAY ALTER THE EFFECTIVENESS. BUT I THINK THAT IF WE THINK ABOUT THESE INTERVENTIONS BEING IN PERPETUAL BETA, THAT'S NOT NECESSARILY A PROBLEM, IT COULD BE AN ADVANTAGE GIVEN THE WAY IN WHICH WE CAN COLLECT DATA FROM mHEALTH TECHNOLOGY. SO WE HAVE A FLOW OF INFORMATION THAT MOVES TO AND FROM THE USER THAT ALLOWS FOR ONGOING COLLECTION OF DATA. I THINK REALLY CRITICALLY WHEN WE'RE LOOKING AT BEHAVIORAL INTERVENTIONS, ALMOST ALWAYS WE'RE COLLECTING THE OUTCOME AS WE GO ALONG. SELF-MONITORING IS ALMOST ALWAYS PART OF IT SO WE'RE GETTING THE PRIMARY OUTCOMES. SO THE CAPACITY FOR mHEALTH INTERVENTION PLATFORMS TO COLLECT ONGOING DATA ALLOWS US TO BEGIN TO THINK AB CLINICAL CARE SYSTEMS IN DEPLOYMENT AS ENGINEERING SYSTEMS THAT PERMIT THE EVALUATION OF MULTIPLE CHANGING COMPONENTS AS WELL AS INTERACTIONS BETWEEN THOSE. SO I'M GOING TO PRESENT A VERY SIMPLE SCHEMATIC OF HOW THIS WILL WORK AND THEN A COUPLE OF HOW WE CAN MANAGE SOME OF THE PROBLEMS WITHIN THIS FRAMEWORK. SO ON THE VERTICAL AXIS HERE WE HAVE EFFECT SIZE AND CONFIDENCE INTERVALS AND THEN TIME MOVING ALONG THE HORIZONTAL AXIS. IF WE IMAGINE THAT IN DEPLOYMENT WE'RE DEPLOYING AN mHEALTH INTERVENTION VERSION 1.0, YOU CAN PUT A CONTROL ARM IN HERE AS WELL BUT I WON'T COMPLICATE THE DISCUSSION HERE WITH THAT. SO THEN, IMAGINE .2 WE HAVE A SECOND mHEALTH INTERVENTION, YOU SEE THE CONFIDENCE INTERVALS NARROW THERE A BIT. THE SECOND ONE IS DEPLOYED, TIME .3, A THIRD IS DEPLOYED, TIME POINT FOUR YOU BEGIN TO SEE THE FIRST ONE DIFFERENTIATES ITSELF FROM THE FOURTH. SO IT CAN BE DROPPED. THERE'S EQUIPOISE BETWEEN VERSION TWO, THREE AND FOUR BUT THE FIRST ONE HAS SHOWN ITSELF LESS EFFECTIVE DROPPED. TIME .5 THERE'S NO EQUIPOISE THERE, IT'S LESS EFFECTIVE AND IT CAN BE DROPPED. THE POINT IS THAT ESSENTIALLY THIS ALLOWS -- THIS BEGINS TO BREAK DOWN THE DIFFERENCE BETWEEN AN RCT, BETWEEN HAVING SOMETHING THAT SUPPOSEDLY CONFIRMATORY AN DONE AND QUALITATIVE IMPROVEMENT. I DON'T THINK WITH mHEALTH INTERVENTIONS WE NEED TO NECESSARILY MAKE THAT DISTINCTION ANY MORE. THERE ARE SOME PROBLEMS THAT WITH THIS, ONE MIGHT BE FOR EXAMPLE TRANSITIVITY. IF WE HAVE FOR EXAMPLE A SITUATION WHERE TWO IS BETTER THAN ONE, THREE BETTER THAN TWO, FOUR IS BETTER THAN THREE, CAN WE CONCLUDE 4 IS BETTER THAN 3? POPULATIONS CHANGE OVER TIME. EARLY ADOPTERS ARE DIFFERENT THAN LATER ADOPTERS. BUT I THINK THAT IN THIS CASE IT IS POSSIBLE TO TAKE PROGNOSTIC FACTORS LIKE BASELINE DEMOGRAPHICS, USE PATTERN, OTHER DATA COLLECTED AS PART OF THIS AND USE THOSE TO EVALUATE THE VALIDITY OF THIS TRANSITIVITY INFERENCE AND IF IN FACT THERE ARE CHANGES IN THE POPULATION TO CONTROL FOR THAT -- THOSE POTENTIAL BIASES STATISTICALLY. SO ANOTHER ISSUE IN DEPLOYMENT MORE IN REAL WORLD SETTINGS IS WHEN YOU ENROLL SOMEBODY IN A CLINICAL TRIAL PEOPLE EXPECT TO BE RANDOMIZED TO ONE THING AND NOT HAVE A CHOICE. IN THE REAL WORLD PEOPLE WANT TO HAVE A CHOICE. THEY WANT PREFERENCE, AT LEAST THEY WANT THE ILLUSION OF IT. AND SO HOW CAN YOU -- HOW CAN YOU MANAGE THAT? I THINK THAT TOO CAN BE MANAGED THROUGH EQUIPOISE STRATIFIED RANDOMIZATION. IF THE AVAILABLE TREATMENTS WHETHER EQUIPOISE, IF PEOPLE ARE ALLOWED TO CHOOSE WHAT THEY PREFER, THEN RANDOMIZATION CAN OCCUR WITHIN THAT EQUIPOISE STRATUM AND PREFERENCE CAN BE CONVERTED INTO A PRE-RANDOMIZATION FACTOR THAT CAN BE STATISTICALLY CONTROLLED. SO WHAT I'M SUGGESTING HERE REALLY IS THAT I THINK THIS IDEA OF THE RCT BEING SOMETHING CONFIRMATORY ENDS AND THEN IT'S SIMPLY LET LOOSE IN THE ENVIRONMENT, WE DON'T NEED THAT. WHAT WE CAN DO IS BEGIN TO THINK ABOUT THE DEPLOYMENT OF THESE THINGS AS IN TERMS OF MONITORING, EVALUATION, WE CAN CONTINUE TO COLLECT DATA AS THESE THINGS MATURE. I THINK THAT WE CAN MANAGE -- WE CAN DEVELOP METHODS OF MANAGING THAT. BREAKS DOWN DISTINCTION BETWEEN FEEL MONITORING AN CLINICAL IMPROVEMENT, IT INB CLIEWDZ PROTECTIONS HERE. SO I THINK THAT THIS IS ALSO A FLEXIBLE FRAMEWORK, SOMETHING THAT CAN INCORPORATE SMART OR OTHER METHODOLOGIES. [APPLAUSE] >> SO WENDY IS INSTRUCTING ME ON ALL SORTS OF THINGS BEFORE WE START ON THE DISCUSSION. SO ONE, EVEN THOUGH I HAVE BEEN IN TECHNOLOGY FOR A WHILE I HAVE NEVER TWEETED BUT APPARENTLY SOME PEOPLE DO. SO IF YOU'RE ON THE WEBINAR AND YOU WOULD LIKE TO TWEET YOUR QUESTION YOU CAN. I PERSONALLY NEVER THOUGHT ANYBODY WOULD CARE ABOUT ANYTHING I WAS DOING IN LIFE SO I DIDN'T THINK IT NECESSARY TO TWEET ANYTHING ABOUT IT. THE SECOND ISSUE IS THAT THE MICS ARE ON AND IF YOU LEAVE THEM ON THEY'RE VOICE ACTIVATE SOD WHEN YOU START TALKING THEY EVENTUALLY CATCH UP TO YOU AND START AMPLIFYING. SO I WILL OPEN IT UP FOR DISCUSSION AND QUESTIONS. I THINK THOSE HAVE BEEN FOUR INTERESTING WAYS OF THINKING ABOUT EVALUATION OF MOBILE TECHNOLOGIES. WE'LL START US OFF. BOB -- AND I'M GOING TRY TO KEEP UP WITH HANDS BUT I MAY NOT DO THAT SO WELL. BUT BOB I THINK I SAW FIRST. >> HEY, DO WE NEED TO ROW RE INTRODUCE OURSELVES AS WE'RE SPEAKING FOR THE PEOPLE -- THIS IS BOB EVANS FROM GOOGLE. I HAD A QUESTION FOR SUSAN ABOUT ADAPTIVE INTERVENTIONS. EVEN THOUGH YOU'RE RANDOMIZING EACH CRITICAL DECISION POINT, ARE YOU STILL BUILDING ANY WEIGHTED DECISION TREE BEFORE THE EXPERIMENTAL STUDY GETS UNDERWAY SO THAT YOU HAVE A PROBABILITY? AND HOW DOES THAT IMPACT CHOICES TO RANDOMIZE AT THIS POINT? >> GOOD POINT. THE TRIALS THAT I'M INVOLVED WITH ALREADY, USUALLY THERE WERE ETHICAL OR STRONG SCIENTIFIC CONSIDERATIONS. WHICH FOR EXAMPLE YOU WOULD NOT CONSIDER THE SAME TREATMENT OPTIONS FOR EARLY RESPONDERS AS YOU WOULD FOR EARLY NON-RESPONDERS. SO THE CLASSIC -- THE TREATMENT OPTIONS ARE DIFFERENT DEPENDING HOW THE INDIVIDUAL WAS PROGRESSING THROUGH TIME. ONE SET OF TREATMENT OPTIONS, SAY THE EARLY RESPONDERS ARE GETTING MORE MAINTENANCE-TYPE OPTIONS BEING CONSIDERED. WHEREAS EARLY NON-RESPONDERS ARE SAYING WE NEED TO CHANGE TREATMENT, WE NEED TO DO SOMETHING ABOUT THIS. SO U YOU'RE LOOKING AT SWITCHES OF TREATMENT, SO ON. SO THE EXPERIMENTAL DESIGN ITSELF CAN INVOLVE DIFFERENT OPTIONS FOR DIFFERENT KINDS OF PEOPLE SO IT INVOLVES TAILORING. IN ADDITION YOU HAVE RANDOMIZATION. YOU WANT TO COLLECT OTHER INFORMATION AS WE ALWAYS DO IN THE CONDUCT OF THE TRIAL WHICH ALLOW YOU DISCOVER FOR EXAMPLE, MAYBE PEOPLE WHO ARE EXPERIENCING CRAVING AND IN A CERTAIN LOCATION SHOULD BE GETTING ONE OF THE TREATMENT OPTIONS PREFERENTIALLY OVER OTHER TREATMENT OPTIONS. SO EVEN THOUGH -- SO LET ME BE MORE PRECISE, SUPPOSE YOU KNOW THAT WHEN SOMEONE IS IN A CERTAIN LOCATION, YOU WANT A -- A DANGEROUS LOCATION UP RANDOMIZE TO ONE OF SEVERAL OPTIONS. YOU COLLECT IT. IT COULD BE IN THE ANALYSIS OF THE DATA YOU DISCOVER THEIR LEVEL OF CRAVING IS ALSO IMPORTANT IN DECIDING WHICH OF THOSE OPTIONS SHOULD BE SELECTED. SO THERE'S INDIVIDUALIZATION BUILT INTO THE DESIGN, A PRIORI AND YOU DISCOVER INDIVIDUALIZATION AS YOU ANALYZE THE DATA. (OFF MIC) >> IT DEPENDS ON THE SETTING. IF YOU HAVE A GROUP OF PEOPLE MOVING THROUGH, THE DESIGN OF THE TRIAL IS FIXED AND ANALYSIS OCCURS WITH A BATCH OF TRAINING SET. MACHINE LEARNING. HOWEVER IF YOU'RE LOOKING AT ONE POINT IN TIME IT'S MORE REINFORCEMENT LEARNING TYPE ALGORITHM. THESE COULD BE BAYESIAN, OFTEN THEY'RE NOT, THEY'RE HEIDI MENTIONAL PROBLEMS. IT DEPENDS ON THE SETTING. >> MY COMMENT GETS DOWN TO THE DIFFERENCE BETWEEN EMPERIMENTAL DESIGN AND INTERVENTION DESIGN. I WANT TO MAKE SURE IT'S CLEAR TO EVERYBODY WHEN SUSAN TALKED ABOUT RANDOMIZATION SHE WAS NOT TALKING ABOUT INCLUDING RANDOMIZATION IN THE INTERVENTION. THE RANDOMIZATION IS A PART OF THE DESIGN AND THE IDEA IS TO COME UP WITH A SET OF DECISION RULES THAT YOU USE LATER IN THE INTERVENTION. IN THE -- WHEN -- ONCE YOU CHOOSE THE FINAL INTERVENTION THAT INTERVENTION WOULD NOT INVOLVE RANDOMIZATION, I WANT TO MAKE IT CLEAR. >> IT WOULD BE DIFFERENT FOR DIFFERENT PEOPLE BECAUSE IT DEPENDS HOW THEIR SITUATION IS EVOLVING. SO HOW I'M TREATING IS DIFFERENCE FROM HOW YOU'RE TREATED, SO ON. (OFF MIC) >> SO I'M THINKING OF WHAT I HAVEN'T USED BUT I KNOW THE COMPUTER SCIENTISTS WHO WORK IN THIS AREA. THIS IS ARTIFICIAL INTELLIGENCE. THEY -- THERE SEEM'S YOU HAVE TO DO A LOT OF APPROXIMATE MAIGS TO GO DOWN THAT ROAD. YEAH. >> BARBARA >> MIDDLEMAN AT NIH. ONE QUESTION FOR DAVID, AN INSTRUMENTAL QUESTION. WHEN YOU TALKED EQUIPOISE STRATIFIED RANDOMIZATION. WHY DO YOU WANT TO TEST THE DIFFERENCE BETWEEN OPTIONS YOU THOUGHT A PRIORI WERE EQUAL? >> I THINK WHAT I'M SAYING IS WHEN PATIENTS HAVE PREFERENCES YOU DON'T NECESSARILY HAVE TO FORCE THEM TO BE RANDOMIZED TO SOMETHING THEY DON'T WANT. SO SAY YOU HAVE FOUR DIFFERENT INTERVENTIONS AND TWO OR THREE ARE ACCEPTABLE TO THE PATIENT, ONE IS NOT. YOU CAN RANDOMIZE WITHIN THOSE THREE. AND THAT -- THAT PREFERENCE CAN BECOME A PRE-RANDOMIZATION VARIABLE AND CONTROLLED FOR. >> I HAVE TO TALK MORE OFFLINE. I JUST DONE UNDERSTAND. >> SUPPOSE YOU'RE THE KIND OF PERSON, SUPPOSE YOU'RE DEPRESSED AND YOU DO NOT LIKE TALK THERAPY. IF HIS OPTIONS FOR THE DEPRESSED PERSON ARE COGNITIVE BEHAVIORAL THERAPY OR SSRI OR SSNI OR MAYBE SOME OTHER MEDICATION, YOU AS A PATIENT -- YOU AS A PARTICIPANT COULD SAY I'M ONLY INTERESTED IN TAKING A PILL. SO DON'T RANDOMIZE ME TO THIS COGNITIVE BEHAVIORAL THERAPY. ONLY RANDOMIZE ME BETWEEN THOSE THREE MEDICATIONS. THAT'S WHAT -- >> I UNDERSTAND THAT PART OF IT BUT SEEMS TO ME THAT THAT FROM VIEDZ YOU WITH ENORMOUS SKEW GOING INTO THE STUDY WHERE THE POPULATION WHO WAS ELECTING NEVER TO TALK AN ONLY TO TAKE SMEDZ SUBSTANTIALLY DIFFERENT. >> I GUESS IT WOULD FALL APART IF EVERYBODY HAD THE SAME PREFERENCES. BUT TYPICALLY THEY WON'T. SO YOU CAN USE THOSE AS PRE-RANDOMIZATION VARIABLES AN CONTROL FOR THAT. >> IN PREFERENCES HERE ARE LIKE TAILORING VARIABLES. THEY'RE WAY TO INDIVIDUALIZE THE TREATMENT TO THE PERSON. SO WHEN YOU THINK CLINICAL PRACTICE, IS IT WORTHWHILE TO KNOW HOW CBT WOULD WORK ON SOMEONE WHO IS COMPLETELY OPPOSED TO TAKING CBT. SO IT'S LIKE A VARIABLE THAT WOULD BE USED TO INDIVIDUALIZE THE TREATMENT FOR DEPRESENTATION. >> I WAS GOING TO SAY, I THOUGHT YOU MIGHT BE ASKING ABOUT THAT RATIONALE FOR THE CONCEPT OF EQUIPOISE. BECAUSE WE WOULDN'T DO A TRT CLINICAL TRIAL IF WE DIDN'T THINK -- CLINICAL TRIAL IF WE DIDN'T THINK WE WERE AT EQUIPOISE BUT YOUR POINT IS THAT WHAT IS EQUIPOISE DIFFERS FOR DIFFERENT PEOPLE. >> THE OTHER QUESTION I HAVE IS ABOUT THE NATURE OF THE ADAPTIVE PROGRAM THAT PROVIDE MONITORING AND FEEDBACK, PROVIDES A CERTAIN ENVIRONMENT FOR A PATIENT PARTICIPATING THAT HAS POTENTIAL TO MAKE YOU FEEL ATTENDED TO, CARED FOR, AND THAT PER SE MAYBE AN INTERVENTION IN AND OF ITSELF. SO HOW DO YOU DISTINGUISH BETWEEN THE EFFECT OF BEING IN SUCH A TRIAL WITH THE EFFECT OF THE INTERVENTION ITSELF? PARTICULARLY IF ALL THE PATIENTS IN THE TRIAL HAVE THIS ADVANTAGE OF BEING CARED FOR AND ATTENDED TO SO THAT'S GOING TO BE IN THE BACKGROUND FOR EVERYBODY. >> THAT'S SOMETHING I HAVE BEEN DOING THINKING ABOUT. I THINK WE NEED TO DRAW A DISTINCTION BETWEEN MONITORING THAT WOULD ALWAYS BE DONE AS PART OF THE INTERVENTION AN MONITORING THAT'S BEING DONE JUST FOR THE PURPOSE OF THE EXPERIMENT. THERE ARE A LOT OF SITUATIONS WHERE A CERTAIN AM OF MONITORING WOULD BE DONE IN THE COURSE OF AN ADAPTIVE INTERVENTION AND THAT MONITORING IS PART OF WHAT FORMS THE BASIS FOR THE ADAPTATION OF THE INTERVENTION. FOR EXAMPLE, IF YOU HAVE SOMEONE IN ADAPTIVE INTERVENTION, ALCOHOLIC IN ADAPTIVE INTERVENTION, YOU MAY MONITOR THEIR DRINKING DAILY AND VE VIEW IT, THE INTERVAL MIGHT BE EVERY COUPLE OF WEEKS. IF THEY'RE DRINKING A CERTAIN LEVEL YOU MAY STEP UP OR DOWN THE INTERVENTION. IF THAT MONITORING IS PART OF THE INTERVENTION THE ACTIVITY IS REALLY NOT AN ISSUE BECAUSE THAT'S ALL -- YOU HAVE TO THINK AS PAR OF THE INTERVENTION. SO I THINK IT HELPS TO DRAW THAT DISTINCTION IT'S NOT AN ISSUE OF PEOPLE FALSELY FEELING CARED FOR IF THAT'S AN INTEGRAL PART OF THE INTERVENTION. WE'RE MOVING TOWARD A LOT OF INTERVENTIONS WHERE THAT MEASUREMENT IS IN mHEALTH. >> >> CAN I RESPOND, THAT'S PART OF THE INTERVENTION, I'LL ADD TO WHAT YOU'RE SAYING THAT THE QUESTION IS MORE ONE OF MECHANISM THEN. IS -- SO YOU CAN IMAGINE IT'S ONGOING SELF-MONITORING AS PART OF AN INTERVENTION AND THERE MAYBE A NUMBER OF OTHER TASKS THAT THE PERSON IS EXPECTED TO PERFORM. IT MAYBE THAT ALL OF THAT IS DRIVEN BY MONITORING. IN FACT IN A COUPLE OF THINGS WE'RE DEVELOPING WE HAVE BEEN SURPRISED HOW EFFECTIVE MONITORING ALONE IS. THAT DOESN'T MEAN IT'S NOT EFFECTIVE, IT MEANS THAT IT MAYBE YOU CAN FIGURE THAT OUT. >> I WAS ASKING SPECIFICALLY IN THE CONTEXT OF AN INTERVENTION WHICH IS BEING TAILORED AND OPTIMIZED FOR INDIVIDUAL PATIENTS. THE SEND OF BEING INDIVIDUALLY TAKEN CARE OF WHEN YOU REALIZE THE EFFORT IS DIRECTED AT THE BEST THING FOR YOU AS THE SUBJECT. >> I THINK IF YOU HAVE ACHIEVED THAT, THAT'S -- YOU HAVE A VERY GOOD INTERVENTION. >> THE BEST THING IS THE PATIENT NOT THE SUBJECT. IT'S IMPORTANT THE DRAW THE DICTION BECAUSE IF ALL THAT EFFORT IS PART OF THE INTERVENTION THEN YOU DONE HAVE A CON FOUND WHEN YOU GO TO EVALUATE THE INTERVENTION LATER. IT'S TRUE YOU CAN'T TELL WHETHER IT'S SAY THE DRUG YOU'RE GIVING OR MONITORING OR THE ASPECT OF YOUR INTERVENTION, THAT'S TRUE IN ANY RCT. EVALUATING ANY PART OF THE PACKAGE, SO I DON'T SEE IT AS ANY DIFFERENT. >> I DON'T KNOW WHERE THE MIKE IS. >> MISHA PAVEL, NATIONAL SCIENCE FOUNDATION. I WAS DELIGHTED TO HEAR LINDA'S DESCRIPTION OF THE ENGINEERING APPROACH. AN ENGINEERING APPROACH YOU NEED SEVERAL COMPONENTS TO MAKE IT WORK. ONE IS YOU HAVE TO DEFINE THE OBSERVABLE WHICH IS WE DO. CONSTRAINT WHICH IS ARE RECEIVE SPECIFIED BUT WE NEED TO SPECIFY OBJECTIVE FUNCTION AND MOST WE HAVE TO HAVE AN UNDERLYING MODEL AND THE MODEL IS COMPUTATIONAL OR MATHEMATICAL. AS I HEARD THE FIRST THREE COMPONENTS SOMEWHAT ADDRESSED T MODELING ASPECT HAS NOT BEEN BROUGHT UP. I WAS WONDERING IF YOU AND THE REST OF THE PANEL CAN COMMENT ON THAT. >> I'LL START SINCE THAT WAS ABOUT MY PRESENTATION. FIRST I'LL SAY THE MULTI-PHASE OPTIMIZATION STRATEGY IS A WORK IN PROGRESS. I DON'T MEAN THAT IT'S COMPLETELY NAILED DOWN AT THIS POINT, IT'S AN AREA OF METHODOLOGICAL INQUIRY BUT YOU NEED AN OBJECTIVE FUNCTION, THAT'S PART OF WHAT I'M WORKING ON. I HAVE BEEN -- THERE'S TWO AREAS WHERE I'M WORKING. ONE IS IN TRYING TO ENCOURAGE BEHAVIORAL SCIENTIST TO USE FACTORIAL EXPERIMENTS IN THE WAY THEY'RE USED IN ENGINEERING. IN ENGINEERING WE CAN ALMOST NEVER GET AWAY WITH THOSE IN A FACTORIAL DESIGN IN THE BEHAVIORAL SCIENCES BECAUSE WE HAVE SO MUCH VARIABILITY. BUT YOU CAN USE ESTIMATED EFFECT SIZES AS A BASIS FOR MAKING DECISIONS. THE OTHER AREA WHERE I HAVE BEEN WORKING IS COLLABORATING WITH A CHEMICAL ENGINEER THAT SOME OF YOU HERE KNOW AND HAVE HER SPEAKS, DANIEL RIVERACH WE HAVE BEEN WORKING ON THE IDEA OF TRYING TO COLLECT EMA OR INTENSIVE LONGITUDINAL DATA ON PEOPLE. VERY SIMILAR TO WHAT DAVID IS DOING, EXCEPT WE'RE SORT OF LAYERING ON TO IT THE IDEA OF AN N OF ONE EXPERIMENT. IF YOU CAN THEN DEFINE THE WORKING OF THE INTERVENTION AS A DYNAMICCAL SYSTEM YOU CAN APPLY AN ENGINEERING CONTROL TO OPTIMIZE THE INTERVENTION THATCH'S MY ANSWER TO YOUR QUESTION. A WORK IN PROGRESS FOR SURE. >> ANOTHER POINT YOU CAN ALSO THINK OF THE DATA AS A MODEL. THIS IS OFTEN THE CASE. I THINK IN A SETTING IN WHICH WE HAVE SO LITTLE -- WE DON'T HAVE PHYSICAL LAWS. WE DO HAVE SOME BUT THEY'RE VERY -- THEY'RE MOSTLY QUALITATIVE. SO I THINK THE DATA IS A MODEL IS CRUCIAL HERE, THAT CONCEPT. MORE EMPIRICAL MODELS. >> WHO HAS THE MIKE NOW? OKAY. >> I HAVE A QUESTION RELATED TO (INAUDIBLE) WITH COLUMBIA UNIVERSITY. BASICALLY TALKING ABOUT ALWAYS BEING CONTINUOUS DATA AN OPTIMIZATION BEING THE KEY AN REDUCING THE TIME TO GET THE EVIDENCE. SO ONE POSSIBLY MODEL THEY HAVE SEEN IN OTHER AREAS IS CROWD SOURCING. FOR EXAMPLE, GOOGLE JUST RELEASED GOOGLE PLUS, AND HAVE 5 MILLION USERS WITH EXPONENTIAL ADOPTION. THE THING IS HOW COULD WE DO TO GET CROWD SOURCING mHEALTH. THE REASON BEING THAT FOR EXAMPLE I WAS TALKING TO MR. PATEL FROM THE ADA, I WAS LIKE I WANT TO RELEASE SOMETHING, I WANT TO CROWD SOURCE AND SEE WHAT WORKS, WHAT DOESN'T, OPTIMIZE IT AND GET RESULTS FAST. BUT THEN MAYBE THE FDA WANTS TO PRIOR EVIDENCE THIS WORKS. BUT I'M JUST STARTING THE OPTIMIZATION PROCESS. HOW CAN WE DO -- SO WE ALLOW CROWD SOURCING TO REACH MOBILE HEALTH, WE CAPTURE THE ADVANTAGES THAT ALL THESE OTHER AREAS ARE GETTING. >> SO I HAVE BEEN TRYING -- AMAZON.COM HAS THIS THING WHERE YOU CAN HIRE PEOPLE TO RUN YOUR LITTLE TASK. NTERK. SO I WAS -- I ACTUALLY STARTED A PROJECT WHERE WE WERE TRYING TO LOOK AT DECISION-MAKING WITH MENTAL HEALTH. THIS IS -- I DON'T HAVE AS MUCH POSITIVE -- IT TURNS OUT IT'S MORE COME PLIT CAD, HARD FOR REGULAR PEOPLE TO IMAGINE WHAT IT'S LIKE TO HAVE SEVERE DEPRESSION. SO I'M NOT -- I THINK MAYBE FOR SOME SMALLER PROBLEMS MAYBE WE CAN USE MTER WHICH IS MORE A CROWD SOURCE, IT'S FANTASTIC. BUT I'M NOT SURE IF WITH HEALTH IT'S VERY HARD -- WHEN SOMEONE HAS A CHRONIC ILLNESS EVEN IF YOU GET THEM AT ONE POINT WHEN THEY FIRST DEVELOP IT, THEY'RE UNDERSTANDING OF WHAT'S POSSIBLE FOR THEM IS VERY DIFFERENT THAN IT IS A YEAR LATER AND TO THAT ILLNESS. THEY CHANGE THEIR PREFERENCES, THEIR UNDERSTANDING OF WHAT THEY WANT TO ACHIEVE. AND SO IT'S VERY HARD TO USE A MORE CROWD SOURCING UNLESS YOU CAN JUST GET THOSE PEOPLE THAT HAVE THAT DISORDER AT THE SAME POINT IN TIME. BECAUSE YOU HAVE ENORMOUS HETEROGENEITY HOW LONG THEY HAVE HAD THE DISORDER AS WELL. IT AFFECTS THEIR PREFERENCES AS TO WHAT THEY THINK THEY CAN ACHIEVE. >> THE OTHER THING, I THINK YOUR ASKING ABOUT AT THE END OF THE DAY IF YOU WANT TO GO THE FDA YOU'RE GOING TO NEED A RANDOMIZED CONTROL TRIAL. I THINK IT'S WHY IT MAKES SENSE TO THINK ABOUT THE DIFFERENCE BETWEEN OPTIMIZING AND WHAT YOU DO TO GET TO GREAT AND WHAT YOU YOU DO TO ACTUALLY TEST AND HAVE THE EVIDENCE THAT WILL BE CONSIDERED AS EVIDENCE BY OTHER KINDS OF BODIES. I THINK WITH THE FDA THEY'RE GOING TO WANT AN RCT. WITH JOURNALS WE MAY HAVE A LITTLE BIT MORE PLAY IN CHANGING SOME OF THOSE STANDARDS BUT YOU WOULDN'T WANT TO START WITH THE RCT. YOU MIGHT DO CROWD SOURCING FIRST. (OFF MIC) >> FOR MOST MARKETING SURVEILLANCE YOU CAN, FOR GETTING AN INTERVENTION APPROVED YOU PROBABLY WANT TO DO IT FIRST, USE WHAT YOU LEARN, PUT IT IN AN RCT. IT MIGHT HELP YOU DESIGN A BETTER INTERVENTION THAN YOU WOULD IF YOU STARTED WITH AN RCT. >> DO YOU WANT TO RESPOND IN >> CAN I MAKE A COMMENT? WE HAVE THE FDA HERE. NOT THE WHOLE FDA HERE BUT WE HAVE THE FDA SPEAK. >> WE HAVE IMPORTANT PEOPLE FROM THE FDA HERE. I DO WANT TO SAY PLEASE DO NOT SPEAK WITHOUT A MICROPHONE, IT'S RUDE TO OTHER WEBINAR GUESTS SO PLEASE WAIT FOR THE MIC. >> I THINK REALLY GOOD QUESTIONS AN SOME OF THE THINGS -- SORRY. (INAUDIBLE) PATEL FROM THE FDA. YOU PROBABLY HAVE SEEN WHAT WE PUT OUT ON MOBILE APPSCH IT'S HARD MAKE A GENERAL POLICY STATEMENT HERE TALKING SPECIFIC APPLICATION OR SPECIFIC SITUATION. IN GENERAL THERE ARE MECHANISMS THAT FDA HAS TO TO ALLOW FOR INVESTIGATIONAL PRODUCTS TO BE OUT THERE AND DO -- COLLECT INFORMATION AS THEY GET PROVEN AND AS THEY GET -- ALLOW MECHANISMS TO COLLECT INFORMATION TO HAVE THE FINAL PRODUCT WHEN YOU'RE MARKETING IT TO BE AT THAT EXPECTATION OF SAFE EFFECTIVENESS. THAT'S THE BOTTOM LINE HIGH LEVEL PICTURE. THE DISCUSSION ABOUT RCT VERSUS EVIDENCE I THINK MORE LEAN TOWARDS GETTING TO END POINT WHERE EVIDENCE SHOWS EFFICACY OR EFFECTIVENESS AND SAFETY OF WHATEVER SOLUTIONS YOU PROVIDE. THAT'S REALLY MORE IMPORTANT TO FOCUS ON THAT ASPECT AND THEN ACTUALLY COMING ONE A PROCESS AN MECHANISM, THIS DISCUSSION IS INTERESTING BECAUSE YOU'RE FOCUSING ON THE NEW AREAS OR NEW WAYS THAT ONE CAN COME UP WITH EVIDENCE THAT IS SAFE AND EFFECTIVE. THAT'S SOMETHING WE'RE ALSO INTERESTED IN. THIS DISCUSSION IS VERY FRUITFUL FOR THAT. AND I ENCOURAGE CONTINUING TO DO THIS. DICK, I GUESS YOU HAVE THE MIC MIKE NOW, RIGHT? >> -- THE MIC NOW, RIGHT? >> DICK KATZ WITH GW. THE POTENTIAL FOR mHEALTH AND GETTING TO GREAT INVOLVES MULTIPLE COMPONENTS. THOUGH IT SHOULD BE PATIENT-CENTERED FOR IT TO MAXIMIZE ITS IMPACT EFFECTIVENESS WE HAVE TO ENGAGE AND OPTIMIZE, ADAPT TO THE CASE MANAGERS, THE PRIMARY CARE PROVIDERS OR OTHERS IN THAT SYSTEM. SO ADAPTATION, EXPERIMENTAL DESIGN, APPLICATION AND INTERVENTION ALSO NEEDS TO MAKE SURE THAT WE ADAPT TO INDIVIDUALS WHO ARE TRYING TO ASSIST WITH THE HEALTHCARE, TO REALLY MAXIMIZE THE IMPACT. SUCH THAT FOR ME IT'S ABOUT WORK FLOW. IF I EACH NOT ABLE TO LOOK AT THE DATA AND GIVE THE FEEDBACK OR MY CASE MANAGER DOESN'T -- HAS TOO MANY PATIENTS CAN'T LOOK AT IT OR WHATEVER WE'RE GOING TO NOT GET TO THE HIGHEST LEVEL. SO I WONDERED HOW WHETHER IT'S CHOICES OF EQUIPOISE, WHICH SYSTEM I WOULD LIKE TO USE IN MY CLINIC OR ADAPTATION OF OH, WE BETTER CHANGE PROVIDERS START USING IT BETTER. ALSO SOMETHING YOU PROBABLY WANT TO THINK ABOUT IN THE DESIGN. >> WHEN LINDA AND I -- WHEN I -- LINDA WAS FIRST STARTING TO WORK ON THE FACTORIAL DESIGN, ONE OF THE FACTORS WE THOUGHT, ONE THING WE THOUGHT WAS REALLY IMPORTANT WAS NOT ONLY HOW SHOULD SOME OF THE FACTORS COULD BE ASPECTS OF THE CONTEXT, THAT IS, IS IT A COMMUNITY CLINIC VERSUS A SPECIFICITY CLINIC? DOES TRAINING OF THE STAFF ?F IN ORDER IT COULD BE FOR SOME -- THAT THE TRAINING OF THE STAFF CERTAIN TRAINING MAYBE YOU REQUIRE CERTAIN LEVEL OF TRAINING IN ORDER TO EVEN HAVE THIS INTERVENTION. THAT WOULD BE A FACTOR THAT YOU WOULD BE INTERESTED IN AS ONE OF THE COMPONENTS. SO WHEN I SAY FACTOR I MEAN A COMPONENT OF THE INTERVENTION. IF YOU THINK ABOUT CHARACTERISTICS OF THE CONTEXT OR THE STAFF OR THE WORKPLACE, THESE COULD ALL BE COMPONENTS THAT ARE TESTED AS PART OF THE INTERVENTION. >> ABSOLUTELY. SO THERE'S A LOT OF DIFFERENT COMPONENTS, IT CAN BE COMPONENTS WITHIN THE INTERVENTION, COMPONENTS THAT PERTAIN TO IMPLEMENTATION, THEY CAN BE COMPONENTS THAT PERTAIN TO ADHERENCE AS WELL. REALLY, ANYTHING THAT INFLUENCES THE OUTCOME OF AN INTERVENTION CAN BE CONSIDERED A COMPONENT. ALSO IN TERMS OF OPTIMIZATION, IT WOULD BE POSSIBLE TO OPTIMIZE THE INTERVENTION FOR IMPLEMENTATION PURPOSES TOO. FOR EXAMPLE, IF -- WELL, YOU CAN IMAGINE THAT -- LET'S SAY -- I MEAN, I DONE KNOW MUCH ABOUT HEALTHCARE IMPLEMENTATION SO WHAT I'M ABOUT TO SAY IS PROBABLY NAIVE BUT SUPPOSE YOU HAD HEALTH -- WHOEVER IS IMPLEMENTING THE HEALTHCARE RATE HOW DIFFICULT THEY FELT THE INTERVENTION WAS TO IMPLEMENT, YOU COULD TRY TO OPTIMIZE IT BY GETTING THE MOST EFFECTIVE INTERVENTION YOU CAN GET RATED BELOW SOME LEVEL ON THAT SCALE. SO THAT'S JUST OFF THE TOP OF MY HEAD, I'M SURE THERE'S OTHER BETTER WAYS TO INCLUDE THAT AS A CONSIDERATION. BUT IT CAN BE INCLUDED AS FACTOR IN A DESIGN, AND/OR A CONSIDERATION IN DECISION MAKING >> YOUR POINT IS WELL TAKEN BECAUSE WE WAN THESE INTERVENTIONS TO BE USED SO WE HAVE TO ENGAGE WITH THEM. I THINK THIS GETS TRICKY, WHEN WE USE PATIENT PREFERENCE DESIGNS OR END USER PREFERENCE DESIGNS, THERE'S THE IMPLICIT ASSUMPTION THAT PEOPLE KNOW WHAT THEY PREFER. I THINK PREFERENCES ARE OFTEN VERY FLUID WHERE PEOPLE CAN'T KNOW UNTIL THEY HAVE TRIED SOMETHING. I THINK THAT'S ONE OF THE THINGS THAT MAKES IT HARD TO KNOW HOW TO GO ABOUT THAT KIND OF ADAPTATION. I THINK AT THE END OF THE DAY IT'S REASONABLE TO ASK THEM BUT IT'S ALSO REASONABLE TO HAVE HYPOTHESES AND TEST THINGS IN A FRACTIONAL FACTORIAL FOR EXAMPLE, TO SEE WHAT ACTUALLY WORKS. >> LET ME FOLLOW UP ON THAT. OPT PATIENT PREFERENCE ISSUE, GOOD POINT, INTERESTING TO HEAR HOW YOU MANAGE THIS. BECAUSE PATIENT PREFERENCE DOESN'T ALWAYS ALIGN WITH WHAT'S MOST EFFECTIVE, YOUR THEORY HOW YOU THINK BEHAVIOR CHANGE OCCURRED, SO YOU GET PEOPLE WHO ARE SAYING THIS IS WHAT I LIKE, VERSUS YOU KNOWING THIS IS WHAT I THINK IS PROBABLY MOST EFFECTIVE, HOW DO YOU DEAL WITH THAT CONFLICT? >> I HAVE KIND OF TWO ANSWERS TO THIS. ONE IS WHAT I DO IN MY RESEARCH, THE OTHER IS WHAT I DO CLINICALLY. SO IN MY RESEARCH WHEN WE STUDY THAT QUESTION, THIS IS PART OF THE REASON I'M NOT SO VERY HIGH ON PREFERENCES BECAUSE BASICALLY WHAT WE FIND IS IN OUR HANDS THINGS THAT PEOPLE PREFER AREN'T NECESSARILY THE MOST EFFECTIVE. IF WE ASK THEM AFTER THE FACT WHAT THEY PREFER, OFTEN THEIR PREFERENCES HAVE CHANGED IN LINE WITH WHATEVER IT WAS THEY GOT. WHEN I'M THINKING ABOUT THIS CLINICALLY IN TERMS OF EVIDENCE-BASED PRACTICE, I THINK IT'S WHY SOME OF THE MODELS OF EVIDENCE-BASED PRACTICE SAY THE FIRST THING YOU NEED TO DO IS FIGURE OUT THE OPTIMUM INTERVENTION AN IMPLEMENT THAT. START WITH THE BEST EVIDENCE-BASED APPROACH BUT THEN YOU HAVE TO ADAPT. YOU HAVE TO ANALYZE AND ADJUST. I THINK OFTEN WHAT YOU'RE DOING WHEN YOU GIVE SOMEBODY AN INTERVENTION, YOU'RE ACCOMMODATING THEIR PREFERENCE AND YOU MAY WIND UP GIVING THEM SOMETHING THEY PREFER THAT YOU DON'T THINK IS THE BEST EVIDENCE BASED INTERVENTION. WHAT YOU'RE BANKING ON IS YOUR PRESERVING THE RELATIONSHIP. OFTEN IT'S BEEN A CONTEXT TO HAVE THEM EXPERIENCE WHAT YOU'RE GIVING THEM AND BE WILLING TO TAKE IT ON YOUR RECOMMENDATION THAT THEY'LL GET TO THE EVIDENCE BASED ONE NEXT. THAT'S VERY MUCH NOT AN EXPERIMENTAL DESIGN. >> CAN I JUST FOLLOW-UP ON THAT? BONNIE SAID SOMETHING WHEN SHE WAS GIVING HER TALK. IT'S A POINT YOU LEARN IN INTERVENTION SCIENCE 101 BUT WORTH REPEATING. IN AN RRT THE INTERVENTION HAS TO BE THE SAME AT THE END OF THE TRIAL AS IT WAS AT THE BEGINNING. I THINK THERE IS A LOT OF CONFUSION ABOUT ADAPTIVE INTERVENTIONS AND HOW THIS DOVE TAILS WITH RCT. I HOPE IT'S CLEAR THAT WITH ADAPTIVE INTERVENTION, YES, YOU ULTIMATELY WOULD EVALUATE ADAPTIVE INTERVENTION USING AN RCT. WHERE THE TREATMENT GROUP IS -- WE'RE IN THE TREATMENT GROUP, THE INTERVENTION IS THE SET OF DECISION RULES AN TAILORING VARIABLES DECIDED A PRIORI AND THAT SET OF TAILORING VARIABLES AND DESIETION YOU -- DECISION RULES DOES DOESN'T CHANGE ALONG THE CURSE OF THE RCT. PEOPLE I HAVE BEEN NOTICING IN MY TRAVEL SEEM THE THINK IF YOU SAY YOU HAVE AN ADAPTIVE +"VERVENTION IT'S A LICENSE TO] AN RCT. I WAS ON A CONFERENCE CALL -- I WON'T GO INTO THAT. ANYWAY, IT'S JUST -- I THINK IT'S IMPORTANT TO DRAW THAT DISTINCTION, ADAPTIVE INTERVENTION IS AN INTERVENTION. IT HAS A SET OF PRINCIPLES. THAT SET OF PRINCIPLES HAS TO BE ADHERED TO THROUGHOUT THE ENTIRE RCT, IF YOU EVALUATE USING AN RCT. >> THIS IS A TERRIFIC SESSION, IT'S SO INTERESTING TO HEAR ALL OF THIS DISCUSSED. WHAT I'M WONDERING, HOW ARE WE GOING TO TRAIN THE NEXT GENERATION OF FOLKS WHO UNDERSTAND THIS HYBRID WAY OF THINKING? DO WE RECRUIT OUT OF ENGINEERING SCHOOLS AND BRING THEM TO OUR HEALTH SYSTEMS? DO WE RECRUIT OTHER HEALTH SCHOOLS AND HAVE THEM RECRUIT WITH ENGINEER? MAYBE I'LL PUT MISHA ON THE SPOT ON THIS ONE AS WELL. THIS IS EXTRAORDINARILY IMPORTANT FOR US TO THINK ABOUT HOW TO DO H. THIS CRISIS WE'RE SEEING IN THE HEALTHCARE SYSTEM AS FAR AS CONTINUING TO DO WHAT WE HAVE DONE, SPENDING MORE AND MORE HAS TO BE SOLVED BY SOMETHING DISRUPTIVE. I DON'T HAVE AN ANSWER TO THIS BUT WHAT ARE YOUR INSIGHTS AS FAR AS WHETHER WHERE WE FIND THE BEST EDUCATIONAL PATHWAYS AND WHAT DO YOU THINK THE NEXT STEPS THOSE OF US WHO ARE -- INFLUENCE THIS SHOULD TAKE? >> I THINK AT LEAST FROM MY PERSPECTIVE WHERE -- I MEAN, I'M NOT AN ENGINEER, THAT'S SAYING THE OBVIOUS BUT I'M TRYING TO INCORPORATE IDEAS FROM ENGINEERING AND INTO THE BEHAVIORAL SCIENCES. I HAVE BEEN COLLABORATING WITH ENGINEERS. IT'S A PROBLEM BECAUSE THERE ARE STUDENTS WHO ARE INTERESTED IN THIS AND THERE AREN'T REALLY TRAINING PROGRAMS. I WOULD LOVE TO SEE JUST AS THERE ARE INDUSTRIAL ENGINEERING TRAINING PROGRAMS, I WOULD LOVE TO SEE THE ENGINEERING BEHAVIORAL INTERVENTIONS BE A SUBSPECIALTY WITHIN ENGINEERING AND THERE COULD BE TRAINING PROGRAMS IN THAT AREA. I WISH THAT WOULD HAPPEN, IT DOESN'T HAVE TO HAPPEN. >> ONE THING I HAVE TRIED TO DO AS A RESEARCHER IS I HAVE MADE AN ENORMOUS EFFORT. I WAS FOR GNAT NIH GAVE ME A KO-2 TO LEARN ABOUT COMPUTER SCIENCE AN ENGINEERING. SO NIH ACTUALLY FACILITATES THIS BY KO-2 AWARDS OR KO-1 AWARDS. THIS IS ONE WAY. K-24, K-23 I FORGET THE OTHER CLINICIANS. >> DANIEL RIVERA HAS A K-25 A CHEMICAL ENGINEER AS I SAID BEFORE. HE'S SORT OF RETOOLING TO ALSO HAVE A LINE OF RESEARCH IN THE BEHAVIORAL SCIENCES THAT'S ONE. I HAVE RECRUITED AND BECAUSE OF THAT NOW MY POST-DOC, BOTH MY POST-DOCS ARE COMPUTER SCIENTISTS. YOU CAN DO THAT. I HAVE WRITTEN GRANTS WITH COMPUTER SCIENTISTS WHO WORK IN ARTIFICIAL INTELLIGENCE SO THEY DON'T KNOW -- THEY'RE NOT AS SOPHISTICATED AS SCIENTISTS WE HAVE HERE IN TERMS OF HEALTH BEHAVIOR THAT'S WAY TO GET IN BECAUSE I LEARNED ENOUGH OF THEIR LANGUAGE SO THAT I COULD COMMUNICATE BETWEEN THEM AND THE CLINICIANS. SO I THINK THESE ARE ALL NIH IS REALLY DOING A LOT. THEY'RE TRYING HARD I THINK TO FACILITATE >> CURRENTLY THROUGHOUT NIH MANY NIH FOLKS HERE ARE INVOLVED WITH IT BUT WE DO ACTUALLY BRING IN A QUARTER OF CLASSROOM ENGINEERS, A QUARTER OF COMPUTER SCIENTISTS AN BEHAVIORAL AND AUTOMATIC NURSING DOCTOR MD NURSE AND THEY ACTUALLY HAVE TO WORK TOGETHER ON PROJECTS. IT'S AN AMAZING PROCESS, WE HAD A FIRST ONE IN JUNE. IT TOOK A WHILE FOR THE LANGUAGE TO DEVELOP. THERE ARE MULTIPLE WAYS TO DO IT, THEY ENJOY IT IN A WAY THAT WAS JUST -- I MEAN, WHAT I HAVE BEEN STRUCK WITH, WE HAVE A MEETING IN JUNE, THEY SPENT A WEEK TOGETHER. WE HAVE CORE GRANT -- FOUR GRANT PROPOSALS COMING TO NIH FROM THESE SWRIEWN YOUR INVESTIGATOR TEAMS. WHICH IS KIND OF AND I MAZING FOR RAY. SORT OF RANDOM ASSIGNMENT TO A TEAM. THEY SO ENJOY THIS. THIS IS A BUDDING INDUSTRY AREA. WE HAVE HEALTH TECHNOLOGY ENGINEERING IS THAT CORRECT? >> IT IS A PROGRAM PAIRING MEDICAL STUDENTS WHO SIGN UP FOR IT WITH ENGINEERS. SO THEY DO PART OF THEIR TRAINING I THINK TAXES ON AN EXTRA YEAR THEY DEVELOP SOLUTIONS FOR PARTICULAR HEALTH PROBLEMS. SO IT'S A BUDDING PROGRAM, TAUGHT BY PREPEOPLE IN PREVENTIVE MEDICINE AND PEOPLE IN SOCIOLOGY AND PEOPLE IN ENGINEERINGR– AND WE HAVE A LONG WAY TO GO AS WE DEVELOP IT. JUST STARTING ITS FIRST YEAR BUT IT'S VERY EXCITING. THERE IS A START. >> I'M DO NEE (INAUDIBLE) FROM THE UNIVERSITY OF SOUTHERN CALIFORNIA. >> CARNEY ALSO HAD A BUDDING PROGRAM. >> THIS IS MISHA PAVEL SINCE I WAS CHALLENGED. NSF, ONE OF THE OBJECTIVES, ONE OF THE BROAD IMPACTS WHICH WE EVER WEIGHT GRANTS, GRANT PROPOSALS, IS TEACHING EDUCATION. NOW, THE WELL BEING PROGRAM IS PARTICULARLY FOCUSED ON THIS MULTI-DISCIPLINARY -- WE CALL IT TRANSDISCIPLINARY NOW. EDUCATION AND RESEARCH. AND WHEN I GIVE A TALK ABOUT THIS, I EMPHASIZE THE SEE ISSUE IN MAKING THIS SUCCESSFUL IS BRINGING PEOPLE TOGETHER. IT'S NOT ENOUGH. IT'S NOT ENOUGH THAT ENGINEERS AND CLINICIANS TALK THE SAME LANGUAGE. THEY ACTUALLY HAVE TO UNDERSTAND EACH OTHER'S CULTURES. AND THAT IS THE KEY AND THE MOST DIFFICULT PART OF THE -- OF ACHIEVING THIS KIND OF MULTI-DISCIPLINARY AND TRANSDISCIPLINARY WORK. IT USUALLY TAKES MANY MEETINGS BEFORE PEOPLE ACTUALLY APPRECIATE WHAT THE TERMINOLOGY MEANS IN THE DIFFERENT DISCIPLINES. WE DO THAT BY SORT OF ENCOURAGING, MAYBE MORE THAN ENCOURAGING, PEOPLE TO REALLY HAVE DEVELOP VERY DEEP PLANS OF HOW THIS IS GOING TO BE EXECUTED AND WE APPRECIATE Y'ALL ON PIs INNOVATIVE APPROACHES TO THIS. IT'S NOT AN EASY TASK, IT'S A VERY DIFFICULT TASK. AND BUT WE CANNOT MAKE PROGRESS WITHOUT IT. I'M GLAD WE BROUGHT THIS UP. >> GARRETT (INAUDIBLE) WITH WHO. I WANT TO COME BACK TO TRAINING, JUST TO AGAIN KIND OF LOOK AT DISRUPTIVE APPROACHES THAT ARE OUT THERE ALREADY. AND JUST AS AN EXAMPLE ONLINE VIDEOS THAT PEOPLE HAVE MADE AVAILABLE IN A RANGE OF DIFFERENT TOPICS AND I'M THINKING OF SOLOMON, CAN'S ACADEMY, MANY OF YOU MAY HAVE SEEN RECENTLY ON TED, BUT I'M THINKING WE MIGHT HAVE A PARTICULAR ISSUE THAT I NEED TO UNDERSTAND, WELL, WE'LL LOOK AT THE LITERATURE BUT ALSO ONLINE VIDEOS WHO PROVIDE SUCCINCT, CONCISE, AND I'M LOOKING AT THE OPEN COURSE WARE APPROACH THAT MIT AND OTHER UNIVERSITIES HAVE USED TO MAKE AVAILABLE ONLINE LECTURES. IT'S AN AMAZING RESOURCE. AMAZING. THERE MAYBE AN OPPORTUNITY TO LOOK AT SOME OF THE UNIVERSITIES THAT HAVE mHEALTH TRAINING COURSE, NORWAY, THINKING OF MIT, COLUMBIA, OTHERS THAT ARE MAKING AVAILABLE COURSES. THERE ARE OTHERS. I HAD HEARD ABOUT THE NIH TRAINING COURSE BUT I CAN REMEMBER WHY I DIDN'T FIT INTO IT FOR SOME REASON. I WAS EXCITED AT POSSIBILITY OF IT BUT AGAIN, I DIDN'T KNOW WHETHER IN FACT IT WAS AN OPEN SYSTEM THAT I CAN ATTEND. AT A LATER POINT LOOKING AT LECTURES BUT IT PROBABLY, IT WILL, THAT'S GREAT. WE NEED TO LOOK AT THAT, ESPECIALLY AT WHO AND I'M IN A PART OF WHO THAT'S ABOUT TRAINING, RESEARCHERS AND DEVELOPING COUNTRIES AND LOW RESOURCE SETTINGS, WE CAN'T FLY EVERYBODY EVERYWHERE. WE NEED ACCESS TO TRAINING TOOLS THAT WE CAN MAKE AVAILABLE TO THEM IN THEIR OWN SETTING AND OWN RESOURCE CONSTRAINED ENVIRONMENTS, AT LECTURES, TRAINING TOOLS AND CURRICULUM THAT'S SHARED WITH THEM ARE REALLY VALUABLE FOR US. AND REALLY VALUABLE IN LOW RESOURCE SETTINGS. SO I WOULD ENCOURAGE PEOPLE TO THINK ABOUT HOW WE MAKE IT THOSE KINDS OF THINGS AVAILABLE AND WE WOULD BE HAPPY TO HELP FACILITATE THAT TRANSFER TO GET TO SOME OF THOSE PEOPLE IN LOW INCOME SETTINGS, LOW -- DEVELOPING COUNTRIES WHERE A LOT OF INNOVATIONS ARE TAKING PLACE. BUT THERE ISN'T ACCESS TO THE SORT OF MANY EXPERT DOMAIN SYSTEMS ABOUT RESEARCH METHODOLOGIES FOR EXAMPLE. >> QUESTIONS, COMMENTS? >> QUESTION ON THE ISSUE OF TRAINING, SEEMED LIKE PART OF WHAT THE -- YOU HAVE DESCRIBED AND WITH THIS -- THE mHEALTH TRAINING INSTITUTE IS ADDRESSING THE ISSUE OF THOSE OF US WHO ARE ALREADY IN THE FIELD TO UPDATE OURSELVES. BUT IT ALSO SEEMS LIKE WE REALLY ARE AT THE POINT WHERE THERE'S A NEW GENERATION COMING AND WE HAVE THE OPPORTUNITY TO MAKE THEM FAR BETTER QUALIFIED RESEARCHERS THAN ANY OF US IN THIS ROOM CK BECAUSE BECAUSE WE CAN START IN THE BEGINNING. THIS IS NOT UNIQUE, THIS HAS HAPPENED, CERTAINLY WITH PSYCHONEUROLOGY, PEOPLE REALIZE THEY NEED TO SET UP TRAINING PROGRAMS FOR STUDENTS THAT WOULD INTEGRATE IMMUNOLOGY AND ENDOCRINOLOGY AND BEHAVIORAL SCIENCES. BUT I GUESS I WONDER, IS THIS SOMETHING AS WE BEGIN THINKING ABOUT THE NEXT GENERATION, IS THIS SOMETHING WHERE WE SHOULD PUT SOME SORT OF CONCERTED EFFORT INTO THINKING ABOUT WHAT THESE TRAINING PROGRAMS SHOULD BE? DO WE JUST LET EVERYBODY AT DIFFERENT INSTITUTIONS DO WHAT THEY WANT TO DO AND SEE WHAT BUBBLES UP AND SEEMS TO WORK? I'M WONDERING DO PEOPLE HAVE THOUGHTS ABOUT -- BECAUSE I THINK WE'RE AT A POINT WHERE THIS IS -- I MEAN IT SOUNDS LIKE DIFFERENT INSTITUTIONS ARE STARTING TO BUILD THESE PROGRAMS AND I KNOW THERE'S A LOT OF OTHER PEOPLE WHO ARE STARTING TO THINK ABOUT IT. HOW DO WE MAKE SURE THE NEXT GENERATION BENEFITS AS MUCH AS POSSIBLE FROM THE COLLECTIVE BUDDY VERSE WISDOM OF PEOPLE WHO COME BEFORE. >> THERE'S PUBLIC HEALTH BIOINFORMATICS. (INAUDIBLE). (OFF MIC) >> MISHA PAVEL, NATIONAL SCIENCE FOUNDATIONCH I WOULD LIKE TO CHANGE GEAR AS LITTLE BIT. WHAT WE HEARD FROM BOTH PANELS THIS MORNING IS A DESCRIPTION OF VARIOUS APPROACHES THAT IMPROVE EFFICIENCY OF PANELS BUT IMPLICITLY I THINK WE ARE STILL FOCUSING ON EVALUATION USING STATISTICAL SIGNIFICANCE. I WAS WONDERING WHETHER THAT IS THE KIND OF GOAL THAT WE WERE -- WE ARE AIMING FOR IN THE FUTURE. FROM MY PERSPECTIVE THE IMPORTANT ASPECT OF BEING ABLE TO CHANGE THE FUTURE HEALTHCARE IS BY DECISIONS THAT WE BASE ON THE RESULTS OF EVIDENCE OF TRIALS AND THAT MEANS WE HAVE TO INCORPORATE UTILITY, VOLUME AND EXPECTED VALUE OF SOME OTHER OBJECTIVE MEASURES. UTILITY OF THESE INTERVENTIONS. I WONDER IF YOU WOULD CARE TO COMMENT. >> ONE OF THE REASONS WHY MANY OF PEOPLE INTERESTED IN H THAT SMART TRIAL IS BECAUSE OF THE COST ISSUES. SO IT'S EXACTLY WHAT LINDA WAS SAYING. IF YOU OPTIMIZE AN INTERVENTION UP TO A CERTAIN COST. SO THAT IS A UTILITY, IT MAY NOT BE THE MORE THEORETICAL UTILITY BUT HAVING A OPTIMIZATION PROBLEM PROVIDES -- LIKE HAVING A UTILITY FUNCTION. >> ONE OF THE MAIN ISSUES THE UTILITY FOR THE PATIENT. WE'RE TALKING PATIENT-CENTRICCH WE HAVE TO INCORPORATE EWE UTILITIES OF THE PATIENT. THE VALUES THAT THE PATIENT BRINGS WILL ADJUST THE COST. THAT WAS -- I WAS WONDERING IF BOB WANTED TO COMMENT ON THIS AS WELL. >> THERE'S OFTEN OUTCOMES ARE FUNCTIONAL OUTCOMES SO THEY'RE PATIENT-CENTERED OUTCOMES. IN MENTAL HEALTH OF COURSE THEY'RE PATIENT CENTERED OUTCOMES, OFTEN THEY ARE. BUT CAN I HOLD A JOB? THAT'S AN OUTCOME. CAN I GET OUT OF BED? CAN I TAKE CARE OF MY CHILD. (OFF MIC) >> IF YOU EVALUATE THE RESULT OF THE TRIAL, WHETHER THE RESULTS ARE STATISTICALLY SIGNIFICANT THAT IS NOT SUFFICIENT. WHAT WE WANT TO MAKE SURE, THE -- IT'S NOT JUST THAT ONE GROUP GETS THE mHEALTH INTERVENTION IS STATISTICALLY SIGNIFICANTLY EASIER TO EMPLOY, WHAT HAVE YOU. BUT THAT THE CHANGE IS LARGE ENOUGH TO MAY RECOLLECT A DIFFERENCE FOR SOCIETY. >> MY PRESENTATION I TRIED TO DRAW A DISTINCTION BETWEEN EVALUATION AND OPTIMIZATION. AND IT'S CERTAINLY TRUE THAT AN INTERVENTION CAN HAVE A STATISTICALLY SIGNIFICANT EFFECT THAT'S NOT -- HASN'T BEEN OPTIMIZED AND MAY NOT BE LARGE ENOUGH TO BE MEANINGFUL. I ALSO WANT TO COMMENT THAT -- WELL, I WANT TO ASK YOU I GUESS, ARE YOU -- IS YOUR COMMENT THAT WE SHOULDN'T USE HYPOTHESIS TESTING AT ALL OR THAT WE SHOULDN'T NECESSARILY BE CHAINED TO P OF .05 BECAUSE THE LATTER DRIVES ME CRAZY, THAT PEOPLE ACT LIKE THAT'S THE 11th COMMANDMENT. IT ISN'T, SUSAN. [LAUGHTER] >> WHEN YOU THINK OF IT AS A DECISION THEN UNDER RISK, THEN YOU CAN TAKE WHAT SEEMS TO ME TO BE A RATIONAL APPROACH TO WEIGHING TYPE 1 VERSUS TYPE 2 ERRORS. >> I AGREE THE STATISTICAL INFERENCE IS STILL IMPORTANT. IF I'M AT KEISER AND SHOULD USE mHEALTH AS A COMPONENT IN MY CARE FOR THE POPULATION, I WOULD LIKE TO KNOW WHAT IMPACT OF THAT VALUE IS, RATHER THAN WHETHER IT'S STATISTICALLY SIGNIFICANT AT PO .05. >> BUT SEEMS LIKE THE OTHER THING YOU MAYBE TALKING ABOUT IS PRE-SPECIFYING A LEVEL OF CLAM SIGNIFICANCE -- CLINICAL SIGNIFICANCE. BUT I THINK THIS MAYBE AN AREA WE NEED TO THINK THROUGH BECAUSE IF WE'RE DEALING WITH INDIVIDUAL CASES LOOKING FOR A STANDARD METRIC OF CLINICAL SIGNIFICANCE MAKES SENSE. IF WE'RE THINKING IN TERMS OF POPULATION HEALTH AND WE WANT SMALL CHANGES IN LARGE NUMBERS OF PEOPLE, THAT MAYBE A QUITE DIFFERENT CRITERION. TO THE EXTENT THAT IT'S GOING TO WIND UP BEING CONSUMERS WHO ADMINISTER THESE mHEALTH THINGS TO THEMSELVES, THAT MIGHT BE A MORE APPROPRIATE METRIC. >> I WANT TO QUICKLY GO BACK, SORRY ABOUT THAT. DAVID MOORE HAD ASKED HOW CAN YOU HELP THE NEXT GENERATION. THE PREVIOUS DISCUSSION ABOUT TRAINING. I THINK I'M PART OF THAT. I JUST GOT MY DEGREE IN 2008. I WOULD FEEL REMISNOT TO MENTION BEYOND TRAINING WE NEED CAREER OPTIONS, WE NEED TO GET FUNDED TO DO THIS WORK T. PARADOX WE'RE IN IS A LOT OF THINGS WE WOULD PROPOSE USING IN AN RO-1 OR WHATEVER IT IS BY NATURE HAVE TO CONCLUDE THESE ITERATIVE DEVELOPMENT PHASES AND REVIEW COMMITTEES ARE UNCOMFORTABLE WITH THAT SORT OF THING. MY HOPE IS MAYBE SOME OF THE ELDER FOLKS IN THIS ROOM CAN HELP AND OBVIOUSLY ANY OTHER PROCESS ANNIE OTHER AREA MENTIONED, I'M GET OG THE POINT WHERE NEXT GENERATION CAN GO INTO AND NOT GET A CAREER IN THIS AREA BY VIRTUE OF GETTING FUNDED. >> I'M SURPRISED BILL RILEY ISN'T SAYING THE NHLBI HAS A SPECIAL RO-1 FOR INTERVENTION DEVELOPMENT WHICH YOU CAN'T GET FUNDED UNLESS YOU PROPOSE ITERATION UP FRONT. >> THANK YOU, BONNIE. [LAUGHTER] >> WITH THE RO-1s I WANT TO ADD TRAINING GRANTS, MORE FOLKS COMING THROUGH THE PIPELINE, I THINK MANY OF US WHEN YOU THINK ABOUT WHAT PATH YOU WHEN ON FOR YOUR RESEARCH CAREER, A LOT OF WHAT YOU'RE EXPOSED TO, SO YOU HAVE TRAINING PROGRAMS THAT ALLOW STUDENTS TO BE EXPOSED TO THE DEVELOPMENT OF TECHNOLOGY WILL HELP A LOT. >> THE OTHER THING I'LL MENTION IN ADDITION TO NHLBI PIECE OF THIS, THE COMMON FUND INITIATIVE IN THE mHEALTH PROCESS DOES FOCUS A GREAT DEAL ON THAT GAP WE HAVE IN TERMS OF AN ITERATIVE INTERVENTION DEVELOPMENT BEFORE WE GET TO THE EVALUATION OF IT AND TRYING TO BOLSTER THAT. IT'S ANOTHER THING TO GO INTO THAT SITE AND SAY YOUR PIECE ABOUT BUT IT'S A GOOD POINT. EVEN BEYOND THAT THERE'S MORE WE NEED TO DO BECAUSE YOU'RE RIGHT. THE TYPICAL SITUATION WE SEE COMING THROUGH REVIEW I THINK IS THAT PEOPLE LOOK AT THAT AND GO YOU SHOULD ALREADY HAVE YOUR INTERVENTION DEFINED, WHY IS IT NOT DEFINED YET? AND GETTING REVIEWERS TO UNDERSTAND THAT THERE'S SOME DEGREE OF INTERVENTION DEVELOPMENT THAT NEEDS TO HAPPEN. THERE'S SOME LEAP OF FAITH THEY HAVE TO TAKE AND FUNDERS NEED TO TAKE IN ORDER TO GO I DON'T KNOW WHERE YOU'RE GOING TO END UP AT THE END OF THE DEVELOPMENT PROCESS. WE HAVE TO TAKE SOME LEAP OF FAITH THAT'S GOOD ENOUGH YOU CAN EVALUATE IT AND WE'LL HAVE DECENT DATA FROM IT. (OFF MIC) >> WE HAVE TIME FOR ONE MORE QUESTION. (OFF MIC) (OFF MIC) >> IT'S LIKE THIS CLOUD OVER YOU. WHEN IT COMES TO MICROPHONES. WE'RE GOING THE GET THIS RIGHT. >> IS THAT BETTER? GARRETT MEHL, WHO. MAYBE A DEEPER VOICE WILL HELP. MY QUESTION WAS, WHEN WE THINK ABOUT THE KINDS OF EVIDENCE THAT WE WOULD LIKE TO SEE COMING UP THE PIPELINE, WE THINK OF DIFFERENCE KINDS OF STAKEHOLDERS THAT WOULD BE WANTING THAT EVIDENCE. OF COURSE AND DIFFERENT KINDS OF STAKEHOLDERS INVOLVED IN PERHAPS INFLUENCING THE KINDS OF EVIDENCE GENERATED. I'M WONDERING IF IF FOR EXAMPLE ADMINISTRATIVE HEALTH. THEY WANT TO KNOW WHAT ARE THE COSTS, WHAT ARE THE -- SO THOSE ARE EVALUATION QUESTIONS BUT THEY'RE ALSO RESEARCH QUESTIONS COMPARING THIS SYSTEM TO ANOTHER SYSTEM. COMPARED TO THE STANDARD OF CARE RIGHT NOW. AND I'M WONDERING NUMBER ONE, IF THE KINDS OF THINGS LIKE COMPARATIVE EFFECTIVENESS RESEARCH APPROACHES ARE ONES THAT PEOPLE WHERE SOME DIFFERENT STAKEHOLDERS ARE INVOLVED IN THAT PROCESS OF FORMULATING EARLY ON WHAT KINDS OF OUTCOMES AND STUDY DESIGNS WOULD BE APPROPRIATE. WONDERING IF ANY OF YOU HAVE BEEN INVOLVED IN THAT AND WHETHER THAT PLAYS INTO ANY OF THIS STUDY DESIGN QUESTIONS THAT YOU'RE ALL HACKLING. IT'S AN OPTIMIZATION QUESTION PARTLY. BUT IT'S ALSO A -- IT'S BIGGER THAN THAT, I THINK. >> YEAH. I'M INVOLVED WITH SOME PEOPLE AT THE UNIVERSITY OF WISCONSIN AND WE'RE WORKING ON DEVELOPING AN OPTIMIZED CLINIC BASED SMOKING CESSATION INTERVENTION. WE'RE -- AND AS PART OF THIS, I'M DEVELOPING SOME METHODOLOGICAL IDEAS. WIIVE STRUGGLING WITH WHAT THE OPTIMIZATION CRITERION SHOULD BE AND HOW TO HANDLE THAT. WHERE WE MAY END UP IS IS PUBLISHING THE -- ENOUGH INFORMATION THAT UP TO A POINT YOU COULD SELECT THE COMPONENTS YOU WANTED TO OPTIMIZE THE CRITERION YOU WANT. IT COULD BE IN BUN CLINIC SETTING TIME IS REALLY IMPORTANT. SO YOU WANT TO SELECT THE COMPONENTS THAT GAVE YOU THE MOST EFFECTIVE INTERVENTION YOU COULD GET THAT COULD BE DELIVERED UNDER SOME AMOUNT OF TIME. SO I THINK IT IS POSSIBLE WE CAN MOVE TOWARD COLLECTING EVIDENCE THAT WOULD ENABLE PEOPLE TO IDENTIFY THEIR OWN OPTIMIZATION CRITERION AND POSSIBLY SELECT AN INTERVENTION THAT WOULD WORK FOR THEM. >> GIVEN TIME, LINDA, CAN YOU SUMMARIZE A BIT? >> I FEEL LIKE WE HAVE TALKED ABOUT SO MUCH HERE AND SO MUCH DURING THE DISCUSSION THAT I'M NOT SURE HOW TO SUMMARIZE IT. OBVIOUSLY I HAVE MY OWN BIASES ABOUT THIS. BUT ONE THING THAT CAME OUT IN THE DISCUSSION THAT I THOUGHT WAS REALLY INTERESTING IS THE TRAINING ISSUE. I'M ENGAGED IN TRAINING MYSELF AND I THINK THAT'S REALLY IMPORTANT ONE, SOMETHING THAT I HOPE WE PURSUE. SORT OF LIKE TO TURN IT OVER TO THE OTHER PANELISTS TO GIVE THEM A CHANCE TO SUM UP ALSO. >> I THINK I RAISE THE QUESTION, I THINK THE EDUCATION ISSUES ARE SOMETHING THAT I HAVE BEEN THINKING ABOUT AND I ALSO ANYBODY HERE WHO HAS ANY INFORMATION ABOUT PROGRAMS BEING DEVELOPED I WOULD BE VERY INTERESTED TO TALK WITH YOU AFTERWARDS. I THINK I GUESS ONE THING I TAKE AWAY IS THERE'S STILL SEEMS TO BE I'M NOT ENTIRELY CLEAR AT THIS POINT WITH THESE TECHNOLOGIES HOW MUCH OF THE EVALUATION -- I MEAN, MAYBE FOR THE FDA THERE NEEDS TO BE A CLEAR END POINT WHEN SOMETHING IS APPROVED BUT IN PRACTICE, HOW MUCH OF A DIFFERENCE -- DISTINCTION THERE IS ONCE YOU HAVE AN EVALUATION? SEEMS TO ME THAT WITH THESE TECHNOLOGIES THAT ARE CONTINUING TO EVOLVE WE REALLY HAVE AN OPPORTUNITY TO THINK ABOUT HOW THESE SYSTEMS OF DEPLOYMENT CAN ALLOW US TO MONITOR THESE AND CONTINUE TO LEARN FROM THESE IN ONGOING BASIS IN WAYS THAT CAN BOTH INCLUDE OPTIMIZATION AS WELL AS MONITORING FOR EFFECTIVENESS AND SAFETY, THE KINDS OF THINGS WE TRADITIONALLY NEED TO DO TO PROTECT THE PATIENTS AND THE STAKEHOLDERS IN OUR CARE SYSTEM. >> THE MAIN THING I GOT WAS BACK WITH THE TRAINING ISSUE. IT IS IMPORTANT FOR FUNDING AGENCIES TO HAVE A LOT OF PATIENCE WHEN YOU FUND INTERDISCIPLINARY GROUPS AND THEY'RE TRYING TO LEARN EACH OTHER'S CULTURES. IT TAKES YEARS TO GET THAT STRAIGHT. IT'S ONLY CLOSE OTHER THE END OF THE FIVE YEAR GRANT THAT YOU ENUP BEING AS PRODUCTIVE IN THE SENSE NOT INDIVIDUALLY YOU CAN BE PRODUCTIVE. I'M TALK JOINTLY PRODUCTIVE. IT'S AT END OF THE FIVE YEAR GRANT YOU'RE JOINTLY PRODUCTIVE. WE NEED TO REMEMBER THAT IN TERMS OF FUNDING THESE PROJECTS THAT YOU HAVE TO TAKE THESE CHANCES. IN OTHER WORDS, I'M MAKING A PLEA FOR TIME THOUGH WE HAVE BEEN SAYING THAT WE HAVE TO DO THINGS FAST HERE. >> I'M TAKING AWAY A COUPLE OF THINGS BEING ABLE TO OPTIMIZE TO DIFFERENCE CRITERIA. THAT WOULD BE A HUGE ASSET TO KNOW HOW TO DO THAT. IT'S NOT ONLY TRAINING THE YOUNGER FOLK, THE TRAINING ALL OF US WHO ARE OUT IN THE FIELD ABOUT HOW TO DO TEAM SCIENCE. IF WE WANT mHEALTH TO MOVE QUICKLY WE NEED TO LEARN HOW TO COMMUNICATE WITH THE DESIGN FOLKS, WITH THE SOFTWARE ENGINEER FOLKS WITH FOLKS WHO ARE -- AND WITH THE FOLKS IN THE IMPLEMENTATION END OF THINGS IN THE SYSTEM AND IT'S THE POINT THAT MISHA WAS MAKING THAT WE BRING NOT ONLY DIFFERENT METHODOLOGIES BUT VERY DIFFERENT WORLD VIEWS AND CULTURAL BIASES AND I'LL PUT IN A PLUG FOR SOMETHING WE HAD ONLINE, WE DEVELOPED WITH ARRA FUNDS THROUGH SUPPLEMENT TO THE CTSA AND ALSO WITH SOME FUNDING FROM OBSSR. THERE IS A WEBSITE CALLED TEAM SCIENCE.NET THAT HAS INTERACTIVE ONLINE TOOLS THAT LET PEOPLE WALK THROUGH THE EXPERIENCE OF DESIGNING AN INTERDISCIPLINARY PROJECT. AS JUNIOR INVESTIGATOR SENIOR INVESTIGATOR OR RESEARCH DEVELOPMENT OFFICER. THERE ARE INTERVIEWS WITH PEOPLE DOING TEAM SCIENCECH THOSE OUT AND ABOUT NOW ALSO NEED SOME TRAINING IN THIS AREA THAT WE SURE DIDN'T GET WHEN WE WERE IN GRADUATE SCHOOL. I THINK EVEN ARTICULATED THE PROBLEM YET. I GUESS THE LAST THING, I'LL LOOK FOR THE PANEL OVER THE BREAK, I'M INTERESTED IN THE QUESTION ABOUT GATE KEEPERS AN POLICY MAKERS WHO TELL US WHETHER EVIDENCE IS ENOUGH. WHO TELL US WHETHER WE'RE THERE YET. WHETHER WE REALLY SEE THAT THERE ARE POSSIBILITIES FOR CHANG THERE. IT'S ONE THING TO EDUCATE VIEWERS OF RESEARCH BUT AT THE END OF THE DAY THERE ARE PROFOUND ISSUES ABOUT WHATEVER WE DO OR DECIDE HERE HAVE AN IMPACT UPSTAIRS. >> THIS IS A GREAT -- YOU WANT TO MAKE A COMMENT? I THINK WE HAVE TO REMEMBER WE'RE GOING TO HAVE A QUICK BREAK FOR LUNCH RIGHT AFTER WE HAVE THE COMMENT BUT WE HAVE THE LAST SEGMENT OF OUR DAY, THIS IS PART OF THE CONVERSATION. >> RIGHT. SO EXCELLENT QUESTION AND I THINK TEEING UP, THE SENOR JUST PUBLISHED A DOCUMENT CALLED FACTORS TO CONSIDER WHEN MAKING BENEFIT RISK DETERMINATION FOR MEDICAL DEVICES AND PRE-MARKET REVIEW. IT ACTUALLY TOUCHES ON NOT VERY DEEP BUT VERY HIGH LEVEL ON SOME HYPOTHETICAL CASES AS WELL AS REAL CASES AND FACTORS THAT ONE WOULD WANT TO CONSIDER WHAT THE AGENCY HAS BEEN SEEING, WHAT THE AGENCY IS LOOKING AT. OPEN FOR PUBLIC COMMENT. IT'S A GREAT AVENUE FOR AT LEAST THE GROUP HERE TO PROVIDE FEEDBACK. IN ADDITION TO THE -- THIS DOCUMENT, IT IS IN DROP FORM AND WE'RE LOOKING FOR COMMENTS, IS A WILLINGNESS TO CHANGE THE AGENCY. WE'RE ALSO STRUGGLING WITH THE SAME THING. WE SEE THIS AREA BEING HEADING IN THIS DIRECTION. WE SEE EVIDENCE COMING IN IN DIFFERENT FORMS. I THINK WE'RE LOOKING FOR HOW DO WE GET PREPARED FOR AS EVIDENCE COMES IN. HOW DO WE GET BETTER PREPARED TO ASSESS. SPEAKING OF WHICH I'M SETTING UP A WORKSHOP 12th AND 13th OF SEPTEMBER, ON THE 13th IT'S SET UP HALF DAY TALKING ABOUT CLINICAL DECISION SUPPORT SYSTEMS. AS PART OF THE MOBILE APPS WHEN YOU LOOK AT MOBIL APPS MOST ARE CLINICAL DECISION SUPPORT SYSTEMS OM OF SOME SORT, SOME LEVEL. SO WE'RE ASKING THE QUESTION, WE ASKED IN WHAT FACTORS SHOULD ONE CONSIDER WHEN YOU TAKE THIS WHOLE BIG FIELD AND DICE IT UP. IS THERE -- ARE THERE ANY THEMES ONE SHOULD TAKE INTO CONSIDERATION TO EVALUATE THE RISK? AND IS IT THE RIGHT EVIDENCE? IS IT THE RIGHT EFFICACY? THE RIGHT SAFETY? AND WE ARE SEEKING THAT FEEDBACK. AND I THINK IF YOU GUYS CAN ATEN IT WILL BE GREAT. ALSO WE'RE ASKING HOW SHOULD THE AGENCY ASSESS THESE ONCE YOU HAVE THIS BROAD GUIDELINE SET UP. I THINK THAT'S SOMETHING TO KEEP IN MIND. THE DOCKETS ARE OPEN FOR MOBILE APPS GUIDANCE AND THIS GUIDANCE. BASICALLY THEY ARE GUIDANCE, DOCUMENTS THAT PORTRAY THE AGENCY'S THINKING AND WE'RE SEEKING FEEDBACK. REALLY APPRECIATE IF YOU GUYS CAN PROVIDE THAT SO WE CAN AT LEAST ADJUST AND TAYLOR TO WHAT THE LAW ALLOWS US TO DO AND HEAD US IN THE RIGHT DIRECTION. >> LET'S THANK THE PANEL. [APPLAUSE] >> SO WE WILL BE BACK FOR OUR WEBINAR FOLKS THANK YOU, KEEP YOUR COMMENTS COMING AND THERE'S A TWITTER DOWN LINK BUT WE WILL BE BACK AT 12:30. SO THE ROOM HERE IS GOING TO RUSH OUT GRAB SOME FOOD AND COME BACK FOR MORE CONVERSATION. ALSO THINK ABOUT YOU HAVE A LOT OF RESPONSIBILITY NOW. WE'RE GIVING YOU LOTS OF OPPORTUNITY TO PROVIDE FEEDBACK AND MULTIPLE INVENTORY VENUES. LET'S KEEP GOING. THANKS. I HAVE ONE BIT OF HOUSEKEEPING BEFORE WE GET TO THE NEXT WONDERFUL PANEL. THE NCIALG AND EXCITEMENT HERE IS AMAZING -- THE ENERGY AN EXCITEMENT IS AMAZING AND WHEN YOU HEAR THE PANEL YOU WILL BE FURTHER ENERGIZED IF YOU CAN BELIEVE IT. ONE THING I'M GOING TO SAY, I HATE TO DO IT BECAUSE IT'S LOGISTICS BUT I KNOW MANY MAYBE CATCHING CHAOS TO GET OUT OF HERE TODAY. CAN WE ALL STOP -- YOU MAYBE CATCHING CABS TO GO BACK TO WHEREVER YOU WANT TO BE TONIGHT. I'M GOING TO SEND A PAD AROUND, PUT YOUR NAME AND WHEN YOU NEED A CAB, IF YOU NEED IT. THIS AFTERNOON, THIS EVENING, LET US KNOW AND WE'LL WORK ON GETTING SOME CABS SET UP. (OFF MIC) >> WE CAN TALK TO YOU ALL ABOUT KIND OF TIME BUT -- (OFF MIC) >> WITH METRO. I DON'T KNOW. HALF HOUR? HALF AN HOUR, IF YOU WANT TO GO TO NATIONAL AND TAKE THE METRO IT'S HALF AN HOUR, EASY TO GET TO. NOW, OW NEXT SESSION IS IS AUDIE ATIENZA. HELP AND HUMAN SERVICES. >> WE LEARNED RECENT ASPECTS OF MOBILE HEALTH AND MO MOBILE HEALTH RESEARCH. ENGAGING CONVERSATIONS ABOUT THE INs AN OUTS OF RESEARCH DESIGN. NOW WE'RE COMING TO THE ASPECTS OF WE GET ALL THIS DATA, SOME PEOPLE SAY CALL THIS FIELD QUANTIFIED SELF WHICH YOU MONITOR PEOPLE 24/7, USING MULTIPLE DEVICE, SOME CALL IT REAL TIME DATA CAPTURE, SOME ECOLOGICAL ASSESSMENT OR EXPERIENCE METHOD DOINGS BUT YOU CAN GET JUST -- METHODOLOGIES BUT YOU CAN GET LOADS OF DATA. THE QUESTION IS, AS WE -- AS THE PLANNING COMMITTEE WAS THINKING ABOUT THIS, HOW DO YOU MAKE SENSE OF THIS DATA, HOW DO YOU EXTRACT THE VALUABLE PIECES OF INFORMATION TO MAKE IT USABLE FOR NOT ONLY USERS BUT CLINICS AND O -- COMMITTEE CLI ANYTHINGS AND OTHERSCH AS WE TALK ABOUT THIS, THIS AREA OF DATA MINING, ARTIFICIAL INTELLIGENCE AND LOOKING AT MACHINE LEARNING BECAME A CRITICAL ASPECT OF TRYING TO UNDERSTAND HOW TO MOVE THIS AREA FORWARD IN AN EMPIRICAL DATA-DRIVEN WAY. SO WE HAVE A PANEL THAT WILL BE FOCUSING ON THIS. ALSO PART OF THIS GROUP PRESENTING TODAY IS CRAIG LEFAE FROM RTI AND YOUTH OF SOUTH FLORIDA AS WELL AS RICHARD KATZ FROM GEORGE WASHINGTON UNIVERSITY. TODAY WE'RE PLEASED TO HAVE WITH US HAND HONORED TO CHAIR THIS SESSION IS AMY ABEARNATHY, A MEDICAL ONCOLOGIST AT DUKE UNIVERSITY. SHE'S FOCUSED A PALLIATIVE CARE M.D. FIRST AN FOREMOST BUT IS INTERESTED IN AREA OF PATIENT REPORTED OUTCOMES OF THE MEDICAL SYSTEM AND HOW TO IMPROVE IT WITH HEALTH IT. LET ME IS HAVE AMY, GIVE A WARM WELCOME TO AMY. [APPLAUSE] >> I'M AMY. RATHER THAN TALK ABOUT MY RESEARCH PER SE, WHAT I THOUGHT I WOULD DO IS INTRODUCE THIS PANEL BY TRYING TO PUT SINCE MAKING IN A BROADER CONTEXT AND BRING IT BACK DOWN AGAIN. FIRST IN ORDER IS WE NEED AMY'S SLIDES. I'M ASSUMING THERE IS A -- TALKING AND TECHING. BASICALLY REALLY AS WE THINK ABOUT WHERE WE ARE IN HEALTHCARE TODAY, WHAT I WANTED TO DO IS THINK ABOUT A NUMBER OF CONVERGING THEMES IN HEALTHCARE THAT ARE REALLY CENTER STAGE RIGHT NOW. SPECIFICALLY, WE'RE TALKING ABOUT A NUMBER OF THINGS ON THE POLICY LEVEL AS WELL AS DIRECTLY ON THE PATIENT CARE LEVEL THAT ARE GREATLY INFLUENCED AND READY FOR IN HEALTH. THIS IS ARE THE CONVERGING THEMES OF HEALTHCARE REDESIGN, HEALTHCARE QUALITY, COMPARATIVE EFFECTIVENESS, PERSONALIZED MEDICINE, PATIENT CENTERED CARE AND RAPID LEARNING. IF YOU LOCK YOU MIGHT SAY THEY DON'T HAVE MUCH TO DO WITH EACH OTHER. HOW DO WE REORGANIZE AND OPTIMIZE THE HEALTHCARE SYSTEM IN ORDER TO MAKE IT MORE EFFICIENT, MORE PATIENT-CENTERED AND GET THE RIGHT INTERVENTION TO THE RIGHT PATIENTS AT THE RIGHT TIME, HEALTHCARE QUALITY, HOW DO WE MAKE SURE WHAT'S HAPPENING FOR ANY INDIVIDUAL PATIENT, ANY PARTICULAR INTERVENTION IS DOING WHAT WE INTEND IT TO DO. WE HAVE PATIENT CENTERED CARE, MAKING SURE PATIENTS ARE APPROPRIATELY EMPOWERED WITH THAN SYSTEM AND LINES UP WITH THAT YOU ARE VALUES AN GETS INTERVENTIONS TO THE PATIENT AT THE RIGHT TIME. PERSONALIZED MEDICINE, RIGHT TIME, RIGHT PATIENT USING SOPHISTICATED SCIENCE PLUS EVERYTHING ELSE WE KNOW ABOUT THIS PARTICULAR INDIVIDUAL, LOTS OF VARIABLES COMING IN AT ONE TIME, RICH DATA SETS, DATA SOURCES COMPARATIVE EFFECTIVENESS RESEARCH, HOW WE FIGURE YET AGAIN, NOW WITHIN THE CONTEXT OF ALSO COST EQUALITY AND SPEAKING TO PAYERS, PATIENT, CONSUMERS OF ALL TYPES. REALLY ALL THIS IS ABOUT BETTER DECISION MAKING AND DATA-DRIVEN CARE. WHEN WE THINK ABOUT WITHIN THAT CON SECTION WE NEED THREE CORE ELEMENTS. WE HAVE TO HAVE EVIDENCE. WE NEED TO KNOW WHAT WORKS, WE NEED DATA. IT NEEDS TO BE LINKED DATA WITH INPUTS AND OUTPUTS. WE HAVE TO HAVE ANALYTICS AND SENSE MAKING AND ABILITY TO GET RESULTS AND REPORT IT IS PEOPLE WHO NEED IT AND WE HAVE TO DEFINE WITHIN THE SYSTEM A NUMBER OF WAYS OF THINKING ABOUT VALUE. BOTH AT THE SYSTEMS LEVEL AND SYSTEMS ORIENTATION AS WELL AS AT THE PATIENT LEVEL. REALLY AS WE START TALKING ABOUT THESE APPROACHES I WANT TO POINT OUT THE PERSPECTIVE IS FUNDAMENTAL. SO IF I HAVE SAID ALL THESE ARE DIFFERENT VERSIONS OF EXACTLY THE SAME STORY ESPECIALLY AS WE THINK ABOUT DATA INFORMATICS AND HEALTH REALLY HEALTHCARE REDESIGN AN QUALITY ARE THE SYSTEMS-BASED PERSPECTIVE. PATIENT-CENTERED CARE AND PERSONALIZED MEDICINE MIGHT BE SEEN AS THE PATIENT-CENTERED PERSPECTIVE, A DIFFERENT LENS. COMPARATIVE EFFECTIVENESS RESEARCH IN ITS IOM MONIKER BASICALLY IS THE INDIVIDUAL PATIENT AND SYSTEMS PERSPECTIVE SIMULTANEOUSLY. SO AS WE THINK ABOUT SENSE MAKING AND DIFFERENT WAYS OF TRYING TO WORK OUR WAY THROUGH THE MORE RAS OF DATA THAT WILL BE SITTING IN FRONT OF US WE START OFF AT THE INDIVIDUAL PATIENT WE CAN THINK ABOUT HOW WE GET INFORMATION TO CARE FOR INDIVIDUALS OR DERIVED FROM INDIVIDUALS HOW WE INTEGRATE IT AT THE CLINIC OR SYSTEMS LEVEL ACROSS HEALTH SYSTEM TO DO HEALTHCARE REDESIGN AN QUALITY OR NATIONAL SOCIETAL LEVEL AS WE THINK ABOUT POPULATION HEALTH, PUBLIC HEALTH AND TAKING CARE OF WHOLE SOCIETIES. IN 2007 YOU HEARD IDA MENTION EARLIER THE INSTITUTE OF MEDICINE PUT FORTH THE CONCEPT OF A RAPID LEARNING SYSTEM WHICH I THINK IS REALLY IF YOU THINK ABOUT THE PINNACLE OF THE EVIDENCE BASED MEDICINE OR EVIDENCE DEVELOPMENT SYSTEM AS A WAY OF WRAPPING THIS UP TOGETHER. WHEREBY THE CARE OF THIS INDIVIDUAL PERSON IS INFORMED BY ALL PEOPLE WHO COME BEFORE HER, SIMILAR CHARACTERISTICS, AND HER CARE IS REINVESTED INTO AN OVERARCHING GROUP OF COORDINATED DATABASES AN DATA SETS THAT ARE CONTINUOUSLY AND EVOLUTIONARILY MINED SO THAT WE HAVE A PROGRESSIVE ITERATIVE DEVELOPMENT OF LEARNING WITHIN THE SYSTEM. THIS HAS TO HAPPEN WITHIN THE CONTEXT OF STANDARDS GOVERNANCE AN TRUST IN PRIVACY AND NEVER WANT TO FORGET HOW IMPORTANT THIS IS. ONE REASON I BROUGHT IT UP EARLIER, IT'S AN AREA IF WE DONE GET RIGHT IT CAN GET AWAY FROM US AND IT'S VERY HARD TO WIN TRUST BACK FROM THE PUBLIC. BUT IF WE'RE THOUGHTFUL, INFORMATION FROM MOBILE HEALTH TECHNOLOGY SOLUTIONS ABOUT ONE INDIVIDUAL PATIENT IS PARSED OUT TO SOLVE ALL THESE DIFFERENT PARTS OF THE CONVERGING THEMES. AS WE MOVE INTO SENSE MAKING HERE IS THE OH NO. I BORROWED THIS SLIDE FROM BILL STEAD. WE'RE TALKING ABOUT THE HUMAN COGNITIVE CAPACITY, MY ABILITY AS A DOCTOR TO DO THIS AT ANY PARTICULAR TIME IS MAXED OUT. I YET HAVE MANY, MANY VARIABLES COMING AT ME CONTINUOUSLY AMONG WHICH I HAVE TO MAKE SENSE OF THIS. SO THIS IS WHERE DIAGNOSTICS ANALYTICS AND NEW SOLUTIONS ARE CRITICAL TO MOVE THIS FORWARD AS WE HAVE ALL THESE INTERVENING DATA COMING IN AT US. SO IN HOUSE SOLUTIONS AND SENSE MAKING ARE CRITICAL. AS WE THINK ABOUT SENSE MAKING I LIKE TO HIGHLIGHT WE HAVE TWO BIG AREAS. WE NEEM TO HELP USERS AND A HALF GATE THAT COGNITIVE OVERLOAD AND WE TALK AB USERS IN THE CONTEXT OF THOSE OF US DOING EVIDENCED DEVELOPMENT, USER AT THE POINT OF CARE. I ALSO WANT TO REMIND US USERS ARE HEALTH SYSTEMS POLICY MAKERS, AS WE THINK ABOUT DIFFERENT WAYS SINCE THE DATA WE WANT TO MAKE SURE WE'RE THINKING ABOUT THE DIFFERENT USER PERSPECTIVES AND ALSO THE SOLUTIONS CAN HAVE PATIENT LEVEL AND POPULATION LEVEL IMPACT. WE'LL HEAR FOUR DIFFERENT APPROACHES FIRST MONITORING SOLUTIONS AND TANZEEM WILL TELL US WHAT SHE'S WORKING ON TO HELP UNDERSTANDING HOW HUMAN HEALTH WE CAN DEFINE WHO IS SICK AND NOT SICK. ANALYTIC APPROACHES AN BEHAVIOR CHANGE APPROACHES THROUGH EXAMPLES THAT ALLOW US TO STUDY WITHIN REAL TIME HOW TO MAKE SENSE OF THESE DATA THROUGH EXAMPLES THAT GREG NORMAN WILL GIVE YOU. HOW DO WE MAKE SENSE OF CONTINUOUS PHYSIOLOGIC MONITORING DATA. MASSIVE INFORMATION COMING AT US IN REAL TIME AND WE NEED TO UNDERSTAND THOSE PATTERNS AT THAT TIME POINT OF CARE TO FIGURE OUT WHEN THERE'S MEANINGFUL TRENDS IN PATTERNS AND VLADMIR WILL TELL YOU MORE ABOUT THAT. FINALLY PLACES LOTS OF WORK IS HAPPENING NOW IS HOW TO PUT IT TOGETHER AND DEMONSTRATE WHAT THIS LOOKS LIKE ON THE GROUND. RAPID LEARNING SOLUTIONS THAT RECOGNIZE IT'S mHEALTH SINCE MAKING WITHIN THE CONTEXT OF REAL SYSTEMS THAT HAVE GOT ALL THE LOGISTICAL POLITICAL AND CULTURAL CHALLENGES THAT WE HAVE TO NAVIGATE THROUGH FIGURING HOW TO WORK THROUGH SYSTEMS CHALLENGING ON THE GROUND. ONE THING THAT I WORK ON, WE SEE EXAMPLES OF mHEALTH SOLUTIONS AN CLINICAL TRIALS FROM RICHARD KATZ AND ALSO SOME EXAMPLES IN SOCIAL MARKETING. WITH THAT, I WOULD LIKE TO TAKE TO THE FIRST TALK. [APPLAUSE] >> AS YOU SEE WITH VARIOUS PANELISTS, IT IS TRULY A INTERDISCIPLINARY FIELD THAT REQUIRES NOT ONLY THE RESEARCHERS, ENGINEERS, INFORMATION SCIENTIST, BEHAVIORAL SCIENTISTS BUT ALSO INDIVIDUAL WHOSE ARE FOCUSED ON THE CLINICAL ASPECTS OF PROVIDERS, PHYSICIANS, AND OTHER INDIVIDUALS WHO FOCUS ON THE CARE OF THE PATIENT. REALLY REQUIRES A TEAM OR DISCIPLINARY APPROACH FOR THIS SO WE TRIED TO IN MANY RESPECTS INFUSE THOSE DIFFERENCE PERSPECTIVES IN THIS MEETING. NEXT WE HAVE TANZEEM CHOUDHURY, IN INFORMATION COMPUTER SCIENCE AT CORNELL UNIVERSITY. >> THANK YOU, EVERYONE, FOR GIVING ME THIS OPPORTUNITY. I'D LIKE TO PRESENT SOME WORK ON HOW WE CAN DO SOCIETAL SCALE SENSING OF HUMAN BEHAVIOR USING MOBILE DEVICES. ONE OF THE REASONS I DIDN'T -- NOW IS A GREAT TIME TO DO THIS, IF WE LOOK AT THE EVOLUTION OF MOBILE SEPSING, PEEP -- SENSING PEOPLE HAVE TO BE WEIRD TO ADOPT THE SENSING AND ACTUALLY EVEN BE WILLING TO BE SUBJECTS. AS THIS IS A SENSING DEVICE I WORKED ON IN 2000 BUT IF YOU LOOK AT EVOLUTION OVER THE YEARS THINGS BECOME SMALLER BUT THINGS WE'RE USING IN SPECIALIZED CUSTOMIZED DEVICE ARE EMBEDDED TO PHONES. WE HAVE MICROPHONES THAT GIVE INFORMATION ABOUT CONVERSATIONS, LOCATION AND ACTIVITY. SO IT'S REALLY GIVES US A RICH PLATFORM TO BE ABLE TO SCARE AND MEASURE BEHAVIOR AT A SOCIETAL SCALE WHERE THOUSANDS AND MILLIONS OF PEOPLE ARE CARRYING THESE DEVICES. WE HAVE BEEN USING THESE TO LOOK AT DIFFERENT LEVELS OF BEHAVIOR. SO LOOKING AT PHYSICAL ACTIVITIES. WHAT KIND OF PHYSICAL ACTIVITIES DO PEOPLE ENGAGE IN IN THEIR DAILY LIVES, THINGS SIMPLE AS WALKING, RUNNING, TAKING STAIRS, TO PLACES THEY VISIT AS WELL AS LOOKING AT THE CONVERSATIONS, HOW THEY INTERACT WITH EACH OTHER. AND I ALSO COLLECTIVELY HOW DO WE FIND CONNECTIONS BETWEEN PEOPLE HOW THEY'RE ENGAGING WITH EACH OTHER, HOW THEY'RE SIMILAR TO EACH OTHER TO MODEL SOCIAL NETWORKS. WHEN WE'RE TALKING ABOUT SENSORS AN LOWER LEVEL KIND OF MAKING HIGH LEVEL INFERENCES FROM LOW LEVEL SENSOR DATA, FROM SENSING AN LEARNING PERSPECTIVE BEING ABLE TO IDENTIFY A CONVERSATION IS A HIGH LEVEL INFERENCE OR BEING ABLE TO IDENTIFY WHETHER SOMEONE IS WALKING OR JOGGING IS HIGH LEVEL INFERENCE BUT IF WE THINK ABOUT REALLY MEASURING HEALTH AND WELL BEING, THIS IS STILL PRETTY LOW-LEVEL INFORMATION. ONE CHALLENGE IS THAT HOW DO WE ACTUALLY TAKE WHAT WE CALL HIGH-LEVEL INFERENCE AND MAKE HIGH LEVEL INFERENCES. IF WE THINK ABOUT PROBLEMS THAT WE'RE LOOKING AT ASSESSING MENTAL HEALTH. IF WE WANT TO DO IT CONTINUE LUSLY SEAMLESSLY IN THE BACKGROUND, PEOPLE AROUND YOU LIKE YOU OR DI LIKE YOU. OR DO YOU FEEL JUST AS GOOD AS OTHER PEOPLE AROUND YOU SOME OF THE THINGS THAT ARE ASKED IN QUESTIONS IF YOU ARE LOOKING AT THE DEPRESSION SURVEY. SO HOW DO WE MAKE CONNECTIONS BETWEEN THE HIGH LEVEL BEHAVIOR AND REALLY HIGH LEVEL BEHAVIOR WE'RE INTERESTED IN. IN GENERATING THESE TYPES OF EVIDENCE, ONE OF THE THINGS WE FIRST NEED TO DO IS BE ABLE TO BRIDGE BETWEEN SENSOR INFORMATION AND HIGH-LEVEL MENTAL STATES. ONE THING WE LOOKED AT IN STUDIES WAS JUST TAKING A SIMPLE MEASURE, HOW MUCH TIME DO PEOPLE SPEND IN CONVERSATIONS WITH OTHER TO SEE IF THERE'S CONNECTION WITH MENTAL HEALTH SCORE. AND CERTAIN DEPRESSION SCALE. ONE OF THE THINGS, THIS IS AGAIN PRELIMINARY SMALL SCALE STUDY WE FOUND THERE WAS A STRONG CONNECTION. I WANT TO EMPHASIZE THAT'S NOT CONCLUSIVE EVIDENCE BUT ONE THING WE BELIEVE THAT IT PROVIDES THAT IF YOU CAN DO MASSIVE SCALE SENSING IT MIGHT BE A WAY OF FILTERING THE POPULATION TO NE WHICH SUBGROUPS ARE MORE SUSCEPTIBLE ARE MORE INTENSIVE. I THINK WAYS OF TRYING TO BRIDGE BETWEEN WHAT WE CAN SENSE AUTOMATICALLY AN WHERE YOU NEED MORE RICHER SOURCES OF INFORMATION AND AN AUTOMATIC WE'RE FINING THE RIGHT PEOPLE TO STUDY AN MEASURE IS SOMETHING WE CAN DO WITH MOBILE DEVICES. BUT AT THE SAME TIME THERE ARE CHALLENGES IN TERMS OF GENERATING EVIDENCE AND STILL IF WE WANT TO REACH OUR GOAL OF REDUCING LENGTH OF STUDIES WE NEED TO ALSO SOLVE SOME TECHNICAL CHALLENGES. AND SOME OF THE TECHNICAL CHAL CHALLENGES TO ENABLE SOCIETY TO BUILD SENSING PEOPLE ALTHOUGH ARE PROVIDING THESE RICH SOURCES OF INFORMATION, ARE VERY DIVERSE, HOW DO WE BUILD SYSTEMS THAT CAN DEAL WITH THE DIVERSITY AND ALSO TEAL WITH SOME OF THE DIFFERENT TYPE OF MISSING. WE TALKED ABOUT MISSINGNESS IN THE DATA. SO ONE IS EVEN IF WE LOOK AT PHYSICAL ACTIVITY PEOPLE ENGAGE IN DIFFERENT TYPES OF PHYSICAL ACTIVITY. SO JUST WALKING AND RUNNING IS NOT ENOUGH. HOW DO WE BUILD SYSTEMS THAT ACTUALLY WILL LEARN THE TYPES OF PHYSICAL ACTIVITY THAT PEOPLE ENGAGE IN ON THE GO THROUGH THE SENSING AND BEING ABLE TO PROMPT THE USERS AT THE RIGHT TIME TO GET THE INFORMATION AND LEARN RELEVANT INFORMATION ABOUT THE LIFESTYLES. AS WELL AS SIMULTANEOUSLY WE CAN LEVERAGE FROM THE COMMUNITY OF USERS REDUCING THE BURDEN IN LEARNING THESE TYPES OF BEHAVIOR. CAN WE LEARN SIMILARITY IN LIFESTYLE, DATA PATTERN TO ACTUALLY BE ABLE TO LEARN PERSONALIZED BEHAVIOR AT A SOCIETAL SCALE. SO BE ABLE TO GET COVERAGE AS WELL AS SCALE SCALE SIMULTANEOUSLY TO BETTER JUDGE EVIDENCE USED IN THIS SYSTEM. FINALLY I WANT TO END WITH THE NOTION OF HOW DO WE FIND NEW WAYS OF GIVING INFORMATION BACK TO THE USER, THIS IS KIND OF INSPIRED BY SOME WORK BY -- THAT CAME OUT OF UNIVERSITY OF WASHINGTON AND WHICH BUILDING UPON IS HOW CAN WE USE THE MOBILE DEVICES GIVING US CONTINUAL AWARENESS, THIS IS SHOWING US SECIAL AWARENESS, YOU HAVE YOUR -- SOCIAL AWARENESS, YOU HAVE YOURSELF REPRESENTED, THE SCHOOL OF FISH IS YOUR SOCIAL LIFE AND THERE'S DIFFERENT LEVELS SO CAN WE COMBINE MULTIPLE ASPECTS OF WELL BEING INTO A DAYS DISPLAY THAT MIGHT GIVE FEEDBACK TO A USER AN NEW WAYS OF CONNECTING TO THE USER TO ENCOURAGE BEHAVIOR CHANGE. WITH THAT I WANT TO THANK YOU. I WANT TO EMPHASIZE THIS WORK IN COLLABORATION WITH ETHAN BURK, WHO IS A DOCTOR AN INVALUABLE MEMBER OF HER TEAM AN STUDENTS AND OTHER FACULTY MEMBERS. THIS IS THE WEBSITE. FEEL FREE TO ASK ME ANY QUESTIONS. [APPLAUSE] >> THANK YOU, TANZEEM. NEXT IN THE RAPID FIRE PRESENTATIONS, THESE ARE FIVE MINUTES EACH, WE HAVE DR. GREG NORMAN FROM UCSD, ASSOCIATE PROFESSOR IN THE DEPARTMENT OF FAMILY AND PREVENTATIVE MEDICINE. ALSO RESEARCHER AT CAL IT TOO. SO GREG. GREAT TO BE HERE. THIS COUPLE OF SLIDES I'LL GO THROUGH ARE REALLY BASED ON A PROPOSAL THAT JAVIER MOVALON AND I WROTE IN RESPONSE LAST YEAR IN RESPONSE TO AN RFA NOT FUNDED AND IT'S KIND OF FUNNY BECAUSE SEEMS LIKE I DEALS WITH ISSUES WE HAVE BEEN TALKING HERE ABOUT REVIEWERS MAYBE NOT GETTING THE CONCEPTS AND CAN THESE TWO GUYS COLLABORATE TOGETHER. I WORK WITH KEVIN PATRICK AND HIS SOLUTION TO EDUCATING AND CROSS VALIDATING MEDICINE AND PLOP US INTO CAL TECH 2 WHICH IS A SCHOOL OF LABORATORY SCIENTISTS SO HE CAME TO OUR LAB, HE'S HEAD OF THE MACHINE PERCEPTION LAB AND THAT STARTED TALKING AB IDEAS AN TAKING THINGS HE'S WORKING ON AND APPLYING TO MEDICAL ISSUES. VERY INTERESTING. HOW TO GET PEOPLE TO BE MORE ACTIVE AND LESS SEDENTARY. WE SPEND WAY TOO MUCH TIME BEING SEDENTARY AND IS ACTUALLY NOW WE'RE FINDING EVIDENCE IT HAS INDEPENDENT RISK FACTOR FROM BEING PHYSICALLY ACTIVE, SO SPENDING THE DAY LIKE THIS IS CONTRIBUTING TO OUR ILL HEALTH. BUT PEOPLE CAN ONLY DO SO MUCH VIGOROUS ACTIVITY. YOU CAN ONLY RUN JOG OR SWIM SO MUCH. SO IF WE CAN THE LIGHT INTENSITY, THE WALKING STANDING MOVING AROUND WILL DECREASE THE SEDENTARY TIME. WE WANT TO NUDGE PEOPLE DOING MORE LIGHT ACTIVITY AND DO MORE MODERATE TO VIGOROUS ACTIVITY BUT WE CAN ONLY DO SO MUCH OF THAT. THAT'S THE PROBLEM WE'RE TRYING TO DEAL WITH, HOW TO NUDGE PEOPLE TOWARDS BEING MORE ACTIVE THROUGHOUT THE DAY MORE. WHEN I THINK OF mHEALTH, MAYBE THI3C6Ö SLIDES HAS BEEN TALKED ABOUT ALREADY, WE'RE GETTING TO THE SATURATION OF IDEAS MAYBE A LITTLE BIT BUT I THINK AS THIS CONVERGENCE OF DATA ANALYTICS, DIFFERENT WIRELESS TECHNOLOGIES AND DIFFERENT TYPES OF BEHAVIOR CHANGE STRATEGIES AN TECHNIQUES WE CALL PERSUASIVE DESIGN, IF YOU GET mHEALTH TO ME IS IMPROVING MEASUREMENT AND IMPROVING INTERVENTION THROUGH THE ALMOST PERFECT STORM OF THESE TYPES OF TECHNOLOGIES AND TOOLS. CONTROL 3 I SHOULD MENTION WHICH IS IN THE TITLE OF THE TALK DEALS WITH INFLUENCING DYNAMICCAL SYSTEMS AND DEALS WITH BRINGING TOGETHER ELEMENTS AN FEEDBACK LOOPS. SO THIS SLIDE CAME FROM A PAPER IF YOU'RE INTERESTED IN LEARNING HOW TO -- ABOUT MACHINE LEARNING AND HOW TO AS A GOOD STARTING PAPER TALK ABOUT MACHINE LEARNING VERSUS TRADITIONAL PROGRAMMING BUT I ALSO THOUGHT IT SORT OF INTRODUCES THE CONCEPT OF WHERE WE HAVE COME FROM, OR AT LEAST WHERE I HAVE COME FROM IN DEVELOPING HEALTH BEHAVIOR CHANGE INTERVENTIONS AN WHERE WE ARE EAR TRYING TO GET TO AND MOVE TO. SO TRADITIONALLY WE HAVE DONE THESE DETERMINISTIC INTERVENTIONS WHERE YOU HAVE TO -- YOU PROGRAM IN SPECIFICALLY THE DIFFERENCE RULES OF THE SYSTEM AND YOU ASSUME THAT ALL THE DATA IS ESSENTIALLY NOISE FREE AND YOU WANT TO DEVELOP MODULES OF THE SYSTEM AND GOING THROUGH CODING AND DEBUGGING IT. YOU HAVE VERSION 1, MAYBE YOU HAVE VERSION 2.0 DOWN THE LINE. WHICH IS A VERY DIFFERENCE APPROACH THAN WHERE WE'RE TRYING TO GET TO. WHICH IS STOCHASTIC, WE'RE DEALING WITH NOISY DATA DATA, TRYING TO LEARN FROM THE DATA. WHAT THE PATTERNS AN PRINCIPLES SHOULD BE IN THE INTERVENTION. THERE'S OOH A PIPELINE OF DATA IS CRITICAL TO THE DEVELOPMENT AND IT'S MORE AN EXPLORATORY PROCESS TRYING IT AND IMPLEMENTING THE PROCEDURE ITERATIVELY. HERE IS THE IDEA OF SORT OF A CLASSIFICATION PIPELINE. THERE SHOULD BE AN ACCELEROMETER THERE BUT IT'S NOT THERE. (OFF MIC) [LAUGHTER] >> I GOT ONE RIGHT HERE. BUT PICTURE >> DONNA SAID THERE SHOULD BE AN ACCELEROMETER EVERYWHERE. THIS IS SHOWING THE PIPELINE OF RAW DATA TO SENSOR DATA AN PARSING OUT THE ATTRIBUTES YOU CAN GET OUT OF THE DATA AND APPLY THE LEARNING ALGORITHM TO CLASSIFY DIFFERENT TYPES OF PHYSICAL ACTIVITY AND TEST IT WITH NEW DATA TO SEE HOW WELL YOUR CLASSIFICATION IS, THAT COULD BE AN ITERATIVE PROCESS IN ITSELF. FINALLY, THIS IDEA OF A HEALTH BEHAVIOR CHANGE INTERVENTION AS A CONTROL LOOP WHERE THE PLANT IS YOUR PARTICIPANT YOUR PEOPLE YOU'RE INTERESTED IN, THAT TERM COMES FROM ENGINEERING WHERE YOU'RE MAYBE WORKING IN A FACTORY PLANT AND TRYING TO OPTIMIZE THE PROCEDURES THERE. AND YOU'RE USING SENSORS THE MAKE INFERENCES ABOUT WHAT THE PERSON IS DOING AND THAT'S THEIR STATE. AND THEN YOU USE ANOTHER TYPE OF MACHINE LEARNING, AN EXAMPLE IS REINFORCEMENT LEARNING TO FIGURE OUT WHAT'S THE OPTIMAL INTERVENTION STRATEGY TO APPLY WHICH IS CALLED THE POLICY AND THEN THE POLICY STRATEGIES, AND THEN YOU CAN SEE HOW THAT ITERATIVELY WORKS THROUGH AND CREATES THE INTERVENTION. SO I'LL STOP THERE. THAT'S ALL I GO. THE EXTRA SLIDES ARE EXTRAS. ANYWAY, THANK YOU. [APPLAUSE] >> THANKS, GREG. YOU HAVE HEARD FROM THE BEHAVIORAL PERSPECTIVE OF MACHINE LEARNING, ARTIFICIAL INTELLIGENCE AND DATA MINING. NEXT WE HAVE VLADMIR SHUSTERMAN CARDIOLOGIST UNIVERSITY OF PITTSBURG WHO ALSO DOES WORK IN THEORETICAL MATHEMATICS AND ARTIFICIAL INTEL SWRENS TO TALK FROM A CLINICAL PERSPECTIVE. >> THANK YOU VERY MUCH. GOOD AFTERNOON. FIRST OF ALL I WOULD LIKE TO THANK ORGANIZERS OF THIS WONDERFUL AND IMPORTANT MEETING. AND THANK THEM FOR INVITING ME TO GIVE ME A CHANCE TO PRESENT OUR WORK THAT OUR GROUP HAS BEEN DOING FOR THE LAST FIVE, TEN YEARS WITH THE GENEROUS SUPPORT FROM NIH. THIS IS A COLLABORATIVE WORK WITH THE UNIVERSITY OF PITTSBURG AND YALE UNIVERSITY SCHOOL OF MEDICINE. OUR WORK IS IN PATIENTS WITH CARDIOVASCULAR DISEASE AN PEOPLE WITH CARDIOVASCULAR RISK FACTORS WHICH IS A BIG PORTION OF THE AMERICAN POPULATION AS YOU KNOW. WE RECORD A LOT OF DATA, CONTINUOUSLY UP TO ABLE CHANNELS EIGHT CHANNELS OF ELECTROCARDIOGRAPHIC DATA,es"uh TO ONE THOUSAND CHANNELS PER SECOND. AND NOW WE ROR CHANGES IN BODY POSITION AND PHYSICAL ACTIVITY AND RESPIRATION AND CHANGES IN PSYCHOLOGICAL STATUS. PATIENT ACTIVATED EVENT. SO A LOT OF DATA. WHEN YOU PROCESS ALL THESE MOUNTAINS OF DATA, THE RECEPTOR DEVELOPED OVER THE LAST TWO, THREE DECADES DOES A GOOD JOB PROCESSING THIS DATA ONLINE AND SOME OF THIS IS NOW WORKING ON MAFORT PHONES. THE PROBLEM IS MOST OF THIS OR ALL OF IT DETECTS THE EVENTS AFTER THEY HAPPENED AND USUALLY IT'S TOO LATE. THE EARLIER WE START THE INTERVENTION THE MORE EFFECTIVE WE ARE. THAT'S WHERE OUR MOST OUR FOCUS IS. I THINK WHERE THE ADVANTAGE OF LONGITUDINAL DATA IS CRITICAL. TO GUF YOU AN IDEA OF THE DATA WE'RE DEALING WITH, HERE IS ONE EXAMPLE. THE TOP OF THE SLIDE YOU SEE BIT AT THE END OF THIS PERIOD GOT A LIFE THREATENING ARRHYTHMIA. OVER 16 HOURS THERE'S LOTS OF CHANGES, THE PATTERN IS COMPLEX. YOU CAN SEE THAT THERE IS SOME CHANGES BUT BECAUSE THE PATTERN IS SO COMPLEX IT'S HARD TO DESCRIBE IN A SINGLE STATISTICAL PARAMETER. IF YOU APPLY 8 PARAMETERS COMMONLY USED FOR TIME SERIOUS STANDARD DEVIATION, SPECK TROAD AND LINEAR METHODS ALL SHOW CHANGES. THE PROBLEM IS THAT THESE CHANGES ARE NOT SPECIFIC THEY OCCUR ALL THE TIME AND NOT VERY USEFUL SO IN ADDITION THEY VARY FROM PATIENT TO PATIENT. HOW CAN WE DEAL WITH THIS PROBLEM? ONE APPROACH WE HAVE BEEN WORKING FOR THE LAST FIVE, TEN YEARS IS TO FINGER PRINT THE PATIENTS. WHETHER WE DO IS APPLY WE DEVELOP AND APPLY PATTERN RECOGNITION BASED APPROACH TO EXTRACT THE COMPONENTS FROM THE SIGNAL OF EACH PATIENT. AND BECAUSE THE SINGLE PRINT PATTERNS ARE ON STRUCK FROM THE STATISTICS OF THE STRUCTURE ITSELF, IT'S TO THIS PARTICULAR PATIENT. ON THE LEFT OF THIS SLIDE YOU CAN SEE THE VECTORS ARE NOT VERY INFORMATIVE IF YOU LOOK AT THEM. BUT IF YOU HAVE THE CHANGES IN THE VECTORS IN THIS PATIENT, THEN WE CAN CLEARLY SEE ON THE RIGHT THAT THE SIGNAL BECOMES UNSTABLE SEVERAL HOURS BEFORE THE EVENT HAPPENED. THIS METH TURNS OUT MUCH MORE SPECIFIC BECAUSE WE TAILORED IT TO ONSET OF CLINICALICALLY IMPORTANT EVENTS. TWO THINGS IMPORTANT FOR APPLYING THIS TECHNOLOGY. ONE TO FIND THE WINDOWS WITH WE NEED TO TRAIN OUR METHOD TO IDENTIFY THE FINGERPRINTS. ADAPTIVE SEG MENTATION IS THE KEY. WE NEED TO KNOW IN OTHER WORDS WHAT EXACTLY WAS A PATIENT DOING, WHAT WAS THE PHYSIOLOGICAL AND PSYCHOLOGICAL ACTIVITY DURING THIS PERIOD AND THAT'S WHY WE ARE NOW RECORDING EXTRA CAR OWE GRAM BUT A SUITE OF OTHER SIGNALS TO IDENTIFY TO CHARACTERIZE THE FINGERPRINT PATTERNS. TO FURTHER IMPROVE ON THIS INFORMATION. THE SECOND THING I WANTED TO MENTION IS THIS COMPUTATION IS VERY INTENSIVE. IF YOU'RE TALKING LONGITUDINAL DATA WE'RE RECORDING OVER DAYS, WEEKS AN MONTHS, IT'S A LOT OF DATA. IT MAKES TO DO THIS INTENSIVE ANALYSIS ON THE INTERNET SERVER. IT DERIVE THIS FINGERPRINT PATTERNS AN SEND THEM TO THE CELL PHONE TO THE APPLICATION AND I'LL SHOW YOU IN THE NEXT SLIDE HOW THE SYSTEM THAT WE DEVELOP ACTUALLY DOES THIS IN REAL TIME. SO HERE IS THE ONE EXAMPLE, IT'S A REAL PATIENT FROM -- REAL DATA FROM A PATIENT WHO CAME TO THE EMERGENCY DEPARTMENT WITH CHEST PAIN, THAT'S A COMMON SCENARIO. FIRST YOU CAN SEE THE ECG IS RECORDED, PROCESSED ON THE CELL PHONE AND MARKED BY EVERYTHING WITHIN NORMAL LIMBS SEND TO THE SERVER, THE SERVER DOES THE FINGERPRINT ANALYSIS AN UPDATES SINCE THE RESULT OF THIS ANALYSIS TO THE CELL PHONE WITHOUT THE PATIENT KNOWING IT. THEN THE NEXT RECORDING FIVE MINUTES LATER YOU CAN SEE THE SAME THING, WITHIN NORMAL LIMITS. BUT IT GIVES THE SERVER A CHANCE TO BETTER ADAPT TO BETTER LOAN THE FINGERPRINT PATTERN IN THIS PATIENT. WHAT HAPPENS NEXT, EIGHT MINUTES LATER THERE'S SUBTLE CHANGES IN THE SEGMENT THAT ARE SO SMALL THAT WOULD BE OVERLOOKED BY STANDARD STATISTICAL METHOD Z BUT BECAUSE WE TRAINED OUR CELL PHONE APPLICATION USING THIS FINGERPRINT PATTERN THEY WOULD BE IMMEDIATELY DETECTED. IT'S POSSIBLE TO SEND IMMEDIATELY AN ALERT TO HEALTHCARE PROVIDER. AT THAT POINT THE HEALTHCARE PROVIDER INITIATES PREVENTATIVE MEASURES THAT WOULD BE MORE EFFICIENT THAN IF THEY START 20 MINUTES LATER WHEN FULL-BLOWN ISCHEMIA DEVELOPED. WE DON'T WANT TO WAIT FOR THIS. SO WHAT WE LOAN FROM THIS EXPERIENCE WE NEED TO DO MORE, COLLECT LIBRARIES OF INDIVIDUAL FINGERPRINT BASIS FOR DIFFERENT TYPES OF PHYSIOLOGICAL AN PSYCHOFIZZ QUO LOGICAL ACTIVITIES TO LEARN MORE ABOUT DIFFERENT PATTERNS. WE NEED LARGER SCALE STUDIES ON THIS PERSONALIZED FINGERPRINT AN METHODS NOT ONLY THE ONE I SHOWED TO YOU BUT WE NEED A SERIES OF PILOT STUDIES BUT CLEARLY MORE NEEDS TO BE DONE. FINALLY WE NEED TO DEVELOP BENCHMARK STANDARDS AND SOFTWARE FOR COMPARATIVE TESTING OF NEW METHODS BECAUSE A LOT OF METHS HAVE BEEN DEVELOPED FOR TIME SERIES ANALYSIS OVER THE LAST TWO, THREE DECADES AND THE PROBLEM IS THAT THE LANGUAGE IS CHANGING SO RAPIDLY THAT IT'S BECOMES VERY DIFFICULT TO KEEP TRACK OF WHAT IS -- HOW NEW METHODS COMPARE TO OLD ONES AND MAYBE WHERE REDUNDANT, MAYBE WE'RE REINVENTING THE WHEEL. IT'S NOT ENOUGH TO SHOW YOUR METHOD WORKS BUT HOW IT COMPARES TO OTHER METHODS. FINALLY, IT'S VERY IMPORTANT THAT WE NEED TO RESOLVE INTEROPERABILITY ISSUES AND STANDARDIZE THAT AND EXCHANGE PROTOCOLS FOR VARIOUS TYPES OF DATA SO THAT EVERYONE IN THE RESEARCH COMMUNITY WILL HAVE ACCESS TO ALL THIS DIFFERENT TYPES OF DATA. THANK YOU VERY MUCH. [APPLAUSE] >> THANK YOU, VLADMIR, AS DON WAS SAYING THE RICHNESS OF THE DATA IS INCREDIBLE, MANY EXAMPLES ILLUSTRATE WHAT CAN BE DONE WITH THAT, WITH PADDED RECOGNITION AND LOOKING AT MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE. I WOULD LIKE TO TURN IT OVER TO AMY AS CHAIR OF THE SESSION. YOU HAVE THE FIRST QUESTION TO THE PANELISTS. >> ONE THING I STRUGGLE WITH IS RISK. AND AS WE THINK ABOUT DEVELOPING NEW SOLUTIONS TO MAKE SENSE OF THE DATA AND PRESENTING THAT INFORMATION TO BE USED HOW DO YOU BEST MANAGE THE RISK THAT YOU MIGHT BE REACHING THE WRONG CONCLUSION AS YOU'RE MAKING SENSE OF THE DATA? >> I THINK ONE OF THE APPROACHES THAT WE HAVE TRIED THE DO, IT REALLY HAS TO BE ITERATIVE AND NOT JUST MAKING A FINAL CONCLUSION FROM THE DATA TO EVEN SEE CAN WE REPLICATE EXISTING RESULTS. IF WE CAN, ARE THERE NEW INFORMATION OR NOVEL MEASURES THAT WE CAN MAKE TO UNDERSTAND A CERTAIN CONDITION BETTER. ALSO I THINK THE FLIP SIDE COLLECTING THE DATA AND PROCESSING THE DATA HAS ITS OWN RISK SO WHAT ARE THE BEST WAYS OF I THINK THERE ARE WAYS TO INNOVATE IN THE SPACE OF COLLECTING THE DATA, THERE IS RISK, I TALKED ABOUT PRESENTING IT BACK TO THE USERS, THERE ARE DEFINITELY RISKS OF THAT PARTICULARLY IF YOU'RE THINKING ABOUT MENTAL HEALTH, YOU DON'T WANT TO TELL THEM THAT OH, YOU'RE MISERABLE TODAY. SO THERE ARE RISKS AT DIFFERENT STAGES AN BREAKING IT DOWN FROM THE DATA RECORDING STAGE TO THE CONCLUSIONS THAT YOU CAN MAKE AND THE INFORMATION THAT YOU PROVIDE BACK. AND REALLY ENGAGING FOR ME PERSONALLY BEHAVIORAL SCIENTISTS AND SIGH CHAI TRYSTS KNOW WHAT ARE THE BEST WAYS OF MAKING INFERENCE AND FEEDING BACK TO THE DOCTOR, THE USER IS CRITICAL. Ir DIFFERENT CONTEXT, WHAT THE LEVEL OF RISK IS. I THINK IT'S NOT A ONE-SHOT THING. ONE HAS TO BE AWARE AND THINK THROUGH AND ITERATIVELY FIND A WAY THAT'S ACCEPTABLE. AND USEFUL. >> VLADMIR. >> I WAS GOING TO ADD, FOR PHYSICAL ACTIVITY RESEARCH THE RISK IS DIFFERENT. WE WOULDN'T WANT TO RELEASE A SYSTEM OR APP OR SOMETHING THAT CLASSIFIES ACTIVITY UNLESS YOU FELT CONFIDENT ABOUT THE ACCURACY AND THEN THERE IS THAT ITERATIVE AND CONTINUING TO GENERALIZE THE WORK FOR THIS POPULATION SEGMENT. BUT THE OTHER THING RELATED TO RISK IN TERMS OF IS THE TRUST THE PARTICIPANTS THE PATIENT HASN'T DEVICE, AND SO IF YOU START GIVING FAULTY INFORMATION OR SAYING YOU HAVEN'T BEEN ACTIVE IN THE LAST TWO HOURS AN ACTUALLY WERE YOU HAVE A RISK OF NOT TRUSTING THE SYSTEM AND NO LONG ERGOING TO USE IT AND SO YOU WANT TO TRY TO AVOID THAT BY HAVING A BULLET PROOF SYSTEM. AND IN ADDITION HAVING A WAY THE PARTICIPANT CAN ACTUALLY ALMOST SELF-ANNOTATE THEIR DATA IN A SOANS THEY SEE SOMETHING DIDN'T GET CAPTURED OR THEY DIDN'T WEAR -- THEY WEREN'T CARRYING THE DEVICE AND WANT TO GET CREDIT FOR I DID THIS OR TOOK MY MEDICATION, WHATEVER YOU'RE TRYING TO WORK ON. SO THEY CAN SELF-LOG INFORMATION NOT THE RISK OF THEM FEELING LIKE THIS IS NOT A VERY USEFUL DEVICE FOR ME. >> ONE THING THAT I WOULD LIKE THE MENTION, THE ADVANTAGE OF DOING mHEALTH AND PERSONALIZED DATA COLLECTION IS THAT IF WE DESIGN FROM THE BEGINNING THAT WE'RE RECORDING SUFFICIENTLY COMPLETE SET OF PHYSIOLOGICAL AND PSYCHOLOGICAL PARAMETERS, THEN WE WILL HAVE TWO ADVANTAGES. ONE IS THAT WE ACTUALLY CAN DO COMPUTER MODELING AND SIMULATION USING THIS DATA. BEFORE WE APPLY IT TO REAL PATIENTS SO IT'S LIKE FULLY BACK. WE CAN SAY WHAT IF WE MISSED DATA FROM SENSORS ONE, TWO AN THREE? HOW ACCURATE IT WOULD BE? OR WHAT IF -- ET CETERA, ET CETERA. WE CAN DO A LOT OF RETROSPECTIVE ANALYSIS BEFORE WE GO TO PATIENTS USING THIS DATA. SO THAT'S ONE THING THAT I WANTED TO MENTION. SECOND THING IS THAT I THINK IT'S VERY IMPORTANT THAT AT THE BEGINNING OF THE DESIGN STAGE, IF OUR TEAM IS TRULY INTERDISCIPLINARY AND DESIGNS THE STUDY NOT JUST TO ANSWER ONE SPECIFIC QUESTION BECAUSE THERE IS A LOT OF TIME, EFFORT AND COST, ET CETERA, ET CETERA, THAT GOES INTO COLLECTING THIS DATA. THEN WE CAN USE ONE STUDY TO ANSWER A HOST OF DIFFERENT QUESTIONS IN PSYCHOPHYSIOLOGY AND IN HEALTH BEHAVIOR, ET CETERA. SO IT SHOULD BE TRULY INTERDISCIPLINARY EFFORT, ESPECIALLY AT THE DESIGN STAGE BECAUSE IT'S VERY DIFFICULT TO MODIFY SOMETHING THAT IS ALREADY ONGOING. IT MAKES MORE SENSE TO DO IT AT THE BEGINNING OF THE DESIGN STAGE. >> DATA EVERYWHERE. QUESTIONS FROM THE GROUP? HOW ABOUT COMMENTS? >> DEBRA UCLA. THIS IS SOMETHING FROM THE PREVIOUS PANEL AS WELL AS FOR THIS ONE. CLEARLY NEEDS TO BE MULTI-DISCIPLINARY. BUT I THINK WE HAVE A MIXTURE OF THINGS GOING ON. SOMETIMES WE'RE TALKING STUDY DESIGN AND SOMETIMES WE'RE TALKING ABOUT WHAT WAS MENTIONED BEFORE, ITERATIVE INTERVENTION DEVELOPMENT PROCESS. YOU'RE NOT DOING THE STUDY DESIGN YET, YOU'RE TRYING TO DEVELOP THE HIGHER LEVEL ACTIVITIES THAT ARE WHAT'S HIGHER LEVEL TO US IS NOT HIGHER LEVEL TO THEM. AND MOVE UP THAT PIPELINE UPSTREAM TO WHAT'S HIGHER-LEVEL FEATURES. THIS IS SOMETHING I WANT TO INJECT IN THE MULTI-DISCIPLINARY TEAMS, IT'S IMPORTANT TO HAVE CLARITY ABOUT WHEN -- WHO IS RESEARCH YOU'RE DOING IT'S DIFFICULT TO DO SIMULTANEOUSLY THE Ph.D. TOPIC AN COMPUTER SCIENCE WHILE YOU'RE DOING THE Ph.D. TOPIC IN BEHAVIOR HEALTH SCIENCE, IT DOESN'T MEAN IT'S NOT PART OF THE SAME PROJECT. BUT TO BE CLEAR ABOUT, AS A COLLEAGUE OF OURS MARGARET BARTINOCI FROM ALWAYS SAYS IS YOU HAVE TO TAKE TURNS. TO BE CLEAR WHOSE TURN IT IS, TO BE USING THE BEST KNOWN WAY OF DOING SOMETHING, SOMETHING THAT A MACHINE LEARNING RESEARCHER COULDN'T GET PUBLISHED, BUT IS THE BEST APPROACH TO DEALING WITH YOUR DATA IN THIS PHASE, AND LETTING THEM USE YOUR DATA AN EXPERIENCE THE DRIVE YOUR ENVELOPE INTO SOMETHING YOU'RE EXPLORING WHEN IT COMES TO THIS QUESTION OF RISK. SO AS WE'RE DOING THESE MULTI-DISCIPLINARY RESEARCH PROJECTS, I THINK WE CAN DO WONDERFUL THINGS TOGETHER BUT THERE'S A NEED TO BE CLEAR THERE AS TO WHEN WHAT WE'RE DOING IS TRYING TO USE STATE-OF-THE-ART VERSUS WHEN WE'RE ACTUALLY PUSHING THE ENVELOPE. >> COMMENTS FROM THE PANEL. >> ONE COMMENT I THINK ALSO IS IMPORTANT IS THE INTERFACING PART. SO FROM A MACHINE LEARNING OR PUBLICATION PERSPECTIVE YOU HAVE BENCHMARK OF ACCURACY, THAT'S ALL YOU CARE ABOUT AND ONE OF THE THINGS, ALMOST IMPOSSIBLE TO BUILD BULLET PROOF SYSTEM SO HOW DO YOU EXPOSE UNCERTAINTY THAT'S SOMETHING YOU DON'T SEE IN MACHINE-LEARNING PAPERS. ONE THING WE HAVE LEARNED THROUGH THIS ENGAGEMENT, THE INTERFACE OF HOW DO YOU TRANSFER KNOWLEDGE IS IMPORTANT AND WHAT MIGHT BE KIND OF THE CRITICAL BENCHMARK IN ONE AREA MIGHT NOT BE AS IMPORTANT AND TRYING TO FIGURE THAT OUT EARLY ON IS ALSO USEFUL. WHAT DO YOU EXPOSE FROM YOUR SYSTEM THAT WILL BE ESSENTIAL OR VERY INFORMATIVE TO THE OTHER DISCIPLINES. >> ONE REASON I'M SO ENAMORED BY WHAT YOU JUST SAID, IT REALLY GETS TO THE HEART OF WHAT I WAS GETTING AT. IN THE WORLD I LIVE IN, THERE'S A BELIEF WE'LL GET TO THE POINT FIVE TEN YEARS FROM NOW WHERE WE PUSH BUTTONS AND THERE'S GOING TO BE MAGICAL ANSWERS. I'M ALWAYS LIKE OKAY, WELL, BUT AS YOU APPROXIMATE MULTI-DISCIPLINARY TEAMS YOU HAVE TO HAVE SOMEBODY WHO IS HELPING TO PUT THAT RISK INTO CONTEXT. AND THE ONE THAT I HAVE BEEN DEALING WITH LATELY IS TREATMENT SEQUENCING FOR CHEMOTHERAPY. WHEREBY THERE IS THIS THEORY THAT WE CAN USE MACHINE LEARNING AND ADAPTIVE LEARNING APPROACHES TO TRY AND FIGURE OUT WHICH CHEMOTHERAPY COMES FIRST, SECOND AND THIRD. BUT WHILE THE COMPUTER SCIENTISTS ARE HELPING US ANSWER THAT QUESTION, YOU HAVE TO HAVE THE CLINICAL SCIENTIST SAYING WELL, HERE IS THE RISK OF IF WE GET THIS WRONG. AND SO YOU HAVE TO HAVE THE WHOLE MULTI-DISCIPLINARY TEAM WORKING TOGETHER SO THEY CAN GOING FORWARD AS A GROUP TO BE IN THE LEARNING SPACE BUT ALSO SAYING HERE IS HOW WE'LL APPLY IT IN A CAREFUL WAY. GREAT PANEL. THANK YOU SO MUCH. MY QUESTION -- DONNA SPROUT, UNIVERSITY OF SOUTHERN CALIFORNIA. MY QUESTION IS LOWER LEVEL FOR TANZEEM. I KEEP GOING AROUND IN THIS LIKE MOUSE TRAP THING OF HOW MANY PHYSICAL ACTIVITIES CAN WE IDENTIFY USE MAG SHEEN LEARNING UNTIL THIS FALLS OUT OF OUR BRAIN. THERE'S SO MANY ACTIVITIES. IF YOU SEE STEVE, HE'S GOT A BUNCH TO IDENTIFY NOW, WE CHOSE NINE BECAUSE YOU CAN GET ALL THE VARIOUS ACTIVITIES THAT ONE DOES AS FAR AS ENERGY EXPENDITURE INTO THOSE NINE BUCKETS. BUT FROM -- I'M IN PEDIATRIC OBESITY, IT'S IMPORTANT TO KNOW WHAT THE KIDS ARE DOING MORE THAN ENERGY EXPENDITURE ALTHOUGH I HAVE TO COME BACK TO IT. IT'S WHERE DO YOU STOP IDENTIFYING BEHAVIORS AND GO BACK TO THE ENERGY EXPENDITURE PARADIGM AB HOW THOSE MESH. I DESIGNED A DIFFERENT QUESTION, PULLING PEOPLE OUT OF THE RIVER UPSTREAM BEFORE THEY GET FAT. I THINK THAT'S KIND OF A QUESTION WHICH WE DON'T KNOW THE ANSWER TO EXACTLY BUT THAT'S A VERY IMPORTANT QUESTION BECAUSE IF YOU LOOK AT A LOT OF THESE MONITORING SYSTEMS, THE FIRST ASSUMPTION THEY MAKE WE'RE GOING TO DECK THESE ACTIVITIES AND WHAT WE'RE SEEING IN OUR DEPLOYMENT IS THAT YOU MIGHT SAY GO TO JIM AN WALKING AND RUNNING BUT THE PERSON GOES DANCING EVERY DAY AND IS AN AVID HIKER. SO HOW DO YOU ACTUALLY GIVE THEM CREDIT FOR THAT. ONE THING THAT THERE HAS BEEN SOME ACTIVITY IN THE LEARNING SPACE IS AS YOU DEPLOY THE SYSTEM THAT YOU CAN TAKE CORE ACTIVITY AND THEN EVOLVE THE SYSTEM TO SAY I SEE SOMETHING NEW GOING ON, IF IT HAPPENS FREQUENTLY ENOUGH OR REPEATEDLY ENOUGH DURATION THIS IS A RELEVANT EVENT AND ENGAGE THE USERS TO GIVE A LABEL AND TRY TO BUILD A PERSONALIZED BUCKET OF ACTIVITIES. THERE'S ALSO A TRADE OF H OFF THAT YOU CANNOT HAVE ALL POSSIBLE ACTIVITIES ON THE PHONE BEING INFERRED. SO ONE TRADE OFFS MIGHT BE THAT YOU HAVE THE MOST RELEVANT SUBSET FOR EACH USER. THERE'S CERTAIN THINGS EVERYONE DOES BUT ADAPTIVE SO I THINK A LOT OF THESE SYSTEMS REALLY HAVE TO SIMULTANEOUSLY PERMIZE AND SCALE AND WHAT'S THE MOST EFFICIENT WAY OF DOING THAT IS A CHALLENGE. AND THAT'S SOMETHING THAT COMPUTER SCIENTISTS ARE WORKING ON AND HAVE TO CONTINUE WORKING ON. >> OTHER QUESTIONS? >> THIS IS COMING IN FROM THE WEB. IT SAYS WHEN THINKING ABOUT WHEN TALKING FINGER PRINTS THIS IS A DEVELOPMENT mHEALTH EFFORTS WITH MENTAL HEALTH, THE LACK OF CONSISTENT EARLY OUTCOME MEASURES SO THE PERSON IS ASKING DO WE NEED TO COME UP WITH CONSENSUS, CAN WE BUILD FINGERPRINT LIBRARIES AROUND MENTAL HEALTH ISSUES. GREAT QUESTION, THINGS IN HUMAN LIFE ARE RELATED TO PSYCHOLOGY AND IMPACT OF NERVOUS SYSTEM. MANY EVENTS, MYOCARDIAL INFARCTION HAVE A COMPONENT OF MENTAL STRESS. A LOT OF FINGERPRINTING WILL BE AND SHOULD BE RELATED TO PSYCHOPHYSIOLOGY. TO CHARACTERIZATION OF WHAT THE NERVOUS SYSTEM IS DOING. THIS QUESTION IS STRAIGHT TO THE POINT. WHEN WE DO DEVELOP METHODS WE HAVE TO DESIGN CERTAIN SET OF TEMPLATES. HOW EXACTLY WE GO INTO CHARACTERIZE THE STATES OF MENTAL STRESS. PSYCHOLOGICAL STRESS. OBVIOUSLY THIS QUESTION IS NOT UP KNOWN TO PSYCHOLOGISTS. IT'S A PART OF OUR COLLABORATIVE STUDY THAT OUR COLLEAGUES AT THE YALE UNIVERSITY SCHOOL OF MEDICINE ARE EXPERTS IN. MORE DATA IS NEEDED BECAUSE WE'RE JUST UNDERSTANDING AND CHECKING DATA LABELING IT, CREATING THIS TEMPLATE TO UNDERSTAND WHAT TYPES OF PATTERNS, PSYCHOPHYSIOLOGICAL PATTERNS ON THERE. >> RELATED QUESTION. IT GETS BACK TO THE ISSUE OF SEGMENTATION AN GREG'S ISSUE OF ITERATIVE DESIGN. YOU HAVE TO THINK ABOUT A CRITERION WHICH TO BASE THE SEGMENTATION ON. CAN YOU TALK A LITTLE BIT ABOUT HOW YOU DECIDE UPON THE STANDARD5'h CRITERION, WHAT MAYBE HELPFUL FOR OTHER RESEARCHERS OR FOR THE RESEARCHERS AT LARGE IN THINKING USING CRITERIA TO SEGMENT TO IDENTIFY THE PATTERNS. AND HOW YOU GO ABOUT THAT PROCESS. >> ANOTHER GREAT QUESTION. BEFORE I ANSWER IT GOES BACK TO TANZEEM'S COMMENT, OUR IDEOLOGY IS THE DESIGN OF THIS SEGMENTATION AND DIFFERENT PATTERN RECK ANYTHING TOOLS HAS TO BE OPEN AND FLEXIBLE ENOUGH TO PLAY AROUND WITH THIS. THERE IS NO ONE SIZE FITS ALL SOLUTION. HOW TO ADOPT THE SEGMENTATION, THERE ARE SEVERAL WAYS OF DOING THIS. RIM SLEEP NON-RIM SLEEP AND EXERCISE, PHYSICAL EXERCISE. AND ET CETERA, ET CETERA, THAT WE CAN CHARACTERIZE MANY THINGS AND WE CAN GO FURTHER TOO FOR EXAMPLE SAY LOW, MODERATE AND HIGH PHYSICAL ACTIVITY AND PSYCH LOGIC CHANGES AND WE CAN USE SUBJECTIVE CHANGES, SUBJECTIVE ASSESSMENT AND SAY CHANGES IN HEART RATE, HOW MUCH IN X, YRK, Z ACTIVITIES. THAT'S ONE THING WE CAN DO. SECOND THING, WE CAN HELP THIS A PRIORI LABELING WITH STATISTICAL TOOLS. WE CAN APPLY MEASUREMENTS OF STANDARD DEVIATION, CO-VARIANTS THAT WAS TALKED ABOUT THIS MORNING. WE CAN APPLY MARKUP MODELS. FOR EXAMPLE WE KNOW WAKEFULNESS ALWAYS IS FOLLOWED BY SLEEP. I'M SIMPLIFYING. IT HAS TO BE A COMBINATION OF STATISTICAL METHODS AND A PRIORI INFORMATION THE MOST EFFECTIVE IN TRAINING TO MAKE THIS TOOL SMALL. IT HAS TO BE FLEXIBLE AN OPEN SOURCE. >> WHEN YOU DECIDE PHYSICAL ACTIVITY HOW DO YOU DECIDE THE STANDARD TO THE PATTERNS OF PHYSICAL ACTIVITY? >> YOU NEED THE TRUTH. >> YOU HAVE THE DATA COMING IN AND IT'S EASY TO ANNOTATE FOR THE TRAINING DATA ANNOTATE AS YOU GO ALONG AND THEN YOU SEE IF WHEN YOU DO THE TRAINING THE SIGNALS YOU'RE GETTING AND THE DECISIONS YOU'RE MAKING, DOES THAT MATCH THE ANNOTATIONS AND THEN WHEN YOU GET TO A CERTAIN AMOUNT OF ACCURACY YOU CAN TRY ON NEW DATA AND SEE IF IT FITS. I WAS GOING TO SAY DO WE HAVE AN EASY STANDARD WE OTHER TRYING TO MEET IN TERMS OF PUBLIC HEALTH GUIDELINES. IS IT A CLEAR CRITERIA ON WHAT'S -- WHAT THE LEVEL OF PHYSICAL ACTIVITY SHOULD BE AND A LOT ABOUT THE DETERMINANTS OF PHYSICAL ACTIVITY. SO THE MODEL -- THOSE MODELS ARE THERE TOO. >> QUESTIONS? >> GARRETT MEHL, WHO. FEELING GOOD ABOUT THAT. THANK YOU. I CAN HEAR YOU. SO MY QUESTION IS REALLY RELATED TO I THINK ALL OF THE PRESENTERS DISCUSSED ISSUES OF PATIENTS. TO SOME EXTENT THAT MAY NOT BE COMPLETELY TRUE BUT I'M WONDERING WHEN YOU HAVE SYSTEMS WHERE THE MONITORING IS OF HEALTHCARE PROVIDERS, DIFFERENT COUNTRIES AND DIFFERENT LEVEL OF -- I DON'T THINK THERE'S ANYTHING DIFFERENT HERE, MAYBE THERE IS BUT I'M WONDERING TO WHAT EXTENT THESE KINDS OF SYSTEMS YOU TALKED ABOUT CAN BE ADAPTED FOR REAL TIME DATA COLLECTION HEALTH WORKERS ARE ENGAGED IN AND THE DIFFERENT ACTIVITY THEY'RE DOING. ADS LISTENING AS YOU REGISTER THOSE KINDS OF THINGS, THESE AREN'T BIOMETRIC SENSORS BUT THESE ARE -- THEY'RE REPORTING ON ACTIVITIES THAT THEY'RE ENGAGED IN, AND THIS IS WHAT WE'RE SEEING IN DEVELOPING COUNTRIES, PEOPLE ARE REPORTING ON, HAVING GONE TO A HOUSEHOLD AND REPORTING DATA AND I'M WONDERING TO WHAT EXTENT CAN THESE KINDS OF SYSTEMS BE ADAPTED FOR THAT ENVIRONMENT AS WELL AND THOSE KINDS OF QUESTIONS. >> WE HAVE SYSTEMS TO MONITOR PROVIDERS AN PATIENTS AND HOW DOES HE MAKE DECISIONS AND ASSIGN AND IN FACT WE MONITOR PROVIDERS WITHIN THE SYSTEM AND OPTIMIZE HEALTHCARE FLOW WITHIN THE SYSTEM AS WE WATCH PROVIDERS MOVE THROUGH BOTH OPTIMIZE PROVIDER EFFICIENCY AS WELL AS CONTACT TIME WITH PATIENTS AN MINIMIZE ACTIVITIES THAT AREN'T CONTRIBUTING DIRECTLY TO PATIENT CARE. THE OTHER THING IS THAT IN ORDER TO MAKE THESE DECISIONS WE MONITOR WHAT DECISIONS CLINICIANS MAKE AND USE THAT AS OUR ASSIGNMENT PROCESS. FOR EXAMPLE, WE'LL HAVE A PATTERN OF PATIENT AND CLINICIAN MONITORING AND WE'LL WATCH TO SEE WHAT PHYSICIANS DO IN THE CONTEXT OF THAT PARTICULAR PATTERN AND ASSIGN IT THAT PARTICULAR DECISION-MAKING AND PUT THINGS INTO BUCKETS BASED ON CLINICAL DECISION MAKING AND START TO OPTIMIZE FROM THERE. IN FACT WE USE LEARNING AND ADAPTIVE SYSTEMS TO OPTIMIZE EFFICIENCY IN THE HEALTHCARE SYSTEM AS WELL AS CLINICAL DECISION MAKING EFFICIENCY. >> I WOULD LIKE TO NOA WHAT AMY SAID, IT'S ALREADY IN OUR SYSTEM THERE IT'S ALREADY THERE, IT SHOULD BE AN IMPORTANT PART OF THE SYSTEM. WITH THE HEALTHCARE PROVIDERS CAN DO, THEY CAN LOG TO THE SERVER ANYWHERE IN THE WORLD FROM PERSONAL COMPUTERS AND MOBILE PHONES AND SEE WHAT'S GOING ON, SEE THE PROGRESS, SEE THE DATA. SEE THE ANALYSIS. THEY ALSO HAVE AN OPTION OF MANUALLY TAKING CORRECTIVE ACTION. IT IT SIMPLIFIES IT AND MAKING THEIR WORK MORE EFFICIENT. YOU CANNOT TRACK 24/7, 7 DAYS A WEEK THOUSANDS OF PEES OF INFORMATION SO THE SOFTWARE DOES ANALYSIS AND GIVES IT TO A HEALTHCARE PROVIDER AND SHOWS WHERE IT THINKS THERE ARE CHANGES THAT REQUIRE ATTENTION. IT IS CONFIGURED IN SUCH A WAY THAT IT CAN EVEN SEND A REMINDER OR ALERT TO PHYSICIAN SMART PHONE SAYING THERE IS SOMETHING UNUSUAL GOING ON. THE PATTERN CLEARLY CHANGED AND YOU BETTER LOOK AT THIS RIGHT NOW. THERE IS A LOT OF FLEXIBILITY THAT THE HEALTHCARE PROVIDERS CAN USE THESE TOOLS, OVERRIDE THESE TOOLS, MANUALLY ADJUST THEM AND GET THE INFORMATION ANY TIME FROM ANYWHERE. >> CAN I ADD ONE THING TO THAT? THAT'S RESOURCE-CONSTRAINED ENVIRONMENTS. WE OFTEN SET UP PARAMETERS OF THE RESOURCE-CONSTRAINED ENVIRONMENT TO BEGIN WITH SO THAT WE'RE OPTIMIZING THE SYSTEM WITHIN THAT PARTICULAR SETTING. SO FOR EXAMPLE, IF WE HAVE ONLY GOT A CERTAIN NUMBER OF CLINICAL VISITS OF A COLLAR KIND OF PROVIDER WE SET UP A SYSTEM SAYING WE ONLY HAVE THE ABILITY TO INTERACT WITH THE PSYCHIATRIST AT 15% F TERKSE AND WE CAN'T CREATE OPTIMIZED SYSTEMS WITH MORE PSYCHIATRY TIME WITHIN THEM THEN AT 15% FTE. SOMETIMES WE DESIGN SYSTEMS THAT ARE UNDOABLE. FROM THE BASIC SENSING AND MODELING PERSPECTIVE, THE BEHAVIOR IS USEFUL FOR PATIENTS AS WELL AS HEALTHCARE PROVIDER HOW MUCH TIME THEY SPEND COMMUTING, INTERACTING WITH PATIENTS, THOSE ARE THE GUTS OF THE SYSTEM IS THE SAME. AND COULD BE APPLIED FOR A DIFFERENT POPULATION. SO IT'S GOING TO BE WHAT ARE THE NEEDS AND BEHAVIORED INFERRED BUT FROM THE SENSING AND THE MODELING A LOT OF THE GUTS CAN REMAIN THE SAME. >> QUESTION. >> LET ME GET TO ONE ISSUE WHICH I BELIEVE IS AT THE HEART OF THIS mHEALTH EVIDENCE. BY THE WAY, THIS IS (INAUDIBLE) FROM UNIVERSITY OF MEMPHIS. WE ARE NOW SEEING THAT WE CAN LEAVE SOME DECISIONS TO MOBILE HEALTH SYSTEM. SO AS VLADMIR PRESENTED, YOU COULD HAVE THIS SYSTEM LIGHTED TO SAY DOES THIS SYSTEM NEED ATTENTION. AND LINDA AND SUSAN'S APPROACH THAT YOU COULD HAVE A SYSTEM MAKE SOME DECISION BASED ON ADAPTIVE INTERVENTION BASED ON SOME OUTCOME THAT IS MEASURED. ARE WILLING TO DELEGATE THE SYSTEMS TO THE MOBILE HEALTH SYSTEM IN THE FUTURE. THERE LIES THIS ISSUE OF CONFIDENCE. IN THE TECHNOLOGY WORLD THERE IS A HUGE PUSH RIGHT NOW AND MAKING THIS HIGH LEVEL INFERENCE FROM THIS DATA, WHETHER IT WAS OR ADDICTIVE BEHAVIOR, OR ABOUT THE HEALTH CONDITION. AT WHAT POINT WHAT IS THE RIGHT EVIDENCE THAT WE SHOULD SEEK FROM IN THE HEALTHCARE SYSTEM PURELY RELATED TO OUTCOME MEASURES BECAUSE THAT'S WHAT IS GOING TO BE ONE OF THE FOUNDATIONS FOR MAKING FUTURE DECISIONS THAT WE WILL HAVE THE RIGHT CONFIDENCE THAT YES, WE CAN USE IT IN OUR INTERVENTIONS OR WE CAN USE IT IN OUR FIELD AND WE'RE GIVING IT OUT TO OUR PATIENTS. >> GREAT COMMENT. YOU'RE LOOKING AT AWTD MATED DECISIONS. AND HOW DO WE MITIGATE RISK AND ENSURE SAFETY. COMMENTS. >> WORKING WITH JAVIER, I FELT LIKE WE'RE GOING TO HAVE THIS AUTOMATED SYSTEM AND NOT NEED HEALTH COUNSEL LORES, AS AN ENGINEER YOU KNOW YOU HAVE TO HAVE A GATEWAY FOR THE HUMAN TO GET INVOLVED, AT WHAT POINT YOU NEED TO CONTACT THE PHYSICIAN OR THE PERSON USING THE SYSTEM CAN GET HOLD OF THE DOCTOR, THE HEALTH COUNSELOR. SO THERE NEEDS TO BE THAT BUILT INTO THE SYSTEM THAT THERE IS ACCESS TO THE HUMAN TO MAKE THOSE DECISIONS. >> THIS IS NOT REPLACEMENT OF CLINICAL CARE, IT'S ENHANCEMENT OF CLINICAL CARE. >> AND TRIAGE, MAYBE 80 TO 90% OF THINGS THAT CAN BE AUTOMATED. HAVING THAT ABILITY TO GET TO THAT WHATEVER PERCENTAGE IT IS THAT NEEDS THE HUMAN INVOLVEMENT. >> I THINK AT SOME LEVEL A LOT OF HEALTH TESTS HAVE FALSE POSITIVE AND NECKTIVE AND YOU MAKE DECISIONS BASED ON UNING WHAT ARE THE FALSE POSITIVE, FAS NEGATIVE RIGHTS AND WE SHALL DO A LOT TO HAVE SAME, HUMANS NEED TO BE INVOLVED. BUT QUANTIFY WHAT FOR THIS LEVEL OF ACCURACY, WHAT ARE YOU GOING TO WHAT MISTAKES ARE YOU GOING TO MAKE TO ALLOW YOU TO KIND OF MAKE THE RIGHT JUDGMENT CALL. >> WITHIN THE CONTEXT OF PCORI METHODOLOGY THERE'S AN APPROACH TO STRUCTURE ESSENTIALLY HOW WE'RE GOING TO ASSIGN WHAT LEVEL OF EVIDENCE IS NEEDED FOR WHAT PARTICULAR QUESTION. THIS HITS TO THE HEART OF THE MATTER OF WHAT YOU'RE ASKING AS WELL. IF THE LEVEL OF RISK IS LOW AND THE COST IS LOW. AND COST OF IMPLEMENTATION IS LOW, THERE'S THE THRESHOLD OF DOING IT, APPLYING TOO MANY SYSTEMATIC EVALUATIONS BECOMES A BIT SILLY. ON THE OTHER HAND, IF THE -- THERE IS A HIGH LEVEL RISK IF THE POPULATION HEALTH IMPACT IS HUGE OR IF THERE'S A SUBSTANTIAL COST TO SOCIETY FOR SOME REASON, THE EXPECTATION UPON US TO GET THIS RIGHT IS HERE AND NOW. I WOULD PROPOSE THAT ONE OF THE THINGS THAT YOU MIGHT CONSIDER DOING AS AN AGENDA POINT FROM THIS MEETING, IS COMING UP WITH THE A STRUCTURED PLAN TO HEALTH IMPACT, POPULATION BASED HEALTH, CO., RISK AND HOW YOU WORK YOU WAY THROUGH THOUGH AND TESTING STRATEGY YOU'RE GOING TO USE FOR INTERVENTIONS BASED ON THAT. >> QUESTION IN THE BACK. >> TOM KERSHNER FROM LEGACY. BEFORE WE GET TO QUESTIONS FROM SAFETY AND ABSOLUTE QUESTIONS ABOUT ACCURACY SENSITIVITY, SPECIFICITY, I THINK THERE'S SUBTLE ISSUES THERE AS WELL. PLUG THE mHEALTH INSTITUTE WENDY TOLD ME I HAD TO BUT ONE WE SAW THERE IS A COMING RESOLUTION WHICH IS COMING BIOLOGICAL SENSORS, IT WON'T BE LONG BEFORE YOU PUT ON A PATCH AND YOU KNOW EVERYTHING ABOUT VITAL SIGNS AND ALL THAT ON YOUR PHONE. THE QUESTION I HAVE IS WHEN IS THAT TOO MUCH INFORMATION? AND TO GIVE AN EXAMPLE, THERE'S CURRENTLY I CONTROVERSIAL DEBATE ABOUT CANCER SCREENING. THE HARMS VERSUS THE BENEFITS OF INCREASINGLY EARLY DETECTION OF THINGS THAT WOULD HAVE BEEN BENIGN CANCERS AND THE TREATMENT REJ MEN, THE SCREENING BEGIN, WHAT HAPPENS WHEN ALL THE NEW BABIES HAVE THEIR VITAL SIGNS AROUND THEIR NECK AND THEIR WHOLE LIFE THEY'RE WAITING FOR -- I MEAN JUST TO BE SILLY BUT THEY'RE WAITING TO THEIR HEART ATTACK, THEY'RE WATCHING THEIR CARDIAC READINGS OR WHATEVER IT IS. AT WHAT POINT IS IT TOO MUCH DATA AND ARE WE ALL WAITING -- SITTING AROUND ALMOST BECOMING PARANOID AS FAR AS SELF-MONITORING GOES? I'M -- I BELIEVE IN THIS STUFF SO DON'T GET ME WRONG. >> WHEN IS THE QUANTIFIED SELF TOO MUCH OF A GOOD THING? >> GREAT QUESTION. I THINK THERE ARE TWO DIFFERENT QUESTIONS. IMBEDDED IN IT. ONE IS TECHNOLOGICAL QUESTION THAT WAS SANTOR'S QUESTION. THE SECOND IS BACKGROUND MEDICAL RATIONALE FOR USING TECH NOL OR NOT USING IT IN CERTAIN POPULATION. THERE ARE TWO DIFFERENT ANSWERS. ONE ABOUT TECHNOLOGY. I THINK THERE'S A GOOD SYSTEM THAT HAS BEEN DEVELOPED AND PROVEN BY A LONG TIME OF USING IN DEVICE DEVELOPMENT. IT'S AN INDUSTRY STANDARD USED BY FDA IT HAS SEVERAL WELL DEFINED STEPS, THE FIRST IMPORTANT STEP IS DESIGN REQUIREMENTS, THAT'S THE CRUCIAL STEP WHERE NOT ONLY ENGINEERS DEFINE IT IS DESIGN BUT THEY HAVE TO HAVE CLINICAL INPUT, STATISTICAL INPUT. THEY HAVE TO KNOW WHAT EXACTLY THEY ARE GOING FOR. THEN THERE ARE SEVERAL STEPS INCLUDING DESIGN REVIEW, NOT GOING TO THE LAST STEP TO FIND OUT THAT (INAUDIBLE) IN THE ROOM. BUT TO MAKE SEVERAL CHECK POINTS ALONG THE WAY. THERE IS A LOT OF SENSE IN ADOPTING THIS SYSTEM FOR TECHNOLOGICAL DEVELOPMENT IN MHEALTH AND WHAT WE'RE TALKING TODAY. THE SECOND QUESTION IS MORE ABOUT MEDICAL SCIENCE. IF YOU CAN OR CANNOT USE SOMETHING. I THINK THIS IS LESS DEFINITIVE THERE IS NO DEFINITIVE ANSWER TO THAT ONE. I THINK WE DISCUSSED DURING THIS -- DURING MORNING TODAY A LOT OF DIFFERENT APPROACHES THAT WE RANDOMIZE DESIGN VERSUS NON-RANDOMIZED DESIGN. SO THAT'S -- I THINK THAT GOES INTO THAT DIRECTION. AND WE HAVE TO CONTINUE DISCUSSING THIS. >> CAN I ADD ONE PIECE? THIS IS AN ETHICAL QUESTION AS MUCH AS IT IS ANYTHING. AND SO JUST AS WE NEED TO PUT ON OUR AGENDA HOW TO DEAL WITH TRUST AND HOW WE'RE GOING TO ACCOMPLISH THIS UNEXPECTED SET OF PARAMETERS TO EVALUATE WHAT WE HAVE GOT IN FRONT OF US. WE NEED TO PUT ON THE AGENDA HOW TO DEAL WITH ETHICAL QUESTIONS, AND HOW THEY COME UP TO US BEFORE THEY SMACK US IN THE FACE. ONE THING THAT'S LOOMING IS THE M SHELT PARALLEL TO WHAT'S GOING ON IN GENOMIC ME SIN RIGHT NOW. SO IF WE THINK ABOUT THE WORLD OF GENOMIC SIGNATURES AND WHO OWNS YOUR GENOMIC SIGNATURE AND THE DECISION MAKING THAT'S HAPPENING IN THE COURTS ABOUT OUR GENOMIC SIGNATURES BECAUSE WE HADN'TED A SOCIETY FIGURED HOW TO HAVE THAT DISCUSSION TOO MUCH BEFORE THEN, I THINK REALLY WE'RE GOING HAVE THE SAME ISSUE BECAUSE WE'RE GOING TO HAVE FINGERPRINTS. AND WE'RE GOING HAVE FINGERPRINTS OF PHYSIOLOGY IN FRONT OF US IN A WAY THAT WE REALLY HAVEN'T GARNERED THE CONVERSATION. SO AGAIN, I'M LOOKING TO AGENDA MAKING AND I SUGGEST THIS IS AN IMPORTANT QUESTION. >> FINGERPRINTS IN CONJUNCTION WITH GPS LOCATION OF WHERE YOU ARE AND WHAT YOU'RE DOING SO BRAVE NEW WORLD, YES. IT'S AN IMPORTANT ISSUE. MISHA. >> MISHA PAVEL, NSF. THIS QUESTION IS REALLY VERY CLOSELY RELATED TO PREVIOUS ONE BUT MORE TECHNICAL. SUPPOSE YOU DEVELOP A DETECTOR THAT DETECTS ANOMALIES WITH 98% SPECIFICITY. BUT BECAUSE YOU MONITOR THIS HUGE AMOUNT OF DATA, AND THE PREVALENCE OF THE EVENTS YOU'RE TRYING TO DETECT SO LOW, ONE PERSON, MORE THAN 50% OF YOUR DETECTION WILL BE FALSE ALARMS. IF YOU'RE A PILOT AND I PUT ON YOUR AIRPLANE A DEVICE THAT'S 50% FALSE ALARP, THEY TURN IT OFF. >> THE CLASSIC EXAMPLE WE SEE OF THIS RIGHT NOW IS I TURN OFF MY -- I TURN OFF THE ADVERSE EVENT ALERTS IN MY CLINICAL DECISION SUPPORT SYSTEMS BECAUSE TELLING ME THIS PATIENT IS ON ASPIRIN AND I I'M PRESCRIBING PIE BIEW PRO FEN DOES NOT HELP ME IN THE CARE OF THIS PARTICULAR INDIVIDUAL. THERE'S SO MANY IN FRONT OF ME THAT I DONE HAVE TIME TO DEAL WITH THEM. CLINICS HAVE BEEN TURNING THEM OFF FOR A DECADE. IBUPROFEN I PERSONALLY THINK THIS IS A THING TO GET OUR HEADS AROUND. >> SAME HERE. IN CARDIOLOGY, VIRTUALLY ALL BEDSIDE MONITORS HAVE ALARM SYSTEMS AND VIRTUALLY ALL ARE TURNED OFF. THERE'S SO MANY FALSE ALARMS. BUT IF YOU TALK TO ANY PHYSICIAN USING THEM OR NUR PRACTITIONER THEY WILL TELL YOU THAT WE WANT THIS TECHNOLOGY. WE WANT IT TO BE SMARTER. >> I THINK TO DO THIS, I THINK WHAT I HAVE BEEN TOLD IS THAT THERE'S A BREWING BATTLE BETWEEN PHYSICIANS AND INDUSTRY. WHEREAS FOR THE PAST, WHAT IS IT, TEN, 15 YEARS WE SEE THE PHARMA COMMERCIALS TELLING PEOPLE TO TALK TO THEIR DOCTOR ABOUT A DRUG, IT'S NOT LONG BEFORE WE'RE GOING TO SEE COMMERCIALS SAYING TALK TO YOUR DOCTOR ABOUT THAT SUPER CHEAP CARDIOLOGY APP THEY CAN USE INSTEAD OF $1,500 WORKUP. SO THERE'S GOING TO BE QUESTIONS ABOUT PHYSICIAN CREDIBILITY VERSUS MUCH CHEAPER AUTOMATED SOLUTIONS. SO I DON'T KNOW WHAT THE ANSWER IS BUT TO ME IT SEEMS LIKE IT'S -- WELL, A LITTLE BIT SCARY. >> I'M THE ONCOLOGIST IN THE GROUP, I'LL TELL YOU IT'S TODAY. >> WE SEE THE FUTURE AND THE FUTURE IS TODAY. >> ONE COMMENT I WANTED TO MAKE IN RESPONSE, I THINK IT'S ALSO NOT ONLY JUST AN ALERT BUT THE CONTEXT OF THE ALERT AND WHEN YOU GENERATE THE ALER. ONE OF THE THINGS IN ELDER CARE, IT'S MED COMPLIANCE, THE TIME WHEN YOU ACTUALLY GIVE THE ALERT IS RELEVANT. SO I THINK THERE IS A LOT OF RM FOR INNOVATION NOT ONLY JUST -- EVEN IF YOU'RE 50%, IF YOU DO IT AT THE RIGHT MOMENT OR TIME OR CONTEXT, IT MIGHT BE MUCH MORE EFFECTIVE THAN IF YOU JUST RANDOMLY GENERATE AN ALERT. >> ABSOLUTELY. MY POINT ASKING THAT QUESTION IS I THINK THIS IS ONE OF THE URGENT TOPICS WE NEED TO RESEARCH VERY CAREFULLY. HUMANS DON'T STILL MAKE THOSE ERRORS. CAN WE USE OUR UNDERSTANDING OR OUR COGNITIVE SYSTEMS OF M.D.s AND CLINICIANS TO ACTUALLY ENABLE THE TECHNOLOGY BEHAVE IN A SIMILAR WAY. >> IN RESPONSE TO WHAT YOU MENTION IN RESPONSE TO WHAT YOU MENTION NOW, THAT'S CALLED -- IT'S ALERTS THAT ARE CUT OFF AND UNFORTUNATELY IN CARDIOVASCULAR SETTINGS. FOR EXAMPLE, THE SPES AT SPECIFICITY OF THAT PARTICULAR SYSTEM. >> RICHARD KRAVITZ, UC DAVIS. THE CONVERSATION IS BEGINNING TO MOVE IN THIS DIRECTION. I WANT TO CHANNEL IT FURTHER FOR A MOMENT. BY WAY OF INTRODUCTION, IT'S NOT AN EXTRAORDINARILY HYPERBOLLIC STATEMENT TO SAY IF WE CARE ABOUT THE FUTURE OF THIS COUNTRY, WE SHALL TALK HARDLY ANYTHING ELSE EXCEPT MENDING THE MEDICARE COST CURVE. THAT'S HOW WE'LL GO BANKRUPT IN THE NEXT 25 YEARS. WE HAVE HEARD ABOUT OPERATING WITHIN CONSTRAINTS BUT WHAT WAYS HAVE YOU THOUGHT OF WHICH APPLICATIONS YOU DESCRIBED COULD SAVE THE HEALTHCARE SYSTEM MONEY? >> I WOULD LIKE TO OPEN THAT UP TO THE GROUP BECAUSE IT HAS FASCINATING QUESTION. >> COLUMBIA, UNIVERSITY. GREAT QUESTION. RELATE ODD THE RISK IT WILL RANS THAT YOU WERE SAYING. IF I PLAY WHICH HE IS AGAINST A MACHINE, 90% OF THE TIME THE MACHINE WILL BEAT ME. SO DECISION MAKING THINGS WE'RE CLINICS AND WE HAVE TO ADMIT IT OR YOU PUT THE CLINICIAN AGAINST THE MACHINE THE MACHINE HAS THE SAME EFFECTIVENESS AND IF IT DOES NOT IT'S THE CRITERIA NOT INCORPORATEED IT RIGS AND YOU'LL HAVE 90% WITH THE CLINICIAN, TOO EXPENSIVE TO USE THE MACHINE. SO WE SHOULD SEEK STANDARD OF COMFORT LEVELS IN WHICH WE THINK THE MACHINE CAN DO AS MUCH AS WE CAN, IF NOT WE PASS TO IT THE TECH ANYTHING BUZZ THEY CAN DO SOMETHING EXTRA INCLUDING REDUCED COSTS. >> ADDITIONAL COMMENTS? >> IN OUR RAPID LEARNING SYSTEM ANALYSES, EVERY SINGLE ONE OF OUR SYSTEMS INCLUDES COST AS A VARIABLE WITHIN THE SYSTEM. NOW AT THE MOMENT IT'S NOT BECAUSE WE'RE OPTIMIZING ON COST BUT BECAUSE WE ACTUALLY SHOW BACK TO OUR HEALTH SYSTEM WHAT THE COST OF SOMEONE INTERVENTION STRATEGY IS VERSUS THE OTHER. IN REAL TIME SO THEY CAN SEE WHAT WE'RE DOING AT THE SAME TIME THAT WE CAN SEE IT. BUT WHAT THAT'S DONE IS IMPROVE THE CONVERSATION ABOUT HAVING LINKED COST DATA AVAILABLE TO US WHICH HAD BEEN HISTORICALLY HIGHLY PROPRIETARY INFORMATION, VERY DIFFICULT TO GET. SO NOW BECAUSE OF THE FACT THAT WE HAVE CREATED FEEDBACK LOOPS TO THE HEALTH SYSTEM THAT'S ACTUALLY STARTED TO CREATE THE FEEDBACK LOOP BACK TO THE RAPID LEARNING SYSTEM THAT INDEED HAVING COST DATA IS A OKAY THING. THEN SHOWING HOW WE USE IT IN A RESPONSIBLE MANNER. ONE THING IS TO BUILD IT IN AND CREATE COMMUNICATION LINKS TO YOUR STAKEHOLDERS. THE SECOND IS THAT ONE OF THE BIGGEST ISSUES IS FIGURING OUT HOW TO HONE HEALTHCARE SO THE PEOPLE WHO NEED THE INTERVENTION GET THE INTERVENTION AND THE PEOPLE WHO DON'T NEED INTERVENTION DON'T GET. AND CERTAINLY THAT'S ONE OF THE WAYS IN CANCER BOTH TO REMOVE THE CHALLENGE WHERE WE HAVE GOT LOTS OF PEOPLE TREATED IN WAYS THEY SHOULDN'T BE AND WE'RE NOT NECESSARILY TARGETING THE PEOPLE WHO NEED IT. SO WHAT WE HAVE BEEN TRYING TO DO IS IN A STEP WISE MANNER BOTH SHOW HOW WE HAVE BEEN ABLE TO HONE INTERVENTIONS AS WELL AS COST SAYINGS ALONG THE WAY. KEEP WANTING TO WORK ON THE PROBLEM. >> WE HAVE ACTUALLY TIME FOR ONE BRIEF QUESTION. >> I'LL MAKE A COMMENT, YOU CAN ANALYZE THE BIGGEST PEOPLE THAT ARE COSTING US THE MOST. AND THUS YOU CAN FOCUS ON THOSE TO TRY TO REDUCE THOSE COSTS WHETHER IT BE HEART FAILURE REMISSIONS FOR ME OR OTHER PEOPLE WHO MAKE THE NON-ADHERENCE WITH MEDICATIONS WHO WILL BECOME RESISTANT TO DRUGS OR SUCH SO THAT YOU CAN FOCUS IN AN REDUCE THOSE COSTS. >> THANK YOU. WE'RE RUNNING OUT OF TIME BUT I WANT TO GIVE THE PANELISTS ONE BRIEF COMMENT TO END THE SESSION. SO START WITH VLADMIR. >> THANK YOU. MY COMMENT IS DIRECTLY RELATED TO WHAT YOU JUST SAID AND I WANT TO SAY THAT I AGREE 100%. OUR VISION IS THAT WE NEED TO -- THERE ARE TWO THINGS. ONE IS COST REDUCTION BECAUSE WE WILL REDUCE THE NUMBER OF ADMISSIONS AND READMISSIONS, AND WE CAN DO MUCH MORE WITH PATIENT AT HOME IN TERMS OF MORATORIUM, IN TERM OF TELEMEDICINE. BUT THERE IS A BIGGER PICTURE THAT TRADITIONALLY IN CARDIOLOGY WE HAVE A SUITE OF DIFFERENT DEVICES. ONE FOR (INDISCERNIBLE), ONE FOR EVENT MONITORING, ONE MORE LOOP MONITORING, ANOTHER FOR STRESS TEST. AND ALL OF THESE DEVICES BECOME A CHALLENGE HOW TO GET THE DATA, TALK TO EACH OTHER AND HOW TO GET ALL THIS DATA TO THE CLINICIANS. WHAT OUR SYSTEM DOES IS WE'RE TRYING TO DEVELOP A NEW PARADIGM WHERE IT WAS ONE SIMPLE DEVICE. WHICH IS VERY FLEXIBLE. YOU CAN DO ALL OF THESE THINGS AND YOU CAN MAKE THIS DEVICE REALLY INEXPENSIVE AND YOU CAN MAKE IT PERSONAL. SO THAT ALL THE INFORMATION THAT PATIENT IS COLLECTING OVER VARIETY OF DIFFERENT TESTS OVER HIS OR HER LIFE WILL BE AVAILABLE FOR TRAINING THE SYSTEM AND ALSO FOR THESE CLINICS TO MAKE A BETTER INFORMED DECISION, SO TO OPTIMIZE THE HEALTHCARE. >> THANK YOU, VLADMIR. >> GREG, TWO SECONDS. >> WORKING ON TIME HERE. >> I'M JUST IN A STAGE OF TRYING TO GET A SYSTEM THAT WORKS SO I HAVEN'T THOUGHT ABOUT BIGGER ISSUES OF COST AND ETHICAL ISSUES BUT ALSO SAY THAT I'M HAVE BEEN TRYING TO BE IN THE BUSINESS OF PREVENTION SO REDUCING COSTS IS MORE ABOUT PREVENTION AND HARD TO PUT RETURN ON INVESTMENT. >> TAN SCENE. >> I WOULD LIKE TO ECHO GREG. A BIG OPPORTUNITY IS EARLY DEK AND PREVENTION AND PROVIDING FEEDBACK AND NUDGES TO KEEP PEOPLE IN A HEALTHY LIFE PATTERN SO THAT PEOPLE DON'T NEED TO GO THROUGH EXPENSIVE MEDICAL -- >> I'M GOING TO THROW THE CURVE BALL AT THE END AND SAY THE HIGHLY DATA COMPLEX ISSUE WE NEED TO DEAL WITH WHILE PREVENTION IS IMPORTANT, THE REAL ONE IS MULTI-MORBIDITY. SO 85% OF MEDICARE BENEFICIARIES HAVE FIVE OR MORE DISEASES, THAT'S DEALING WITH THE ISSUE OF FIVER MORE ILLNESSES AT ONE TIME IN ANY ONE INDIVIDUAL. IF WE'RE GOING TO THINK ABOUT COMPLEX DATA SYSTEMS WE HAVE TO WAVE THROUGH, THAT'S WHERE THE LAND MINE IS. >> GREAT, AS WE SAW IN THIS SESSION, THESE DATA MINING DATA AGGREGATION PATTERN RECOGNITIONS, AS THE COMPUTER SCIENTISTS AND THE MULTI-DISCIPLINARY TEAMS DEVELOP THEM, THEY HAVE VERY IMPORTANT IMPLICATIONS FOR POLICY FOR REIMBURSEMENT, FOR SAFETY, FOR MANY THINGS BECAUSE THEY ARE THE BUILDING BLOCKS TO WHICH THE EVIDENCE WILL BE BASED. SO PLEASE GIVE ME A ROUND OF APPLAUSE TO THE PANELISTS. [APPLAUSE] >> OKAY. EVERYONE, WE HAD ORIGINALLY SCHEDULED THE LAST PANEL RIGHT NOW BUT IT LOOKS LIKE EVERYBODY NEEDS A BREAK. YOU HAVE 15 MINUTES, WEBINAR FOLKS WE'RE BACK AT 2:15. DON'T MAKE ME CHASE YOU BACK IN HERE. THOSE THAT NEEDED A CAB, I FORGOT TO ASK YOU WHAT AIRPLANE. TELL XAVIER ON THE WAY OUT PLEASE. SO I THINK WE HAVE A MOUSE KEEPING ANNOUNCEMENT FOR REIMBURSEMENT ANNOUNCEMENT FOR PEOPLE IN THE ROOM? >> AS I SENT MANY MY EMAIL IN CASE YOU KIN CHECK IT YESTERDAY, IF YOU HAVE YOUR REIMBURSEMENT FORMS READY WITH THE RECEIPTS YOU'RE READY TO SIGN IT OFF, I'M READY -- YOU CAN GIVE IT TO ME. YOU CAN PUT EVERYTHING IN THERE AND I CAN TAKE IT. IF NOT YOU CAN CORRESPOND VIA POSTAL MAIL. THANK YOU. >> I NODE NO INTRODUCTION I SUPPOSE. THANK YOU FOR STAYING FOR THIS EXCITING PAN ON INFRASTRUCTURE. IT'S CALLED INFRASTRUCTURE BECAUSE IT COVERS A LARGE AREA. AND THE PERSON WHO IS GOING TO HEAD THIS IS DEBRA US ESTRIN AND I HAVE A GREAT PRIVILEGE TO INTRODUCE HER. SHE IS A PIONEER AND INNOVATOR AND SCIENTIST DEVELOPING INTERESTING REVOLUTIONARY TECHNOLOGY FROM SYSTEMS TO MOBILE TECHNOLOGY. OBVIOUSLY I DON'T HAVE TIME TO TELL YOU ABOUT HER ESPECIALLY HER CAREER STARTED IN THE 7TH GRADE WHEN SHE PROBABLY IRRITATED THE TEACHERS BECAUSE SHE WAS SO SMART AND SCIENTIFICALLY ORIENTED ALREADY THEN. AND DURING THE BRIEF INTERVALS WHEN SHE'S NOT TRAVELING, SHE IS PROFESSOR OF COMPUTER SCIENCE AT THE UNIVERSITY OF CALIFORNIA AND HOLDS A CHAIR IN COMPUTER NETWORKS. SHE WAS FOUNDING -- I'M GETTING THE CUT OFF. DID YOU SEE THAT? THAT'S WHAT I MEAN. ANYWAY, ONE OF THE GREAT ACCOMPLISHMENT WAS SHE IS FOUNDING DIRECTOR OF THE NSF CENTER, $40 MILLION CENTER ON (INAUDIBLE) NETWORK SENSING WHICH STARTED THIS REVOLUTION TEN YEARS AGO. DEBRA. [APPLAUSE] >> I DO. I WON'T DO IT IN FRONT OF THE WEBINAR BUT I WILL TELL YOU MY STORY LATER ABOUT 10TH GRADE STORIES, NOT A 7TH GRADE STORY. SO I HAVE BEEN TRYING TO FIGURE OUT ALL DAY WHY WE WERE CALLED INFRASTRUCTURE INNOVATION AND HOW I COULD TURN THAT TO OUR ADVANTAGE IN TERMS OF DEFIEBING WHAT WE HAVE TO SAY AT THIS LATE DATE. THIS LATE TIME IN THE DAY. AND IT JUST CAME TO ME, I WROTE IT ON MY NOTE PAD. INFRASTRUCTURE IS SHARED. HIGHWAYS, INTERNET, SO I WILL TURN THIS INTO A TAKING WHAT OUR DISCUSSION IS ABOUT ABOUT WHAT WE WANT THE FIELD TO BE ABLE TO DO, WHAT WE WANT NIH TO FUND, WHAT WE WANT KEVIN TO PUBLISH INTO ALSO OPERATE TO MOVE THINGS FORWARD BETTER AND FASTER BY BUILDING MORE EFFECTIVE INFRASTRUCTURE. SO I WAS ASKED TO DO A COUPLE OF THINGS IN MY FEW MINUTES, ONE TO REVIEW WHAT I HAD SAID IN MY POSITION PAPER. I BASICALLY SAID FOUR WORDS. MODULARITY SHARING ANALYTICS AND ITERATION. THAT IS HOW WE CAN BUILD EFFECTIVE INFRASTRUCTURE IN THIS SPACE. WE HAVE BEEN HEARING ALL DAY LONG ABOUT THE NEED FOR ITERATIVE INTERVENTION DESIGN. WE CAN'T GET IT RIGHT ALL AT ONCE, WE'RE NOT GOING TO DO A STUDY DESIGN THAT DECIDES WHAT THE TECHNOLOGY UNDERPINNINGS ARE GOING TO LOOK LIKE. WE NEED TO DEVELOP THAT UNDERPINNENING AN ITERATIVE WAY WITH REAL HEALTH SCIENTISTS AND REAL PEOPLE AND DO IT IN A WAY THAT SERVE MORRIS THAN ONE STUDY AT A TIME. WE WANT TO DO IT IN A WAY THAT COME POANTSZ ARE MODULAR. ONE THING WE HAVE HEARD THROUGHOUT THE DAY IS THE RANGE OF HEALTH CONDITIONS AND DEMOGRAPHICS AND COHORTS TO WHICH THIS CAN APPLY. IN ORDER TO DO A LOT OF SHARED LEARNING AND BUILD A INFRASTRUCTURE THAT'S GOING TO APPLY ACROSS THAT RANGE AND IN A WAY THAT'S COST EFFECTIVE, WE NEED THE COMPONENTS OF THOSE INTERVENTIONS TO BE MODULAR SO THEY CAN BE MIXED AND MATCHED. SO MODULARITY TREMENDOUSLY IMPORTANT. MOD LAYER THE IS THERE SO WE CAN SHARE. WE OFTEN TALK ABOUT SHAIRG DATA WHICH IS TREMENDOUSLY IMPORTANT BUT THAT'S NOT WHAT I'M TALKING ABOUT. I'M TALKING SHARING METHODS AND TOOLS. SOFTWARE AND TECHNIQUES. SO WE WANT THAT MODULARITY SO WE CAN DO THE SHARING. AND THOSE TWO REALLY COME TOGETHER AND WE HAVE VERY IMPORTANT PRECEDENCE OF THAT FOR ALL THIS BY LOOKING AT THE INTERNET AND HOW IT EVOLVED. A THIRD THICK IS WHETHER WE CALL IT ANALYTIC OR OTHER TERMS SIMILARLY TO ALL BEHIND THIS BEING ABLE TO BUILD A SYSTEM, THIS LEARNING HEALTHCARE SYSTEM AND THIS ITERATIVE IMPROVEMENT. BECAUSE THESE INTERVENTIONS ARE ON A DIGITAL PLANE FROM THE EXPERIENCE SAMPLING EMA, OBSERVATION DAILY LIVING, WE HAVE THIS INCREDIBLE OPPORTUNITY TO TAKE THE FACT THAT THIS IS ALL ON A DIGITAL PLANE AND BE CONTINUOUSLY MEASURING AND EVALUATING. SO EVALUATION SUSPECT JUST PART OF SOME STEP IN A PROCESS WHERE YOU EVALUATE YOUR CONTINUOUSLY MEASURING AND IMPROVING. AND THIS WAS DONE LONG BEFORE GOGGLE STARTED IT BUT THESE DAYS GOOGLE IS AN EXAMPLE HOW DO THEY CONTINUOUSLY IMPROVE HOW SEARCH WORKS, BY THAT PROCESS OF CONTINUALLY MEASURING HOW THE SYSTEM IS WORKING AND IMPROVING AND WE JUST HAD A SESSION ABOUT MACHINE LEARNING AND REALLY ALL THOSE ALGORITHMS ARE DESIGNED IN A WAY TO CONTINUOUSLY -- IF YOU TAKE CONTINUOUS MEASUREMENT OFF HOW YOU'RE DOING YOU CAN IMPROVE HOW YOU'RE DOING SO WE HAVE AN OPPORTUNITY THROUGH HAVING THESE DIGITAL MEASUREMENTS OF THE WAY PEOPLE TAKE CARE OF THEMSELVES AN ADHERE OR DON'T AND WHAT WORKS IN TERMS OF SUPPORTING BEHAVIOR CHANGE TO BUILD THE SYSTEM THAT LEVERAGES CONTINUOUS MONITORING. IT ALSO MAKES IT ME SIKER IN TERMS OF NOT JUST A DESIGN SOMETHING DEPLOY IT, EVALUATE IT. ALL THOSE THREE ARE ENTERMIXED. AND SO ANALYTICS AND ITERATION THEN COME TOGETHER JUST AS MODULARITY AND SHARING DO. SO FOR ME IT'S THOSE LESSONS FROM HOW THE INTERNET EVOLVED THE BEST OF THOSE LESSONS THAT COME TOGETHER AND SUPPORTING A WAY TO PURSUE THIS INFRASTRUCTURE TOGETHER IS BY DOING THAT IN A MODULAR SHARED WAY AND TAKING ADVANTAGE OF ANALYTICS TO SUPPORT AND DRIVE OUR ITERATION. THAT'S WHAT I WANTED TO QUICKLY PUT OUT FROM MY POSITION PAPER AND THEN AS I ASK THE REST OF THE PANELISTS I WANT TO DO A DIF WITH WHAT HAS BEEN SAID TODAY AND MENTION A COUPLE OF OTHER THINGS THAT ARE NEW IN THIS. SO, AS BACKGROUND, MANY MECHANISMS WE HAVE BEEN HEARING ABOUT TODAY APPLY ACROSS MOBILE HEALTH APPLICATIONS SO WE SEE NUMBERS OF SYSTEMS COMING UP THAT CAN BE SCRIPTED AND YOU'LL HEAR ABOUT THEM TODAY. TO SUPPORT PROCESSES AND DATA COLLECTION WHETHER EXPERIENCE SAMPLING OR AUTOMATED THAT WORK ACROSS APPLICATIONS. GIVES US A BROADER BASE ON WHICH TO DO THIS SHARED LEARNING. IN THE CONTEXT OF HAVING THESE SCRIPTABLE SYSTEMS THAT WORK ACROSS THAT MOTIVATED US AND A NUMBER OF US ARE IN THE ROOM, MAYBE ONE PERSON TO YOUR TABLES. SECRET EVENT OR PRIZE BUT NOW GOING PUBLIC, THIS INCREASING EFFORT TO GET THE COMMUNITY TO WORK IN AN OPEN SPACE. ALONG THE COLLABORATOR WE OPENED mHEALTH AND THANKS TO SOME CONTRIBUTIONS FROM SOME OF YOU IN THE ROOM WITH FUNDENING PARTICULAR FROM ROBERT WOOD JOHNSON FOUNDATION IN CALIFORNIA HEALTHCARE FOUNDATION BEING INCUBATED BY THE TIDE SENOR WE'RE LAUNCHING A NON-PROFIT CALLED OPEN mHEALTH TO BRING TOGETHER TO THE IMMUNITY THAT SHARED INFRASTRUCTURE COMPONENTS. WE ALL KNOW IN THIS ROOM THAT IT'S NOT JUST A MOBILE APP. IT'S NOT JUST A SCRIPTED SET OF AUTOMATED SMS MESSAGE, IT'S ABOUT THE SENSE MAKING OF THE DATA. WE WANT TO BE ABLE TO BRING THE COMMUNITY TOGETHER ACROSS THE TECHNOLOGY AND METHODOLOGY AND CLINICAL CARE TO REALLY FURTHER THE TECHNIQUES THAT WE HAVE AVAILABLE TO US TO PURSUE THIS ON MULTIPLE P FRONTS. SO LET ME MAKE THAT MORE CONCRETE BY GIVING YOU EXAMPLEPS OF THE SORTS OF OPEN MODULAR WORK WE HOPE THE IMMUNITY WILL PRODUCE TOGETHER, SHARE EARLY AND OFTEN AND SHARING MEANS NOT ONLY PROVIDING OTHER THINGS FOR PEOPLE TO USE BUT ADOPTING WHAT OTHERS PRODUCED. SO WHETHER IT'S DETAILS AROUND USAGE ANALYTIC, WHAT HAS WORKED AND WHAT YOU'RE USENING PARTICULAR, INTERVENTION, WE TALKED EARLIER TODAY ABOUT THINGS LIKE INFORMED CONSENT AND IRB STRAJ JIS, ALSO THAT -- STRATEGIES, THAT LEVEL OF METHODOLOGY AND ALSO ABOUT SHARED LIBRARIES, NOT JUST PUBLICATIONS AS WE SAID EARLIER TODAY. IT'S ABOUT HAVING NOT JUST DATA WE MIGHT END UP SHARING OR PUBLICATIONS BASED ON THAT DATA BUT IN FACT LIBRARIES AND ANALYSIS MODULES THAT HAVE STANDARD APIs THAT CAN BE MADE AVAILABLE AND SHARED ACTIVELY. SO IN THIS OPEN mHEALTH CONTEXT THERE'S SO MUCH TO POTENTIALLY DO, VERY IMPORTANT QUESTION HERE THAT ALWAYS IS IN STUDY DESIGN A RESEARCH DESIGN IS WHERE TO START. THERE ARE A NUMBER OF PEOPLE, YOU'LL BE HEARING ABOUT SOME OF THEM WHO HAVE DONE EXCELLENT CREATIVE AND PRAGMATIC WORK STARTING TO ENABLE NEW KINDS OF DATA COLLECTION SO WE DECIDED THAT REALLY THE MOST EFFECTIVE WAY WAS TO STEP ABOVE THAT AND IN THE THEME OF THIS WORKSHOP AS WELL TO FOCUS ON PROVIDING SHARED MODULAR INFRASTRUCTURE IN WHICH WE CAN START TO SHARE OUR ANALYSIS INFORMATION ANALYSIS PRESENTATION AND REUSE AND COMPONENTS IN THAT CASE. OBVIOUSLY MORE TO BE SAID HERE AND I'M SURE WE CAN HAVE MORE DISCUSSION ABOUT IT IN THE DISCUSSION PERIOD. THE NOTION IS THAT WE SHOULD BE ABLE TO LET PEOPLE DO EXPLORETORY DASH BOARDS, RESEARCH DASH BOARDS BY PULLING TOGETHER AND ANALYZING THESE DATA STREAMS WE HEARD ABOUT IN THE LAST SESSION IN A MYRIAD OF WAYS, SHARING THOSE INTERIM RESULTS MUCH FASTER THAN EVEN THOUGH THREE AND A HALF YEAR CYCLES. THESE WOULD BE COMPONENTS OF THINGS THAT WENT INTO YOUR THREE AND A HALF YEAR STUDIES. BUT WOULD HAPPEN MUCH FASTER. WITH THAT I WANT TO INTRODUCE THE PANEL, AS BEFORE WE'LL OPEN UP TO DISCUSSION. TO DO THAT I CHOSE MY FAVORITE OR MOST REPRESENTATIVE SENTENCE OUT OF THEIR POSITION PAPERS. AND I MIGHT NOT HAVE PERFECTLY CAPTURED THEM HERE BUT IT'S PRETTY CLOSE, I HAD THIS WEIRD PDF FILE THEY WAS TRYING TO COPY FROM. I CHANGED THE ORDER FROM WHAT'S IN YOUR PANEL BUT LET ME GIVE YOU A SENSE OF THE LOGIC. FIRST, WE HAVE BOB EVANS FROM GOOGLE WHO WILL TELL US ABOUT A SYSTEM HE'S BUILDING AND A CONTEXT IN WHICH HE'S GOING TO BE DEPLOYING IT AND I SEE WHAT IS VERY EXCITING ABOUT HIS WORK IN THIS SPACE, HE'S REALLY OPENING UP THE ASKING OF THE QUESTIONS. SO CREATED A SYSTEM WHICH INDIVIDUALS TO ASK QUESTIONS OF OURSELVES, NOT JUST PARTICIPATE, THAT HEALTH SCIENTISTS MIGHT POSE TO THEM. SOMETHING STARTED FROM DATA CAPTURE AN PRACTICE INNOVATION MOVING TO CRAIG WHO WILL TALK ABOUT LARGE SCALE SYSTEMS HE'S STARTING TO DO IN THIS SPACE, I THINK STILL LARGELY TEXT MESSAGING BASED WHEREAS BOB IS SMART PHONE-BASED BUT IN THIS CONTEXT REALLY TAKING USABILITY AND EFFECTIVENESS AND USER RESPONSE TO THESE SYSTEMS IN ADDITION THE MORE TRADITIONAL CONCERNS THAT DRIVE PEOPLE'S STUDY DESIGN. THEN THE SECOND HALF OF THE PANEL GETS MORE TOWARD NEW METHODOLOGIES FOR DOING SOME OF THIS MORE OUTCOMES RESEARCH AND EVALUATION. SO WE'LL HEAR FROM SUSAN ABOUT PRACTICE BASED EVIDENCE WHICH IS A METHODOLOGY THAT SEEMS TO HAVE TRE MENS DO POTENTIAL FOR APPLICABILITY TO THE mHEALTH SYSTEMS WE HAVE BEEN TALKING ABOUT AND FROM GARRETT FROM THE WORLD HEALTH ORGANIZATION, ABOUT ESTABLISHING METRICS THAT WILL HELP US USE THESE METHODOLOGIES, THESE CAPTURE APPROACHES TO SCALE UP MOBILE HEALTH INTERVENTIONS AND NOT JUST DOMESTICALLY BUT GLOBALLY AS WELL. [APPLAUSE] >> EXCELLENT INTRODUCTION AS EXPECTED. FIRST SPEAKER WILL BE BOB EVANS WHO WILL TELL US AB HIS, BOB IS A GOOGLE SOFTWARE ENGINEER. >> THANK YOU. I'M HAPPY TO BE HERE. THANK YOU FOR HAVING ME AND THANK YOU TO THE ORGANIZERS FOR THEIR HARD WORK. THIS IS A PAPER WRITTEN BY TWO OF US. STACY LINDEAU FROM UNIVERSITY OF CHICAGO IS NOT HERE TODAY BUT I AM HERE TO TELL YOU ABOUT IT SO I GIVE HER APOLOGY IN ADVANCE HOW I WILL MANGLE HER PROGRAM. ESSENTIALLY WE WANTED TO LOOK AT HOW TO CREATE TWO PIECES OF INFRASTRUCTURE. I HAD BEEN WORKING ON A SYSTEM -- I WAS AT THE NIH A YEAR AND A HALF AGO TALKING TO NIH ABOUT PROTOTYPES BUT IT'S REPLACED ON SOMETHING WE'RE IT RATING ON GOOG M THE LAST YEAR AND A HALF OR SO, SLIGHTLY LESS, TO BUILD A SYSTEM THAT ALLOWS PEOPLE TO DEPLOY SURVEYS RAPIDLY NOT JUST SURVEYS BUT OTHER SENSORS AS WELL, WHAT SENSORS ARE AVAILABLE ON THE PHONE, OUTBOARD SENSORS IS WHERE WE WANT TO GO, THERE'S STUFF THAT'S EXCITING TO ME THAT EVERYONE ELSE IS TALKING ABOUT LIKE MACHINE LEARNING, THINGS LIKE THAT BUT I NEEDED TO BUILD THIS INFRASTRUCTURE FIRST TO BE ABLE TO DO THOSE THINGS. ALONG THE WAY WHAT I REALIZED WAS I HAVE NO IDEA WHAT MOTIVATES PEOPLE TO CHANGE THEIR BEHAVIOR. I REALLY DON'T. NOT EVEN MYSELF. SO BEING A GOOD COMPUTER SCIENTIST I BELIEVE THE SOLUTION OF ANY PROBLEM IS ANOTHER LAYER OF DIRECTION. I BUILT A SYSTEM TO PEOPLE TO FIGURE OUT TO WRITE THEIR OWN EMPERIMENTS TO FIGURE OUT WHAT MOTIVATES PEEP AND HOPEFULLY WE CAN LEARN FROM THAT ARTIFICIAL INTELLIGENCE BACK TO THE MECHANICAL IDEA, TAKE THE RESEARCH STUDIES AN HA MAKE THE BEST PRACTICE TOOLS AVAILABLE TO LAY PEOPLE AND PRACTITIONERS WHO WANT TO BUILD INSTRUMENTS IN THIS PLATFORM. SO I BUILT IT FOR EXPERIMENTATION IN MOBILE HEALTH AND BEHAVIOR CHANGE. MY COLLEAGUE STACY WHO I MET IN APRIL AT THE END OF THIS US A SPI SHUS MEETING, THE OPEN MHEALTH GROUP, I YELLED THAT I'M READY TO TAKE IT OUTSIDE OF GOOGLE WHERE WE TESTED WITH THOUSANDS OF GOOGLERS AN OVER THE CURSE OF FALL WILL ROLL OUT TO 30,000 GOOGLERS SO WE'LL GET SOME SCALEUP CHALLENGES WITH THIS. BUT I SAID WE'RE LOOKING FOR EXTERNAL TESTER, PEOPLE WHO MIGHT WANT TO TRY THIS OUT AND SHE RAISED HER HAND AND SAY YEAH, WE'RE WORKING WITH SOUTH SIDE OF CHICAGO, WE HAVE 60,000 PEOPLE WE WANT TO TRY IT OUT H WITH. EXCELLENT. I CAN DOUBLE NUMBER BEFORE I GET TO THAT NUMBER THIS FALL. SO AS ALL GREAT THINGS GO, IT HASN'T MOVED QUICKLY SO WE HOPE TO PRESENT RESULTS TO YOU AT THIS POINT. BUT THAT'S NOT GOING TO HAPPEN. WHAT WE CAN TELL YOU IS THE CURRENT STATE. LET ME GIVE YOU MORE BACKGROUND ABOUT THE REST OF THIS. PACO AS I MENTIONED IS BASICALLY IN A NUTSHELL A RAPID DEPLOYMENT ENVIRONMENT. YOU CAN CREATE AN EXPERIMENT IN MINUTES. AND PUBLISH IT THROUGH A SET OF PEOPLE WITH ANDROID PHONES, SORRY iPHONE USERS. IF ANYONE WANTS TO CONTRIBUTE TO THAT, PLEASE CONTACT ME AFTERWARDS. BUT THE IDEA IS THAT I REALLY DON'T KNOW WHAT'S GOING TO WORK SO I WOULD RATHER BE EMPIRICALLY DRIVEN, GET SOMETHING OUT THERE, GET FEEDBACK AND IT RATE. THAT'S MY MOTTO. AND THAT'S WHY THESE SLIDES ARE ILL PREPARED BECAUSE I IT WAS A FEW MINUTES LATE, BECAUSE I HAD ANOTHER PLAN SET UP AFTER BEING HERE ALL DAY. I LIKE TO DO THINGS LAST MINUTE. LATE BINDING, INCORPORATED THE LATEST INFORMATION. SO THE IDEA IS NON-PROGRAMMERS CAN BUILD THESE EXPERIMENTS, DEPLOY THEM TO PHONES EASILY. AND WE'RE VERY CONCERNED ABOUT PRIVACY, IT RUNS OFFLINE ON THE PHONE. YOU CAN GO TO THE DESSERT FOR A WEEK IN SEPTEMBER AND RECORD YOUR OBSERVATIONS AND COME BACK AND CLANDESTINELY WHEN YOU COME ON A NETWORK THEY'RE UP LOADED. THEY'RE THE PICTURE OF THE MEAN SCREEN, THERE'S RUNNING EXPERIMENTS YOU'RE PARTICIPATING IN, FINDING EXPERIMENTS THERE'S EXPLORING YOUR DATA WHICH IS A NEW PIECE, MY INTERN JUST DID THIS SUMMER WHERE EAR STARTING TO TRY TO ANSWER QUESTIONS ABOUT HOW CAN PEOPLE WHO ARE NOT STATISTICIANS INTERACT WITH THE DATA MOST SUCCESSFULLY. AND YOU CAN CREATE YOUR OWN. THIS IS NOT JUST RESEARCHERS, THIS IS FOR INDIVIDUALS, THE QUANTIFIED HELPERS AS AUDIE MENTIONED EARLIER. STAY STACY'S PROGRAM IS THE SOUTH SIDE HEALTH AND VITALITY STUDIES, THE GOAL IS MAKE URBAN SHIG HIG THE MOST HEALTHY URBAN ENVIRONMENT IN THE COUNTRY AND THEY WORKED WITH THE COMMUNITY LEADERS YEARNING FOR TECHNOLOGY. THEY BELIEVE ACCESS TO TECHNOLOGY WILL MAKE THEM MUCH MORE EFFECTIVE IN THEIR COMMUNITY. AND THEY HAVE ALSO BEEN DOING MAPPING OF ALL THE ASSETS IN THE COMMUNITY, THEY HAVE HAD PEOPLE OUT WORKING IN THE IMMUNITY. I DON'T KNOW IF I HAVE A PICTURE. YEAH. SO THEY HIRE PEOPLE TO GOUT AND MAP THE HEALTH ASSETS AND OTHER THINGS IN THE ENVIRONMENT TO BILL UP THIS TEST BED AND WHAT THEY WANT TO BE IS A MEDICAL TESTING INFRASTRUCTURE, THE IMMUNITY LEADERS ARE INTO THIS, THEY WANT IT TO EXIST AND PART OF WHAT WE'RE DOING TO TEST THAT TEST BED ROLL OUT A PROGRAM TOGETHER. RIGHT NOW WHAT WE'RE DOING IS NOT TELLING OUR RESULTS BECAUSE WE'RE GETTING IT TOGETHER. WHAT YOU REALIZE WHEN YOU -- THE REASON I HARP ON HCI, HUMAN COMPUTER INTERACTION IS THAT YOU FIND OUT A LOT WHEN YOU TAKE OUT OF CLOSED ENVIRONMENT SHAPE, A BUNCH OF ENGINEERS AT GOOGLE AND GIVE TO PEOPLE WHO AREN'T NECESSARILY TECH SAVVY AND DO QUANTITY TAI ACTIVE OR QUALITATIVE EXPERIMENTAL OBSERVATION AND REALIZE WOW, THIS BREAKS IN MYRRH RAD WAYS I NEVER IMAGINED. WE'RE DOING STUDIES AND WIRE TWO OR THREE WEEKS AWAY WITH ASSET WORKSERS IN THE FIELD, WE HOPE TO SCALE IT UP TO THE SOUTH SIDE TO HELP THEM BUILD EXPERIMENTS TO MAP THE HEALTH. SO THE REAL IDEA IS TO MAKE IT SO YOU CAN MOVE QUICKLY, IT RATE AND GATHER RESULTS AND MAKE BETTER DECISIONS, EXPERIMENTS, INTERVENTIONS, INPUT MECHANISMS. SINCE I HAVE JUST GOTTEN APPROVE TO OPEN SOURCE PACO THIS SUMMER, WE'RE ABOUT TO DO THAT, WITH A NETWORK EFFECT OTHER PEOPLE CAN ADD NEW INPUT INSTRUMENTS, VISUALIZATION INSTRUMENTS, REASONING MODULES IN THE MIDDLE AND PRODUCE AN OVERALL PLATFORM FOR PEOPLE IN THE RESEARCH COMMUNITY AND AS INDIVIDUALS TO WORK MUCH MORE EFFECTIVELY. THANK YOU VERY MUCH. [APPLAUSE] >> THANK YOU VERY MUCH. THE NEXT SPEAKER IS CRAIG LEFEBVRE, RTI INTERNATIONAL AND PUBLIC HEALTH UNIVERSITY OF FLORIDA. THINK' HE'S GOING THE TALK TO US ABOUT SOCIAL MEDIA, INCLUDING MEDICATION ADHERENCE AND STUFF LIKE THAT. >> THANK YOU FOR INVITING ME TO BE HERE, I FIRST HAVE TO SAY THANKS TO NIH AND NHLBI. MY CAREER STARTED BY BEING A POST-DOCTORAL RESEARCH FELLOW IN BEHAVIORAL MEDICINE, FUNDED BY THE NATIONAL HEART LUNG AND BLOOD INSTITUTE. THE QUESTION IS HOW DO YOU TRAIN PEOPLE IN NEW FIELDS, BEHAVIORAL MEDICINE AT THAT POINT WAS A BRAND NEW FIELD AND WHAT NHLBI DID WAS FUND A NUMBER OF TRAINING AND EDUCATION CENTERS AROUND THE COUNTRY. I HAPPEN TO BE AT THE UNIVERSITY OF PITTSBURG SO AFTER MY Ph.D. I WENT ON AND LEARNED ALL ABOUT BEHAVIORAL MEDICINE, TODAY I'M SURE I WOULD BE LEARNING ABOUT MOBILE HEALTH. AFTER THAT, POST-DOCTORAL POSITION NHLBI CAME TO THE RESCUE AGAIN FUNDING WHAT WAS THE FIRST POPULATION BASED PROGRAMS. I WAS INVOLVED WITH THE PAWTUCKET PROGRAM ONE OF THREE BASED PROGRAMS, CAN YOU CHANGE BEHAVIOR T AT SCALE TO REDUCE HEART DISEASE, MORBIDITY AND MORTALITY. SO BETWEEN MY COLLEAGUES AT STANFORD, UNIVERSITY OF MINNESOTA AND MY COLLEAGUES AT BROWN, WHAT I DID THE NEXT EIGHT YEARS IS BEHAVIORAL MEDICINE AT SCALE. WHAT BROUGHT ME TO MOBILE HEALTH ALL THESE YEARS LATER IS THE SAME IDEA OF HOW DO WE USE THESE TECHNOLOGIES TO GET TO SCALE. ONE CONCERN WE CAME HERE WITH TODAY IS WHAT IS THE ROLE OF RANDOMIZED TRIALS. I CAN'T LET THE DAY GO BY WITHOUT NOTING PAUL MIER PASSED AWAY THIS WEEK. HE WAS THE CHAMPION OF RCTs BACK IN THE 50s AND 60s, SO I SAY DO THE STARS ALIGN OR WHAT DOES THAT MEAN? BUT DON'T LET THAT PASS. IN MY WHITE PAPER I WAS ALSO NOTING THAT YOU THINK ABOUT RCTs THERE'S LOTS OF OTHER EVIDENCE BASED WAYS OF CONSIDERING WHETHER OR NOT WE SHOULD BE USING MOBILE HEALTH. SOME OF THOSE EXAMPLES I USE WERE THINGS CALLED OBJECTIVE PERFORMANCE CRITERIA AND PERFORMANCE GOALS AND WHAT WAS MENTIONED TODAY WAS COMPARATIVE EFFECTIVENESS RESEARCH. WE NEED TO THINK HOW THESE TECHNIQUES ARE USED BY THE FDA TO MAKE DECISIONS ABOUT MEDICAL DIAGNOSTIC TOOLS AND OTHER MEDICAL APPLICATIONS CAN ALSO BE SUBSTITUTED AT TIMES WHEN TERRIFIC QUOTE YOU PULLED OUT OF MY PAPER WHEN RCTs ARE BEING OVERPOWERED TO ANSWER OUR QUESTIONS. LAST COUPLE OF MINUTES I WANT TO INTRODUCE TO YOU SIX Ss FOR THINKING ABOUT EVIDENCE BASED MOBILE HEALTH. THE FIRST S IS THE ONE WE TALKED ABOUT TODAY, THE ONE ABOUT SAFETY. I THINK THE SAFETY RISK COMBINATION HAS TO BE ONE OF THOSE ISSUES THAT IS AT THE FOREFRONT OF WHAT WE'RE DOING. THE SECOND ONE I WANT TO TALK ABOUT, MENTION TO YOU ANYWAY, IS THIS NOTION OF SUSTAINABILITY. SOMEONE WHO WORKS IN -- FANCY ELABORATIVE EVALUATIONS AND THAT'S SUCKED OUT OF THE COMMUNITY AND WE'RE LEFT WITH NO INFRASTRUCTURE. AS WERE THAT'S WHERE THE INFRASTRUCTURE IDEAS ARE ARE HERE. HOW DO WE DESIGN SUSTAINABLE EVALUATIONS IS IMPORTANT FOR US IN MOBILE HEALTH WITH THE MOBILITIES WE HAVE WITH MOBILE HEALTH AND NOT JUST THINK ABOUT SUSTAINABLE INTERVENTIONS. WE'RE BRINGING PARADIGMS INTO THESE NEW TECHNOLOGIESK WHETHER IT'S SOCIAL MEDIA, MOBILE HEALTH, RECEIVERS, MESSAGES, THERE'S AN EXCHANGE THAT GOES ON BETWEEN THE TWOOUS YOU WAS AN NOBODY ELSE. THAT DOES A DISSERVICE TO THE TECHNOLOGY. AND DOES A DISSERVICE TO USERS OF TECHNOLOGY WHO ARE DOING MORE. THE WORD SHARING, THE WORD CONVERSATION IS PART OF WHAT WE TALK ABOUT EVERY DAY AND THAT'S PART OF HOW PEOPLE USE THESE TECHNOLOGIES AND WE KNEE TO BE SENSITIVE TO THAT AS WELL. SOLVING PROBLEMS, ARE WE DESIGNING MOBILE HEALTH INTERVENTIONS THAT SOLVE PROBLEMS IN THEIR DAILY LIVES AND NOT SOLVING OUR PROBLEMS BECAUSE OF HEALTHCARE BAN. WE TALK ABOUT PATIENT CENTER AND PEOPLE FOCUSED. HOW DO WE BRING THAT PERSPECTIVE INTO OUR WORK IS IMPORTANT. I LOVE THE COMMENT THIS MORNING ABOUT WHAT HAPPENS WHEN PATIENTS DECIDE WHAT THE RELEVANT OUTCOMES SHOULD BE. IT'S A REALLY INTERESTING IDEA. FINALLY, THE LAST ONE IS SARASOTA. SARASOTA WHERE I ALSO LIVE IS WHERE WE'RE BUILDING THE INFRASTRUCTURE TO DO COMMUNITY BASED AND COMMUNITY WIDE MOBILE HEALTH PROGRAMS. WE'RE BRINGING LOTS OF PEOPLE IN FROM THE PRIVATE SECTOR RIGHT NOW TO HAVE MEETINGS. WE HAVE A MILLION DOLLARS FROM OUR COUNTY SAYING WE WANT TO INVEST IN MAKING SARASOTA COUNTY THE HEALTHIEST BEST PLACE FOR PEOPLE TO LIVE IN THE COUNTRY. AS WE THINK ABOUT INFRASTRUCTURE, AS WE THINK ABOUT SCALING UP, AS WE THINK ABOUT COMMUNITIES AND POPULATION-BASED BEHAVIOR CHANGE, WHAT WE'RE IN THE PROCESS OF DOING IS SIMILAR TO WHAT'S GOING ON IN SOUTH SIDE OF CHICAGO SAYING HOW DO WE BUILD THOSE LABORATORIES THAT PEOPLE DON'T HAVE TO KEEP REINVENTING WHEELS AS THEY EXPAND EFFORTS IN MOBILE HEALTH. THANK YOU. [APPLAUSE] >> THE NEXT SPEAKER IS SUSAN HORN, SENIOR SCIENTIST AT THE INSTITUTE OF CLINICAL RESEARCH AND VICE PRESIDENT FOR VERGE IN INTERNATIONAL INFORMATION SYSTEM IN AT LAKE CITY. IN SALT LAKE CITY. Q. VERY BRIEFLY, I HAVE BEEN WORKING IN THE LAST 20 YEARS OR SO ON A NEW MODEL FOR TRYING TO GET EVIDENCE IN THE ACTUAL PRACTICE OF CARE. RATHER THAN INFLUENCING WHAT TREATMENTS ARE DONE AN TRYING TO GET PEOPLE TO OBEY CERTAIN PROTOCOLS, WHAT WE HAVE DONE IS WORK WITH FRONT LINE CLINICIANS TO HAVE THEM TELL US AND ALSO THEIR PATIENTS WHAT THEY'RE DOING AT WHAT POINTS IN TIME AND WHAT ARE THE EFFECTS OF THAT. SO THE GOAL IS LONGITUDINALLY TO CAPTURE DIFFERENTIAL OUTCOMES OF THE CARE PROCESSES ASSOCIATED WITH NATURALLY OCCURRING VARIATIONS IN TREATMENTS. ADJUSTING FOR SOCIAL DEMOGRAPHIC FACTORS CO-MORBID CONDITIONS AN CO-OCCURRING TREATMENTS. SO WE CAN DRAW CONCLUSIONS ABOUT THE ASSOCIATIONS BETWEEN TREATMENTS AND OUTCOMES. ESSENTIALLY THE MODEL LOOKS LIKE THIS WE WORK WITH FRONT LINE CLINICIANS FOR THEM TO HELP DEFINE OUTCOMES THAT ARE OF INTEREST TO THEM AND THEN ASK THEM FROM THEIR EXPERIENCES LITERATURE, ET CETERA, WHAT ARE THEY DO, WHAT THEY THINK ARE THE OUTCOMES. THAT'S THE PROCESS BOX AND WHAT MAKES PATIENCE DIFFERENT SO TREATMENTS THEY'RE GIVING ACTUALLY INTERACT DIFFERENTLY DEPENDING UPON WHAT THE PATIENTS ARE. IS PICKING UP PSYCHOSOCIAL DEMOGRAPHIC FACTORS, DISEASES, SIGNS AND SYMPTOMS OF THOSE DISEASES, GENETIC INFORMATION AND NEWER STUDIES AT MULTIPLE POINTS IN TIME. ONE BIG CHALLENGE I HAVE FOUND OVER THE YEARS GETTING PEOPLE TO BELIEVE WHICH TREATMENTS ARE ASSOCIATED WITH BETTER OUTCOMES IS TO BE ABLE TO MAKE SURE WHEN THEY SAY HOW DO YOU KNOW IT'S THE TREATMENT AND NOT X THAT YOU HAVE GOT X MEASURED? AND WHAT I FOUND WAS THAT IF WE KNEW A LOT ABOUT HOW SICK THE PATIENT WAS IN DIFFERENT WAYS THAT PATIENTS THINK OF AND CLINICS THINK OF, THAT COULD HELP AVOID SOME OF THE PEOPLE SAYING I'M NOT SURE I BELIEVE THIS SO THEY WON'T IMPLEMENT THE CHANGE UNLESS YOU CAN CONVINCE THEM THAT IT REALLY COULD APPLY TO THEIR KINDS OF PATIENTS. SO WE DEVELOPED A SYSTEM CALLED THE COMPREHENSIVE SEVERITY INDEX LOOKING AT THE PHYSIOLOGIC COMPLEXITY PRESENTED TO MEDICAL PERSONNEL DUE TO THE EXTENT AND INTERACTION OF DISEASES PUT TOGETHER DISEASE-SPECIFIC SYSTEM THAT HAS HUGE AMOUNTS OF DETAILS IN IT. IT ONLY LOOKS AT PATIENTS' SIGNS AN SYMPTOMS, IT DOESN'T USE TREATMENT ADS CRITERIA BECAUSE WE CAN'T WHAT TREATMENT IS BEST FOR SPECIFIC SIGNS AND SYMPTOMS. IT'S CLINICALLY CREDIBLE, COMPREHENSIVE AND CAN MEASURE HOW SICK PEOPLE ARE AT MULTIPLE POINTS IN TIME TO ADDRESS SELECTION BIAS OR CONFOUNDING BY INDICATION QUESTIONS. IT'S THOUSANDS OF THINGS THAT LOOK LIKE THIS. HAVING THIS AVAILABLE IN EACH PRACTICE BASED EVIDENCE STUDY ALLOWED THE PROJECTS TO MOVE AHEAD MORE QUICKLY BECAUSE IF AT THE BEGINNING OF EVERY STUDY YOU HAD TO ASK WHAT ROLE THE POSSIBLE PRIMARY DISEASES AN SECONDARY DIAGNOSES PEOPLE COULD HAVE AND START AT THE BEGINNING OF THAT, THAT TAKE AS LONG PROCESS TO DO. THIS ALLOWS IT TO SPEED UP CONSIDERABLY. ANOTHER BIG DIFFERENCE IS WHO LEADS THEM. IT'S THE FRONT LINE CLINICIANS AN PATIENTS, RATHER THAN THE RESEARCHER. SO WE ASK CLINICS FROM ACROSS THE COUNTRY BECAUSE WE FIND HUGE VARIATION IN TERMS OF PRACTICE AND WHAT PEOPLE ARE DOING IN DIFFERENT PARTS OF THE COUNTRY, TO TELL US WHAT DO THEY THINK ARE THE PATIENT FEATURE, THE TREATMENTS, AND OUTCOMES OF INTEREST TO THEM. WE AGREE UPON DEFINITIONS, I KNOW THAT SOUNDS LIKE IT MIGHT BE IMPOSSIBLE TO DO BUT ACTUALLY WE FOUND THAT ONCE YOU GET PEOPLE TALKING TO EACH OTHER, THEY CAN FIGURE OUT HOW TO STANDARDIZE THEIR DOCUMENTATION AND DEFINE WHAT THEIR REALLY TALKING ABOUT IN TERMS OF WHAT THEIR PATIENTS LOOK LIKE AND WHAT TREATMENTS THEY'RE DOING. WE PILOT TEST OF COURSE THE DATA COLLECTION, DECIDE ON A LAY-OUT SO IT'S VERY EASYTOR PEOPLE TO COLLECT THESE DATA AND THE ACTUAL PRACTICE OF CARE SO IT'S NOT AN ADDON. IT'S ACTUALLY PART OF ROUTINE CLINICAL DOCUMENTATION THAT WE CAN THEN USE ALSO FOR RESEARCH. SO THE SIGNATURE FEATURES OF THIS -- THESE APPROACHES ARE THAT THE HYPOTHESES CAN BE FOCUSED OR BROAD. ALL INTERVENTIONS ARE CONSIDERED TO DETERMINE THE RELATIVE CONTRIBUTIONS OF THEM. THEY'RE VERY BROAD PATIENT SELECTION CRITERIA TO MAXIMIZE THE GENERALIZABILITY. ABLE TO GET DETAILED CHARACTERIZATIONS OF THE PATIENTS AND CLINICAL FEATURES, PATIENT DIFFERENCES AND GET CONTROLLED STATISTICALLY RATHER THAN THROUGH RANDOMIZATION THAT RESULTED IN FASTER CLINICAL BUY IN AN IMPLEMENTATION BECAUSE CLINICS HAVE BEEN THE ONES THAT PUT THIS TOGETHER. THEY ASK THE QUESTIONS AND SAY I'M GOING TO TRY THAT. I THINK IT WILL WORK. QUITE A FEW HAVE BEEN STUDIED IN THESE AREAS -- CHANGES IN RECOMMENDATIONS OF VARIOUS TREATMENT PROCESSES IN DIFFERENT POINTS IN TIME. THANK YOU. [APPLAUSE] >> THANK YOU VERY MUCH, LAST BUT NOT LEAST, GARRETT MEHL IS A SCIENTIST FROM DEPARTMENT OF REPRODUCTIVE HEALTH AND RESEARCH IN THE WORLD HEALTH ORGANIZATION. HE'S GOING TO TELL US ABOUT EXCITING WORK COMBINING OPERATION RESEARCH AND SOCIAL SCIENCE. >> THANKS VERY MUCH. I WANT TO ACKNOWLEDGE MY COLLABORATORS ON THIS WHITE PAPER WE SUBMITTED. BOTH IN THE DEPARTMENT OF REPRODUCTIVE HEALTH AND RESEARCH WHERE I'M BASED IN GENEVA AS WELL AS COLLABORATORS AT GEORGE WASHINGTON UNIVERSITY EARTH INSTITUTE AND HEALTH ALLIANCE. I'M GOING TALK ABOUT A PROCESS THAT WE ARE -- WE HAVE UNDERWAY WHERE WE RECOGNIZE A NEED, IT'S INTERESTING BECAUSE A LOT OF TALK THE WHOLE DAY IS VERY MUCH ABOUT SOME OF THE HIGHLIGHTING SOME OF THE GAPS AND ISSUES THAT WE WERE ALSO SEEING AND WE FELT THAT THERE WAS VALUE IN BEGINNING TO DEVELOP A PROCESS. SORRY, THIS IS TO DETERMINE WHAT KINDS OF EVIDENCE WOULD BE VALUABLE TO WHO AND TO OUR STAKEHOLDERS WHICH ARE THE MINISTRIES OF HEALTH AROUND THE WORLD. THERE'S A CERTAIN THRESHOLD OF EVIDENCE TYPICALLY TRADITIONALLY CONSIDERED WARRANTED TO MAKE RECOMMENDATIONS TO COUNTRIES. WE DIDN'T SEE THAT BECOMING AVAILABLE YET. AND WE WERE CONCERNED BY THAT. WE FELT IT CRITICAL TO BEGIN A PROCESS, HAPPEN HAZARD AS IT MAYBE TO DEVELOP A COMMON SET OF EVALUATION STANDARDS. APPROPRIATE FOR ANY STUDY DESIGN TO MONITOR PROCESS EFFECTS, COSTS, AND ASSESS THE IMPACT HEALTH INTERVENTION PERFORMANCE AN OUTCOMES. THAT WOULD ENABLE AN FACILITATE UNIQUE COMPARISONS AND WOULD PROVIDE SOME LEVEL OF INFORMATION TO INFORM SCALE OF PROCESSES. AS PART OF THE PROCESS LAST YEAR WE CONDUCTED A GLOBAL SURVEY AMONG mHEALTH PROJECT MANAGERS, NOT GOVERNMENT BUT THE MANAGERS THEMSELVES TO BETTER UNDERSTAND THE STATE OF MHEALTH EVALUATION AND RESEARCH AMONG ONGOING PROJECTS AND WE ALSO ENGAGED STAKEHOLDERS INCLUDING mHEALTH PROJECTS IN MEETINGS IN A NUMBER OF GLOBE -- COUPLE OF GLOBAL CONFERENCES TO BETTER UNDERSTAND HOW mHEALTH CONSTITUENCIES WERE VIEWING AND OPERATIONALLIZE UING THE ROLE OF EVIDENCE AND RESEARCH. WE LEARNED OF THESE TWO -- I WANT TO -- NEXT TIME I WANT TO FIGURE OUT HOW TO GET A MACK TO WORK IN THIS. LOOK AT THAT. LET'S GO BACK TO THIS. THAT DIDN'T COME OUT AS IT SHOULD HAVE. MAYBE I'LL TRY TO -- [LAUGHTER] I SAW DEBRA USING A PDF AND THOUGHT THAT'S THE WAY YOU DO IT. WHAT WE DID FIND, WE HAD IT IN OUR SURVEY, WE HAD 70 PROJECTS FROM OVER 45 DIFFERENT ORGANIZATIONS. AND OVER 65% OF THE PROJECTS SURVEYED WERE LESS THAN TWO YEARS OF AGE. AND VERY FEW OF THE PROJECTS WERE REPORTING mHEALTH SOLUTIONS BEING DELIVERED AT SCALE. OVER 80% WERE NOT SCALED UP AND WERE AT BEST PILOT PROJECTS. IN THE SURVEY WE ASKED A SERIES OF QUESTIONS TO BETTER UNDERSTAND THE RIGOR OF THE RESEARCH DESIGN EMPLOYED IN THE VALUATION OF EACH PROJECTS AND GIVEN THAT WHO GUIDELINES ARE BASED ON EVIDENCE, AND CERTAIN THRESHOLDS, IT'S CRITICAL FOR US TO UNDERSTAND HOW WELL THESE PROJECTS ARE GENERATING EVIDENCE. SO WE CREATED AN INDEX BASED ON CRITERIA OF RANDOMIZATION, OF MATCHED OF CONTROL GROUPS, OF NUMBER OF OTHER FAIRLY CRUDE INDICATORS OF RIGOR AND WE COMPARED THOSE PLANNING TO SCALE AND THOSE NOT LANKING TO SCALE AND FOUND INDEX OF 8 THAT REALLY THERE WAS IMPROVEMENT AROUND RIGOR IF USING TRADITIONAL CRITERIA OF RIGOR OF EVIDENCE. WE ENABLED TO PROVIDE US INFORMATION ABOUT THE ASSISTANCE THEY MIGHT NEED TO PROJECTS AND PARTICIPANTS NOTED WHILE WE HAVE BEEN COLLECTING DATA THE PAST THREE YEARS THEY FEEL THEY DON'T HAVE CAPACITY OR GUIDANCE TO KNOW HOW TO ANALYZE THE DATA. THEY NEED GUIDANCE ON WHAT TO DO WITH IT. THEY FELT A NEED TO LEARN FROM WHAT OTHERS ARE DOING AND COMPARE BETWEEN PROJECTS AND ALONG SIMILAR OUTCOME MEASURES AS WELL AS PROCESS INDICATORS. SO IN GENERAL THEY WANTED TO KNOW HOW TO ASSESS THE IMPACT OF THEIR TOOLS. THIS WAS A FAIRLY SOPHISTICATED GROUP OF PEOPLE WHO SUBMITTED. SO THOSE ARE SELF-SELECTED PROJECTS WHO SAID WE WOULD LIKE TO SUBMIT OUR PROJECT TO THE SURVEY. I HIGHLIGHT AS NEED NOT ONLY IN THE SAMPLE THAT WE DISCOVERED BUT GROWING CONSENSUS IN THIS ROOM AND ELSEWHERE CLEARLY AMONG THE ORGANIZERS OF THE MEETING A NEED FOR BETTER EVIDENCE OR BETTER UNDERSTANDING HOW EVIDENCE SHOULD BE GENERATED. AND SO WE FELT THAT THERE SHOULD BE STANDARDIZATION AROUND CERTAIN KINDS OF THINGS ACROSS DIFFERENT STUDY DESIGNS, THAT IS ONE WAY WE COULD TACKLE THIS. SO THAT MIGHT BE STANDARDIZED METRICS FOR MONITORING, AS WELL AS MINIMUM STAN STANDARDS FOR EVALUATION AN IMPACT ASSESSMENT. WITHOUT THESE GUIDELINES PROJECTS WOULD CONTINUE TO COLLECT DATA AND MAY NEVER PROCEED TO ANALYZE A REPORT ON IT AND IT MAYBE NEVER BE AT A LEVEL TO INFORM DECISION MAKERS. AND SO WE FELT THERE WERE BASED ON FINDINGS OF THE SURVEY AND THE STAKEHOLDER MEETINGS THAT WE HAD, THERE WERE A NUMBER OF KINDS OF ISSUES THAT SHOULD BE INCLUDED IN ANY KIND OF TOOL OR GUIDANCE IN THIS AREA. THIS IS NOT EXHAUSTIVE BUT IT BEGINS TO GET AT SOME OF THE THINGS THAT AT LEAST WE WOULD BE LOOKING FOR. IT REALLY -- THE WHITE PAPER ITSELF FOCUSED ON THE COMMON METRICS ACROSS STUDY DESIGNS TO FACILITATE UNIQUE COMPARISONS BUT THERE ARE OTHER ISSUES IN THERE AS WELL. WE WOULD BE INTERESTED TO KNOW THE KINDS OF QUESTIONS OF HOW DIFFERENT PROJECTS ARE FACILITATING AN IMPACT ON THE MDGs AND THAT THEY SHOULD BE ACROSS THE PROCESS INPUT AFFECTS AND OUTCOMES. THE -- WE'RE CURRENTLY ABOUT TO ENGAGE IN A COUPLE OF ADDITIONAL MEETINGS TO BEGIN TO WE HAVE A MEETING WE'RE ORGANIZING WITH COLLEAGUES AT HARVARD TO -- ON eHEALTH. I THINK THOSE OF US IN THE ROOM NOTE WHILE mHEALTH IS WITHIN THE eHEALTH ECOSYSTEM, THEY'RE UNIQUE CHARACTERISTICS IN MHEALTH THAT I THINK WE'RE ALL ACKNOWLEDGING HERE. BUT THAT THERE IS A NEED TO BEGIN TO THINK THROUGH HOW THAT FITS INTO THE LARGERE HEALTH EVALUATION. IT'S GOING TO BE A COMPLEX MEETING BUT ONE THAT WE'RE LOOKING FORWARD TO IN SEPTEMBER. AND WE'RE ALSO ENGAGED WITH A NUMBER OF WELL -- WILL BE ENGAGED WITH A NUMBER OF PROEXPRECTS TO LOOK AT EVALUATION STRAT T JIS FOR MULTIPLE PROJECTS PART OF THE UN INNOVATION WORKING GROUP. THESE ARE PROJECTS TRYING TO GO TO SCALE. WE WANT TO FIND THE COMMONALITIES IN AN EVALUATION TO HELP THEM GET THE SCALE AND USE INFORMATION TO INFORM THAT SCALE UP PROCESS. AND THAT'S -- ANYWAY, THAT WILL COME BACK TO A MEETING SOMETIME LATER EARLY NEXT YEAR THAT WILL BEGIN TO SHARE EXPERIENCES OF THE EVALUATION FRAMEWORKS AND STRATEGIES. [APPLAUSE] >> THANKS VERY MUCH TO THE ORGANIZERS. >> VERY EXCITING BUT WE DO NEED TO KEEP ON TIME. I WOULD LIKE TO ASK DEBRA TO TAKE OVER FOR A MOMENT TO SUMMARIZE THE ISSUES AND LEAD THE DISCUSSION. (OFF MIC) >> I DON'T THINK WE NEED SUMMARY RIGHT NOW, IT -- YOU SORT OF HEARD SOME OF WHERE WE COME FROM AND BRING TO THIS QUESTION, I THINK WE SHOULD TRY TO FOCUS OUR DISCUSSION AROUND INFRASTRUCTURE. SO WHAT CAN WE BRING OUT OF THESE METHODS FROM EVALUATION TO EVIDENCE GENERATION TO EXPLORE TORE DATA COLLECTION WHAT CAN WE BUILD AS PART OF A SHARED INFRASTRUCTURE OR SHOULD WE BUILD AS PART OF A SHARED INFRASTRUCTURE SO I WOULD LIKE TO TRY TO FOCUS OUR DISCUSSIONS IN THAT DIRECTION. THE PANEL JUST SPOKE SO WHAT I WOULD PARTICULARLY LIKE TO DO IS ASK ANY OF THOSE PEOPLE LISTED AS ADDITIONAL WHITE PAPER AUTHORS FIRST IF YOU WOULD LIKE TO CHIME IN ABOUT INFRASTRUCTURE SO JANET, STEVEN, IS HE HERE? NO. CHARLENE AND IDA AND DONNA, IN PARTICULAR ON THIS QUESTION OF WHAT SHOULD WE BE BUILDING INTO AN INFRASTRUCTURE HOW SHOULD WE BE DOING THAT. THEN OPEN IT UP TO OTHERS AS WELL. >> EXCELLENT SUMMARY. >> BUT I'M NOT IN CONTROL OF THE -- [LAUGHTER] >> THANKS FOR A GREAT PANEL. I THINK THE ONE THING I WOULD LIKE TO SAY IS I REALLY APPLAUD DEBRA'S AND IDA'S EFFORT TO WHAT WE NEED RIGHT NOW TO SHARE ARE THE GUTS OF THE THINGS WE DEVELOP WITH ONE ANOTHER. WE DO HAVE TO REINVENT THE WHEEL ANY NUMBER OF TIMES BUT WE SHOULD BE DOING IT TOGETHER. WE'RE NOT FAR ENOUGH ALONG, I FEEL WE NEED TO BE VERY CAUTIOUS IN DOING WHAT YOU SUGGESTED. I DON'T THINK WE'RE READY TO BUILD SHARED METRICS. I DON'T THINK WE'RE -- I THINK WE'RE READY TO SHARE THE BUILDING OF METRICS AND I THINK WE NEED TO BE VERY CAREFUL. >> THAT LEFT THE ROOM SILENT. >> BOB SAID HE HAD AN APP USED TO DESIGN TREATMENT. I DON'T GET IT. I DON'T UNDERSTAND HOW TO USE AN APP TO DESIGN A TREATMENT. THIS IS PART OF BUILDING THAT INFRASTRUCTURE. I DON'T UNDERSTAND HOW YOU CAN USE AN APP TO DESIGN A TREATMENT. IS THAT OKAY IF HE ANSWERS THAT A LITTLE BIT? >> OF COURSE. >> EVEN OUT OF ORDER BUT I WAS GOING TO GET THE THAT. >> IF HE DOESN'T ANSWER SUCCINCTLY I'LL ANSWER. >> IT JUST WORKS. NO. [LAUGHTER] >> TRUST HIM HE'S FROM GOOGLE. (OFF MIC) >> I JUST LEAVE IT ON. SO FOR LIMITED SET OF TREATMENTS NOW, YES. MOSTLY SELF-MONITORING AND SOME MINIMAL FEEDBACK PREDICATED ON THE INPUT SO YOU COULD BUILD PREDICATE RULES FOR THEIR RESPONSES OR THIS SENSOR DATA YOU CAN COLLECT TO OFFER THEM ENCOURAGEMENT OR OTHER OPPORTUNITIES OR STRATEGIES FOR DEALING WITH THE SITUATION. THAT STUFF WE CAN DO NOW BUT THERE'S MANY MORE WE CAN DO THAT ARE MORE ADVANCED SO THAT'S WHY I REACHED OUT. RIGHT NOW WHAT YOU DO, YOU Z AN EXPERIMENT CREATOR HAVE A WEBSITE WHERE YOU FILL OUT A BUNCH OF FORMS INCLUDING STUFF WHERE YOU CAN PUT IN PREDICATE LOGIC TO DO MORE ADVANCED INTERVENTION, YOU SAVE IT, YOU DECIDE WHO BY EMAIL ADDRESS YOU'LL DEPLOY IT TO, THEY REFRESH ONCE THEY HAVE THE APP AN IT DOWNLOADS INTO THE PHONE, THEY CAN OPT TO JOIN THEM OR NOT. >> MOBILE SURVEY MONKEY. Q. DPEPT OPEN ENDED ON BOTH SIDES. SO YOU CAN CONTROL YOURSELF. (OFF MIC) >> IF YOU WANT TO FOLLOW ON THAT SAME TOPIC AS OPPOSED TO BRINGING SOMETHING NEW, RAISE BOTH HANDS. IF YOU WANT TO START A NEW TOPIC WE'LL JUST DO A COUPLE OF BOTH HANDS SO WE HAVE A LITTLE BIT OF, YES. >> THIS IS NEW. THIS IS WILL BACCUS FROM NSF. WONDERING BEYOND NOT BEING EVIL OR DING SOMETHING OUT OF GOODNESS OF YOUR HEART HOW DO WE INCENTIVIZE PEOPLE TO SHARE THEIR TOOLS? WHAT'S THE BUSINESS MODEL? >> MY PERSONAL PERSPECTIVE ON THAT IS I GOT IT OPEN SOURCE BECAUSE NOT SURE HOW IT ALIGNS WITH GOOGLE'S BUSINESS MODEL. BECAUSE I ALSO BELIEVE THE USER SHOULD HAVE COMPLETE CONTROL OVER THE DATA, SHOULD BE COMPLETELY PRIVATE AND SHAI THEY HAVE DECIDE IF THEY WANT TO SHARE IT OR NOT. I'M A NUT AND IT IT TO BE MORE PRIVATE. THE BUSINESS MODEL GETS TRICKY. I HAVE HAIR BRAINED SCHEMES I'M JUST NOW COOKING UP THAT I THINK COULD BE GOOD WAYS TO INCENTIVIZE PEOPLE TO PARTICIPATE. THAT SAID, WHO KNOWS IF THEY'RE ANY GOOD. >> WELL, YOU KNOW, THERE MANY STAGES AT WHICH YOU CAN ASK THAT QUESTION. AS RESEARCHER AN INNOVATORS ALL OF US IN THE ROOM TRYING TO CREATE NEW METHODS AND NEW DESIGNS IN SCIENCE AND ENGINEER ING. YES, THERE ARE INCENTIVES FROM KEEPING THINGS CLOSED. IN THE BIG PICTURE BY SHARING AND HAVING OUR STUFF USED, IT'S ABOUT CITATIONS, NOT PUBLICATION, THE SAME NOTION. I THINK FUNDING AND TENURE PROCESSES AND ALL THAT CAN CONTRIBUTE TO HELP TO REALIGN PEOPLEK TO GET INCENTIVES RIGHT TO PRODUCE THINGS THAT GET USED AND NOT OVERLY FOCUS ON THINGS THAT ARE AT THE RESULT OF SOME LONGER SILOED PROCESS. SO IN THE EARLY STAGES WHERE WE'RE EXPLORING AND DEVELOPING, WHAT INCENTIVIZES US AN PEOPLE ON THE WEBINAR TO SHARE I THINK IS A SOMEWHAT EASIER PROBLEM, IT DOESN'T HAVE TO BE EVERYBODY, YOU JUST NEED THE START TO GET CRITICAL MASS SO THAT WHAT IS AVAILABLE IN THE SHARED AND OPEN SPACE BECOMES THE MORE EFFECTIVE WAY TO GET FARTHER FASTER. OTHER OPEN COMMUNITIES, YOU HAVE REALLY ACTIVE OPEN SOURCE COMMUNITIES, ICE NOT SOME RANDOM RESEARCH GROUP THAT SAYS OUR STUFF IS OPEN SOURCE. SO WE OPEN SOURCE OUR STUFF BUT DOESN'T MAKE US AN OPEN SOURCE COMMUNITY. DEBRA WITH UCLA. WHAT'S GREAT AS OPEN SOURCE COMMUNITY PEOPLE ARE DEVELOPING, PUTTING THINGS IN AND TAKING THEM OUT. THEY'RE NOT JUST SAYING I'M OPEN SOURCE AND IF YOU USE MY SYSTEM THEN YOU'RE PART OF AN OPEN PROJECT. YOU START TO THEN SEE THAT THE TOOLS AND THE MODULES AND THINGS IN THAT OPEN SPACE HAVE MORE VALIDATION THAN ANYTHING YOU CAN DO ON YOUR OWN BECAUSE OF ECONOMY OF SCALE. SO IF ENOUGH PEOPLE GET THE CRITICAL MASS YOU START HAVING IT BE THAT THAT OPEN THING IS THE WAY TO DO THE MORE EXCITING PROJECT FASTER, IT'S GOING TO BE MORE COMPETITIVE WHEN YOU GO TO GET RESEARCH FUNDING OR COMPETING FOR A PROJECT. IF YOU GO THEN TO A COMMERCIAL QUESTION STARTING A COMPANY AND NOT DOING A RESEARCH PROJECT, AND YEARS AGO I DON'T REMEMBER WHAT THE VALUE OF N IS, NOBODY THOUGHT COMMERCIAL ENTERPRISES WOULD USE OPEN SOURCE AS PART OF WHAT THEY BUILD. NOW THEY DO, IT DOESN'T MEAN YOU OPEN EVERYTHING. YOU STILL HAVE SECRET SAUCE BUT YOU PUSH THAT SECRET SAUCE AND STUFF YOU DO SPECIALLY MORE TOWARDS THE EDGE OF THE SYSTEM AND THE COMPONENTS YOU'RE BUILDING IT ON, YOU'RE NOT RESPONSIBLE FOR MAINTAINING THEM AND KEEPING THEM UP TO DATE. IT'S A MORE ECONOMIC WAY TO GO. ONCE YOU START INTO THAT MODE, IT IS SUSTAINING TO HAVE THINGS MORE ECONOMICAL AN MORE ROBUST WHEN DONE IN THE OPEN. >> I WOULD LIKE TO ADD SOMETHING ABOUT GENERAL APPROACH BENEFITS AND COST ANALYSIS. YOU CAN BUILD FANTASTIC SYSTEMS THAT ARE NOT BENEFICIAL TO PEOPLE, NOR DEVELOPERS AND THEY WON'T GET USED. SO THIS IS THE MAIN ISSUE THAT WE NEED TO FOCUS ON. >> THE DIFFERENCE BETWEEN EHEALTH AND mHEALTH IS THERE'S CERTAIN YOU CANNOT AVOID, IF YOU HAVE A MILLION USE OF TEXT MESSAGES A DAY, THAT'S $10,020,000 A MONTH. OPEN SOURCE LIKE MOZILLA OR FIRE FOX YOU CAN DO OPEN SOURCE PEOPLE JUST IMPROVE IT AND THERE IS NO UNDERLYING COST OF MAINTENANCE BUT THE PROBLEM WITH mHEALTH IS YOU HAVE THIS MAINTENANCE THAT WHEN YOU PILE UP A MILLION USERS WHO IS PAYING FOR THAT. SO YOU NEED THE BUSINESS MODEL BEYOND JUST OPEN SOURCE. SO WE SHOULD BE THINKING ABOUT HOW WE COME WITH SOLUTIONS TO SOLVE THIS PROBLEM BECAUSE I HAVE BEEN TRYING TO COME UP. >> AN ISSUE OF USAGE COST. >> A HUGE MAINTENANCE COST TO KEEP SERVERS TO SECURE ALL THIS INFORMATION. SO IT'S NOT THE SAME AS A PIECE OF SOFTWARE AND PEOPLE CONTRIBUTE. WHEN YOU HAVE 10 MILLION USERS IT'S LIKE YOU CANNOT JUST DO IT FROM A UNIVERSITY, YOU HAVE TO UP A REVENUE -- HAVE A REVENUE SOURCE THAT COMPENSATES FOR THAT CASH OUTFLOW. >> I HAD A SIMILAR COMMENT AND DATA SHARING AND PRIVACY ISSUES. NOTE ONLY COST SERVER, ET CETERA, BUT WHERE IS THAT STREAM OF DATA GOING? IF YOU DON'T HAVE -- >> SOMETIMES ALL WE DO IS CREATE A LIST OF ISSUES GOING FORWARD. SO ONE OF THEM ANOTHER ONE IS ISSUE ABOUT USAGE COST AND WHEN YOU'RE DOING IT'S PROBABLY BEFORE YOU GET TO SCALE AND THIS IS SOMETHING THAT'S BEING DEPLOYED BY PROVIDERS AND INSURANCE COMPANIES AND THINGS LIKE THAT TRYING TO DO LARGER PILOTS THERE IS A USAGE COST THAT MIGHT BE COME TO YOUR PARTICIPANTS SO YOU SUPPLEMENT THEM FOR UNLIMITED TEXT MESSAGING AS WELL AS THE OUTPUT MESSAGING. NOW, IN SOME -- SO THAT'S SOMETHING THAT CAN BE AN INHIBITOR TO SCALING AND GETTING PAST SMALL PILOTS AND THAT'S INTERESTING FOR FUNDERS IN WAYS WE ANY ABOUT PUBLIC PRIVATE PARTNERSHIPS TO ENABLE SOME OF THIS BY GETTING MORE COLLABORATION BY THE INDUSTRY THAT SUPPORTS THAT. SO THERE'S INTERESTING OPPORTUNITIES MDNOs AREN'T LOOKED ON FAVORABLY BUT SOMETHING THAT STARTS TO SCALE SOME OF THESE THINGS MAKES SENSE. SO THEN WE WENT TO DIFFERENT ISSUE. AND MAYBE WILL HAVE MORE DISCUSSION WHICH COMES BACK TO THIS ONE OF PRIVACY. THAT'S WHAT YOU'RE BRINGING UP? YOU BRING IT UP IN CONTEXT OF COST WHICH IS WHAT CONFUSED ME. YOU DONE HAVE A MICROPHONE THAT'S WHY YOU CAN'T SPEAK. OKAY. NEVER QUITE SEEN YOU SO TONGUE TIED. >> WENDY TRAINED ME WELL. SO COST IS AN INHIBITOR. IF YOU CAN'T AFFORD THE IT STAFF SERVER INFRASTRUCTURE ET CETERA TO HOLD ON TO YOUR OWN DATA, DO YOU GET IN A FUNNY SITUATION WHERE -- I DON'T KNOW >> WHEN YOU CAN'T AFFORD IT YOURSELF YOU PUT TO IT THE CLOUD AND GIVE UP PRIVACY. >> AN OPEN mHEALTH SOLUTION IS ONE WE OFFER AS A SERVICE AND THAT WOULD BE A GREAT THING BUT THEN NOT THAT THIS IS SOME DOOM AND GLOOM THING BUT HOW DO WE DEAL WITH ISSUES OF WHOSE DATA IS IT, WHO HAS FIRST SAY IN WHAT'S DONE WITH THAT DATA WHAT'S AGGRAVATED. THAT IS ALL ARE LIKE PI CONSIDERINGS SETTING ASIDE PATIENT PRIVACY. WHICH IS ALSO -- >> SO LET'S TAKE ON PRIVACY BUT NOT JUST ONE OF COST. RIGHT? >> A COUPLE OF IDEAS ABOUT THAT, NOT BEING SELF-SERVING TO SUGGEST A SOLUTION THAT PARTIALLY INVOLVES GOOGLE. OKAY. I'M GOING TO. NOT BECAUSE I WORK THERE, HAPPEN TO WORK THERE AND THEY DO THAT. SO ONE PRIVACY IS HUGE ABOUT THIS. I PERSONALLY DON'T BELIEVE PEOPLE WILL PARTICIPATE UNLESS THEY'RE INCENTIVIZED IF YOU HAVE TO PARTICIPATE IN A STUDY THAT'S A WAY AROUND PRIVACY CONCERNS AND THEY TRUST RESEARCHERS AND GETTING $150 OR WHATEVER IT IS OR USE A PHONE FOR SIX WEEKS. FOR INDIVIDUALS APPROXIMATE AND SPREADING THIS WHERE IT SCALES YOU HAVE TO BUILD A SYSTEM AND WHAT I WANT TO DO IS OUR SYSTEM AND I THINK DEBRA WANTS TO DO IN GENERAL IS BUILD A SYSTEM WHERE THE DATA, WHEREVER IT'S STORED CAN BE CONTROLLED BY THE EXPERIMENT OR THE PARTICIPANT AND STORED ENCRYPTED SO THE SERVICE IT'S STORED ON HAS NO ACCESS TO THE DATA. AND THE SIMULATOR THERE ARE WAYS TO DO ENCRYPTED COMPUTATION OVER THE DATA. IT'S CRUDE BUT GETTING BETTER AND BETTER AND SOMEBODY JUST WANT WON A MAJOR SCIENCE AWARD FOR THEIR WORK IN THIS. THERE'S A WAY TO USE FREE SERVICES AS LONG AS THOSE ARE AVAILABLE TO STORE YOUR DATA SECURELY AND THAT'S ONE WAY AROUND THESE ISSUES BUT I DON'T KNOW HOW IRBs FEEL ABOUT THAT AND IF IT WILL WORK. I WON'T WORK FOR ATTORNEYS. IT WON'T WORK FOR THE US GOVERNMENT, I KNOW THAT. >> ANY COMMENT ON PRIVACY FROM (INAUDIBLE)? >> IT'S -- YES, I DO. MAYBE I SHOULD JUST SAY WEAR THE DATABASE, WHERE THE DATA IS STORED, THIS IS PARTICULARLY TRUE BUT WE'RE FINDING IN COUNTRIES THAT WE WORK IN THEY WANT THE DATA STORED IN THEIR OWN COUNTRY. THE CLOUD BECOMES AN ISSUE. THAT NEEDS TO BE TAKEN INTO ACCOUNT. WE ALSO WE NEED TO TAKE INTO ACCOUNT THE -- WHILE THERE MAYBE VALUE IN PVPs IN THIS AREA WHERE WE -- FOR INFRASTRUCTURE, WE ALSO -- YEAH, I THINK WE NEED TO TAKE INTHE ACCOUNT -- >> THE VALUE OF DEVELOPING THE INFRASTRUCTURE THAT SAY MINISTRY OF HEALTH MIGHT USE TO ENGAGE WITH THEIR PROVIDERS AND AND MINISTRIES OF HEALTH. IF THEY DON'T HAVE HISTORY THEY'RE GOING TO SEEK OUT OTHER PRIVACY SECTOR SOLUTIONS. THERE ARE LOTS OF CHALLENGES THAT JUST NEED TO BE I THINK WE'RE STILL LOOKING AT AND TRYING TO UNDERSTAND. CAN I GO INTO A DIFFERENCE TOPIC? >> BACK TO A PREVIOUS TOPIC. >> WE HAVE A QUESTION -- (OFF MIC) >> WE HAVE A QUESTION -- FROM THE WEB. >> WE HAVE A QUESTION FROM THE WEB. THIS IS ABOUT -- THE QUESTION IS ABOUT DIGITAL IDs. SO THEY'RE TALKING ABOUT -- WHAT DO WE THINK OF THE ROLES OF DIGITAL IDs IN mHEALTH AND EXAMPLES OF PUBLICLY FUNDED STRATEGIES OR ONES THAT INCLUDE THE BRITISH NHS UNIQUE PATIENT IDENTIFIER. THERE'S ONE IN INDIA, SO THEY'RE TALKING ABOUT THAT THERE ARE INFRASTRUCTURE THINGS TO BUILD THESE DIGITAL IDs, DOES THE PANEL HAVE THOUGHTS ABOUT THAT? THAT'S A WEB QUESTION. >> I THINK GARRETT WANTS TO. THERE ARE MANY ASPECTS OF THAT BUT ONE THING THAT AS YOU MAY OR MAY NOT NO, INDIA WILL HAVE UNIQUE IDENTIFIERS FOR THE MAJORITY OF THE$Gx POPULATION WITHIN THREE YEARS IF THEY ACHIEVE THEIR TARGETS. THEY'RE INCENTIVIZING THAT SYSTEM. AND THEY'RE ALSO ENABLING MAKING USE OF THAT UNIQUE IDENTIFIER. I THINK THAT'S AN IMPORTANT THING WE SHOULD ALL KNOW ABOUT. I THINK THERE'S A SEPARATE THING THAT WE HAVE mHEALTH PROJECTS AN eHEALTH PROJECTS WITH EACH HAVING INDIVIDUAL IDs THAT ACTUALLY DON'T SPEAK TO EACH OTHER. IT IS AT LEAST IN MANY COUNTRIES THIS IS A TRAGEDY BECAUSE WE CAN'T -- WE DON'T ACTUALLY KNOW WHETHER THIS PERSON HAS BEEN VACCINATED IN THIS DATABASE IS THE SAME PERSON NOT VACCINATED IN THIS DATABASE. WE PROPOSE AD SOLUTION FOR THAT. I WOULD BE HAPPY TO ADD IT TO SOME THINKING THAT MATT BERG AND I AN COUPLE OF OTHERS A UNICEF THOUGHT A LITTLE BIT OF THIS THROUGH. WE THINK GOVERNMENT SHOULD HAVE THAT ID BUT THERE'S A WHOLE RANGE. >> I HAD HAD THE NAIVE THOUGHT. I HAD A NAIVE THOUGHT COMING INTO THIS THAT THIS WASN'T AN mHEALTH PROBLEM BECAUSE WE SHOULD USE SOME OTHER DIGITAL ID MANAGEMENT CONTEXT IS THAT (OFF MIC) >> COUNTRIES THAT HAVE UNIQUE IDENTIFIERS ALREADY THAT CAN BE USED FOR THIS. THERE IS VALUE IN THINKING THROUGH A BRIDGING ID FOR EXAMPLE. >> THERE'S LOTS OF THESE SYSTEMS, SCANDINAVIA HAS A GREAT SYSTEM. YOU GET YOUR SOCIAL SECURITY NUMBER AND FOR THE REST OF THEIR LIFE THEY DO GREAT LONGITUDINAL EPIDEMIOLOGY STUDIES IN SCAND SCANDINAVIA BECAUSE OF THESE THINGS. IN THE U.S. IT GETS SHOT DOWN QUICKLY FOR POLITICAL REASONS AS MUCH AS ANYTHING ELSE. PEOPLE DO NOT LIKE THE IDEA OF ANYBODY HAVING ACCESS TO ALL THEIR INFORMATION WHETHER HEALTH INFORMATION, FINANCIAL INFORMATION OR ANY OTHER KINDS OF INFORMATION THROUGH ANY KIND OF UNIQUE ID WHETHER SOCIAL SECURITY NUMBER OR OTHERWISE. SO IN OTHER PARTS OF THE WORLD IT MIGHT WORK BUT WE HAVE YET TO SEE U.S. POPULATION AND OPINION POLLS SAYING WE'RE REI DO DO THAT EXCEPT ON A VOLUNTARY BAS WHERE PEOPLE OPT IN BUT NOT ON A MANDATED LEVEL. >> DECOUPLING THE NOTION OF ID FROM HAVING ACCESS TO SOMEBODY'S INFORMATION IS AN IMPORTANT BEING THERE TO BRING SOME MORE RATIONAL THOUGHT TO THIS THING. BUT THAT DOES REQUIRE THAT EVEN WE IN OUR PRACTICE HAVE SOME NOTION ABOUT IT IS PATIENT CONTROLLED DATA BECAUSE IT'S CHECKED DOESN'T MEAN IT'S NECESSARILY SHARED, ENCRYPTED, ARE PEOPLE DONE WITH PRIVACY AND READY TO MOVE ON TO ANOTHER TOP ?IK >> (INAUDIBLE) FROM CORNELL UNIVERSITY. ALL THE PRIVACY AND COSTS ARE IMPORTANT, NOT SURE HOW MUCH IT HAS TO BE TIED TO THE SHARED INFRASTRUCTURE AND WHAT YOU WERE TALKING ABOUT IS SENSOR PROCESSING VISUALIZATION, EACH EXPERIMENT HAS ITS OWN COSTS AND PRIVACY GUIDELINES BUT STILL THAT'S SHARED INFRASTRUCTURE WHICH MENTION THAT MOST DONE KNOW WHAT THEY'RE DOING WITH THE DATA IS KEY TO SOLVING THAT PROBLEM. THIS IS WHAT YOU CAN DO WITH IT HERE YOU CAN EASILY USE, IRRESPECTIVE OF PRIVACY OR COST ISSUES. >> SO I ALMOST AGREE WITH YOU. FIRST I THINK WE HAVE A MANTRA AROUND (INAUDIBLE) YOU CANNOT COLLECT DATA IF YOU DON'T HAVE AN INITIAL IDEA OF WHAT YOU'RE GOING TO DO WITH IT. DOESN'T MEAN THAT'S ALL BUT THE NOTION OF CAPTURING DATA AND HAVING NO IDEA WHAT THE DO WITH THE DATA ISN'T A GOOD APPROACH. WHEN YOU THINK ABOUT WHAT YOU DO WITH IT YOU D A BETTER JOB WHAT YOU CAPTURE. I KNOW THAT'S WHAT YOU'RE -- HAVING A CONTEXT WHICH PEOPLE ARE ALREADY HAVE HAVE A SET OF TOOLS TO THINK ABOUT WHAT THEY'RE GOING TO DO WITH THE DATA BEFORE THEY CAPTURE ANYBODY, THAT'S THE FIRST PLACE TO START. AS WITHIN OUR OPEN mHEALTH CONTEXT THAT'S WHY WE'RE STARTING THERE. THAT BEING SAID, IT'S VERY HARD TO BRING IN SECURITY LATER. AND IT'S HARD FOR US TO PULL OUT OF WHERE WE ARE BY AND LARGE IF YOU LOOK AT INTERNET AND COMMERCIAL SPACE AND E-COMMERCE AND SEARCH AND SOCIAL MEDIA, SOME FOLKS MIGHT LIKE TO BE IN A DIFFERENT PLACE WITH RESPECT TO PRIVACY BUT IT'S HARD TO/9œ GET THERE ONCE YOU REALLY HAVE LARGE NETWORK SERVICES THERE. I DO THINK THAT FOR EXAMPLE THE NOTION OF HAVING DATA COME TO REST SOMEPLACE IT IS ENTIRELY PRIVATE ENCRYPTED IN THE CLOUD BUT IN A WAY STILL IS ENTIRELY PRIVATE AND WHETHER YOU COMPUTE OVER THAT ENCRYPTED DATA OR TRY TO BRING THE COMMUTATION CLOSER TO THE DATA AND SHARE THOSE RESULTS THAT IS A STYLE AND SOMETHING THAT HAS TO DO WITH SHARED INFRASTRUCTURE THAT MAYBE IS SOMETHING WE CAN THINK ABOUT IF WE WANT THESE THINGS TO EMERGE COMMERCIALLY WITH A DIFFERENT NOTION OF PRIVACY THAN SOME OF OUR OTHER INTERNET SERVICES HAVE EMERGED. WITH THAT MAYBE WE CAN MOVE ON NO, I CAN'T. GARRETT IS UP NEXT. >> LET'S GO IN THIS ORDER. >>ING THIS MOVING TO SOMETHING DIFFERENT. (OFF MIC) >> SO THE PRIVACY IS THERE AND PUTTING PRIVACY AROUND SENSOR DATA IS HARD BECAUSE YOU'RE SENDING A SIGNAL, SOMETHING TO THAT EFFECT, YOU CAN'T CARRY PRIVACY INFORMATION ALONG WITH IT. BUT IN TERMS OF CHECKING THE DATA AND KNOWING WHAT TO D WITH IT I LOOK TO LIKE THE PEOPLE, AT THE END OF THE PANEL THAT DEVELOPED BEST PRACTICES AND EVERYONE ELSE WHO WORKED ON INTERVENTIONS SHOULD BE THE MODEL THAT DESCRIBES WHAT PARAMETERS NEED TO BE COLLECTED AND HOW THEY NEED TO BE COLLECTED. AND THE TOOLS IN MY PERSONAL OPINION SHOULD BE OPEN-ENDED TO ALLOW IMPLEMENTATION OF BEST PRACTICE THAT ACTUALLY COME FROM THE RESEARCH AND FROM THE FIELD STUDIES AND THEN BE PUT INTO PLACE AND WITH PRIVACY. (OFF MIC) >> THIS IS BACK TO THE METRICS QUESTION. I COMPLETELY AGREE, THERE ARE -- IT'S DIFFICULT AT THIS POINT TO KNOW WHAT THE CORRECT METRICS ARE TO RECOMMEND TO PEOPLE BUT THIS IS WHY WE START THIS CONVERSATION. TRYING TO FIGURE OUT WHAT PEOPLE ARE USING AND CROWD SOURCE AND WE'RE ACTUALLY INTERVIEWING PEOPLE, DOING SURVEYS WITH PEOPLE FIGURING OUT WHAT THEY WOULD RECOMMEND AND HOW THOSE INDICATORS OF METRICS BE USED TO FOR DECISION MAKING OR TO INFLUENCE PARTNERS. SO WE WOULD BE KEEN TO TALK TO YOU ABOUT WHAT YOU'RE USING AN FEED IT INTO THE INFRASTRUCTURE THAT OTHERS ARE DEVELOPING. THE OTHER THING I WANT TO BRING UP, SO IT'S A PROCESS, I THINK IS THE IMPORTANT THING. THE SECOND IS SCALEUP IN SUSTAINABILITY. ISSUES OF HOW DO YOU PAY FOR THINGS AND HOW DO YOU -- BUT ALSO I THINK THAT IT'S A REALLY CRITICAL QUESTION AND SOMETHING WE WERE SEEING THE NEED TONE COURAGE GROUPS AT LEAST IN THE COUNTRIES THAT WE'RE WORKING IN, TO CONSIDER BUSINESS MODELS OR AT LEAST SUSTAINABILITY MODELS FOR THEIR SOLUTIONS. AND THEY SHOULD BE THINKING ABOUT THOSE ISSUES IN ADVANCE OF DEVELOPING THEIR mHEALTH SOLUTION. AND THEY SHOULD BE GATHERING THE EVIDENCE THEY NEED TO ENGAGE AND COON VINCE THEM WHETHER DONORS, MINISTRIES OF HEALTH, THE NGO COMMUNITY OR WHETHER THAT'S THE OTHER FUNDERS OUT THERE. YOU NEED TO GET THE RIGHT EVIDENCE AND FIND THE SOLUTIONS THAT WILL LEAD TO AT LEAST SUSTAINABILITY FOR THOSE INCENTIVES IN OPEN SOURCE, TO BRIEFLY SAY WHEN WE HAVE A PROJECT IN INDIA, AN WE WERE CONVINCING OUR DONOR COMMUNITY WE WERE ABLE THE SAY THIS SOLUTION WE'RE BILLING UPON HAS BEEN USED IN OVER TEN COUNTRIES WITH THIS SIZE POPULATION, AND FOUNDATION THAT HAS SOME LEVEL OF STABILITY BASED ON OPEN SOURCE TOOLS COMPLIANT WITH INTEROPERABLE. THANK YOU. >> THIS IS AN INFRASTRUCTURE ISSUE THAT I THINK IS IMPORTANT. KEVIN PATRICK UC SAN DIEGO. CRAIG WAS TALKING ABOUT SARASOTA, THE PROJECT IN CHICAGO AND WHAT TOM IS DOING IN BALTIMORE. THIS IS THE NOTION, ONE THING THAT EXCITE MESS ABOUT mHEALTH IS HOW IT CENGS TO HEALTH AND PLACE, THIS NOTION WE ARE NOT DISCONNECTED FROM WHERE WE LIVE AND WHERE WE SPEND OUR ENTIRE LIVES. THESE DEVICES ALLOW US TO GET A BETTER UNDERSTANDING OF THESE LIFE TRACES AND OTHER THINGS, EXTRAORDINARY BECAUSE IT UNCOVERS MANY THINGS THAT ARE IMPORTANT. THE RAILROAD CAR AN TRACKS IF YOU THINK WHAT WOULD A RAILROAD CAR BE WITHOUT THE TRACKS, VIE VERSA SO THINKING ABOUT THIS IS PART OF THE INFRASTRUCTURE AND WE NEED TO BE THINKING ABOUT DOING THIS YOU MENTION SOME COUNTRIES WANT TO HAVE ALL THE DATA IN THEIR PARTICULAR COUNTRY AND BEGIN TO UNDERSTAND THAT BECAUSE THEY'RE SENSITIVE ABOUT IT GOING SOMEWHERE ELSE. THAT'S A FUNCTION SOCIAL CULTURAL OTHER ISSUES UNIQUE TO THAT PARTICULAR PLACE AND THAT PARTICULAR POINT IN TIME. AND NOBODY SAID THIS YET SO THIS SHOULD BE PART OF THE AGENDA OF THE mHEALTH INFRASTRUCTURE, THE SETTINGS AND PLACES WITHIN WHICH WE STAND THESE THINGS UP. THAT'S MY POINT. >> THANK YOU. >> THE PANEL LIKE TO RESPOND TO THAT? (OFF MIC) >> THIS IS THE FOLLOW ON TO GARRETT'S POINTS ABOUT OUTCOMES AND TALKENING GENERAL, THE NOTION OF COMMON VARIABLES WE WOULD LIKE TO GET TO, THERE HAVE BEEN MANY ATTEMPTS IN CANCER, PROBABLY THE MOST SUCCESSFUL INITIATIVES FROM CDISC AND THE PHOENIX PROJECT, THOSE OF YOU THAT KNOW THAT, THERE'S MULTIPLE -- SOMEBODY EARLIER SAID IT'S PARTLY BECAUSE YOUR CAREER IS BASED ON COMING UP WITH A METRIC WITH YOUR NAME ON IT, IT'S HARD TO GIVE THAT UP. IF WE WANTBL METRICS OR COMMUNITY METRICS WE NEED TO DEVELOP THEM AS A COMMUNITY AND LEARN HOW TO USE THEM AS A COMMUNITY NOT AS A MANDATE BUT AS SOMETHING YOU WOULD RATHER USE THAN NOT. JUST GETTING TO THAT POINT I THINK IS A COMMUNITY DEDEBRA AND I TALK ABOUT TECHNICAL INFRASTRUCTURE, THE HARDEST IS COMMUNITY INFRASTRUCTURE. THIS IS A GREAT COMMUNITY, READY FORMED IN THIS ROOM, TO ADD THAT TO AGENDA SOMEHOW DO WE START DEFINING POSSIBLE COMMON MEASURES, HOW DO WE EXPERIMENT ON THEM. HOW DO WE REFINE THEM. HOW DO WE SHARE THEM AN PROMOTE THEIR USE. SO THERE ARE VARIOUS INITIATIVES GOING ON AROUND THE WORLD, HOW DO WE BRING THAT INTO AN INFRASTRUCTURE INTO A LINE WE I DON'T THINK THEY'RE GOING TO GET OUT OF THERE, IT'S GOING TO BE A FULL JOURNEY. >> THANK YOU. THIS HAS BEEN VERY INTERESTING DISCUSSION. IF THE PANEL WOULD LIKE TO SUMMARIZE THEIR CONCLUSION FROM THE DISCUSSION. EACH OF YOU HAS -- I WAS TOLD WE GO TO QUARTER TO SO WE HAVE A FEW MORE MINUTES TO DO THIS. (OFF MIC) >> I WAS WOBBING ARE IF THE PANEL WOULD CONSIDER A COUPLE OF Ss HERE. ONE THAT I HEARD WAS SCALABILITY. AND THE OTHER ONE PERHAPS, SECURITY. MAYBE SARASOTA COULD BE PUSHED UP A LITTLE BIT. IS THERE ANY MORE DISCUSSION? (OFF MIC) >> CHARLENE QUINN, UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE. ONE SORT OF SUMMARY I'M GETTING OUT OF THIS DATA THAT I HAD IN MY PAPER IS I HOPE THAT WE BEGIN THE TALK ABOUT WHAT WE'RE DOING AS MOBILE HEALTH SCIENCE AND NOT JUST MOBILE HEALTH AND ALL THE PIECES OF IT. BUT THAT WE REALLY HAVE A SCIENCE THAT WE'RE TALKING ABOUT. AND WHEN I BROUGHT THAT UP IN RELATION TO MY OWN STUDIES PEOPLE SAY YES, FINALLY, SO THE PEOPLE STOPPED TALKING ABOUT THIS AS JUST ANOTHER APP BUT THERE IS A SCIENCE BEHIND IT. AND -- >> YOU MEAN DOING SCIENCE WITH OR THE SCIENCE OF? >> THE SCIENCE OF. THE SCIENCE OF. AND THAT SORT OF SUMMARIZES THEN THERE WOULD BE MULTIPLE DISCIPLINES BEHIND THIS. AND THAT MAYBE WE DO HAVE SIMILAR TO I DON'T KNOW I LIKEN IT TO WHAT'S HAPPENED IN GENOMICS WHERE PEOPLE DO HAVE AN INCENTIVE TO SHARE THINGS, THERE IS AN INCENTIVE TO FUND DATA ANNOTATION. SO THERE'S COMMONALITY ACROSS THE SCIENCE OF MOBILE HEALTH. >> THANK YOU. I WOULD ALSO LIKE TO ADD TO IT ACTUALLY IF YOU WERE LISTENING TO THIS LAST SESSION ESPECIALLY BUT ALL OF THEM, MANY ISSUES ARE BROADER THAN mHEALTH. A LOT OF DISCUSSIONS WE HAD RANGING FROM APPLICATIONS TO INFRASTRUCTURE AND TO SECURITY AND HAVE TO DO WITH NETWORKING AND INFORMATION TECHNOLOGY IN GENERAL. NOT JUST mHEALTH BUT mHEALTH HAS THE ADDITIONAL PERSONAL ASPECT OF COLLECTING DATA. >> THAT FACT IS AN OPPORTUNITY AND A BIT OF A PROBLEM BECAUSE WE CAN QUICKLY GET VERY DIFFUSE AND ALL OVER THE PLACE AND NOT FOCUSED IN. SO IT'S A BIT OF A BALANCING ACT. >> ABSOLUTELY. ANY OTHER COMMENTS? >> TO RESPOND TO YOUR COMMENT CHARLENE, PROPOSALS AN PIECES THAT I THINK ARE IMPORTANT FROM A ENGINEERING PROSPECTIVE THAT I FIND PARTICULARLY INTERESTING ABOUT THIS SPACE SINCE IT'S NOT MY BACKGROUND IN BEHAVIORAL SCIENCE, THE UNDERSTANDING OF ALL OF THE TYPES OF INTERVENTIONS THAT CAN BE DONE. I THINK THAT'S PART OF THE DOMAIN MODEL FOR mHEALTH, WHAT IS PARTICULARLY SUITED TO M AS OPPOSED TO HEALTH IN GENERAL SO IT'S SOMETHING LOCALIZED ON THE PERSON. SO THERE STARTS TO BE A TAXONMY OF THE TYPES OF THINGS THAT WE STUDY IN THIS SPACE AND THAT WE BUILD OUT SUPPORT FOR STUDYING. BY BUILDING OUT THE SUPPORT FOR STUDYING THAT'S ANOTHER SUB DISCIPLINE OF mHEALTH AND HOPEFULLY THEY FEED ON EACH OTHER SO THAT THE INFRASTRUCTURE PIECES AN DISCIPLINE ABOUT BUILDING THAT ALLOWS US TO BUILD UP WHETHER SUBCATEGORIES OR TYPES OF INTERVENTIONS INPUTS, WHATEVER THOSE PIECES ARE, WHICH ARE MOST VALID, EFFECTIVE, EFFICACIOUS AND SO ON. >> AND KEVIN'S POINT ABOUT PLACE IS ANOTHER EXAMPLE, WHEN WHAT YOU'RE DOING AN CONTEXT AN LOCATION COMES INTO PLAY IS ANOTHER OPPORTUNITY >> IT'S HUGE BRINGING THE DISTANCE BACK TO INTERNET. >> ACTUALLY JUST TO FOLLOW FROM WHAT YOU JUST SAID. WE HAVE VERY MUCH THOUGHT OF THE POTENTIAL FOR FEATURES AN FUNCTIONS. SO THE INTENTION BEING A REMINDER AND THERE ARE DIFFERENT WAYS OF REMINDING, FOR EXAMPLE, THERE'S A TAXONOMY THERE. WE DON'T KNOW WHAT THE FULL TAXONOMY IN THE mHEALTH DOMAIN IS. BUT THAT WE KNOW THAT THEY'RE BEING USED IN VARIOUS WAYS TO INFLUENCE PROCESSES ACROSS AND WE KNOW THAT THERE'S SOME IMPACT POTENTIALLY ON THE OUTCOMES, HEALTH OUTCOMES AND EFFICIENCIES. WE NEED TO MODEL THAT. AND I -- THAT'S SOMETHING THAT A COMMUNITY NEEDS TO ENGAIBLG -- ENGAGE IN, IT COULD BE MORE RA PUDLY DONE, THERE'S COULD BE SOME CONSENSUS AROUND WHAT DIFFERENT FEATURES OR FUNCTIONS OR INTENTIONS OR WHATEVER ARE. IN SOME WAYS LIKE -- ANYWAY, YES, I COMPLETELY AGREE WITH YOU, THAT'S SIMILAR TO SOME OF OUR THINKING ON IT AS WELL. >> IT COMES BACK TO MODULARITY. >> RIGHT. >> ALL RIGHT. I THINK I WOULD LIKE TO THANK THE PANELISTS. GIVE THEM A NICE HAND. [APPLAUSE] I THINK THE NEXT SESSION IS AL SHAR'S. >> UNLIKE EVERYBODY ELSE WE GET TO CHANGE OUR PRESENTATION AT THE VERY LAST MINUTE. >> WE GET TO GO NOW FROM A VERY, VERY WIDE-RANGING SESSION, SET OF SESSIONS, THAT WAS REALLY EXCITING, BEFORE WE START LET ME MENTION ONE THING ABOUT IDENTIFIERS BECAUSE IT WAS MENTIONED, TWO THINGS. THERE ARE TWO INTERNATIONAL STANDARDS OF MEDICAL IDENTIFIERS. ONE IS A GENERAL IDENTIFIER AND THE OTHER VOLUNTARY WHICH WAS SET UP SO THAT THE PATIENT CAN WIND UP CONTROLLING THAT. WE HAVE AN EXPERIMENT GOING RIGHT NOW IN LOS ANGELES IN TERMS OF WHETHER OR NOT THAT CAN WIND UP BEING ACCEPTED IN THIS COUNTRY. IF YOU'LL REMEMBER WHEN HIPAA WAS FIRST PASSED THERE WAS A SEGMENT -- A SECTION OF THAT THAT DID CALL FOR A UNIVERSAL IDENTIFIER THAT WITHIN A YEAR WAS REMOVED FROM THE LAW AND SINCE THEN THERE'S BEEN NO GOVERNMENT FUNDING FOR ANY RESEARCH IN THIS COUNTRY OR ON PATIENT IDENTIFIERS. HAVING SAID THAT, WE'RE GOING TO CHANGE THE WAY THIS SESSION WAS DEFINED ORIGINALLY A LITTLE BIT. THE GOAL RIGHT NOW IS TO COME UP WITH WHERE WE'RE GOING FROM HERE. THIS WAS VERY EXCITING, IT'S GREAT, WE NEED AN AGENDA WE NEED A WAY TO FOCUS ON WHAT WE WANT TO WIND UP DOING NEXT, HOW WE'RE GOING TO WIND UP DOING IT. WHAT YOU'RE GOING TO SEE HERE IS A—h5 FIRST SET ARE THINGS THAT I HEARD AND IN TERMS OF TRYING TO HELP US GET FOCUSED, WHAT I WOULD LIKE TO TRY AND DO IS VERY QUICKLY GO THROUGH THESE, THEN ASK PANEL LEADERS TO COMMENT AND ASK YOU TO COMMENT. AFTER THAT COME UP AND WE'LL TRY AN WORK ON HOW WE CAN ACTUALLY NARROW THE AGENDA TO GET THINGS THAT ARE TRULY ACTIONABLE OUT OF HERE. HAVING HAD A VERY LONG ACADEMIC CAREER, IT'S HARD TO GET ACADEMICS TO ACTUALLY FOCUS ON CERTAIN THINGS. AND THAT REALLY IS OUR GOAL. SO WHAT I WANTED TO DO QUICKLY GO THROUGH THE THINGS THAT I HEARD. AND JUST BECAUSE THEY -- WHAT I HEARD WAS DOESN'T MEAN THAT IT IS EITHER THE END ALL OR BE ALL. SO LET ME START OFF WITH THAT. THE FIRST ONE IS THAT IF WE GO FORWARD IT'S NOT FUNDING MORE RESEARCH, WE HAVE ENOUGH FUNDING FOR INDIVIDUAL RESEARCH PROJECTS. IDA MENTIONED LOOKING AT ALTERNATIVES TO PUBLICATIONS, THE TRADITIONAL METHOD OF PUBLICATION IS OBSOLETE AN SLOW FOR WHERE WE'RE GOING AND WE SHALL THINK ABOUT THAT. RATHER THAN TRYING TO LOOK AT WHERE WE'RE GOING IN TERMS OF ACCOMMODATING THE CURRENT LANDSCAPE OF HEALTHCARE, WHAT'S THE FUTURE GOING TO WIND UP BEING? ONE OF THE THINGS I WOUND UP HEARING IS THAT IF WE CAN SATISFY PAYERS THAT THERE'S A VALUE THERE THERE'S A SUSTAINABILITY MODEL WE CAN LOOK AT THAT, NOT WIND UP LACK AT WHAT THE CURRENT MODELS ARE. WE SHOULD LOOK TO SEE WHAT HAPPENS IN OTHER AREAS. CONSUMERRIZATION IS NOT THE MEDICAL ACTIVITY. WHAT DOES IT MEAN WHETHER WHEN YOU PUT THE CONSUMER, THE PATIENT AND THE CENTER, I DON'T MEAN THE CENTER -- I MEAN IN THE CENTER ABLE TO WIND UP CONTROLLING THINGS. WE HAVE TIME AN TIME AGAIN CAME BACK TO WHAT THE ETHICAL QUESTIONS ARE, WHAT ARE THEY. THIS IS A GOOD ONE, THIS WHOLE QUESTION OF ALWAYS BEING IN BETA MIGHT WORK REALLY WELL FOR GOOGLE. I DON'T KNOW HOW WELL IT NECESSARILY WORKS IN MEDICINE WHERE THE END RESULT CAN WIND UP BEING SOMETHING, AN OUTCOME THAT IS TERRIBLY UNFORTUNATE. WHO ELSE DO WE NEED TO WIND UP ENGAGE SOMETHING WE HAVE A ROOM FULL OF PEOPLE WHO ARE REALLY BELIEVE IN THIS. WE KNOW THESE THINGS WORK WE KNOW THESE THINGS ARE GOING -- CAN WIND UP HELPING US. WHO ARE THE SKEPTICS WHO ARE THE PATIENTS NOT HERE, PATIENTS, FUNDERS, PROVIDERS, WE HAD FDA BUT FDA IS BIG. THE WIRELESS COMPANIES, EMPLOYERS, ESPECIALLY THOSE LARGE ONES THAT ARE DOING A LOT OF SELF-FUNDING WHERE THE HEALTH SYSTEM IS INVOLVED. HOW DO WE LEARN THE LANGUAGE OF COLLABORATORS, ONE THING WE HEARD WAS THAT THIS IS A COLLABORATIVE ENVIRONMENT AND THE LANGUAGE THAT AN ENGINEER SPEAKS IS DIFFERENT THAN A LANGUAGE CLINICIAN SPEAKS IS VERY DIFFERENT THAN THE LANGUAGE A COMPUTER SCIENTIST SPEAKS. WE NEED TO START LEARNING HOW DO WE SPEAK THOSE SAME LANGUAGES, HOW DO WE USE TERMINOLOGY IN WAYS THAT HAVE ADVANCED RATHER THAN CONFUSED. FINALLY, WE NEED TO FIND INNOVATIVE WAYS OF DEMONSTRATING VALUE. THAT GOES WITHOUT SAYING. I SAID MY PIECE, I WANTED TO KNOW WHETHER OR NOT THE PANEL LEADERS HAD ANY THINGS TO WIND UP ADDING. THEN WE CAN OPEN UP TO EVERYONE HERE. >> THERE WAS ONE POINT IN THE LAST SESSION ABOUT THE OPEN SOURCE BUILDING BLOCKS. FOR FORMING mHEALTH, BUILDING BLOCKS mHEALTH TOOLS, OPEN SOURCE, THAT'S REALLY IMPORTANT. WHAT WAS IT CALLED, OPEN MHEALTH ORU SOMETHING? >> NO, SHE DIDN'T. I REALLY THINK THAT'S A BIG THING HERE. IT'S -- I THINK ALSO SPEAKING OF SOMEONE WHO IS PRIMARILY WORKED IN OTHER AREAS OF THE CLINICAL FIELD, MOST MEDICAL FIELD IS SO CLOSED AND SO YOU DONE GIVE UP THE DATA ABOUT YOUR PATIENTS, YOU DON'T GIVE -- YOU DONE REARLS THE DATA YOU HAVE DATA FOR A TRIAL YOU DON'T RELEASE THE DATA UNTIL YEARS AFTER THE TRIAL IS OVER, EVERYTHING IS TIEG TO THE CHEST. THIS IS A GREAT OPPORTUNITY IN MEDICINE TO BRING IN A NEW BREATH OF AIFER, A DIFFERENT WAY TO THINK ABOUT INNOVATION AND BE MORE OPEN SOURCE APPROACH. SO I REALLY THINK WE SHOULD -- THAT SHOULD BE A BIG THING HERE. GREAT OPPORTUNITY FOR US. SHE DIDN'T PAY ME. >> I'M GOING TO ASK SOMETHING THAT COULD BE COMPLETELY HERETICAL. ONE OF THE THINGS THAT I DON'T QUITE GET IS WHAT IS mHEALTH AND HOW IT DISTINGUISHES ITSELF FROM THE ELECTRONIC HEALTHCARE AGENDA. THERE'S TWO PARTS OF THE QUESTION LOOKING AT THE RESEARCH AGENDA. ONE IS MAKING SURE THAT AS A COMMUNITY OF RESEARCHERS AND THINKERS, YOU'RE DEFINING THE SCOPE ABOUT WHAT UP TO WORK ON AND MAKING IT CLEAR AMONG YOURSELVES AS A IMMUNITY IN THIS ISSUE OF COLLABORATOR, I WAS THINKING ABOUT COLLABORATORS WITHIN mHEALTH AS WELL AS HAND SHAPE PARTNERS AND COLLABORATORS WHO ARE ON ALL THE STUDS, STUDS OR STUBS, STUBS. ALONG THE EDGES OF WHERE YOU SEE YOUR BOUNDARIES BECAUSE I HAVE KIND OF BEEN AS THE DAY IS PROGRESSED I HAVE SEEN DIFFERENT AREAS OF WHAT I THINK PEOPLE SEE AS BOUNDARIES AND NOT BOUNDARIES AND I THINK IT'S REALLY IMPORTANT THAT YOU DO THAT BECAUSE IT HELPS TO FRAME UP YOUR AGENDA. DOES THAT MAKE SENSE? >> IT ACTUALLY DOES AT LEAST TO ME. >> I'LL ECHO SUSAN'S COMMENT ABOUT OPEN SOURCE, IT WILL HELP US LEARN THE LANGUAGE OF OTHER COLLABORATORS IF IT'S A COMMON AREA ALL OF US THAT COME FROM DIFFERENT AREAS HAVE ACCESS TO THAT, IT WILL HELP US, I MEAN, TODAY IS HELPED ME, MEASURABLY LEARNING ABOUT THE LANGUAGE OF SOME OTHER FIELD, I THINK IT'S AN ONGOING PROCESS THAT AN OPEN SOURCE MENTALITY WILL JUST ENGAGE RATHER THAN PREVENT. >> DOES ANYBODY ELSE WANT TO COMMENT IN TERMS OF REFINING THE DEFINITION IN THE FIELD, WHAT IS mHEALTH? >> THAT'S A VERY GOOD QUESTION WHAT IS mHEALTH DISTINGUISHED FROM eHEALTH OR TELEHEALTH OR ANY OTHER HEALTH THAT ARE OUT THERE. AND FOR THE OPEN mHEALTH PROJECT WE DID HAVE A PROJECT I'M GOING TO PARAPHRASE, NOT SURE I REMEMBER IT TOTALLY. WE DECIDED ON SOMETHING LIKE THE USE OF MOBILE AND INTERNET TECHNOLOGIES IN CONJUNCTION WITH SOCIAL MEDIA TO IMPROVE INDIVIDUAL POPULATION HEALTH. PRETTY BROAD. WE THOUGHT WE DIDN'T WANT TO NARROW IT TOO MUCH, IT IS CERTAIN LAY DEFINITION IN FLUX. RIGID BOUNDARY IS NOT HELPFUL BUT NOT HAVING A BOUNDARY IS NOT HELPFUL. THAT WAS OUR HALF APPROACH PUTTING SOME FRAME ON IT, PUTTING SOME CONSTRAINTS WITHOUT CONSTRAINING TOO MUCH. AND CERTAINLY SOMETHING THAT CAN BE REFINED AND WORKED ON. >> I'M ARTHUR HEREDEK, A YEAR AND A HALF AGO WHEN I CAME TO MOBILE RESEARCH BARRIERS TALK I BELIEVE MOBILE HEALTH IS THE INEVITABILITY OF COMPUTER. THEY WILL HAVE SENSORS, GET SMALLER AND EVENTUALLY BE INSIDE OF US AND WE'LL CALL IT SELF-HELP OR SKIN HEALTH OR IT WILL BE EMBEDDED SO FOR ME I THINK THAT THE DISTINCTION ISN'T VERY CLEAR FROM A COMPUTER SCIENCE STANDPOINT BECAUSE IT'S ALL A CONTINUUM. FROM A PRACTICALITY STANDPOINT THERE IS SOMETHING REAL BECAUSE IT IS FRESH AND MAKES PEOPLE REALIZE THERE'S BEEN -- IF YOU WESTERN PAY AGO TENSION IN COMPUTER SCIENCE, IT IS A DISCRETE JUMP IN TECHNOLOGY AVAILABLE – TO PEOPLE AT LARGE, Ö THAT'S THE DIFFERENCE. >> I'M TAKING THE MIC IN FRONT OF ME. CAN I INTERJECT? I HATE THESE EVENTUAL THINGS, IN THE END WE'RE ALL DEAD THINGS BUT I THINK THERE IS AN INCREDIBLE OPPORTUNITY NOW, IT DOESN'T MAKE THE ANSWER TO YOUR QUESTION EASY BUT IT'S THE PREVALENCE OF PEOPLE HAVING MADE THE CHOICE EXPRESS HER PREFERENCE AND ADOPTED MOBILE PHONES. THEN A FAIR AMOUNT ABOUT WHAT MOBILE HEALTH CAN BE IN THE NEAR TERM IS ABOUT LEVERAGING THE FACT THAT PEOPLE HAVE CHOSEN THAT DEVICE AS A WAY THAT THEY'RE MEDIATING THEIR LIVES SO IT PRESENTS US WITH A PRETTY UNBELIEVABLY AFFORDABLE OPPORTUNITY TO GET DATA AND DELIVER INTERVENTIONS AND SUCH. SO A FAIR AMOUNT FOR ME FOCUSES AROUND THE MOBILE PEOPLE ADOPTED. INCREASINGLY THERE ARE DEVICES THE MOBILE IS LIKELY TO BE THE GATEWAY FOR US. NOW, AS I DID DESCRIBE, THERE'S ALWAYS THE INTERNET ON THE OTHER SIDE OF THAT MOBILE. OUT'S NOT JUST A MOBILE APP, SO IT IS SENORRED AS THE INTERFACE OF INTERVENTION BUT WHEN YOU'RE MAKING SENSE OF IT WITH THE COMMUNITY MODELS BUT mHEALTH IS FOCUSED AROUND THOSE OPPORTUNITIES THAT COME ABOUT WITH THE TECHNOLOGY BEING CARRIED AROUND AND INTERACTING WITH. WHEN YOU'RE GOING AND DOING SOMETHING IT HAS INTERACTION WITH WHAT KIND OF ELECTRONIC HEALTH RECORD EXISTS YOU CAN GET THE REST OF HER CONTEXT, THESE PLACES WHERE IT INTERFACES AND WE HAVE TO BE CAREFUL AND AS MISHA SAYS THE PROBLEMS WE FACE ARE FACED IN OTHER AREAS TOO. SO THEY'RE ALL KINDS OF SLIPPERY SLOABS, THEY CAN GET THE SCOPE CREEP TO BE HUGE. >> CAN I SAY SOMETHING TOO? WE ALL THOUGHT ABOUT THIS A LOT. LIKE DEBRA SAYS, THERE'S A FEAR OF CREEPING BUT THERE'S ALSO A FEAR OF NARROWING AND LIKE BOB SAID I THINK YOU'RE ALL RIGHT. THE FIELD IS EVENTUALLY WILL BE CHIPPED, WE CAN FORGET ALL ABOUT THIS. BUT I MEAN I THINK THE SOUND DOES HAVE ONE THING, IF YOU THINK IN THE SCOPE OF THE WORLD THERE ARE MORE PHONES THAN TOILETS. THIS IS A WAY -- IT'S AN ACCESS TO PEOPLE IN A WAY THAT WE HAVE NEVER HAD ACCESS BEFORE. IT'S A WAY TO GET INFORMATION LIKE DEBRA IS SAYING. YOU CHOSE TO CARRY YOUR PHONE. IN THIS ROOM EVERYBODY HAS ONE AT LEAST ONE, MOST JUST HAVE TWO AT THIS POINT. SO IT'S JUST A VERY, IT'S WHERE THAT'S DIFFERENT. >> THERE'S ANOTHER THING. THAT IN SOME WAYS I HAVE ALWAYS BEEN HESITANT TO TRY AND DEFINE THIS TOO MUCH BECAUSE ONE OF THE THINGS THAT I LIKE ABOUT BEING VAGUE A LITTLE BIT IS THAT WE CAN CHALLENGE THE CONVENTIONAL WISDOM. IT'S A PLACE CONVENTIONAL RESEARCHERS HAVE IGNORED, IT'S HARDER TO WIND UP REFINING, STRUCTURE, ID'S HARDER TO WIND UP CONTROLLING IT. I LIKE THE IDEA OF UNDERMINING THE ESTABLISHMENT THAT'S BECOME SO RIGID IN THE WAY OF WHAT THE TRUTH IS. I THINK THAT USING mHEALTH TO BE ABLE TO UNCOVER NEW TRUTHS IS SOMETHING THAT'S IMPORTANT. I WAIT TO WIND UP LOSING IT BY SAYING WE'VE PUT UP FENCES THAT COME ACROSS. >> CAN I MAKE A SUGGESTION BASED ON WHAT I HAVE JUST HEARD? SO THERE'S TWO PARTS. ONE IS THAT REALLY RIGHT NOW THE MOBILE PHONE CONTEXTUALIZES A CONCEPT WHICH IS THE ABILITY TO COLLECT INFORMATION FROM INDIVIDUALS WHERE THEY ARE IN THEIR OWN UNIQUE ENVIRONMENTS IN A CONSISTENT SETTING. SO RATHER THAN ATTACK IT SO TO A SPECIFIC DEVICE ATTACK IT TO A CONCEPT BECAUSE DEVICES CAN CHANGE AND YOU WANT TO BE A PLATFORM AGNOSTIC IN THAT SENSE. THE OTHER, ACROSS THIS ENTIRE AREA OF RAPIDLY ACCUMULATING DATA WHICH IS SITTING IN LATE PHASE DATA PROGRAMS, RURAL DATA PROGRAMS, THE NUMBER OF DATA PROGRAMS ONGOING AT THE MOMENT IS MIND BOGGLING. FIGURING HOW TO HARMONIZE THAT LANGUAGE SO THAT THEN YOU CAN SAY HERE IS HOW WE FIT IN THAT OVERALL STORY OF WHAT'S GOING ON IN THESE RAPIDLY COMMUNICATING DATA -- RAPIDLY EVOLVING CONCEPTS IN EXPANDING DATA PROGRAMS IS REALLY VERY USEFUL. AND I THINK THAT THERE THE MOBILE PHONE DOES GIVE SOMETHING PALPABLE TO THE CONCEPT THAT YOU CAN INTRODUCE TO THAT OVERALL LANDSCAPE. DOES THAT MAKE SENSE? >> YES, IT DOES. >> JOE (INAUDIBLE) FROM SENOR FOR CONNECTIVE HEALTH AT HARVARD MEDICAL SCHOOL. WE MAYBE BEATING A DEAD HORSE BY THIS POINT BUT I WANTED TO MAKE THE POINT THAT ONE OF THE THINGS THAT MOBILITY OFFERS AND IT ISN'T JUST PHONES, IT'S THE IDEA OF MOBILE COMPUTING, THAT I AT LEAST HAVEN'T SEEN IN 15 YEARS OF BEING IN THE FIELD IS TRUE TIME AND PLACE INDEPENDENCE. AND MY EXAMPLE IS THAT WE -- THE CLINICS IN THE ROOM WILL RESONATE WITH THIS. I THIS THINK WE -- THE STATE OF THE ART MANAGING HYPERTENSION IS TO VISIT YOUR DOCTOR THREE OR FOUR TIMES A YEAR AND HAVE YOUR BLOOD PRESSURE TAKEN IN THE OFFICE. WE DON'T NEED TO THINK THAT WAY. THAT DELIBERATION IS REALLY WHAT MAKES MOBILE EXCITING. NOT ONLY CAN I GATHER DATA POINTS ANY TIME ANYWHERE, I CAN ALSO MESSAGE YOU IN THE MOMENT WHEN YOU NEEDED IT TO MOST. (INAUDIBLE). I DO THINK IT'S NOT JUST ANOTHER KIND OF THING (INAUDIBLE). >> HI. CARRIE (INAUDIBLE) FROM MCKESSON. NOT TO BE A WET PLAIN BLANKET HERE BUT WITH AN HOUR LEFT I KNOW THAT I CAN AT LEAST SPEAK FOR MYSELF PROBABLY THE ORGANIZING COMMITTEE AS WELL, WE WANTED TO GET PEOPLE TOGETHER TO TALK ABOUT HOW WE MIGHT LOOK AT ALTERNATIVES TO A TRADITIONAL RCT TO GET THE TIME FRAME CUT FROM FIVE AND A HALF TO 3 AND A HALF YEARS BY 2016. CLEARLY I'M NOT ACADEMIC BUT I FEEL LIKE THERE'S A LOT OF OPPORTUNITY HERE FOR THE PRACTICAL. NOW I'LL GET INTO THE PRACTICAL PART OF THIS AND THAT'S WHERE WE WANT YOUR EXPERTISE. THAT'S WHAT AL OUTLINED HERE. WHAT DO WE NEED TO DO TO GET HERE THAT'S HOPEFULLY A QUESTION WE CAN TACKLE NEXT HOUR BECAUSE WE HAVE SOMETHING TANGIBLE TO WALK OUT OF THIS MEETING WITH. >> LET ME GET A COUPLE MORE BECAUSE THAT'S A GREAT SEGUE TO NEXT STEPS. >> THIS IS CRAIG LEFEBVRE FROM RTI INTERNATIONAL. TO THE POINT WHAT DOES MOBILE HEALTH OFFER US WHEN YOU THINK OF CONTEXTUALIZING THINGS YOU'RE DHOOG TO SOLVE THEIR PROBLEMS. MOBILE HEALTH IS NOT ABOUT COLLECTING DATA. MOBILE HEALTH IS PEOPLE BEING ABLE TO SOLVE PROBLEMS THEY HAVE. YOU JUST HAPPEN TO BE IN A POSITION MAYBE TO HELP THEM DO THAT. TONY TALKED ABOUT MOBILE HEALTH OR PHONES AS BEING LIFE SIMPLIFIERS AND LIFE NAVIGATORS. THAT'S WHY PEOPLE HAVE THEM IN THEIR POCKET. I THINK THAT THAT AGAIN WHEN YOU THINK ABOUT WHERE THE GUIDING STARS HERE, NEEDS TO BE PART OF THE ISSUE. SO TO GET TO THE ISSUE OF HOW DO WE CUT TWO YEARS OFF OF WHAT WE NEED TO DO. PART OF THE QUESTION THERE, WHAT ARE PEOPLE LOOKING FOR IN TERMS OF WHAT EVIDENCE THEY'RE WILLING TO ACCEPT. THE ONLY REASON MOST DO AN RCT IS BECAUSE THEY'RE TOLD TO. NOT BECAUSE THEY WANT TO. SO HOW DO YOU SIMPLIFY AND HELP PEOPLE NAVIGATE THROUGH THIS PROCESS OF ALL THE OTHER THINGS THAT GO INTO THEIR PROFESSIONAL CAREERS, DOING A NON-RCT TRIAL IS OKAY TO DO AND WE'LL GIVE YOU TENURE AN PROFESSORSHIP FOR DOING THAT. DICK KATZ FROM GW AGAIN. WHEN IT COMES TO WHERE TO GO NEXT AND WHAT OUR RESEARCH AGENDA IS TO HOW TO MOVE THIS ALONG, WE'RE HERE TO MAKE A DIFFERENCE. AND THERE'S STRENGTH IN NUMBERS. I THINK WE HAVE TO START COLLABORATING. WE HAVE THIS CONNECTED HERE AT ONE PLACE, ANOTHER HERE, ANOTHER HERE, (LOST AUDIO ?CHTS) (LOST AUDIO)p CETERA, THERE'S A LOT OF TOPICS THAT MAKE A DIFFERENCE BUT IF WE DO A LITTLE BITS AND PIECES WE'RE NOT GOING TO CONVINCE THE HEALTHCARE SYSTEM THAT WE WERE GOING TO MIC A DIFFERENCE FOR A WHILE. -- TO MAKE A DIFFERENCE FOR A WHILE. >> ONE GROUP I MIGHT ADD IS ACADEMIA, IT'S WELL REPRESENTED IN THIS ROOM BUT I WOULD PUT IN ACADEMIA AS WHAT YOU CALLED IT, AL, ESTABLISHMENT ACADEMIA. AND I THINK TO CHARLENE'S POINT THERE NEEDS TO BE DEFINED A SCIENCE OF mHEALTH THAT'S VERY IMPORTANT. I'M NOT SURE mHEALTH IS SEEN AS A SCIENCE, IT'S SEEN AS SOMETHING COOL AND FUN BUT THERE IS A SCIENCE, IT'S A VERY IMPORTANT SCIENCE. THE IMPORTANCE OF DEFINING THAT IMPACTS ALSO ON HOW WE DEVELOP THE FIELD AS A SCIENCE, CAPACITY OF SCIENCES IN mHEALTH. THAT GETS BACK TO THE POINT OF REVIEWERS AND GRANTS AND FUNDING SO A CONCRETE ACTION ITEM I WONDER ABOUT IS WOULD NIH HAVE A TRANS-NIH SPECIAL REVIEW PANEL ON mHEALTH THAT HAS REVIEWERS THAT ARE SENSITIVE TO THESE ISSUES THAT WILL REVIEW THE PROPOSALS IN WAYS THAT ARE CONSUMMATE TO THE NEEDS OF M HEALTH LIKE SPECIAL PANELS ON STEM CELLS OR WHAT NOT. IS THAT A WAY TO MAYBE GET AWAY FROM THE HARSH REVIEWS PEOPLE GET? IN RCT IT'S EXPECTED BUT NOT REALLY THE RIGHT THING TO DO. >> IT WOULD BE FANTASTIC TO COME OUT OF HERE WITH BASIC SCIENTIFIC QUESTIONS THAT AS OPPOSED TO DEVELOPMENT QUESTIONS. EVERYBODY UNDERSTANDS THE DEVELOPMENT ISSUE. SCIENTIFICS ARE HARDER TO CAPTURE. >> THOSE SCIENTIFIC QUESTIONS ALONG WITH THAT TO SAY TO APPROPRIATELY ASSESS PROPOSALS THAT ANSWER THESE SCIENTIFIC QUESTIONS, THESE THE EXPERTISE NEEDED TO THE REVIEW PANEL AND THEREFORE DRIVE THE COMMUNITIES THAT EVALUATE PROPOSALS. >> CAN I -- THE STATISTICIANS ARE GOING TO HAVE TO LEAVE THE TWO STATISTICIANS, CAN I JUST MAKE A STATEMENT BEFORE WE LEAVE? I'M CONCERNED. I'M CONCERNED, IT'S COMMON IN THE CLINICAL FIELD TO USE THE RCT AS THE BAD BOY AND BEAT IT UP. YOU NEED TO REMEMBER ONE REASON IT SAKES SO LONG TO GET OUR WORK OUT IS IT TAKES FOREVER TO RAMP UP BECAUSE WE DON'T HAVE OPEN SHARING OF IDEAS, OPEN SHARING OF TOOLS. SO ON. THAT TAKES IT FOREVER JUST TO RAMP UP. THE SECOND REASON IS THAT PUBLICATION TAKES FOREVER. RCT TAKE AS LISTENING TIME TO PUBLISH. >> RCT MAKES IT QUICKER. >> IF ANYTHING AND THERE'S THE NEED FOR GOOD PROXIMAL OUTCOMES. THAT'S ANOTHER REASON WHY IT TAKES US FOREVER. THAT'S NOT THE RCT FAULT EITHER. YOU CAN HAVE A PROXIMAL OUTCOME IN RCT. WE NEED TO BE CAREFUL, LET'S NOT THROW THE BABY OUT WITH THE BATH WATER HERE. THERE'S REASONS IT TAKES FOREVER TO GET SOMEWHERE. THE RCT ITSELF WITH PROXIMAL OUTCOME RANDOMIZATION IS NOT AN AWFUL THING WITH A PROXIMAL OUTCOME. YOU CAN HAVE STUDIES THAT LAST FOUR MONTHS. THIS IS NOT A BIG DEAL BUT WHEN IT TAKES A AREA TO PUBLISH OR TWO YEARS THAT'S A BIG DEAL. OR A YEAR TO RAMP UP THAT'S A BIG DEAL. RECRUITMENT TIME. THAT'S THE THING THAT SHOULD BE HERE, ALSO THAT YOU CAN RECRUIT MUCH QUICKER WITH THESE TECHNOLOGIES. >> LET ME USE THIS AS A SEGUE. DO YOU WANT TO COME UP AND -- >> I THINK MOST OF US HAVE HEARD FROM SEVERAL OF US THAT THIS WAS EXTREMELY VALUABLE DISCUSSION THAT I HAD TODAY AND WHAT WE WANT TO DO NOW IN THIS NEXT IS TO DECIDE HOW WE CARRY THIS THROUGH. AFTER WE ARE DONE WITH THIS MEETING THE FASHION THE VISION THAT SEVERAL OF US HAVE EXPRESSED THAT WE DO BELIEVE IN mHEALTH AND WE WANT IT TO MOVE FORWARD WHAT ARE ACTIONS WE HAVE TO DEFINE TO GET US THERE. THE LAST THING WE WANT IS WE HAVE ENERGETIC DISCUSSIONS TODAY AND MOMENT WE WALK OUT THE ROOM THERE ARE SOME COLLABORATIONS AND THAT'S BASICALLY IT. IT IS NOT GOING TO GET THERE, ALL OF US REALLY WANT TO BE THERE. SO THERE ARE A FEW CONCRETE ACTION ITEMS THAT WE WOULD LIKE TO DEFINE FROM HERE. ON WORDS. THERE ARE THREE MAYOR AREAS OF FOCUS THAT I STARTED WITH. ONE WAS INFRASTRUCTURE, SECOND WAS DESIGN, THIRD WAS ANALYTIC METHODS. ALL DESIGNED TO REDUCE THE TIME TO RIGOROUSLY EVALUATE MHEALTH-BASED INTERVENTIONS OR TREATMENTS IN THREE AND A HALF YEARS AND GET IT TO PUBLICATION. SO DURING THE DISCUSSION IT EMERGED THERE ARE TWO OTHER IDIOTS WHICH ALSO NEED TO BE MAJOR FOCUS OFFERS ON RIGHT. ONE IS TRAINING AN EDUCATION. NOT REALLY OFF THOSE WHO ARE ENTERING THIS WORK FORCE THROUGH BU ALSO THOSE RESEARCHERS LIKE US. SEVERAL ARE TRAINED IN ONE DI PLIN BUT IF WE WANT TO WORK IN THIS HEALTH AREA WHICH INHERENTLY IS A MULTI-DISCIPLINARY FIELD, HOW DO WE DEVELOP THE RIGHT TRAINING PROGRAM. BOTH RESEARCHERS AND THOSE ENTERING THE WORK FORCE ARE EDUCATED. THEN WE HEARD THE PUBLICATION. AND THIS PUBLICATION PIPELINE IS BECOMING EFFICIENT IN SOME AREAS BUT IF WE WANT TO GET THERE TO OUR GOAL OF 3.5 YEARS, THAT NEEDS TO BE MADE MORE EFFICIENT. WE HEARD SOME IDEAS YESTERDAY BUT WE NEED TO DWOIPT, DEFINE THE RIGHT ACTION ITEMS SO WE CAN DISSEMINATE IT AND MAKE IT HAPPEN. FOR EACH MAJOR FOCUS AREAS WE WANT TO DEFINE THE RESEARCHERS AND ALL ACTS WE SHOULD BE TAKING AND THEN WE WANT TO DISSEMINATE THAT IN THE LARGER COMMUNITY. VIA PUBLICATIONS IN JURY ROOMS, mHEALTH SUMMIT, VIA PANELLIZATIONS AT NATIONAL INTERNATIONAL MEETINGS. AND IN VARIOUS DISCIPLINES THAT HOLD THESE MEETINGS SO THAT WHAT WE DISCUSS IN THIS ROOM AFTER WE CRYSTALLIZE OUR TARGETS, THAT CAN BECOME INDICATOR TO A LOT MANY PEOPLE AND WE AS A COMMUNITY CAN COME TO A CONSENSUS. IN TERMS OF THE CONCRETE ACTION ITEMS WE WANT TO CALL UPON SOME TO LEAD AND CONTRIBUTE TO THIS PUBLICATION. WHAT DO WE WANT IN THIS PUBLICATION? OUR VISION FOR THIS PUBLICATION IS THIS DEFINES THE RESEARCH MEANING THAT PUBLICATIONS THAT FOLLOW THIS SHOULD BE ABLE TO REFER TO IT AND SAY WE HAVE SOLVED THIS PROBLEM WHICH WAS OUTLINED HERE IN THIS PUBLICATION. SO WE HAVE -- THE AGENDA WAS OR THE CHARGE TIME FROM DESIGN TO PUBLICATION TO THREE AND A HALF YEARS AND THE FOCUS ELSEWHERE FROM PUBLICATION PIPELINE AND DESIGN AND MATTERS AND WE HAVE COME UP WITH THIS OTHER METHOD WHICH REDUCE THIS IS TIME FROM THIS TO THIS, FOR mHEALTH BASED INTERVENTIONS SO THAT'S WHAT WE WANT THIS PUBLICATION TO SERVE AS, IN MULTIPLE DISCIPLINES AND VENUES SO THAT IT CAN BE DISSEMINATED WIDELY. WE ALSO WANT TO CALL UPON YOU TO HELP AT VARIOUS CONFERENCES AND WE ALSO LAKE YOUR HELP IN FURTHER REFINING THE RESEARCH AGENCY. FOR THOSE WHO WE WOULD LIKE TO CALL UPON TO HELP LEAD THIS EFFORT, ENGAGE THE IMMUNITY OF PARTICIPANTS HERE AN THOSE ON THE WEB, TO CONTRIBUTE TO DEVELOPMENT OF RESEARCH AGENDA, AND CALL UPON YOU TO CONTRIBUTE TO THIS. SO IN ORDER TO MAKE THIS HAPPEN, WE'RE GOING TO SET UP A COLLABORATION SITE AND WILL EMAIL YOU SHORTLY ABOUT IT. AND THAT WILL MAKE IT EASIER TO SHARE IDEAS AND DEVELOP THIS AGENDA GOING FORWARD. SO IF YOU FEEL AS STRONGLY ABOUT ANY OF THESE FOCUS AREAS, FEEL FREE TO EMAIL ME IF YOU CONSIDER MAKING A MORE ACTIVE CONTRIBUTION TOWARD DEVELOPMENT OF THIS AGENDA AS WE MOVE FORWARD. ANY SUGGESTIONS FOR VENUES WE SHOULD DISSEMINATE THE OUTCOME OF THIS MEETING, EMAIL US AT YOUR EARLIEST CONVENIENCE. SO THOSE ARE THE CONCRETE ACTION ITEMS SO PLEASE WATCH FOR EMAILS FROM THE ORGANIZING COMMITTEE. AND WOULD LIKE TO CONCLUDE BY SAYING BE THE NEW BEGINNING FOR MAKING mHEALTH A SCIENCE. THANK YOU. [APPLAUSE] (OFF MIC) >> BEFORE WE SHUT THAT DOWN MAYBE WE CAN HAVE A DISCUSSION ABOUT THESE THINGS HOW BIS TO DO IT. >> WERE WE TALKING ABOUT HOW TO SHORTEN THE TIME LINE THE THREE AND A HALF YEARS? IS THAT THE POINT OF DISCUSSION? THE CANOPY? OKAY I'LL TAKE THE OPPORTUNITY TO KIND OF MAYBE REPEAT SOME OF THE THINGS -- DALLAS (INAUDIBLE) FROM UCLA SCHOOL OF MEDICINE. I'M A BEHAVIORAL SCIENTIST INTERVENTIONIST. SO FOR ME LOOKING AT mHEALTH THERE'S TWO WAYS TO PUT A SPIN ON THE EVIDENCE. ONE IS WAYS TO GENERATE EVIDENCE FOR mHEALTH INTERVENTION AN ANOTHER IS mHEALTH IS A TOOL TO GENERATE ALL KINDS OF INTERVENTIONS AND IF THAT EVIDENCE PRACTICE BASED EVIDENCE IN TERMS OF BREAKING THE RCT OR NOT I'M A FAN OF THE RCT AND IT'S GREAT. I THINK WE CAN GET OUR EVIDENCE OUT THERE QUICKER BECAUSE THE mHEALTH TOOLS IN OTHER WORDS MOBILE PHONE AS A PARTICIPANT IN AN INTERVENTION AND PROVIDER DELIVERING THE INTERVENTION WE GET DOSE, EXPOSURE, MONITOR, IN REAL TIME, VERY MUCH CLOSER TO THE PROXIMAL BEHAVIORS THAT ARE HAPPENING. SO MAYBE I HAVE A RCT FUNDED BUT IT'S POSSIBLE TO GET A STRONG SIGNAL IN THE FIRST YEAR OF THE IMPLEMENTATION BECAUSE I HAVEN'T (INAUDIBLE) GOOD EXPOSURE DOSE INFORMATION SO I DON'T HAVE TO WAIT TO PUBLISH THE RESULTS. THE OTHER THING I WOULD LIKE THE SAY IS THAT EARLIER IT WAS SAID THE RCT SHOULD BE CONCEIVED OF AS A CONFIRMATORY THING AND OFTEN THE ASSUMPTION IS THAT ONCE WE COMPLETE THE RCT, NOW THE INTERVENTION IS NOW EVIDENCE BASED AND VALIDATED AND IF WE DISSEMINATE IT IT WILL CONTINUE TO BE EFFICACIOUS FOR EFFECTIVE IMPLEMENTATION WHICH IS I BELIEVE A FALLACY. BECAUSE IN REALITY IS, ONCE IT LEAVES MY SHOP AND GOES OUT TO THE WORLD, IT'S BEING CONSTANTLY CHANGED, ADAPTED, ALWAYS A PERPETUAL BETA BUT MY COLLABORATORS CALL THE BCB, -- CBC DOES MONITORING AND EVALUATION OF PARTICIPANTS AN PROVIDERS BUT IF WE CAN EVALUATE TOOLS INTO PREPARATION OF OUR INTERVENTION AND DISSEMINATION OF THEM, THEN WE HAVE ONGOING MONITORING EVALUATION, WE HAVE ONGOING PRACTICE-BASED EVIDENCE AND THIS EXTENDS NOT ONLY FROM THE RCT BUT BEYOND THE 20 TO 30 YEAR POTENTIAL LIFE CYCLE OF AN INTERVENTION. (INAUDIBLE). FROM >> CAN I USE MY MIC? JUST I THINK THIS IS TERRIFIC. JUST TO SAY I THINK IT'S AN– VERY NICE SUMMARY OF SOME ACTION AREAS AND THAT AT LEAST INDIVIDUALLY I WOULD ABSOLUTELY EMBRACE. AND BE VERY EXCITED TO PARTICIPATE IN. I JUST WANT TO ADD COUPLE OF THINGS, ONE IS THAT THERE ARE SOME OTHER AGENCIES AT THE GLOBAL LEVEL ALSO KEEN TO -- IN THIS AREA AND I THINK WOULD BE VERY EXCITED ABOUT THE OUTCOMES OF THIS MEETING. AND WOULD HAVE THEIR OWN WAYS OF ENGAGING. AND I THINK I CAN THINK OF A COUPLE OF DONORS AS WELL THAT COULD ALSO -- WOULD I THINK BE VERY KEEN TO COLLABORATE IN THIS AREA. SOME UN AGENCIES AN TECHNICAL AGENCIES SO JUST THE ADD THAT IN. ALSO THIS -- I THINK IT HAS BEEN A VERY DOMESTICALLY FOCUSED MEETING AND THERE'S NOTHING WRONG WITH THAT BECAUSE I THINK THERE'S A LOT OF THE INTERESTING INNOVATIONS IN SCIENCE ARE COMING OUT OF THIS. I THINK THERE ARE OTHER KINDS OF DIFFERENT INNOVATIONS, METHODOLOGICAL AND IN mHEALTH COMING OUT OF THE GLOBAL. AND I THINK IT WOULD BE VALUABLE TO BRING THOSE TWO COMMUNITIES TOGETHER. AND JUST TO SAY WE AT WHO WOULD BE PROUD TO PLAY WHATEVER ROLE WE CAN IN THIS AND LOOK FORWARD TO IT. >> THANK YOU. >> THIS IS CRAIG LEFEBVRE AGAIN, TWO SUGGESTIONS TO PUSH THIS ALL FORWARD. ONE WOULD BE I THINK THE IDEA OF POST-DOCTORAL TRAINING FELLOWSHIPS WOULD BE REALLY SMART WAY OF JUMP STARTING CAREER DEVELOPMENT AND TRAINING IN THIS AREA. BECAUSE THE PROCESS OF DEVELOPING COURSES AND GRADUATE MAJORS IN UNIVERSITIES AND COLLEGES WILL TAKE SEVERAL YEARS. SECOND THING, WHY NOT CREATE A MOBILE HEALTH JURY ROOM FOR THIS IMMUNITY TO SUPPORT AND PUBLIC AS A WAY OF ALSO EXPERIMENTING WITH DIFFERENT WAYS OF SPEEDING UP GOOD RESEARCH PROGRAMS. >> SO I JUST WANTED TO TALK ABOUT WAYS TO SHORTEN THE TIME LINE TO PUBLICATION REITERATE SOMETHING DALLAS SAID. WE BOTH WROTE IN TIME FOR THE DEVELOPMENT OF THE APPLICATION WHICH I THINK WE DO NEED TO THINK ABOUT HOW WE FUND THE DEVELOPMENT BECAUSE IT'S NOT FUNDABLE OR ATTRACTIVE IN A GRANT. WHAT WE DID WRITE IN, OUR TYPICAL PHASE SCALEUP IS DEVELOP THE APPLICATION PILOT AT A SITE AND THE PILOT LOOKS SUCCESSFUL, DO IT AT MULTIPLE SITES. I DO BELIEVE THERE'S ENOUGH SITE VARIABILITY TO REALLY UNDERSTAND THE IMPACT OF THE TECHNOLOGY YOU HAVE TO DO IT AT MULTIPLE SITES. BUT HAVING THAT INTERMEDIATE PUBLICATION IS NOT YOUR LANSETT PUBLICATION BUT INFORM ACTIVE TO OTHERS IN THE FIELD IS REALLY IMPORTANT. AND SOMETHING THAT WE CAN DELIVER MORE QUICKLY THAN WE CAN DELIVER OUR MULTI-STAGE -- MULTI-SITE STUDY. SO THAT IS SOMETHING WE'RE TRYING TO DO TO INFORM OUR PRACTICE BUT HOPEFULLY CONTRIBUTING TO THE PRACTICE OTHERS ARE DOING AND ALSO AGAIN, WE TALKED ABOUT THIS EARLIER BUT WORTH REITERATING HERE DOCUMENTING WAYS TO PURSUE THIS, HAVING INTERMEDIATE OUTCOMES ON THE LOGIC PATHWAY TO OUR LARGER GOALS AN COMING UP WITH STUDIES THAT ARE LOOK AT THOSE INTERMEDIATE OUTCOMES AS A WAY TO SHORTEN OUR STUDY TIME ARE ALSO ONE POSSIBILITY. >> SEEMS TO ME WE'RE REVERTING TO THE THINGS THAT ARE COMFORTABLE TO US SO WE'RE REVERTING TO DOING TRIALS THAT IF WE'RE MAKING SMALL CHANGES IN TRIAL DESIGNS ARE TRADITIONAL IN CONCEPTION. WE'RE TALK ABOUT ACADEMIA AS IT EXISTS AND HOW WE CAN TWEAK HOW WE MOVE FORWARD IN ACADEMIA, HOW WE TWEAK HOW WE MOVE FORWARD IN PUBLICATION AN TRADITIONAL PUBLICATIONS BUT THAT MOST OF THESE THINGS ARE NOT GAME CHANGERS. SO I THINK PART OF THE CHALLENGE IS WHAT WOULD BE THE GAME CHANGERS. I WOULD SUBMIT THERE'S A LOT OF STAKEHOLDERS IN THIS UNIVERSE THAT AREN'T IN THIS ROOM. I DONE SAY THAT BY WAY OF CRITICISM, THERE'S NOT THAT MUCH ROOM IN THE ROOM. HOWEVER, AS WE TALKED ABOUT THE POSSIBILITY OF GAME CHANGING, WE HAVE TO THINK ABOUT HOW TO ENGAGE THEM NOT AS SUBJECTS, NOT AS SECOND TIER PARTICIPANTS BUT HOW TO CHANGE THE DESIGN OF THE DISCOURSE. SO THERE'S DIFFERENT WAYS TO DO THAT. I CAN'T TELL YOU WHICH ARE WORKING BECAUSE IF ANY KNEW WHICH WOULD WORK THAT WOULD BE THE ONE WE'RE DOING. PART OF WHAT THE ONUS ON EVERYBODY IN THE ROOM AND THE BIGGER COMMUNITY IS ACTUALLY BILL A IMMUNITY AND BUILD A REPOSITORY, I DONE MEAN ANYTIME PHYSICAL SENSE OR NECESSARILY IN A TRADITIONAL SENSE BECAUSE PLACE WHERE PEOPLE'S THOUGHTS GATHER, TOOLS GATHER PEEPING'S -- PEOPLE'S DATA GATHER AND YOU COLLECTIVELY COME ONE A WAY WORKING AS A COMMUNITY SO THAT IT ISN'T SO PIECE MEAL. BECAUSE STAKEHOLDERS ARE DIVERSE THEY WILL LOOK IN PLACES THAT ARE THE ONES THEY'RE USED THE LOOKING IN SO YOU HAVE TO MAKE A NEW PLACE THAT EVERYBODY IS GOING TO NOW KNOW TO LOOK FOR THE THINGS THAT WILL BE SHARED. PEOPLE COMMENTED EARLIER TODAY PROBLEMS WITH VOCABULARY, PROBLEMS WITH NOSOLOGY, TAXONOMY THAT'S TRUE. BUT WILLING PEOPLE CAN LEARN OTHER LANGUAGES, SOME MORTAL LENNED THAN OTHERS BUT WILLINGNESS GOES A LONG WAY. SO I THINK SOMETHING ALLUDED TODAY TO THE GENOMICS IMMUNITY. THEY HAD THE MEETING IN THE BAHAMAS HOW THEY'RE GOING TO OPERATE IN THE COMMUNITY, THEY USED MORAL SWAITION TO TELL PEOPLE IF THEY DID NOT BEHAVE THE WAY THEY WERE SUPPOSED TO THEY WEREN'T GOING TO GET AIR TIME, NO REAL ESTATE JOURNALS, THEY WEREN'T GOING TO GET SPEAKING ENGAGEMENTS, THIS WASN'T WRITTEN DOWN LIKE A BAD PERSON OR BAD ACTOR YOU'RE NOT GOING TO BE PART OF THE IMMUNITY BUT THERE WAS A LOT OF MORAL PRESSURE. I THINK IN A GOOD SENSE YOU HAVE THE OPPORTUNITY TO BUILD THAT COMMUNITY NOW. AND I SAY YOU BECAUSE I'M HERE BECAUSE I DIRECT THE PUBLIC PRIVATE PARTNERSHIP PROGRAM AT THE NIH, NOT BECAUSE I'M A PRINCIPLE IN THE mHEALTH ENVIRONMENT. I THINK PART OF THIS IS YOU'RE ONLY COOL AS A HAMMER EVERYTHING THAT LOOKS LIKE A NAIL, TRY PUBLIC PRIVATE PARTNERSHIPS AS WAY OF GETTING FOLKS IN INDUSTRY, FOLKS IN GOVERNMENT, IN POLICY, FOLKS IN ACADEMIA, THE FOLKS IN HEALTH CENTERS DELIVERY AND CARE, FOLKS IN THE U.S., NOT IN THE U.S. TOGETHER. AND REALLY SIT DOWN AND WORK UP THE NUTS AND BOLLS HOW TO ENGAGE WITH ONE ANOTHER. YOU HAVE AN OPPORTUNITY BECAUSE THE FIELD IS YOU CAN TO MAKE A HUGE DIFFERENCE TO THE TRAJECTORY OVER TIME. THAT'S MY SCREAM. WE CAN GO ON FROM THERE. >> DONNA (INAUDIBLE) UNIVERSITY OF SOUTHERN CALIFORNIA. I COULDN'T HAVE SAID THAT BETTER. I WON'T TRY. THREE THING IS THOUGHT OF, THE FIRST ONE I'LL SAY LAST BECAUSE IT'S HERETICAL. BUT ONE THING WE TALKED A NUMBER OF TIMES ABOUT GRANT REVIEWERS AT DIFFERENT INSTITUTES WHO ARE NOT SAVVY ABOUT MOBILE HEALTH. I KNOW IN THE INSTITUTIONAL REVIEW BOARD WORLD SOME TRAIN PEOPLE IN TRANSDISCIPLINARY RESEARCH BEFORE THEY REVIEW THE GRANTS. I DON'T KNOW WHY WE COULDN'T HAVE SOME SHARED -- WE COULD POOL OUR RESIEWRSES AND MAKE SURE WE HAD TRAINING AVAILABLE FOR GRANT REVIEWERS. I'M HAPPY TO COME TO WASHINGTON AND DO THAT. I THINK IT WOULD BE A HUGE SERVICE AND WOULD BE -- IT WOULD MAKE REVIEWING FUN FOR THOSE ON STUDY SECTIONS. THAT'S ONE THING, WE CAN DO THAT AS A COMMUNITY, ANOTHER THING WE MIGHT WANT TO CONSIDER, ANY OF YOU SEEN THAT JOURNAL WHERE THEY COME TO YOU AND VIDEOTAPE YOUR METHOD SECTION? IT'S TRUE, THEY WANT TO COME VIDEO MY LAB BUT WHAT I THINK IT WOULD BE GREAT TO RETHINK HOW WE PUBLISH SOME OF THIS STUFF. BECAUSE TRADITIONAL METHOD SECTION IS NOT HELPING YOU REPLICATE ANY MORE. AND WE ARE MOBILE SO WHY AROUND WE USING IT TO PUBLISH? WE THINK WHAT A PUBLICATION LOOKS LIKE, HERE COMES ONE WHERE YOU MIGHT THROW STUFF AT ME. SO JUST BOWNGS OFF WHAT YOU SAID EARLIER, YOUR NAME IS BETHANY, WHICH I THOUGHT WAS REALLY SMART, WHEN WE'RE PUTTING TOGETHER THESE TEAMS, AND LEARNING ONE ANOTHER'S LANGUAGE I HAVE SEEN SO MANY TEAMS AROUND ME GO REALLY WELL AND I HAVE SEEN EQUALLY AM OF THESE TEAMS FAIL. I HAVE SEEN GREAT MOBILE IDEA -- MOBILE HEALTH IDEAS MARCH FURTHER AND I HAVE SEEN AN EQUAL NUMBER FAIL. WHY DONE WE SLICE UP THESE GRANTS AND MOVE IT AROUND DIFFERENTLY? WHY DONE REHAVE GO, NO GO MOMENTS SO THAT IF I HAVE A FIVE YEAR GRANT FOR SOMETHING IN YEAR 2 DEAD IN THE WATER, I'LL SPEND THE MONEY FOR SURE. BUT THERE MIGHT BE ANOTHER WAY OF STRUCTURING THESE GRANTS TO MOVE THE FIELD FORWARD A LITTLE BIT MORE QUICKLY. YOU MAY THROW THINGS THAT WILL TAKE MY (INAUDIBLE). >> SO IN TERMS OF PRACTICAL SOLUTIONS. AMY ABEARNATHY, DUKE. IN TERMS OF PRACTICAL SOLUTIONS, DESPITE ALL MY CONVERSATIONS ABOUT ETHICS, ET CETERA, I'M A PROCESS AND OUTCOMES GIRL. AND ONE OF THE THINGS THAT STRUCK ME IN THIS LAST BIT OF CONVERSATION IS IN TRYING TO INITIATE NEW DISCIPLINE AND GOING FORWARD, SOMETHING THAT WE RECENTLY HAD TO DO ELSEWHERE. THREE THINGS THAT WE HAVE DONE VERY EFFICIENT, ONE IS YOUR GOAL IS THREE YEAR CLINICAL TRIAL TO EVIDENCE THAT COULD RIVAL THE RCT. ONE IS TO WORK TO DESIGN TOGETHER A TEAM, A STREAMLINED VERSION OF THAT STORY. THAT MEAN AS QUESTION EVERYBODY AGREED UPON, THE PROCESS OF WORKING TOGETHER TO SAY WHAT'S THE QUESTION, WHETHER TEASE OUTCOME. YOU HAVE O HAVE TWO PARALLELS. AND THE LOGISTICS IS WHAT IT TAKES TO TO IT. SO HOW THE TEAM SITS DONE AND DOES THAT NOT BUZZ -- IT ALLOWS YOU TO FIGURE OUT WHAT IT TAKES TO GET DONE BUT TO BUILD THE NEWS STORY. THE OTHER PART IS GOOD OLE FASHIONED PUBLICITY AND GENERATING THE STORY THAT GOES WITH IT. THAT'S THE FIRST THING, JUST NEED TO THEN MAKE THE AGENDA OF LET'S DO THIS. SECOND IS PROCESS MAPPING. WE HAVE TALKED ABOUT THE USE OF INFORMATION AND INFORMATION TECHNOLOGY TODAY IN ALL KINDS OF SOLUTIONS FOR HEALTHCARE BUT ALSO USE THAT FOR THE SOLUTION TO GETTING TO THE ANSWER OF HOW CAN YOU FIGURE OUT WHAT MHEALTH CAN DO FOR YOU IN EVIDENCE DEVELOPMENT, WHAT ARE THE STEPS IN WHERE ARE LOOPS YOU FALL INTO, ET CETERA A. WE HAVE DONE THIS ACROSS THE COOPERATIVE GROUPS AND IT'S HELPING WITH EFFICIENCY IN CLINICAL TRIALS COOPERATIVE GROUPS. THE THIRD IS DESIGNING IN A SET OF OUTCOMES. SO TBHEERT GOING TO ACCEPT RESEARCH DESIGN AT mHEALTH THAT TAKE TEN YEARS, WHEN WE'RE NOT HITTING THE THREE AND A HALF YEAR BENCHMARK WHAT IS IS CAUSING US PROBLEMS GETTING TO THAT. COMING TOGETHER AS A COMMUNITY SAYING HERE ARE OUR EXPECTATIONS AMONG OURSELVES AND WHAT THESE OUTCOMES LOOK LIKE AND FIGURING OUT WHEN YOU COULDN'T HIT WHAT IT THE PROBLEM WAS. IT EMPHASIZES WHENND WHERE, AND NOT THE TECHNOLOGY, THAT'S WHAT DIFFERENTIATES mHEALTH, IT'S NOT THE TECHNOLOGY CONTINUUM, WE'RE MANY SURE THINGS WHEN AN WHERE THEY OCCUR. MY UNDERSTANDING IS TO AN ALMOST EXCLUSIVE EXTENT WHAT WE HAVE DONE IS MORE MOMENTARY ASSESSMENT. IT'S ECOLOGICAL IN THE SENSE THAT IT HAPPENS TO OCCUR, WE KNOW IT'S OCCURRING OUT THERE. BUT THE BEST WE DO IS WE MIGHT SAY WHERE ARE YOU. I SUGGEST WHAT I TO THINK OUTSIDE THE DEVICE, OUTSIDE THE BOX, OUTSIDE THE MOBILE DEVICE WE CAN DO MORE AND WE HAVE MORE DATA THAN JUST AVAILABLE TO US ON THE PHONE. SPECIFICALLY GOING BACK TO WHAT KEVIN PA I PAPA TRICK MENTIONED WITH GEOGRAPHIC INFORMATION SYSTEMS THAT'S A SOPHISTICATED AREA OF SCIENCE AND DATA METHODOLOGY, THESE KNOWN PHONES WITH GPS AND OTHER FEATURES PROVIDE A PHYSICAL LINK TO WHERE PEOPLE ARE IN THEIR PHYSICAL CONTEXT. THIS GIVES US A WHOLE LINK TO ANOTHER WORLD OF DATA SO FOR INSTANCE NOT JUST WHETHER OR NOT A PERSON IS NEAR THE BAR OR A PERSON IS IN A CERTAIN PART OF THE CITY WITH A CERTAIN TYPE OF POLICY SO WE THINK OF COMMUNITY LEVEL OUTCOMES ASKING COMMUNITY LEVEL HEALTH AS A FUNCTION OF PEOPLE MOVING THROUGH IT FOR SURE. BUT I THINK WE SHALL NOT JUST FOCUS ON WHAT THE PHONES CAN COLLECT BUT ALSO THE WAY WE CAN USE THAT DATA TO CLOSE THE LOOP ON THE THOUGHT. I THINK IT WAS BARBARA MENTIONED POLICY AND OTHER STAKEHOLDERS. PART OF WHAT WE'RE DOENING DC IS WE HAVE COMMUNITY GROUPS WITH OUR KIDS WONDERING WHAT CAN I DO, THEY'RE GOING AROUND WITH CAMERA PHONES AND TAKING PICTURES OF THE WAY TOBACCO IS MARKETED. TAKE YOUR COMMUNITY BACK. SHOW YOUR FRIENDS AND FAMILY HOW THE INDUSTRY IS MEASHING TO YOU. WE CAN GIVE THEM A GIS MAP AND THE POWER TO PRINT THAT IT AND THEY CAN TAKE TO IT THEIR LOCAL COUNCIL MEMBER, AND THEY CAN SAY LOOK AT WHAT -- WE CAN EMPOWER PEOPLE ON THAT LEVEL SO THERE'S WAYS TO CONNECT TO POLICY THAT'S TOTALLY SEPARATE THAN OUR TYPICAL QUESTIONS SURROUNDING BEHAVIORAL OUTCOMES, ET CETERA. >> OKAY. I THINK THOSE ARE VERY INTERESTING DISCUSSIONS. SO TO STREAMLINE IT A LITTLE BIT MORE, WHAT I WOULD SUGGEST IS WHEN WE HEAR YOUR COMMENTS IT WOULD BE NIETS TO PULL IT IN TERMS OF WHAT IS IT THAT WE COULD DO GOING FORWARD, IT WOULD BE EVEN NICER IF YOU COULD SAY HERE IS HOW YOU COULD CRIB OR HOW YOU PLAN TO CONTRIBUTE IN MOVING THAT AGENDA FORWARD. AGAIN I WOULD LIKE TO BRING US BACK TO THE GOAL, WE'D LIKE TO ACHIEVE OR MAKE mHEALTH A SCIENCE FROM HERE ONWARD. WHAT ARE THE WAYS WE CAN GET THERE AND USE THIS AND HOW DO YOU PLAN TO HELP. >> (INDISCERNIBLE) COLUMBIA UNIVERSITY. I HAVE HEARD A LOT OF QUESTIONS AN ANSWERS TODAY. FIRST THING I REACH AS CONCLUSION, NOT SURE IF WE WANT THE REDUCE THE TIME FROM FIVE TO THREE YEARS BUT WHAT IT SEEMS TO ME IS CLEAR IS THE CURRENT RESEARCH APPROACH IS OBSOLETE AND WE HAVE TO COME UP WITH BETTER APPROACHES. I HAVE MADE IN MY WIPE PAPERS HOW TO APPROVE THE ACCEPTABILITY IES IN A NUMBER OF MENTIONS BUT WHAT I WOULD LIKE TO TALK IS WHEN WE DEFINE MHEALTH NOT ONLY AS A SCIENCE BUT THE ULTIMATE GOAL OF HEALTH IS TO IMPROVE HEALTH OUTCOMES. IT'S EVIDENT. I WAS PROVIDED EVERYTHING, EVIDENCE IS NOT THE SAME AS MHEALTH OUTCOMES IN THE REAL WORLD BECAUSE I HAVE DEMONSTRATED IN A NUMBER OF TRIALS THAT I COULD IMPROVE OUTCOMES AND I PROVIDE THAT EVIDENCE BUT THAT EVIDENCE HAS NOT TRANSLATED TIME PRO. OF OUTCOMES IN THE REAL WORLD. SO WHEN WE TALK ABOUT THE STAKEHOLDERS TO ENGAGE WE HAVE TO ENGAGE STAKEHOLDERS THAT CONSIDER THE EVIDENCE BUT CAN ALSO PROVIDE OUTCOMES IN THE REAL WORLD. I BELIEVE DEALING WITH THE PRIVATE SECTOR HERE, IT'S FOR PROFIT, I UNDERSTAND THAT FOR EXAMPLE (INDISCERNIBLE) THEY DON'T CARE ABOUT THE PATIENT BUT THAT IT'S ABOUT THE DIVIDENDS, RIGHT? BUT THERE'S SOME PEOPLE WHO ARE ACTUALLY IN THE PUBLIC SECTOR OR PRIVATE SECTOR THAT THEY CARE. ESPECIALLY SOCIAL ENTREPRENEURS THAT HAVE WHAT THEY CALL DOUBLE BOTTOM LINE SOCIAL IMPACT WHICH WILL BE THE HEALTHCARE OUTCOMES AND FINANCIAL IMPACT. I THINK WE BEGIN TO THINK ABOUT EVIDENCE TO IMPROVE HEALTH OUTCOMES WE HAVE TO ENGAGE SOCIAL ENTREPRENEURS INTO THE EQUATION, THAT GOES BACK TO THE QUESTION THAT AL SHAR WAS SAYING, WHO SHOULD BE INVOLVED IN THIS KIND OF CONVERSATION. I THINK WE SHOULD INCLUDE THESE KIND OF PEOPLE. FINALLY IN TERMS OF MAKING SOMETHING MOVE FORWARD HERE, I THINK THERE'S A LOT OF PEOPLE WITH A LOT OF DIFFERENT INTERESTS AND WHAT I WOULD LIKE TO UNDERSTAND IS WHO HAS THE SAME INTEREST AS I DO SO I COULD TRY TO ENGAGE WITH THEM. SO THERE WAS A WAY WHICH WE CAN ALSO DIVIDE BY INTEREST AND AFFINITIES. THAT WOULD BE VERY INTERESTING. GREATER METRICS OF CLINICAL DIMENSIONS, OR PEEP WHO ARE INTERESTED IN METHODS AND RESOURCES, IN PRIVACY. THAT -- >> OKAY. WE'RE RUNNING OUT OF TIME HERE SO I WANT TO MAKE SURE EVERYBODY TALKS BUT SANTOSH SAID BEFORE WE'LL WORK ON SOCIAL NETWORKING THAT WE CAN SHARE THESE COMMON INTERESTS AND ALSO KEEP THESE QUESTIONS, KEEP WORKING ON SOLUTIONS TO THESE QUESTIONS. >> CHARLENE QUINN UNIVERSITY OF MARYLAND, WHAT I WOULD LIKE TO SEE COME OUT OF TODAY IS A STATEMENT THAT COMES FROM THIS GROUP. BECAUSE WE NOT ONLY WENT THROUGH THE PROCESS OF ABSTRACT AND BEING SELECTED, THERE WAS A LOT OF PUB PUBLICITY IF YOU WERE PARTICIPATING IN THAT. WE HAVE THE POWER OF THE SPONSORS OF THIS MEETING. SO I FEEL LIKE WE OUGHT TO TAKE ADVANTAGE OF THAT OPPORTUNITY. AND WHAT I WOULD LIKE TO SEE IN THAT STATEMENT, I THINK PEOPLE WANT TO HEAR FROM US A AGREEMENT AMONG US WHAT WE THINK THE MAJOR AREAS WHERE MOBILE HEALTH SCIENCE AND I THINK WE OUGHT TO START USING THAT TERM, MAY CONTRIBUTE TO IMPROVING HEALTH. SO NOT -- WE'RE NOT JUST HERE TO TALK ABOUT IF IT WORKS BUT HOW DOES IT WORK. THAT LINKS BACK TO THE DISCUSSIONS ABOUT BEHAVIOR. PATIENT ENGAGEMENT, HOW DO WE TRANSLATE FINDINGS TO THE LAY POPULATION, TO PROVIDERS. I WOULD HOPE WE COULD COME OUT OF THIS MEETING WITH A STATEMENT LIKE THAT BECAUSE I THINK THAT WOULD BE VERY POWERFUL FOR US. AND THEN WE CAN MOVE FORWARD WITH THESE OTHER PIECES. I'M SPENDING THE NIGHT. >> I'LL START DRAFTING. (OFF MIC) >> I MEAN I THINK THIS IS YOUR FIELD. WE'RE HERE TO SIMULATE IT BUT I THINK WHAT CHARLENE IS SAYING IS YOU NEED TO OWN IT. SO I MEAN, I THINK THAT THAT'S A QUESTION. THOUGHTS? YEAH, NAY, YES? SHOULD WE DO THIS? >> YES. >> WITHIN TWO WEEKS? >> TWO MONTHS. >> MONTHS? JESUS. NO. (OFF MIC) >> A DRAFT IN TWO WEEKS, HOWEVER LONG IT TAKES TO -- YES? (OFF MIC) >> WHAT? (OFF MIC) >> OKAY. SO -- YOU RECRUIT TO OTHER PEOPLE WHO WANT TO MAKE THIS HAPPEN. >> CAN I ADD MY COMMENT BECAUSE IT'S RELEVANT. THIS IS (INAUDIBLE). GOOD TIME SO I DON'T WANT TO CO-OPT WHAT I'M DOING SO IF I GO ASTRAY CORRECT ME AND IS A NO. I AGREE PLEATLY. WE SHOULD GET OPERATIONAL AND MAKE SOMETHING ACTIONABLE COME OUT OF THIS IN THE NEXT 25 MINUTES. SO BEING A COMPUTER SCIENTIST I BELIEVE NOT ONLY IN THE DIRECTION I BELIEVE IN RECOURSE SO I THINK WE SHOULD GIVE EVERYONE IN THIS ROOM A MOBILE PHONE AND CREATE A SELF-MONITORING STUDY ARE WE MAKING PROGRESS TO FURTHER MOBILE HEALTH SCIENCE. TONGUE IN CHEEK BUT SERIOUS. WHY NOT USE BEST PRACTICES WE HAVE EVOLVED AND SELF-MONITORING AN MOBILE HEALTH DEVICE AND OTHER THINGS AND TAKE WHATEVER WE DO AFTER I TALK A SET OF GOALS AND MEASURE HOW ACTIONABLE WE ARE TOWARD THIS. WE'RE EATING OUR OWN DOG, WE'RE BOOTSTRAPPING IT AND PRACTICING. WE HAVE BEST PRACTICES AROUND THESE INTERVENTIONS AN SELF-MONITORING AND WE HAVE ALL GOT PHONES. WHY DONE WE USE THAT IN THAT ACTION AND START MEASURING IT. >> WE CAN SMS. >> FIRST I WANT TO APOLOGIZE, I HAD TO GO BACK TO THE OFFICE FOR A COUPLE OF HOURS SO WHAT I'M SAYING IS MINUS A BIG CHUNK OF WHAT HAPPENED BETWEEN 1:30 AND NOW. I WANTED TO REFLECT BACK ON MY JOB AND HOW THIS RELATES TO WHAT'S GOING ON AND MAYBE INTRODUCE A LITTLE CYNICISM. SINCE I HAVE COME TO THE NIH I'M A MEMBER OF SEVERAL COUNCILS AN GUEST OF OTHER COUNSEL COUNCILS SO THE LAST FEW MONTHS I WATCHED A BUNCH OF COUNCIL DELIBERATION DELIBERATIONS. THE PROBLEM YOU MIGHT FACE IS WHAT YOU'RE EXPECTING IS YOU'RE EXPECTING THE COUNCIL'S TO ADAPT TO YOUR LEVEL OF CREATIVITY. YOU'RE SAYING STUDY SECTIONS AND THESE COUNSELS REALIZE THE CREATIVITY OF OUR WORK. I HOPE THEY DO. THE PROBLEM IS THESE GROUPS ARE VERY HETEROGENEOUS. AN INSTITUTE LIKE THE HEART LUNG AND BLOOD INSTITUTE COUNCIL IS VERY HETEROGENEOUS GROUP OF PEOPLE. AND THEY'RE CYNICAL. SO YOU HAVE TO DO THINGS THAT PERSUADE THEM WHAT YOU'RE DOING IN mHEALTH IS SOMETHING THAT'S GOING TO CHANGE QUALITY OF LIFE EXPERIENCE AN YEARS OF LIFE AN QUALITY OF LIFE IN A WAY COMPREHENSIVE TO THEM. I'M A LITTLE WORRIED ABOUT SOME DISCONNECT TODAY THAT WHAT I THINK THAT NEXT STEP OUGHT TO BE IS THE DEMONSTRATION THAT WHAT YOU'RE DOING IN mHEALTH PROVIDES EVIDENCE, USING RULES THAT THESE PEOPLE UNDERSTAND THAT YOU HAVE A BIG EFFECT ON HEALTH OUTCOMES SO PEOPLE ARE INTERESTED IN THESE PROBLEMS, WE KNOW A LOT ABOUT CONTROLLING BLOOD PRESSURE OR A LOT ABOUT CONTROLLING BLOOD CHOLESTEROL BUT NOT HITTING THE MARK BECAUSE ADHERENCE IS POOR. AND THE CREATIVITY IN THIS IMMUNITY HAS THE POTENTIAL TO REALLY CHANGE THAT OR IN THE HIV IS INCREASING EVIDENCE THAT WE CAN MAKE BIG PROGRESS AGAINST NUMBER OF INCIDENT CASES OF HIV IF WE'RE ABLE TO GET PEOPLE TO ADHERE TO THESE PROTOCOLS THAT WE'RE NOT HITTING THAT MARK. SO I GUESS WHAT I'M SAYING IS WHAT I HAD HOPED THAT I WOULD HEAR TODAY IS THE EMERGENCE OF THESE -- OF THIS EVIDENCE THAT YOU'RE CHANGING THESE IMPORTANT HEALTH OUTCOMES. (OFF MIC) >> YEAH. ONE OTHER REACTION I HAD, THERE'S TREMENDOUS CREATIVITY IN THIS GROUP, IT'S VERY CLEAR, AND THERE'S TREMENDOUS EXCITEMENT ABOUT POTENTIAL AND IT SEEMS TO ME THAT WE NEED TO GET BEYOND THAT. SO OF COURSE THESE TECHNOLOGIES PROVIDE OPPORTUNITIES TO ADVANCE THESE FIELDS. SO NOW LET'S GET ON TO IT. EVEN IN THE RESEARCH DESIGN, THE RESEARCH DESIGN SECTION IS WONDERFUL TODAY. THOSE WERE GREAT PRESENTATIONS. I HAVE A LITTLE BIT OF CONCERN THAT WHEN YOU GO TO -- ARCANE REVIEW SYSTEM, THE REVIEW SYSTEM IS OUT OF DATE SO THE DIFFICULTY AND YOU HEAR THIS IN THE COUNCIL DISCUSSIONS, SO THERE WAS A GREAT COMMENT MADE THIS MORNING THEY SAID SOMEBODY SAID WE OUGHT TO TEND VARIABILITY BECAUSE WHEN YOU SEE A GRANT REVIEWED AND THERE'S BUNCH OF DISCOURSE BUT A LOT OF VARIABILITY THAT MEAN AS LOT OF CREATIVE APPLICATION. THE REALITY IS WITH WHERE WE ARE NOW, AGAIN, HAVING SAT THROUGH THESE COUNCIL SESSIONS I KNOW PEOPLE DON'T WANT TO HEAR THIS BUT FIRST GRANTS ARE TRIAGED THEN OTHERS ARE SCORED AN TYPICALLY RANK ORDERED BY PERCENTILE AND THE REALITY IS IN THE NEXT YEAR IF YOU'RE NOT BELOW THE 10TH PERCENT YOU'RE PROBABLY NOT GOING TO GET FUNDED. SO THAT IT'S GOING TO BE TOUGH THE NEXT YEAR OR TWO. THAT MEANS THAT PROBABLY THINGS HAVE TO CONFORM THE RULES THESE REVIEWERS UNDERSTAND. >> MY COMMENT BEFORE WAS CONFIRMING EVERYTHING SO MY TURN. SO I'M NOT GOING TO RESPOND TO WHAT YOU SAID BECAUSE IT'S -- I'M SURE ALL TRUE BUT A LITTLE -- BUT TOO DEPRESSING. BUT SO I WANTED TO PUT OUT A CALL, I THINK A STATEMENT WE SHOULD HAVE A HEADLINE WE HAVE. I ALSO THINK WE'RE READY TO BUILD AND SHARE TOOLS BEFORE YOU GUYS FIGURE OUT HOW TO FUND US DIFFERENTLY. SO WE'D LIKE TO PUT OUT A CALL FOR KNOWING WHAT COMPONENTS, TOOLS, MODULES, PEOPLE AROUND HERE WOULD LIKE TO HAVE BOTH IN THE COMMERCIAL CONTEXT AS WELL AS IN THE ACADEMIC ONE AS WELL AS WHAT YOU HAVE TO PROVIDE. THAT WE CAN START MODULARIZING WHAT WE DO. AND START IN A MORE GRASSROOTS WAY BY ACTUALLY SHARING AGAIN BOTH IN THE OPEN AS WELL AS IN THE COMMERCIAL CONTEXT SO I THINK WE CAN LEAVE HERE NOT -- I'M A LITTLE SCARED BY CONSORTIA AND DIVIDING UP, I DON'T THINK WE'RE QUITE LARGE ENOUGH TO DIVIDE. WE SHOULD BE SHARING AS MUCH AS POSSIBLE AND TRY TO PURSUE WITH DOING SOME OF THESE THINGS AND THE WAY WE DO LITTLE THINGS TO GET PILOT DATA TO GET YOUR PROPOSAL FUNDED. LET'S TRY TO WITH THE RESOURCES THAT WE HAVE START TO MODULARIZE, OPEN UP AND SHARE. >> I THINK TO BUILD ON DEBRA'S IDEA, WENDY NILSEN FROM NIH, DICK HAD A GREAT IDEA ABOUT WE CAN WORK TOGETHER. WE CAN BUILD ON EACH OTHER'S WORK BUT ALSO IN THESE TRIALS. I THINK GARRETT'S PORTRAIT OF WHAT HAPPENS IN mHEALTH TRIALS HOW LITTLE EVIDENCE THERE IS, WE SEE AT THE mHEALTH SUMMIT AGAIN AND AGAIN DEATH BY PILOT BECAUSE WE DO THESE LITTLE THINGS ALL OVER SO IT'S ANOTHER WAY YOU CAN THINK AB COMING TOGETHER TO WORK. I >> I WOULD LIKE TO ACTUALLY RESPOND OR SUPPORT WHAT BOB WAS SAYING BECAUSE I'M A LITTLE BIT CONCERN ABOUT THE STRATEGY THAT WE SHORTEN THE TIME TO PUBLICATION. IT'S ACADEMIC. WE HAVE TO ESTABLISH OBJECTIVE FUNCTION AND WHY ARE WE SUGGESTING ONE, QUALITY OF LIFE. HEALTHCARE OUTCOMES. THOSE ARE MEASURES OF METRICS THAT WE NEED TO USE IN ORDER THE MAKE IT USEFUL FOR THE MEDICAL COMMUNITY NOT JUST FOR -- IN TERMS OF FUNDING BUT IN TERMS OF ACCEPTANCE BY THE VAs, THIS IS KEY, I WOULD BE VERY DISAPPOINTED IF OUR HEADLINE DOESN'T HAVE A FOCUS THAT GOES WELL BEYOND PUBLICATION. >> MISHA, OUTCOME. MANY ACTION ITEM, WHAT'S THE ACTION ITEM? >> FOR EXAMPLE, IMPROVING QUALITY OF LIFE. AND MEASURING QUALITY OF LIFE. AS OPPOSED TO PUBLICATION. OUTCOME MEASURES. >> YES. OUTCOME MEASURES WOULD BE ONE OF THE RESEARCH ISSUES. I THINK THE CURRENT APPROACH IS TO MEASURING QUALITY OF LIFE. LIMITED TO METRICS THAT HAVE BEEN DEVELOPED ET CETERA AND A LOT OF WORK WENT INTO THAT, THIS MOBILE HEALTH FOR EXAMPLE CAN ACTUALLY MAYBE HAVE A HUGE IMPACT OF UNDERSTANDING THE QUALITY OF LIFE OF THE PERSON WHO OWNS THE MOBILE PHONE. I JUST WANT TO RAISE A LITTLE BIT MORE POINTY-HEADED ISSUE THAN WE HAVE BEEN TALKING AB. THAT IS -- TALKING ABOUT. ONE THING WE TALKED ABOUT -- BILL AND I TALKED A LITTLE BIT AS A BEHAVIORAL SCIENTIST I THINK I GET MORE AND MORE IMPRESSED WITH THIS. WE DON'T REALLY UNDERSTAND HOW PEOPLE INTERACT WITH THESE TECHNOLOGIES. WE HAVE -- THERE'S A LOT OF BEHAVIORAL THEORY ABOUT BEHAVIOR CHANGE BUT I PERSONALLY AM IMPRESSED AT HOW I BUILD SOMETHING AND THEN PEOPLE DO SOMETHING COMPLETELY DIFFERENT WITH IT THAN I EXPECT. THEY MIGHT DO SOMETHING ACTUALLY QUITE USEFUL. BUT IT'S NOT WHAT WE EXPECTED. SO I THINK THAT -- ON THE BEHAVIORAL SIDE A LOT OF OUR THEORIES, THEY'RE BREAKING DOWN AT THIS POINT. ONE IMPORTANT DIRECTION TO MOVE THE FIELD FORWARD IS TO UPDATE OUR BEHAVIORAL THEORY TO TAKE INTO ACCOUNT HOW PEOPLE INTERACT WITH THE TECHNOLOGIES THAT WE HAVE NOW AND SO WE TALK AD LITTLE BIT ABOUT IT BUT IF PEOPLE WANT TO OR ARE INTERESTED WORKING ON THAT I WOULD BE HAPPY TO TRY TO ORGANIZE THAT. >> AL SHAR. THAT DOESN'T CHANGE THE QUESTION. I THINK THERE'S -- I HEARD A CONSENSUS THAT WE SHOULD COME OUT WITH A HEADLINE. A HEADLINE ISN'T A RESEARCH PROGRAM. IT IS A STATEMENT OF COMMONALITY AND BELIEF. WE BELIEVE THAT THE SCIENCE OF MOBILE HEALTH CAN IMPROVE THE QUALITY OF LIFE OR SOMETHING LIKE THAT. FOR CONSUMERS PATIENTS, WORLD, WHATEVER. I'M COME BACK TO SAYING THAT. WE SHOULD COME OUT WITH SOMETHING BECAUSE PEOPLE THIS DID GENERATE EXCITEMENT NOT ONLY IN THIS RM BUT OTHER PEOPLE AND IF WE CAN COME OUT WITH SOMETHING THAT SETS THE STAGE FOR DEVELOPING AN AGENDA WHEREVER IT GOES, WE HAVE DONE SOMETHING WE NEEDED TO BE DOING. (OFF MIC) >> A PRIMARY IS BECAUSE I I WAS NEVER A MATHEMATICIAN AT SCHOOL. I GOT A C FOR MY MATH. SO I HAVE BEEN LEARNING A LOT TODAY. THE SECOND REASON IS BECAUSE WE REPRESENT THE INTERESTS OF 830 MOBILE NETWORK OPERATORS AROUND THE WORLD. WHEN WE WORK WITH THE OPERATORS THEY'RE INTERESTED IN MOBILE HEALTH. VERY INTERESTED IN DOING TRIALS IN MOBILE HEALTH BUT NOT TRIALS ON 150 OR EVEN A THOUSAND OR EVEN 10,000 PEOPLE. WE HAVE OPERATORS KNOCKING ON OUR DOOR AT THE MOMENT SAYING CAN YOU HELP US DO TRIALS OF 80 MILLION PEOPLE. SO MAY I ASK TO YOU AND TO REITERATE A NUMBER OF COMMENTS TODAY HOW CAN WE DEFINE AN ENGAGEMENT STRATEGY FOR INDUSTRY. THAT'S REALLY THE THAT'S THE CRITICAL POINT BECAUSE AFTER 8 HOURS OR ASSESSING WHAT'S GOING ON WHAT CONVERSATION IS, I CAN CONFIDENTLY SAY A BUSINESS DISCONNECT BETWEEN WHAT OPERATORS ARE WANTING TO DEVELOP EVIDENCE AND WHAT SEEMS TO BE THE TOPIC OF CONVERSATION OR THE MESSAGES COMING OUT OF TODAY. DEFINEING A PROCESS FOR ENGAGEMENT WANTING TO DO SOMETHING IN THAT REGARD AND TO BRING THE OPERATORS TO THE TABLE TO BRING THE OPERATORS TO THE CONVERSATION. >> CARRIE BROCK FROM THE MCKESSON FOUNDATION. WET BLANKET DEBRA O YOU HAVE TO BEAR WITH ME, TO PIGGY BACK ON WHAT YOU WERE SAYING WE'RE ALSO MISSING PAYERS AT THE TABLE. UNITED DOESN'T CARE IF U I'M HAPPY, THEY DIDN'T CARE IF WE GO IN THE ED UNNECESSARILY COSTING THEM THOUSANDS OF DOLLARS, IT'S IMPORTANT TO KEEP IN MIND WHO PAYS FOR THIS. AND HOW WE'LL SCALE IT IF WE DON'T FIGURE OUT WHO IS GOING TO PAY FOR IT AN MARC I TOTALLY GET YOUR POINT. I HEAR YOU, SO I JUST THINK IF YES EAR GOING TO PUT A STATEMENTING TO IMPROVING QUALITY OF LIFE, IT SOUNDS VERY NOBLE AND WONDERFUL, WE NEED COST SAVINGS OR QUANTIFIABLE HEALTH OUTCOMES ULTIMATELY. >> I WAS GOING TO SUGGEST WE MAKE QUALITY OF LIFE A SPECIFIED QUANTIFIED STATEMENT ABOUT THINGS LIKE PUT CONTROL OF HEALTHCARE IN THE USERS HANDS, MAKE IT MORE IMMEDIATE, REDUCE OVERALL SYSTEMATIC COSTS OF DELIVERING THAT HEALTHCARE BY VIRTUE OF LOCALITY AN IMMEDIACY. THE STATEMENT IS NOT BAD AS THE FIRST PREPOSITIONAL PHRASE BUT THEN GO ON AND GET PEOPLE INTERESTED IN WHY THEY SHOULD FUND IT. >> GARRETT MEHL, WHO. THERE'S AN INTERESTING SITUATION HERE I THINK A LOT OF US ARE ACADEMICALLY INCLINED COMING FROM ACADEMIC DISCIPLINES. PARTLY BECAUSE THAT'S WHAT THE MEETING IS ABOUT. THE OTHER INTERESTING THING HERE IS IT'S HELD BY HELL BY NIH AND RWG SOME OF THE SUGGESTIONS TRYING TO GET NIH OR TO GET -- TO THINK ABOUT THINGS DIFFERENTLY. BECAUSE WE SEE SOME OF THE CHALLENGES OR OBSTACLES THAT ARE THERE. A COUPLE OF PEOPLE MENTIONED THERE ARE OTHER KINDS OF DONORS. OR FUNDERS OR AGENCIES OUT THERE THAT ARE WANTING TO FUND, WHETHER THE PRIVATE SECTOR, VENTURE CAPITALISTS, THE -- AND THOSE ARE GROUPS OUT THERE THAT WERE NOT ACTUALLY TAKING INTO ACCOUNT, WE'RE TAKING INTO ACCOUNT THE REVIEW PROCESS THAT NIH GOES THROUGH. AND THAT WE WANT TO ENSURE THEY SEE THAT AS THE MU METHOD IS A VALID THING TO REVIEW. THE REALITY IS WE NEED -- THERE NEEDS -- WE NEED TO THINK OF THESE OTHER KINDS OF REVENUE STREAMS. AND MAKE USE OF THAT. I THINK OF THIS IN TERMS OF LOW INCOME COUNTRY SETTINGS, WHEN A PARTICULAR DONOR, I DON'T EVEN WANT -- I MEAN, SO WELCOME TRUSTS, SORRY, WANTED US TO HAVE A COMMERCIALIZATION STRATEGY FOR A GRANT. COMMERCIALIZATION STRATEGY? YOU'RE KIDDING. OKAY. WE COULDN'T DO THAT BUT WE SAID THAT WE WANTED TO GET TO A SUSTAINABILITY MODEL THAT HAD LARGE IMPACT. THEY WERE COMFORTABLE AND EXCITED ABOUT THAT. THEY DINT WAN THE IP OR GRANT TO GO TO WASTE. THAT'S AN IMPORTANT DISTINCTION. THE DONORS HAVE AN IMPORTANT ROLE IN THIS, AS TO THE -- DO THE ACADEMIC JURY ROOMS, THE JOURNALS CAN BEGIN TO -- I KNOW ID YOU WERE ROVED IN CLINICAL TRIALS REGISTRY, JOURNALS SAID WE WON'T WE WILL INCENTIVIZE YOU IN THE CLINICAL TRIALS REGISTRY AND WILL NOT PUBLISH IF YOU'RE NOT IN THAT REGISTRY. THAT'S A STICK. BUT THE IMPORTANT THING HERE IS THAT THERE ARE DIFFERENT KINDS OF MECHANISMS. AND A JOURNAL COULD SAY WILL ACCEPT YOUR CITATION ON AN ONLINE WEBSITE THAT IS A COMMUNITY WEBSITE FOCUSED ON EVIDENCE OF mHEALTH AND WILL ALLOW YOU TO SITE THAT IN A -- CITE THAT AND ALLOW YOU TO PUBLISH BITS AN PIECES IN ADVANCE OF A REAL JOURNAL PUBLICATION. AND THAT BEGINS TO GET THAT INFORMATION EVIDENCE OUT TO THE IMMUNITIES MORE QUICKLY. WE NEED TO LOOK AT THOSE MECHANISMS AN CARROTS AND STICKS IN WAYS OF INCENTIVIZING SPEED AND EFFICIENCY IN THE -- IN THIS IMMUNITY BECAUSE I THINK GOING STRAIGHT -- THE PUBLICATION, IT IS STILL GOING TO BE FIVE YEARS, IT'S GOING TO BE A LONG TIME. YOU HAVE TO HAVE OTHER WAYS OF GETTING OUT THERE. I WOULD ENCOURAGE US TO CONSIDER THE ACADEMIC JOURNALS THE DONORS BEING PART OF THAT ECOSYSTEM TO MAKE THINGS MORE EFFICIENT AND THAT WE LOOK AT OTHER REVENUE STREAMS. AND EMBRACE PARTS OF THEM THINKING ABOUT HOW THEY PLAY A ROLE IN SCALING THESE SOLUTIONS AND ENSURING THAT THERE IS IMPACT DERIVED FROM THIS. LAST THING, CHARLENE A STATEMENT COMING OUT IN THIS MEETING IS CRITICAL. I DON'T HAVE MUCH TIME BUT I FEEL IT'S VALUABLE ENOUGH THAT I WOULD BE HAPPY TO PLAY SOME ROLE IN (INAUDIBLE). >> MAYBE A QUICK I THINK THIS TIES IN JUST AS AN EXAMPLE, I MENTIONED A SECOND AGO THE DIFFERENCE BETWEEN WHAT WE USUALLY THINK OF AS INDIVIDUAL LEVEL HEALTH OUTCOMES AND COMMUNITY LEVEL. EXAMPLE I CAN THINK OF IS ANOTHER STREAM, IS FDA, SPECIFICALLY THE CENTER FOR TOBACCO PRODUCTS EMPOWERED TO REGULATE THE TOBACCO INDUSTRY, A BIG CHANGER IN THE WORLD OF TOBACCO. THEY HAVE MORE MONEY AND DEPENDING ON THE WAY THINGS PLAY OUT THEY COULD BE A BETTER SOURCE FOR FUNDING THAN AT LEAST IN„4 FOR A LITTLE WHILE. ONE OF THEIR INTERESTS IS FOR INSTANCE THE WAY PRODUCTS ARE MARKETED, THEY HAVE TOUGH TARGETS HOW DO PEOPLE PERCEIVE PRODUCTS AND THEY HAVE A LOT OF INTEREST IN USING MOBILE TECH NOLOGY AFFECT BEHAVIORAL CHANGE BEHAVIORAL OUTCOME CHANGE BUT EMPOWERING INDIVIDUALS TO USE THEIR DEVICES TO UNDERSTAND NOT JUST THEIR HEALTH BUT THE FACTORS RELATED TO HEALTH IN THEIR COMMUNITY. SO I WOULD JUST I GUESS TIE MY COMMENTS TOGETHER BY SAYING THAT I REALLY LIKE THE IDEA OF PATIENT-CENTERED. AND I LIKE THE IDEA OF EMPOWERING INDIVIDUALS THROUGH mHEALTH. AND I JUST WOULD SUGGEST THAT WE SAY SOMETHING BEYOND EMPOWERING INDIVIDUALS TO TAKE CONTROL OF THEIR OWN HEALTH BUT EMPOWERING INDIVIDUALS TO TAKE CONTROL OF THEIR OWN HEALTH AND THE HEALTH OF HAIR IMMUNITIES THEY LIVE IN. -- COMMUNITIES THEY LIVE IN. >> ONE THING I WOULD IMPART FROM THE PLANNING COMMITTEE, THREE THING, EVIDENCE, EVIDENCE, EVIDENCE. SO AS WE TALK ABOUT THESE STATEMENTS AN THOSE SORTS OF THINGS WHAT DROVE THE PLANNING COMMITTEE TO GET TOGETHER, DROVE IT TO HAVE THIS MEETING WAS NOT THAT WE'RE NOT GETTING EVIDENCE OUT QUICK ENOUGH BUT THAT THE TECHNOLOGY IS MOVING FASTER THAN THE EVIDENCE CAN KEEP UP WITH. SO THE DRIVE FOR US THROUGHOUT THIS WAS WHAT ARE THE BARRIERS TO WHY WE DON'T HAVE MORE EVIDENCE ABOUT THE EFFECTIVENESS OF mHEALTH SOLUTIONS THAN WE CURRENTLY DO. PART OF THE STREAMLINING EFFORT IS NOT SO WE CAN DO IT FAST BUT BECAUSE WE THOUGHT IT WAS A CRITICAL BARRIER NOT GETTING EVIDENCE OUT BECAUSE IT WAS TAKING TOO LONG FOR IT TO BE DONE AND WE NEED TO GET IT OUT FASTER. SO WHATEVER WE DO IN TERMS OF A STATEMENT WHAT WE'RE TRYING TO DO IS DEVELOP THE EVIDENCE THAT WILL PERSUADE COUNCIL IN STUDY SECTIONS PAYERS AND THOSE SORTS OF GROUPS ALONG THE WAY. >> BILL THAT WAS THE PERFECT THING, WE WANT TO FACILITATE FOR THE DISCUSSION. WE WOULD LIKE TO HELP IF YOU ALL FEEL THAT'S USEFUL. SORRY, IT'S GETTING TIRED. USEFUL, IS THIS A USEFUL THING? CHARLENE MADE AN OFFER TO WORK ON SOMETHING, I HAVE HEARD GARRETT SECOND IT. AND DEBRA. SO WE HAVE A SMALL GROUP THAT WILL GO, WE'LL START WORK ON THAT, THAT CAN GO AROUND. THERE'S OTHER PUBLICATIONS IDEAS AND I THINK WE CAN START TO USE -- LET'S USE MOBILE OR STATIONARY WEB TO WORK ON HOW DO WE GET SOME DISSEMINATION STRATEGIES BECAUSE I THINK WE CAN -- THERE'S NOT ENOUGH MONEY, THERE'S NOT ENOUGH TIME. BUT I THINK LIKE BILL WAS SAYING WE ALL SAID WE ALL BELIEVE THIS WILL CHANGE, THIS IS A GAME CHANGER, BUT AT THIS POINT WE NEED THE DATA AND WE NEED TO DATA OUT THERE NOW. IT DOESN'T COME FROM A COUNTRY THAT DOESN'T HAVE RESOURCES OR COMING FROM A COUNTRY WHICH SPENDS OODLES OF RESOURCES. WE ALL THINK WE NEED THE DATA TO SHOW THAT WHAT WE'RE WORKING ON IS EFFECTIVE. I THINK THAT'S GOING TO IMPROVE IT FOR OUR PATIENTS, WHETHER THEY'RE PATIENTS, PEOPLE OR CONSUMERS, IT'S GOING TO IMPROVE IT FOR PAYER, GOING TO IMPROVE IT FOR STUDY SECTION. AND COUNSEL. I THINK THE DATA IS GOING TO MAKE PEOPLE BELIEVE AND MAKE IT THE GAME CHANGER WE THINK IT'S GOING TO BE. SO I'LL LET SANTOSH END IT HERE. >> ANYBODY WANTS TO WORK ON THIS STATEMENT WE'LL GET TOGETHER TOMORROW MORNING FOR BREAKFAST AT 7:00. COME ON. FARMERS DAUGHTER HERE. YOU GOT TO JOIN MAINE 7:30. 7:30. >> TIME TO BE NEGOTIATED AND WHEN WE GET OFFLINE WE CAN WORK ON IT. >> EMAIL SANTOSH THEN. >> EMAIL SANTOSH. >> ALL RIGHT. SO THAT BRINGS US TO CONCLUDE THIS EXCITING MEETING AND THIS NEW BEGINNING TO MAKE mHEALTH A SCIENCE. WE'LL BE IN TOUCH OVER EMAIL. THANK YOU FOR YOUR PARTICIPATION. [APPLAUSE]y