GOOD MORNING EVERYBODY. I'M JERRY MENIKOFF RESEARCH DIRECTOR AND I WELCOME YOU TO OHRP SECOND WORKSHOP. WELCOME TO ALL OF THOSE JOINING ONLINE; ALL THE CREDIT GOES TO THREE AND EXTRAORDINAIRE PEOPLE, WHO ARE ALL HERE SO THANK THEM. SO GETTING INTO THE TOPIC. --- WE ARE CURRENTLY LIVING IN A WORLD WHERE PREVIOUSLY UNIMAGINED AMOUNTS OF DATA IS BEING GENERATED FOR RESEARCH PURPOSES AND SOMETIMES IT IS ACCESSED THROUGH SHARED DATABASE IN ONE SOLUTION CAN BE INFORMATION COLLECTED AS PART OF HAND A DIFFERENT N THE OTHER - EXAMPLE MEMBERS OF THE PUBLIC THEMSELVES GENERATE LARCHMONT ABEYTA THROUGH THE USE OF HEALTH MONITORING, OTHER APPS, GPS LOCATION SYSTEMS, SOCIAL MEDIA AND OTHER PROBLEMS RELATED TO MOBILE DEVICES AND THAT DATA IS STORED AND SHARED AND USED IN A VARIETY OF WAYS. --- THERE'S A TREMENDOUS POTENTIAL TO USE THESE VAST COLLECTIONS OF DATA TO LEARN MORE ABOUT HEALTH AND BEHAVIOR OF THE POPULATION FOR THE BENEFIT OF ALL OF US AS INDIVIDUALS. HEALTH-RELATED BIG DATA RESEARCH PROMISES NEW INSIGHTS IN TREATMENTS FOR A VARIETY OF CONDITIONS AND DISEASES; IN ADDITION TO INNOVATED WAYS TO SUPPORT AND MAINTAIN GOOD HEALTH. --- HOWEVER CURRENTLY THERE IS NO CONSENSUS ABOUT APPROPRIATE WAYS TO COLLECT, STORE AND SHARE THIS TYPE OF DATA. SO THE QUESTION IS, WHAT ARE THE APPROPRIATE TRADE-OFFS >> FOR OUR ONLINE VIEWERS THE URL IS POSTED IN THE SLIDE; AS A REMINDER THE ARCHIVE VIDEOS AND WORKSHOP SUMMARY OF TODAY'S EVENT WILL BE AVAILABLE ON OHRP WEBSITE IN THE NEAR FUTURE. WITHOUT FURTHER DELAY I WOULD LIKE TO INVITE THE MODERATOR FOR SESSION 1, MS. JODI DANIEL OF CROWELL & MORIUNG, LLP, TO TAKE THIS EVENT FURTHER. GOOD MORNING ANYBODY WE HAVE A BIG TASK TODAY AND I'M EXCITED TO MODERATE THE FIRST PANEL; WE WERE HAVING AN INTERESTING CONVERSATION THAT WILL ENSUE FROM OUR TALK SO WE ARE STARTING THE PANEL FOCUSED ON THE GENERAL LANDSCAPE OF PRIVACY AND ETHICAL CONSIDERATIONS RELATED TO LARGE-SCALE OPEN DATA SHARING AND PUBLIC TRUST AND SOME OF THE TOPICS THAT THE PANELISTS HAVE BEEN ASKED TO ADDRESS OUR QUESTIONS ABOUT EIGHT OWNERSHIP, TYPES OF GOALS OF DATA RESEARCH AND WHAT NEEDS TO BE DONE TO MAINTAIN PUBLIC TRUST IN HEALTH RESEARCH AND WHO ARE THE RESPONSIBLE PARTIES IN THIS. I WILL QUICKLY INTRODUCE A PANELIST AND THEN WE WILL GET STARTED. --- SO FIRST WE HAVE JACOB METCALF, A TECHNOLOGY RESEARCH AND CONSULTING SPECIALIST IN ARTIFICIAL INTELLIGENCE AND HE IS A RESEARCHER AT THE -- RESEARCH INSTITUTE WHERE HE STUDIES OUT DATA ETHICS PRACTICES ARE EMERGING ENVIRONMENTS THAT HAVE NOT BEEN PREVIOUSLY GRAPPLED AND NEXT WE HAVE BRENDA LEONG, A SENIOR COUNSEL AND DIRECTOR OF STRATEGY A PRIVACY FOUM AND SHARE STRATEGIC PLANNING AND ORGANIZATIONAL GOALS AS WELL AS MANAGING FUTURE PRIVACY FIRM PORTFOLIO BIOMETRICS; SHE WORKS ON ETHICS AND PRIVACY ISSUES ASSOCIATED WITH ARTIFICIAL INTELLIGENCES WHILE. AND SHE WORKS IN INDUSTRY STANDARDS IN COLLABORATION OF PRIVACY CONCERNS. --- AND FINALLY WE HAVE DOCTOR CINNAMON BLOSS, AN ASSOCIATE PROFESSOR IN THE DEPARTMENT OF PSYCHIATRY IN FAMILY MEDICINE AND PUBLIC HEALTH, DIVISION OF HEALTH POLICY AT THE UNIVERSITY OF CALIFORNIA SAN DIEGO; HER CURRENT RESEARCH FOCUSES ON THE INDIVIDUAL AND SOCIETAL IMPACT FOR EMERGING BIOMEDICAL TECHNOLOGIES AND HER BACKGROUND IS IN CLINICAL PSYCHOLOGY THE PHYSICAL GENETIC MEDICINE, BIOMEDICAL ETHICS , AND HEALTH POLICY. I WILL BE A MODERATOR FOR THE SESSION AND I MYSELF AM A HEALTHCARE LAWYER AND A HEALTH POLICY EXPERT AND I'VE BEEN FOCUSING ON DIGITAL HEALTH AND HEALTH PRIVACY ISSUE SINCE 2000, WHEN I WORKED A DEPARTMENT OF HEALTH AND HUMAN SERVICES. I ALSO SERVED AS THE DIRECTOR OF POLICY AT THE OFFICE OF THE NATIONAL COORDINATOR FOR HEALTH IT AT HHS FOR 10 YEARS, SO WORKING LOTS OF ISSUES RELATED TO DATA, DATA USING DATA PRIVACY PROTECTION AND THE INTERSECTION OF DIGITAL HEALTH AND INNOVATION AND RESEARCH. --- I CURRENTLY SERVE AS A PARTNER AT THE LAW FIRM OF CROWELL & MORING WHERE I HELP COMPANIES NAVIGATE A RECORD THREE LANDSCAPE IN PROVIDING EXPERT INFORMATION AND ADVICE TO POLICYMAKERS AS THEY ARE THINKING ABOUT ADAPTING THOSE WORLDS TO SUPPORT INNOVATION IN HEALTH CARE AND RESEARCH. --- THE STATED GOALS OF THE SESSION WERE TO EXPLORE THE PROBLEM OF THE PROTECTION IN A DATA RICH WORLD; TO EXPLORE TENSION THAT EXISTS BETWEEN SOCIETAL GOOD THAT COULD COME FROM BIG DATA RESEARCH AND THE REAL AND PERCEIVED RISK TO INDIVIDUALS AND TO DISCUSS THE PUBLIC'S PERCEPTION ABOUT THE RAW DATA SHARING AND THERE IS A LOT ACTING HERE SO WE SHOULD HAVE A VERY ROBUST DISCUSSION. I LOOKED AT THIS IN QUESTION SOME OF THE PREMISES AND WE HAD SOME INTERESTING CONVERSATIONS ABOUT WHETHER PRIVACY IS ACTUALLY A PROBLEM AND HOW DO WE DEFINE PRIVACY. AND HOW DO MAKE SURE THAT THEIR BUILDING TRUST AND IMPORTING DATA USES AN IMPORTANT TRUST IN HOW WE ARE USING THAT DATA. THERE ARE LOTS OF DIFFERENT FRAMERS THAT WE WOULD BE TALKING ABOUT ABOUT HOW WE THINK ABOUT PRIVACY AND DATA PROTECTION AND HOW THE REGULATORY ENVIRONMENT IMPACTS THAT CONTEXT. --- SO LAST OF COVER AND LOTS OF REALLY INTERESTING CONVERSATIONS. WE ARE GOING TO HAVE OUR PANELISTS TALKED FOR ABOUT 20 MINUTES TO INTRODUCE THESE TOPICS AND THAT WILL HAVE AN HOUR-LONG DISCUSSION AFTER THAT SO WITH THAT I WILL TURN IT OVER TO JAKE TO BEGIN OUR DISCUSSION. THANK YOU VERY MUCH. >> JACOB METCALF: THANKS JODI. SO I WAS ASKED TO PROVIDE SOME FRAMING CONCEPTS TO DISCUSS BROADLY -- -- -- AROUND ETHIC AND PRIVACY IN BIG DATA RESEARCH BOTH IN THE ACADEMY AND INDUSTRY. AND SOME GOING TO FOCUS ON ACTUALLY THE JUNCTION BETWEEN THOSE TWO. AND HOW BIG DATA RESEARCH IN THE ACADEMY BECOMES MOBILE WITHIN THE INDUSTRY IN A WAY THAT CAUSES UNEXPECTED HARMS. --- I HEAR (CORRECTION) I'M HERE REPRESENTING PERVASIVE DATA -- NSF GRANT, 7 CO-PIS, INCLUDING MICHAEL ZIMMER WHO IS HERE TODAY AND TALKING ABOUT HOW STAKEHOLDERS ARE DISCUSSING DATA SIGNS AND PROPAGATING BEST PRACTICES. THE GROUP IS BUILDING A COLLABORATIVE DATA RESEARCH PROJECTS IN MANY DIFFERENT AREAS AROUND DATA RESEARCH. ESPECIALLY AS DATA SCIENTISTS CONSIDER MORE ROBUSTLY HOW TO DO ETHICAL AND RESPONSIBLE RESEARCH PRACTICES. --- WHY DO WE SAY "PERVASIVE DATA PRACTICES"? IT DOESN'T DESCRIBE WHAT IS RELEVANT ABOUT DATA RESEARCH; PERVASIVE DATA IS A MORE ACCURATE TERM. AND THAT IS BECAUSE PERVASIVE DESCRIBES THAT OTHER BRIDGES MULTIPLE DIMENSIONS OF THE PERSON'S LIFE. WE TEND TO EXPERIENCE OURSELVES AS HAVING MULTIPLE DIFFERENT PERSONS IN THE SENSE OF WHO WE ARE WITH OUR DOCTOR, WITH OUR LAWYER, WITH OUR TEACHERS, WITH OUR CHILDREN AND WE TEND TO NOT THINK THAT THOSE THINGS HAVE OVERLAP; THE MACHINES DON'T CARE. MACHINES WANT TO DOUBLE ALL THAT TOGETHER AND UNLESS WE HAVE CLEAR AND ROBUST POLICY DISTINCTIONS THOSE WILL BE ANALYZED TOGETHER SO WE WILL LOOK AT WHAT HAPPENS WHEN THOSE DISTINCTIONS COLLAPSE. --- IT IS ALSO THE NATURE OF MACHINE LEARNING TO JUMP DOMAINS. AND THE REAL ECONOMIC VALUE -- THE INTELLECTUAL AND ECONOMIC VALUE OF DATA ANALYTICS ESPECIALLY WHEN MACHINE LEARNING IS INVOLVED IS THE ABILITY TO MAKE INFERENCES ABOUT ONE PART OF YOUR LIFE FROM ANOTHER PART OF YOUR LIFE. AND IN PARTICULAR TO TAKE CHEAP DATA AND MAKE EXPENSIVE INFERENCES; THE ECONOMIC ENGINE OF MACHINE LEARNING IS DEFINED DATA THAT IS CHEAP TO GET OR FREE AND MAKE INFERENCES ABOUT ASPECT OF YOUR LIFE THAT ARE VERY HARD OR EXPENSIVE TO GET. VALUABLE. --- AND THEN WE SEE THE SPEED AND SCALE OF DATA ANALYTICS, PARTICULARLY WHEN MACHINE LEARNING IS INVOLVED; CHANGES IN THE DIMENSIONALITY OF RISK. WHAT WE FIND IN MANY CASES IS THAT THE FRAMEWORK OF INDIVIDUAL PRIVACY DOESN'T HOVER -- IT'S NOT THAT IT'S INAPPLICABLE BUT RATHER THAT THERE ARE AN NUMBER OF TYPES OF HARM THAT CAN OCCUR AND CAN BE CAPTURED THROUGH AN INDIVIDUAL FRAME AND WE NEED TO BE LOOKING AT DOWNSTREAM CONSEQUENCES TO COMMUNITIES AND THE SOCIETY AT LARGE. --- NOW, WHY -- IS BIG DATA SO HARD FOR RESEARCH ETHICS TO GET A HANDLE ON? MY CONTENTION IS THAT IT'S BECAUSE MOST OF THE TECHNIQUES, TOOLS, METHODS AND CONCEPTS OF DATA RESEARCH ARE POORLY COMPATIBLE WITH OUR INSTITUTIONS, ARCHONS IS AN RULES AROUND RESEARCH ETHICS AND THAT IS BECAUSE PERVASIVE DATA SCIENCE USES THESE METHODS AND TYPES OF DATA THAT LARGELY EXEMPT IT. MOST PERVASIVE DATA SCIENCE RESEARCH USES PRE-EXISTING DATA SETS; USES PUBLIC DATA SETS, OFTEN MEANS BORROWED OR GLEANED FROM SERVICES AND USES DATA DE-IDENTIFIED. IN MOST CASES DATA SCIENTISTS ARE NOT INVOLVED IN THE COLLECTION OF DATA AND THEREFORE ARE NOT THAT CLOSELY -- THEY DON'T DWELL, THEIR METHODS ARE NOT CLOSE TO KEY MOMENTS OF RESEARCH ETHICS. IT COMES AFTER THE DATA HAS ALREADY BEEN COLLECTED AND ALMOST UNIVERSALLY THEIR RESEARCH IS EXEMPT UNDER THE COMMON RULE. --- PERVASIVE DATA IS -- -- DOES NOT REQUIRE INTERVENTION TYPICALLY SO THE COMMON RULE SPECIFICALLY DESCRIBES RESEARCH IS SOMETHING THAT REQUIRES AN INTERVENTION IN A PERSON'S LIFE; DATA SCIENTIST TYPICALLY AREN'T DRAWING BLOOD OR GIVING INTERVIEWS. THEY ARE HANDLING DATA THAT ALREADY EXISTS SOMEWHERE IN THE WORLD THAT SOMEONE ELSE ALREADY COLLECTED. --- AND ODDLY, DATA SCIENCE TOOLS BECAUSE THEY ARE SO PROLIFIC AND EASY TO ACCESS ON THE INTERNET MEANS THAT ANYONE CAN BE A RESEARCHER; SO WE HAVE IN A WAY THAT WE DON'T HAVE PROLIFERATION OF THEORETICAL PHYSICISTS OR CLINICAL PSYCHOLOGISTS, WE HAVE A PROLIFERATION OF PEOPLE WITH ACCESS TO THE SCIENTIFIC TOOLS. AND SO RELYING ON INSTITUTIONAL REVIEW BOARDS MISSES A LOT OF THE SCIENCE THAT HAPPENS. MANY PEOPLE ARE NOT IN AN INSTITUTION. --- THIS IS FROM A SURVEY THAT MICHAEL HAS BEEN WORKING ON, PRELIMINARY RESEARCH, A -- CLOUD OF NSF-FUNDED DATA SCIENCE IS USED TO DESCRIBE METHODS. THESE ARE TERMS THAT WE WILL SEE QUITE OFTEN IN THE RESEARCH PROJECT THAT WE HAVE BEEN STUDYING, CLICKSTREAM, TWITTER FEED, GPS DATA, APP DATA, PRIVATE USER, A WARD CLOUD OF WHAT DOES PERVASIVE DATA RESEARCH MAKE USE OF. --- EXCUSE ME VERY CRUDE GRAPHICS; EASIER TO MAKE THEM SIMPLE RATHERN THAN PRETTY. --- THE KEY POINT I THINK, I CLAIM -- OF RISK, AND DATA SCIENCE RESEARCH PARTICULARLY AS IT MOVES FROM THE ACADEMY TO INDUSTRY INVOLVES MODELING. SO, TRADITIONAL RESEARCH METHODS -- WE STUDY POPULATIONS THAT PREDICTS WHAT THE POPULATION WILL DO. WE INTERVIEW INDIVIDUALS AND DRAW BLOOD FROM INDIVIDUALS, WE STUDY THEIR BEHAVIOR IN A PARTICULAR SETTING; BUT THE GOAL IS THE MODELS THAT GET PRODUCED IN TRADITIONAL RESEARCH TELLS US ABOUT POPULATIONS. HOWEVER THE EXACT SAME TOOLS YOU WOULD USE TO MODEL THAT POPULATION WITH BIG DATA, THAT EXACT SAME MODEL -- A MACHINE LEARNING MODEL IS A SET OF WAYS IS AS IF SUCH AND SUCH HAPPENS WAIT A PRODUCTION IN THIS WEIGHT AND IF THIS OTHER THING HAPPENS WEIGHT IT THIS OTHER WAY. THAT EXACT SAME SET OF MATHEMATICAL WEIGHTS PLUGS INTO TARGETING AN INTERVENTION TOOLS; THE DISTINCTION IS THAT WE WOULD FIND IN TRADITIONAL RESEARCH PROJECTS BETWEEN STUDY AND INTERVENE -- THAT DISTINCTION IS INOPERABLE IF YOU ARE USING MACHINE LEARNING TOOLS. IT'S THE EXACT SAME WAY OF MATHEMATICAL WAYS THAT YOU WOULD USE TO UNDERSTANDING POPULATION WHICH ALLOWS YOU TO TARGET AN INDIVIDUAL. --- SO, MODELS REALLY BECOME THE CRUX OF PRIVACY. IT'S NOT JUST TRYING TO PROTECT -- WE SHOULD NOT JUST TRY TO PROTECT THIS PERSON FROM HARM IN THE RESEARCH METHOD -- BUT WE SHOULD ALSO BE PAYING ATTENTION TO WHAT HAPPENS TO THIS MODEL SUCH THAT IT NOW ALLOWS YOU TO TARGET OTHER INDIVIDUALS IN A LIVE FASHION IN THE REAL WORLD. --- THIS IS WHAT I CALL THE "VALUE RATCHET" OF MACHINE LEARNING. WE FIND BEHAVIOR THROUGH SENSORS OR SIGNALS -- MOST OF THE TIME IT IS YOUR PHONE IN YOUR POCKET -- BUT THERE'S MANY SENSORS IN THE WORLD AND WE COLLECT INFORMATION ABOUT BEHAVIOR FROM SENSORS AND IT IS A SIGNAL THAT GETS INTEGRATED INTO A DATA PROFILE, ANALYZED IN ORDER TO MAKE A PREDICTION IN ORDER TO LEVERAGE BEHAVIOR. --- EVERY TURN OF THIS RATCHET CREATES ECONOMIC OR SCIENTIFIC VALUE. LET ME SHOW YOU. --- THE DATA SUBJECT UNDERSTANDS THEIR BEHAVIOR AND OFTEN UNDERSTAND THE SENSOR; THINK ABOUT A FITNESS TRACKER ON THE WRIST. MANY PEOPLE LOVE THEM AND THEY LOOK AT THEM EVERY DAY, YOU KNOW WHAT YOUR FITNESS TRACKER IS DOING BECAUSE IT FEEDS BACK TO YOU, THE BEHAVIOR SIGNAL LOOP IS CLEAR TO YOU. WHAT YOU DON'T SEE IS THE REST OF THE VALUE RATCHET, THE REST OF THE ECONOMY. YOU DON'T SEE THE INTEGRATION INTO A DATA PROFILE THAT INCLUDES A BUNCH OF OTHER INFORMATION ABOUT YOU. YOU DON'T SEE HOW IT IS THAT THE AGREEMENT BETWEEN THE FITNESS TRACKER MANUFACTURER AT THE INSURANCE COMPANY THAT YOUR EMPLOYER SIGNED YOU UP FOR INTEGRATES INTO THE PLATFORM AND MAKE PREDICTION ABOUT THE HEALTH AND HOW IT WILL SHAPE YOUR INSURANCE NEXT YEAR. THE ANALYSIS HAPPENS IN ORDER TO GET THE PREDICTION AND THIS IS THE ECONOMIC VALUABLE PORTION OF THE ADDICTIVE ECOSYSTEM YET IT REMAINS OPAQUE TO THE INDIVIDUAL. --- THE VALUE IS IN THE ABILITY TO LEVERAGE THIS INFORMATION TO GET CHANGES IN BEHAVIOR. THIS IS WHAT WE EXPERIENCE AS "CREEPY": WHEN WE SAY THE DATA COMPANY DID THIS THING THAT FEELS CREEPY WHAT WE MEAN IS THAT WE THOUGHT WE UNDERSTOOD WHAT WAS GOING ON, BUT WE DID NOT SEE THE REST OF THE RATCHET. WE DO NOT SEE HOW THEY WERE GETTING TO ECONOMIC VALUE. --- THIS IS AN IMPORTANT PART OF UNDERSTANDING RESEARCH ETHICS RISK EVEN THOUGH THIS IS NOT HOW RESEARCH HAPPENS SO THIS IS A TECHNICAL STACK AS THEY CALL IT IN INDUSTRY. IN INDUSTRY YOU MOVE FROM SENSOR DATA COLLECTION TO A DATA LAKE; TECH COMPANIES SEEM TO BE MAGICAL BUT IN THE WAY THEY MANAGE DISASTERS AND THE DATA LAKE IS THE MAIN DISASTER AND THAT IS JUST THE COLLECTIVE OF AN ARMS AMOUNTS OF SENSOR DATA THAT IS COMING IN, IT IS ALL POOLED TOGETHER AND A LOT OF TECHNICAL TERMS ARE PICKED UP FROM THE DROPS IN THE LAKE THAT ARE RELEVANT; YOU GET THE DATA LAKE AND OFTEN THERE ARE HOLDINGS OF THAT ENGINEERS THAT MANAGE WHAT THE LAKE IS DOING; DATA SCIENTIST WANT TO FIND IN A SMALL UNIT A DATASET BUT REALLY WHAT THEY ARE INTERESTED IS THIS FEATURE SET. SO A DATASET MIGHT COME FROM MANY DIFFERENT PLACES AND WHAT THEY ARE INTERESTED ARE THE FEATURES. --- IF YOU'RE A ROW IN THE DATA SET, THE FEATURE IS A COLUMN, THE GO LOCATION, AGE, ECONOMIC DATA; THEY WANT TO FIND THE FEATURES ACROSS MULTIPLE DATA SETS AND FEET THAT INTO A LEARNING ALGORITHM IN ORDER TO PROVE SOME MODEL. SO THIS IS THE ACTIVITY OF DATA SIGNS IN A TECHNOLOGY COMPANY. THIS IS WHAT IS A DATA SCIENTIST IS DOING ALL DAY, TRYING TO DEVELOP AN ECONOMICALLY VALUABLE PREDICTION THAT IS OPTIMIZED TO USE THE FEWEST NUMBERS OF ELECTRONS TO MAKE A PREDICTION BASED ON THE SMALLEST POSSIBLE FEATURE SET. AND WE WANT TRADITIONALLY EFFICIENT PRODUCTIONS THAT PRODUCE ECONOMIC VALUE. --- ULTIMATELY WHAT THAT MODEL AS OPPOSED TO DO IS BECOME THE PRODUCTIONIZED MODEL WHICH IS FED LIKE DATA IN ORDER TO PRODUCE THE PRODUCT WERE SENT TO DECOUPLE. IMPORTANTLY, EVERY STEP DOWN THIS CHAIN PRODUCES MORE ECONOMIC VALUE BUT IT ALSO LOSES CONTEXT. AS IT GETS FURTHER AWAY FROM SENSOR AND DATA COLLECTION, ETHICALLY RELEVANT CONTEXT IS STRIPPED; THE MODEL IS A HIGHLY ABSTRACT SET OF MATHEMATICAL WEIGHT AND PREDICTED SYSTEM. THE WHOLE POINT OF MAKING A MODEL IS TO STRIP CONTEXT; THESE TECHNICAL SYSTEMS AREN'T CAPABLE OF HANDLING CONTEXT AS A SORT OF METADATA TYPICALLY THAT YOU WOULD WANT TO HAVE ON HAND IF YOU WERE TO MAKE A ROBUST ETHICAL DECISION ABOUT WHETHER THIS PRODUCT OR SERVICE SHOULD BE OUT IN THE WORLD. --- MOST OF OUR ETHICAL CONTROLS ARE IN THAT SENSOR OR DATA COLLECTION BUT ALL THE ECONOMIC VALUE IS MUCH FURTHER DOWN THE CHAIN OF TECHNICAL SYSTEM. AND AS YOU HAVE LESS CONTEXTY, YOU BECOME MORE PORTABLE. --- SO THIS MODEL CAN BE USED FOR MANY DIFFERENT THINGS, IN MANY DIFFERENT CONTEXTS; IT'S THE MOST ABSTRACT COMPONENT OF THE SYSTEM. --- I'M GOING TO PERSUADE YOU THAT PAPER ANALITICA (SOUNDS LIKE) IS A HEALTH RESEARCH DATA SCANDAL; WE TALKED ABOUT THIS AS ZUCKERBERG IN FRONT OF CONGRESS SWEATING BULLETS; IT WAS A SIGNIFICANT PRIVACY VIOLATION AND IT HARMED INDIVIDUALS. IT'S NOT LEGITIMATE TO USE A FRAMING OF INDIVIDUAL PRIVACY TO UNDERSTAND WHAT HAPPENED BUT IN THE BACKGROUND THE ECONOMIC RATCHET OF MOVING FROM INDIVIDUALS TO POPULATIONS TO PREDICT AND LEVERAGE INDIVIDUALS IS REALLY WHERE THE MEAT OF THE STORY IS. --- YOU HAVE TO UNDERSTAND A LITTLE BIT ABOUT THE POLITICAL CONTEXT. I'M GOING TO DO MY BEST TO BE ENTIRELY NONPARTISAN ABOUT THIS BUT THE -- A CRITICAL PART OF THE STORY IS A RACE BETWEEN THE TWO MAJOR POLITICAL PARTIES IN THE UNITED STATES TO ACHIEVE DOMINANCE IN ELECTORAL DATA SYSTEMS. --- ON THE DATE OF THE ELECTION, 2012, MITT ROMNEY'S CAMPAIGN GET OUT THE VOTE EFFORT COLLAPSED; THE TECHNICAL SYSTEM FAILED AFTER AN ANNOYANCE INVESTMENT FROM THE REPUBLICAN PARTY AND ITS DONORS IN ORDER TO CATCH UP TO THE OBAMA CAMPAIGN WITH THESE NEW DATA SYSTEMS. --- AND THE ROMNEY SYSTEM ON THE DAY OF THE ELECTION COLLAPSED; THEY COULD NOT DO & THE GET OUT THE VOTE DOORKNOCKING CAMPAIGN THAT HAD BEEN PLANNED FOR. AND MANY OF THE LEADERS IN THE REPUBLICAN PARTY SAID, THIS IS WHY WE LOST THIS CAMPAIGN. THEY WERE QUITE CONFIDENT THAT THEY WOULD WIN AND THE NOW DIFFERENCE WOULD BE KNOCKING ON DOORS AND PHONE CALLS AND THEY THOUGHT THAT WAS THE CAUSE OF THE OBAMA VICTORY. --- IN THAT CONTEXT, IMMEDIATELY AFTER THE ELECTION AS POLITICAL PARTIES ALWAYS DO THEY HELD A POSTMORTEM MEETING FOR THEIR LEADERSHIP AND DONORS AND THINKERS AND CONSULTANTS SO THEY ALL GOT TOGETHER TO DISCUSS WHAT WENT WELL AND WHAT DID NOT. AT THAT MEETING IN NEW YORK IN JANUARY, 2013, WERE REBECCA AND ROBERT MERCER. ROBERT MERCER IS ONE OF THE LEADING DONORS TO CONSERVATIVE POLITICS IN THE UNITED STATES. BUT IT IS CRITICALLY IMPORTANT TO UNDERSTAND WHO HE IS; HE INVENTED THE ALGORITHM IN HEDGE FUND THE AND HE BECAME QUITE WEALTHY, ONE TINY VICTORY TIMES A TRILLION; TINY FRACTIONS OF THE TRADE ADVANTAGE AND THAT TREMENDOUS SCALE AND SPEED -- SO SHE BELIEVES IN TINY VICTORIES AT A TIME. THAT IS HIS PHILOSOPHY AND THAT IS WHAT HE IS GOOD AT USING MATH TO GET TINY VICTORIES AT A TIME, AT SCALE. --- REBECCA MERCER IS HIS DAUGHTER AND SHE IS REALLY THE ONE THAT YOU WILL FIND INTERACTING WITH MEDIA SO SHE IS THE FACE OF THE FAMILY WHEREAS HE IS MUCH MORE QUIET. ALSO THERE WAS ALEXANDER NIX, THE CEO OF STRATEGIC COMMUNICATIONS LABORATORY, THE BRITISH FIRM WITH PRIMARY CLIENTS ARE THE AMERICAN DEPARTMENT OF DEFENSE, BRITISH MI-5, MI-6, PROVIDING DATA DRIVEN ELECTORAL INFORMATION SO THEY HAVE SIGNIFICANT EDITORIAL DATA OPERATION GOING ON AROUND THE WORLD AND THEY HAVE A LOT OF CUTTING-EDGE TECHNIQUES. ALSO THERE WAS STEVE BANNON, AHEAD OF -- NEWS. ANOTHER COMMUNICATIONS STRATEGIES FOR CONSERVATIVE POLITICS AND MARK BLOCK WHO IS -- CAMPAIGN MANAGER BUT HE ALSO RUNS A LOT OF CONSERVATIVE POLITICAL ACTION EFFORTS. THESE ARE THE PEOPLE THAT MADE PAPER ANALITCA; IT WAS THE DECISION OF THE MERGERS TO INVEST IN INTELLECTUAL PROPERTY WITH (DID NOT UNDERSTAND) IN CHARGE OF COLLECTING DATA FOR HIS POLITICAL ACTION COMMITTEES. TO FIND TWO OF THE MODELS. --- THE CAMBRIDGE ANALYTICA IS A SHELL COMPANY HOLDING INTELLECTUAL PROPERTY FOR USE IN THE UNITED STATES AND WHAT DOES THAT HAVE TO DO WITH RESEARCH? AT THAT SAME TIME IN THE WORLD -- WHEN THIS GROUP OF HIGH-POWERED POLITICAL THINKERS SAID WE NEED A NEW WAY OF DOING POLITICAL DATA AND A NEW KIND OF SYSTEM THERE IS THIS HAPPENING IN ENGLAND AT THE UNIVERSITY OF CAMBRIDGE PSYCHOMETRIC CENTER. PSYCHOMETRIC IS THE MEASURING OF PSYCHOLOGICAL AND PSYCHIATRIC TRAITS QUANTITATIVELY PARTICULARLY THROUGH DATA OPERATIONS SO MICHAEL KACZYNSKI AND DAVID STILLWELL ARE THE BIG PLAYERS; KACZYNSKI IS NOW AT STANFORD. --- THIS IS A MONTH AFTER THE CAMBRIDGE ANAYTICA AND MEETING, AND THE IDEA WAS TO TAKE PERSONALITY CHARACTERISTICS AND CORRELATE THEM WITH ACTIVITY IN SOCIAL MEDIA HOURS, THE PRIMARY FRAMEWORK IS OCEAN -- (DID NOT UNDERSTAND) -- OPENNESS, HOW ORDERLY ARE YOU? HOW SOCIAL ARE YOU? AGREEABLENESS IS HOW YOU FEEL ABOUT CONFLICTS AND EROTICISM IS HOW DO YOU FEEL ABOUT THREATS. --- THE FFM, FIVE FEATURE MODEL -- THER'S ALSO A 13-FEATURE MODEL, AND LOTS OF DEBATE IN THE PSYCHOLOGICAL IMMUNITY ABOUT THE VALIDITY OF THESE STRUCTURES AND HOW MANY TRAITS DO WE ALL REALLY HAVE BUT THE GENERAL IDEA IS THAT WE ALL HAVE A MATRIX OF QUANTIFIABLE TRAITS AND EVERY HUMAN IS MEASURABLE IN THAT WAY AND THESE ARE TRAITS THAT ARE USEFUL IN MANY DIFFERENT CONTEXTS BUT IN PARTICULAR IN PSYCHIATRI. --- SO THESE TYPES OF MEASURES ARE INTEGRATED WITH THE DSM 5; THESE ARE PSYCHIATRIC CATEGORIES AND THEY GET USED A LOT OTHER THINGS THAT ARE NOT EXPLICITLY CLINICAL BUT FUNDAMENTALLY THEY ARE WELL VALIDATED PSYCHIATRIC TOOLS. --- WHAT DO THEY DO? THEY COLLECT YOUR YOUR "LIKES," THE THINGS YOU LIKE ON FACEBOOK AND THEY ARE ABLE TO PREDICT WITH SIGNIFICANT ACCURACY AND NUMBER OF PSYCHOLOGICAL AND DEMOGRAPHICS TRAITS; ARE YOU CHRISTIAN OR ISLAMIC? DO YOU SMOKE CIGARETTES? WHAT IS YOUR SEXUAL PREFERENCE? YOUR GENDER? HOW YOU VOTE? THE DIVERSITY OF YOUR FRIENDSHIP NETWORK? YOUR AGE? SO BASED ON SIMPLY THE THINGS THAT YOU LIKE ON FACEBOOK THEY ARE ABLE TO MEASURE THAT. --- KACZYNSKI SAYS AMIR 10 LIKES MEASURES CHARACTER TRAITS BETTER THAN THE AVERAGE COWORKER AND WITH MORE THAN 300 LIKES YOU CAN PREDICT WHAT A -- YOU CAN PREDICT A PERSON'S BEHAVIOR BETTER THAN THEY CAN THEMSELVES. WHAT YOU PUT ON FACEBOOK IS A GOOD PREDICTOR OF WHAT YOU DO AND WHO YOU ARE; SO HOW DO THEY DO THIS? VIRAL QUIZZES. HEY, TAKE OUR SUPER FUN PERSONALITY TEST. API -- APPLICATION PROGRAMMING INTERFACE, WHAT ALLOWS THE PARTIES TO GET DATA ON FACEBOOK -- AT THE TIME FACEBOOK'S API POLICY ALLOW THAT AS SOON AS YOU HIT THE "NEAT, SURE THING" BUTTON, WHAT THAT DOES IS THE CAMBRIDGE SECOND METRIC CENTER PEOPLE GET ALL OF YOUR FRIENDS AND THEIR LIKES AND YOUR LIKES. I IS FED THROUGH PSYCHOLOGICAL METRICS AND GETS BACK TO YOU BUT THE IMPORTANT THING IS THAT IT BECOMES PART OF THE DATABASE. SO ALL OF YOUR FRIENDS AND ALL OF YOUR LIKE BECOME PART OF THAT DATABASE AND BECOMES PART OF THE PREDICTIVE VALUE. --- REALLY WHAT IS VERY IMPORTANT IS THAT THIS BECOMES ECONOMICALLY VALUABLE IN FACEBOOK'S OTHER FEATURE OF CUSTOM AUDIENCES. WELL, NIX AND BANNON HEARD ABOUT THIS, AND APPROACH THESE RESEARCHERS AT CAMBRIDGE AND SAID, HEY, CAN WE LICENSE MODEL? STILWELL AND KACZYNSKI DECLINED THAT THEY GO ME TO BE RIGHT DOWN THE HALL TO THEIR COLLEAGUE ALEXANDER COGAN AND SAY, CAN YOU DUPLICATE THIS RESEARCH. HE SAID. THIS IS NOT A GET RICH QUICK SCHEME. XC1 KACZYNSKI SETS OUT AND INCORPORATES GLOBAL SCIENCES LABORATORY AND SETS OUT TO DO THE SAME VIRAL QUIZZES INVOLVING THIS AND USING -- WORKERS SO AMAZON MECHANICAL -- IS A CHEAP WAY TO GET PEOPLE TO DO MECHANICAL THINGS AND HE VIOLATES THE POLICIES IN MTURK (SOUNDS LIKE) WORKERS AND FACEBOOK. --- MEANWHILE COGAN AND HIS RESEARCH COLLIES ARE LOOKING APPROVAL; THE UK RESEARCH COMMITTEE AT CAMBRIDGE SAYS NO WAY AND THIS REVIEW, FIRST TIME THEY REFUSED AN APPLICATION; IT IS HIGHLY PROBLEMATIC, FACEBOOK POLICIES NOT CONDUCIVE TO ETHICAL RESEARCH AND YOU SHOULD NOT BE DOING THIS AND THE RESULT OF THIS IS THAT NO ACADEMICAL RESEARCH HAS EVER BEEN PUBLISHED. THIS OF COURSE BECAME AN ENORMOUS INTERNATIONAL CLINICAL INCIDENT BUT NO ACADEMIC RESEARCH EVER CAME OF IT. BUT HERE'S COGAN SAYING, THIS RESEARCH WAS IN FACT APPROVED BY THE INSITUTION -- IRB -- IN ONE DAY, EXEMPT FROM ETHICAL COLLECTIONS AND I HAVE THE COLLECTION OF DATA SCIENTIST USING THE TERM "EXEMPT" ON TWITTER. A COMMON MISTAKE TO SAY THAT EXEMPT MEANS RESPONSIBLE; EXEMPT MEANS IT'S EDITING FALLS UNDER THE RULES AS DEFINED IN THE COMMON RULE. --- WHY IS THIS ALL USEFUL? IT IS USEFUL BECAUSE FACEBOOK HAS A TOOL THAT ENABLES YOU TO FIND PROXY PROFILES AND THIS MAKES SENSE IF YOU ARE SELLING SHOES; IF I'M A SNEAKER STORE, I WANT TO FIND A SNEAKER STORE, TARGETING MEN 18-30; HOWEVER KNOWING THESE PSYCHOLOGICAL PROFILES ALLOW A POLITICAL ADVERTISER TO FIND PEOPLE BASED ON THEIR PERSONALITY AND YOUR LOCATION. AND TO CREATE PSYCHOLOGICALLY TAILORED CONTENT. --- CAMBRIDGE ANALYTICAL WHILE WORKING FOR THE COMPTON PAIN CLAIM TO TEST HUNDRED AND 175,000 UNIQUE TARGET ADS; THE EXACT MODEL, TO PREDICT WHAT KIND OF POLITICAL ADVERTISING WAS USEFUL FOR YOU INDIVIDUALLY. WHAT WE WILL SEE OVER AND OVER AGAIN IS THESE HALLMARKS OF THE RESEARCH ETHICS SCANDAL, METRICS JUMPING FROM THE MAINS FROM PSYCHIATRI TO SOCIAL MEDIA TO ELECTORAL DATA AND THAT IS BECAUSE OF THE VALUE OF THE MODEL TO TARGET INDIVIDUALS; MODELS ARE NO LONGER ABOUT POPULATIONS; THEY ARE ABOUT TARGETING INDIVIDUALS. --- WE ALSO SEE THINGS LIKE (DID NOT UNDERSTAND) -- BLURRED LINES BETWEEN ACADEMIC AND COMMERCIAL, ABUSE OF MTURK WORKERS, THIS IS A STRONG SIGNAL OF UNETHICAL PRACTICES YET IT IS VERY COMMON; DECEPTIVE AND OPAQUE RECRUITING TACTICS, AND MOST IMPORTANTLY THE DOWNSTREAM EFFECTS OF RESEARCH LIKE THIS ARE NEARLY IMPOSSIBLE TO IMAGINE BECAUSE THE MODELS ARE HIGHLY PORTABLE AND FAR MORE VALUABLE IN THE DAT. --- WE PAY ATTENTION TO THE DATA THAT NO ATTENTION TO THE MODELS AND THEREFORE IT BECOMES VERY HARD TO TRACK THE CONSEQUENCES OF THIS STUFF ONCE IT IS OUT IN THE WORLD. I WANT TO ACKNOWLEDGE THE FOLKS WHO CONTRIBUTED TO THIS RESEARCH. THANK YOU. >> JODI DANIEL: BRENDA, WOULD YOU LIKE TO STEP UP? THANK YOU. >> BRENDA LEONG: THANK YOU, GOOD MORNING. WOW, THAT WOULD BE A HARD ACT TO FOLLOW. A LITTLE BIT ABOUT US IF YOU WANT TO TALK MORE, MY BOSS ASKED YOU TO COME BECAUSE HE HAD THE CHANCE TO GO TO IRELAND AND HAPPILY TOOK IT SO I'LL WILL BE TALKING ABOUT SOME INTERESTING ISSUES AND THE TOPIC I WILL TALK ABOUT A STRIKING A BALANCE BETWEEN THE DATA RISK ANALYSIS AFTER RESEARCH AND AFTER THE PRESENTATION WERE HEARD I'M NOT SURE THERE IS A BALANCE BUT WE WILL SEE IF WE CAN FIND WAYS TO PROTECT THE BENEFITS OF SOME OF THE RESEARCH THAT CAN BE DONE AND MAYBE IDENTIFY OR PREVENT SOME OF THESE KINDS OF HARMS. --- SO AS JAKE VERY ADEQUATELY COVERED, THERE IS A LOT OF TENSION BETWEEN THE RESEARCHER CAN BE DONE AND THE PRIVACY RESPONSIBILITIES OR THE PUBLIC TRUST ISSUES THAT ARE BECOMING MORE PARENTS IN PEOPLE'S UNDERSTANDING OF WHAT IS HAPPENING RELEASE THEIR AWARENESS THAT SOMETHING IS HAPPENING BEHIND THE SCENES THAT IS THE LAST PIECE OF THE CYCLE THAT WAS ON ONE OF THE SLIDES, THE CREEPY FACTOR OF WHAT PEOPLE ARE STARTING TO REALIZE, THE REST OF THE ICEBERG THAT THEY MAY NOT BE AWARE OF BASED ON SECONDARY USES OF DATA AND THINGS LIKE THAT. --- THE SUM OF THE FIRST ISSUES THAT WE HAVE TO CONSIDER ARE IN THIS NEW CONTEXT, WHAT IS RESEARCH? THE IDEA THAT THESE LEGAL CONSIDERATIONS THAT ARE NOW COMING INTO PLAY IN GPR, OR CCPA -- THE CALIFORNIA LAW OR THE EUROPEAN PRIVACY DETECTION LAWS, AND HOW THOSE WILL NEED TO BE INCORPORATED MOVING FORWARD AND IT IS A CONSTANTLY EVOLVING ENVIRONMENT RIGHT NOW IN THAT LEGAL CONTEXT PARTICULARLY AS THESE NEW LAWS THAT ARE JUST COMING INTO EFFECT OR ABOUT TO COME INTO EFFECT IN THE CASE OF CALIFORNIA --AND THE GUIDANCE WE'RE GETTING FROM THE REVELATORY AGENCIES ABOUT DEFINITIONS AND CATEGORIES AND THINGS LIKE THAT SO FOR EXAMPLE THERE IS A REASON RULING IN CALIFORNIA THAT CLARIFIED OR EXPANDED THE DEFINITION OF WHAT A DATA BROKER IS TO THE EXTENT THAT THERE IS SOME BELIEVE THAT ALMOST ANY COMPANY THAT COLLECTS OR USES DATA AT ALL ONLY CONSIDERED A DATA BROKER. YOU CAN HAVE YOUR OWN YOUR OWN OPINION ABOUT WHETHER THAT IS A GOOD THING, AND THEN MOVING FORWARD IN TERMS OF WHAT PROTECTIONS MIGHT BE IMPOSED. --- SO, WE HONESTLY HAVE TO EXPAND OUR UNDERSTANDING AND RESEARCH FROM THE TRADITIONAL PUBLISHED PEER REVIEW, ACADEMIC CYCLE WHERE THE PROPOSALS OUR OFFER AND THAT IT IS COLLECTED AND AS IT WAS POINTED OUT THE MODEL IS BUILT AND NOW THE DATA IS BEING COLLECTED BY VIRTUE OF OUR LIVING OUR LIVES IN A SEMI-DIGITAL WAY; AND WHAT IS BEING DONE BY THAT EITHER BY ACADEMIA OR INDUSTRY THEMSELVES AND PHASES LIKE COMPATIBLE BUSINESS PURPOSES IN PRODUCT DEVELOPMENT ARE SOME OF THE CONCERNS AROUND THAT. --- PART OF WHAT WE DO IS TRYING TO DEFINE THE LINES BETWEEN RESEARCH AND PRODUCT DEVELOPMENT -- IT IS NOT ALWAYS CLEAR THAT THE COMPANY IS DOING IT OR THE PEOPLE IS DOING IT AS THEY ARE DOING IT WHEN IT IS DIRECTED TARGETED TOWARDS A NEW PRODUCT OR A BETTER PRODUCT OR SERVICE OR FEATURE OR WHEN IT IS IN FACT WHAT MIGHT BE MORE GENEROUSLY CALLED PURE RESEARCH; IT COULD BE THE CASE WITH MEDICAL DATA THAT CAN BE EASY TO UNDERSTAND OR CONSUMER DATA AND BEHAVIORAL PROFILING AND TARGETING AND THINGS LIKE THAT. --- WINNERS EACH SUBJECT A DIFFERENT LEGISLATORS REQUIREMENT? AND DOES IT DEPEND ON WHERE THE DATA CAME FROM? THAT IS A DISTINCTION WE ARE MAKING WHETHER THE DATA CAME FROM INDUSTRY OR A SPECIFIC COLLECTION PROCESS FOR THE PURPOSE OF RESEARCH. IS THAT REALLY THE DISTINCTION THAT SHOULD MATTER MOVING FORWARD? IN TERMS OF HOW WE DEFINE THESE ISSUES AND WHERE WE DRAW THE LINE FOR WHAT LEGISLATION OR WHAT REGULATORY REQUIREMENT SHOULD APPLY. AND DOES IT DEPEND ON WHERE THE RESEARCH IS CONDUCTED? WHETHER AT THE COMPANY OR BY AN ACADEMIC IN A RESEARCH CENTER AND ULTIMATELY WHAT THE RESEARCHER IS USED TO; FOR EXAMPLE IN CAMBRIDGE ANALYTICA THERE WERE NO REPORTS THAT REPUBLISH. IS THAT PART OF THE CRITERIA TEST FOR WHEN SOMETHING IS RESEARCH AND HOW IT SHOULD BE COVERED? THERE IS CLEARLY A LOT OF SKEPTICISM BOTH IN THE PUBLIC FROM POLICYMAKERS AND JUST GENERALLY, COMPARED TO THE PAST SORT OF TRUST IN THE RESEARCH PROCESS WHERE TRADITIONAL ACADEMIC RESEARCH CARRIES A CERTAIN PATINA OF HIGH -- DONE IN AN ACADEMIC SETTING FOR SOCIAL PURPOSES TO ACHIEVE NEW INSIGHTS AND ADVANCED OUR UNDERSTANDING IN THE FIELD AND THINGS LIKE THAT. AND NOW THERE IS EXTENDED HARM TO THAT REPUTATION AND LACK OF TRUST FROM CONSUMERS GENERALLY FROM POLICYMAKERS AND OTHER ADVOCATES ABOUT THAT. --- SO, I'M GOING TO DISCUSS A LITTLE MORE IN DETAIL TODAY SOME OF THE HARMS; AGAIN THE FOUNDATION WAS LAID IN THE PREVIOUS PRESENTATION AND HOW THAT MIGHT LOOK BUT I WILL WALK THROUGH SOME OF THE DIFFERENCES IN SPECIFIC WAYS THAT HARMS CAN OCCUR THAT WE HAVEN'T THOUGHT OF IN THE PAST THAT EXCEED THE CONCERNS ABOUT THE HARM IN THE PROCESS TO THE INDIVIDUAL OF THE COLLECTION OF DATA WHICH IS THE TYPICAL IRB CONSIDERATION, WHEN EVALUATING A PROJECT. --- THERE ARE MULTIPLE CATEGORIES OF HARM. THE FIRST OUR OPPORTUNITY HARMS LIKE EMPLOYMENT OPPORTUNITIES, EITHER BEING AWARE OF EMPLOYMENT JOBS BECAUSE OF ACCESS TO SEARCH ALGORITHMS THAT PEOPLE MAY NOT BE AWARE OF; IF LINKEDIN SEND YOU A LIST OF HERE ARE THESE JOBS AVAILABLE OR IF YOU SUBSCRIBE TO A PARTICULAR JOB SEARCH PLATFORM THAT PROVIDES YOU A DAILY OR WEEKLY LIST OF JOB OPPORTUNITIES TAILORED TO YOU, WHAT GOES INTO THAT SELECTION? WHAT ARE YOU NOT SEEN? THERE'S EXAMPLES OF PEOPLE WHO DO A SEARCH ON A SITE THAT IDENTIFY THE FEMALE AND THE SEARCH RESULTS THEY GET BACK IN TERMS OF JOB OPPORTUNITIES TYPICALLY TEND TO BE CORRELATED AT LOWER LEVELS OF EMPLOYMENT OR OPPORTUNITY THAN A MAN WITH THE SAME SKILL SET MIGHT GET WITH THE SAME SEARCH. SO THOSE KINDS OF BEHIND THE SCENES OF ARITHMETIC IMPACTS THAT PEOPLE ARE NOT AWARE OF. --- LIKEWISE HOUSING AND EDUCATION AND FELLOWSHIPS AND GRANTS ALL OF THOSE KINDS OF THINGS THAT ARE IN HIS SEARCH PLATFORM IN THE SENSE THAT YOU ARE NOT GOING TO BE ABLE TO GO OUT AND FIND ALL OF THEM ON YOUR OWN. YOU ARE GOING TO RELY ON ALERT SYSTEMS RESEARCH ENGINES OR THE DESCRIPTION OPPORTUNITIES TO TRY TO MAKE YOU AWARE WHEN THOSE OPPORTUNITIES COME ALONG AND WHAT SHOWS UP IN YOUR ACTUAL INBOX IS GOING TO VARY GREATLY. AND THEN FINALLY INSURANCE AND SOCIAL BENEFITS CAN BE SOME OF THE MOST IMPACTFUL OF THESE OPPORTUNITY COSTS FOR INDIVIDUALS AT THE INDIVIDUAL LEVEL. --- AND THEN THERE'S THE ECONOMIC IMPACT OF CREDIT DISCRIMINATION, DIFFERENTIAL PRICING AND AGAIN NARROWING OF CHOICES; YOU ARE SIMPLY NOT OFFERED SOME OF THE ECONOMIC OPPORTUNITIES WITHOUT KNOWING WHY. --- THEN THERE'S THE LESS IDENTIFIABLE HARMS WHICH INCLUDE SOCIAL DETRIMENT AND THINGS LIKE NETWORK BUBBLES WHICH I THINK WE SAW SOME OF THE EXAMPLES IN THINGS LIKE VOTING PATTERNS AND SOCIAL ENGAGEMENT, PARTICIPATING IN MARCH IS AN ALL OF THOSE KINDS OF MEDIA INFERENCES THAT I THINK WE ALL ARE FAMILIAR WITH OR READ ABOUT; HOW PEOPLE SELF SELECT AND ARE EITHER ALGORITHMICALLY FURTHER NARROWED. WITHOUT A CONCERTED EFFORT ON THEIR PART TO COMBAT OR PREVENT THAT. --- SIGNATURE HARMS AND CONSTRAINTS BASED ON BIAS, LOSS OF LIBERTY, INDIVIDUAL GROUP SURVEILLANCE AND ALL WAY UP TO INCARCERATION OR UNEQUAL TREATMENT WITHIN THE CRIMINAL JUSTICE SYSTEM ARE EXAMPLES OF SOCIAL DETRIMENT. SO THERE ARE ALSO HARMS THAT ARE VERY SPECIFIC TO AI AND MACHINE LEARNING PROCESSES BEYOND THOSE THAT LIKE YOU TO DIVIDE THAT ARE BROADLY TRUE OF DATA, THE DATA AND THE DIGITAL WORLD. GENERALLY WE HAD A WHITE PAPER COMING UP TOMORROW THAT HAS THIS CONVERSATION IN IT THAT TALKS ABOUT THE IDEA THAT IN AN AI MACHINE DRIVEN WORLD ACCESS THE DATA IS NOT REQUIRED; OUR GENERAL UNDERSTANDING OF HARMS IN THE SENSE OF BREACHES OR GETTING INFORMATION -- IS ACTUALLY GETTING ACCESS TO THEIR DATA OR THEIR DATA SET; THAT REALLY IS NOT A REQUIREMENT MACHINE LEARNING BECAUSE OF SOME OF THE EXAMPLES WE'VE SEEN. IF YU HAVE THE MODEL OR INFORMATION ABOUT THE MODEL YOU CAN DO SOME OF THE HEART EVEN WITHOUT THAT. SO SOME OF THE EXAMPLES OF THAT ARE THESE MEMBERSHIP INFERENCES WHICH IS TO SAY THAT YOU CAN TAKE SOME OF THE OUTPUTS OF A DATA SET UNDER MACHINE MODEL AND FIGURE OUT WHOSE DATA WAS INCLUDED OR WHETHER A SPECIFIC INDIVIDUAL'S DATA WAS IN THATDATA SET ORIGINALLY WHICH IS OBVIOUSLY A VIOLATION OF THE PARTICULAR PRIVACY AND PROBABLY OF THE TERMS OF THE COLLECTION AND DATA SET USE OF THE MODEL. --- SOMETIMES THIS CAN BE DIDD (SOUNDS LIKE) BACK TO A PERSON; YOU WHERE IT IS CREATING A PI OF SOME KIND IN THE RECORD AND WE ARE CREATING A MODEL SO EVEN THOUGH YOU MAY NOT HAVE ACCESS TO THE CODING OR THE INTELLECTUAL PROPERTY ASPECTS OF THE OTHER INSIGHTS INTO THE MODEL DESIGN YOU CAN ACTUALLY USE THE OUTPUTS OF THE MODEL TO RE-CREATE THE MODEL ITSELF AND THE PHOTO HERE'S AN EXAMPLE WHERE USING THE OUTPUTS FROM IT THEY WERE ABLE TO RE-CREATE -- USING THE OUTPUTS THEY RETREATED THE FACE, IT IS IMPRECISE BUT IT WAS MATCHED TO THE PERSON AND CAN BE USED TO IDENTIFY THE DATA. --- THE OTHER TYPE OF MACHINE LEARNING SPECIFIC HARMS OTHER WAYS THAT THESE SYSTEMS CAN BE ATTACKED WHICH AGAIN HAS MORE TO DO WITH EXTERNAL ACTIVITIES THAN IT DOES ACCESS TO DATA SETS OR ACCESS TO THE MODEL ITSELF. BUT THERE ARE WAYS TO MANIPULATE THE MODEL DIRECTLY; ONE OF THE TERMS IS CALLED POISONING WHERE YOU INSERT MALICIOUS DATA INTO THE TRAINING DATA DURING THE DEVELOPMENT PHASE IN ORDER TO SUBVERT THE OUTPUTS AT A LATER POINT IN TIME. SO THIS WOULD BE CREATING BIAS WITHIN THE DATA SET INTENTIONALLY WHICH WOULD DISCRIMINATE AGAINST GROUPS BASED ON PARTICULAR FACTORS AND IT COULD BE PROTECTED CLASSES LIKE RACE OR GENDER BUT CAN ALSO BE GEOGRAPHIC OR VOTER PROFILES OR SHOPPING PREFERENCES OR ANY NUMBER OF THINGS THAT YOU ARE ATTEMPTING TO BIAS BY USING THIS TECHNIQUS. --- EVASION IS PUTTING INPUTS INTO THE MODEL WITH THE INTENT TO HAVE IT (DID NOT UNDERSTAND) -- YOU MIGHT HAVE SEEN THINGS ABOUT THE IMAGE RECOGNITION SYSTEMS IN THE DEVELOPMENT OF AUTONOMOUS VEHICLES WHERE THEY CAN RECOGNIZE STOP SIGNS AND ONE OF THE WAYS IN WHICH YOU CAN BREAK THE SYSTEM IS SIMPLE TO PUT A SMALL REFLECTIVE SQUARE STICKER ON THE STOP SIGN AND THAT INTERRUPTS THE PATTERN RECOGNITION SUFFICIENTLY THAT IT NO LONGER RECOGNIZES IT AS A STOP SIGN WHICH OF COURSE IS A PRETTY BIG DEAL IF YOUR CAR IS DRIVING AUTONOMOUSLY AND IT NEEDS TO KNOW THAT IT NEEDS TO STOP AT THAT POINT. --- COLLECTED HARMS ARE PEOPLE WHO ARE NOT INCLUDED IN THE TRAINING SETS AND POTENTIALLY NOT SUBJECTED TO THE MODEL PROCESSING AT ALL BUT INFO ABOUT THEM CAN BE INFERRED ANYWAY AND THIS AGAIN TIES BACK TO THE EXAMPLE THAT WE ALREADY SAW WHERE WE HAVE MODELS THAT HAVE INCREASING POWERFUL PREDICTIVE ABILITY AND IMPACT THOSE ENTIRELY OUTSIDE THE MODEL EOSYSTEM SO NOT ONLY DID THE FRIENDS OF THE PEOPLE WHO TOOK THE QUIZ GET IMPACTED BECAUSE THEY ARE NOW WERE SUCKED INTO THAT ECOSYSTEM OF THE PARTICULAR USE CASE BUT ALSO ANYBODY ANYWHERE THAT CAN BE IDENTIFIED IN SOME WAY AND CORRELATED WITH OTHER MEMBERS OF THE DATA SET WHERE PROFILING IS ALREADY KNOWN; IT PROBABLY DOES NOT TAKE MANY SPECIFIC DATA POINTS; THEY MAY NOT BE ON FACEBOOK SO YOU MAY NOT USE IT IN TERMS OF LIKE BUT MAYBE THEY ARE IN SOME OTHER ECOSYSTEMS OR OTHER SOCIAL MEDIA PLATFORMS OR OTHER SETS LIKE A FREQUENT SHOPPER CLUB OR THINGS LIKE THAT WHERE THE DATA CAN BE CORRELATED WITH OTHER PEOPLE WHOSE PATTERNS ARE READY IN THE DATA SET AND HAVE BEEN MODELED AND INFERENCES CAN BE MADE. THIS CAN ALSO BE DANGEROUS IN APPLICATIONS THAT ARE OTHERWISE SORT OF CONSIDERED BENEFICIAL JUST BECAUSE THE VARIATION UNDER THE (DID NOT UNDERSTAND) PERFORMANCE; AN EXAMPLE OF SOME OF THE OCCUPATIONAL OR PHYSICAL THERAPY SYSTEM DESIGNED TO WORK WITH CHILDREN FOR EXAMPLE AND IF VOICE RECOGNITION OR VOICE PATTERN IS USED TO IDENTIFY CHARACTERISTICS OF AUTISTIC CHILDREN OVER A SPECIFIC DATA SET AND THEN THOSE MODELS ARE USED OUTSIDE OF THAT FOR OTHER CHILDREN IN A NON-AWARE ENVIRONMENT, THAT THEY ARE BEING EVALUATED, THEY'RE JUST SPEAKING IN A PLACE AND ACTUALLY DATED BEING EXTRACTED AND IDENTIFIED AS POTENTIALLY ARTISTIC OR SOMEWHERE IN THE AT THE SICK SPECTRUM AND WHAT THE IMPLICATIONS OF THAT MIGHT BE. --- SO THERE NEEDS TO BE A BENEFIT RISK ANALYSIS HONESTLY FOR HOW TO DEAL WITH THESE CONCERNS IN ORDER TO DETERMINE WHAT THE HARMS MIGHT BE AND WHAT THE BENEFITS MIGHT BE AND HOW TO JUDGE WHEN IT IS WORTH PURSUING THE PROCESS ANYWAY AND THIS IS BOTH THE LEGAL AND ETHICAL EVALUATIONS THAT CAN TAKE PLACE, THIS IS JUST A PARTICULAR WAY OF DESIGNING THAT, NOT A MAGIC BULLET. IT IS FROM WORK WE DID ON SMART CITIES IN OPEN DATA AND DECIDING WHAT THE CITY WILL BE MAKING AVAILABLE TO THE PUBLIC AND THEY WANTED TO HAVE THE OPEN DATA AS MUCH AS POSSIBLE BUT BALANCING RISK AGAINST VALUE IN TERMS OF WHAT DATA TO MAKE AVAILABLE. ---1 WE DID ANOTHER PAPER LAST YEAR CALL BEYOND SPRING ABILITY WHICH IS ABOUT MANAGING RISK IN MACHINE LEARNING MODELS; THIS IS THE MEAT OF THE ENTIRE CONCEPT OF BEYOND IRB PROJECT; WE ALSO HAVE A PROJECT -- PART OF THE REVIEW HAS A TARGET ANALYSIS AND NIKE WON'T GO THROUGH DETAIL NOW, BUT HOW TO DESIGN WAS ORIGINALLY CREATED FORM PARTICULAR USE AND NOW MIGHT BE REPURPOSED FOR OTHER USES OR JUST WHETHER IT IS IN FACT APPROPRIATE FOR ITS USE CASE AS IT IS BEING IMPLEMENT IT; THERE'S ALWAYS TRADE-OFF, THERE'S NO SUCH THING AS A FAIR AND BALANCED SYSTEM. IT ISS JUST WHAT THE DEFINITION OF FAIRNESS IS. FOR EXAMPLE THE CRIMINAL SENTENCINGS GOING; THERE SOME TRADE-OFFS THAT ARE MORE COMPLEX THAN A LOT OF THE MEDIA REPRESENTATION REFLECTS AND THERE ARE CHOICES THAT EVERY DESIGN IS TO MAKE AND THAT IS NOT A DEFENSE ABOUT WHETHER IT IS GOOD ABOUT; THAT IS JUST TO SAY THAT THOSE CONFLICTS OCCUR IN ALMOST ANY MODEL AND CAN BE DOCUMENTED AND IDENTIFIED. --- IN PARTICULAR THE MATERIALITY OR THE SIGNIFICANCE TO INDIVIDUALS VERSUS THE VALUE OF SOCIAL GROUPS CONVERSELY THE HARM TO SOCIAL GROUPS VERSUS THE RIGHT OF INDIVIDUALS MAKE HIS HEART SO THERE MAY BE ASPECTS OF THE DESIGN THAT BENEFIT AN INDIVIDUAL THAT MIGHT CAUSE BROADER HARM COLLECTIVELY BECAUSE OF THE WAY IT IDENTIFIED PATTERN PROFILING THAT CAN BE USED FOR INFERENCES FOR UNRELATED INDIVIDUALS; THERE IS ALSO THE CONCEPT THAT THERE IS SOME SOCIAL GOOD TO BE DONE BY GROUP PATTERN IDENTIFICATION FOR HEALTH DATA AND THINGS LIKE THAT BUT IT MIGHT CAUSE A RECIPROCAL INDIVIDUAL HARMED BECAUSE SOMEONE WITH A RARE DISEASE OR SOMEONE WITH SOME OTHER SENSITIVE DATA ASPECT COULD BE IDENTIFIED AS PART OF THAT GROUP WITHOUT HAVING EITHER PARTICIPATED OR HAD THAT INTENTION. --- SO, SOME EXAMPLES OF THIS ARE FACIAL RECOGNITION, USED A LOT. AND WHAT THE STRATEGIES MIGHT BE FOR THAT. SOME PLACES ARE HAVING BANS; THERE IS A TALK IN CONGRESS OF MORATORIUMS, A SHORT-TERM BAN, AND WILL TRY TO FIGURE OUT HOW TO REGULATE THIS. WHAT THE EIGHT REACTIONS MIGHT BE IF WE ARE GOING TO HAVE THE SYSTEMS. CAN WE MODEL THEM AFTER OTHER WAYS THAT WE HAVE TREATED SEMI-DANGEROUS THINGS BEFORE? SUCH AS THE CONTROL AND RELEASE AND SALE OF DANGEROUS CHEMICALS; NOT BANNED EVEN THOUGH THEY HAVE SIGNIFICANT NEGATIVE POSSIBILITIES BUT IT IS CONTROLLED AND TRACKED AND REGULATED SPECIFICALLY. --- CRIMINAL JUSTICE I HAVE ALREADY MENTIONED. THERE ARE TOOLS TO ADJUST THESE. PET STANDS FOR PRIVACY ENHANCING TECHNOLOGIES; INTERMEDIATE AND FEDERATED TESTING. THERE ARE TOOLS AND PEOPLE WORKING HARD TO DO THIS RESPONSIBLY AND MONITOR IT AND CONTINUOUSLY AND EFFECTIVELY CHALLENGE MODEL EXECUTION THROUGHOUT ITS LIFECYCLE SO THEY CAN IDENTIFY TRENDS AND PROBLEMS AND HIGHLIGHT SOME OF THE CONCERNS THAT THESE CASES SHOW AND ALLOW THE PROCESS TO CONTINUE FOR THE BENEFITS WITHOUT THE EXPLOITATION ON THE BACKEND FOR MINIMIZING WHEN THAT IS HAPPENING IN ORDER TO PROTECT INDIVIDUALS AND HELP DRIVE THE POLICY THEY CAN CONTROL THAT. THAT IS ALL LIKE YOU HAVE TIME FOR RIGHT NOW. THANK YOU VERY MUCH. >> [APPLAUSE] GOOD MORNING. THANK YOU VERY MUCH TO MS. D AND YVONNE FOR THE INVITATION AND I'VE ALREADY LEARNED A LOT THIS MORNING. NOW I'M GOING TO TALK ABOUT SOMETHING A BIT DIFFERENT FROM THE FIRST TWO SPEAKERS; NOT GOING TO TALK ABOUT DATA OR POLICY, I'M GOING TO FOCUS MORE ON PEOPLE. AND QUESTIONS ABOUT THE EXTENT WE UNDERSTAND HOW PEOPLE ARE THINKING ABOUT PRIVACY IN THIS DAY AND AGE SO I WOULD PROPOSE TO YOU THAT WE TALK ABOUT PRIVACY A LOT. IT'S VERY DIFFICULT TO READ THE NEWSPAPER OR LISTEN TO THE RADIO OR TALK WITH FRIENDS OR COLLEAGUES WITHOUT A TOPIC RELATED TO PRIVACY, SECURITY, DATA BREACHES COMING UP. BUT WHAT DOES THAT REALLY MEAN? AND I WOULD INVITE YOU TO TAKE THIS OPPORTUNITY TO THINK ABOUT WHAT'S PRIVACY MEAN TO YOU. --- I GOT VERY INTERESTED IN THIS IDEA A FEW YEARS AGO AND STARTED SORT OF ON A TOUR OF PEOPLE THE I TALKED TO AND ASKING THEM, WHAT IS PRIVACY MEAN TO YOU? WHAT HAPPENS IN MY HOUSE -- THIS IS A FEMALE IN HER 20S -- PERSONAL THOUGHTS AND POSSESSIONS, A SIXTH-GRADE GIRL. PLACES WHERE I HAVE A BAND-AID, A FIVE-YEAR-OLD. NAKED BODY, WOMAN IN HER 30S; WHAT'S IN MY MIND, A 12TH GRADER. AT PRIVACY POLICIES NOW IN HIS 40S. --- MY RIGHT AS A CITIZEN, A 12TH GRADER; THINKS I CAN CONTROL, NOT SPILLING OUT EVERYTHING ON INSTAGRAM EIGHTH-GRADE MALE. THINGS I LOCK IN SECURE. MY INTERNET ACTIVITY, WOMAN IN HER 20S; DOES ANYBODY THINK THE INTERNET HAS PRIVACY? THE POTTY, A THREE-YEAR-OLD. BEING ANONYMOUS IN THE ROWD AND WOMAN WHO 30S AND THIS IS ONE OF MY FAVORITES, THE GOVERNMENT DOES NOT RESPECT IT, SEVENTH GRADE MALE, ONE OF MY FAVORITES . AND ACTUALLY THIS IS A 10TH GRADE BOY ACTUALLY NOTHING IS PRIVATE. --- THESE CONVERSATIONS HAVE GIVEN ME A LOT TO THINK ABOUT AS A SOCIAL SCIENCE RESEARCHER AND A CLINICAL PSYCHOLOGIST AND THAT'S HOW I ORIGINALLY TRAINED; I GOT INTERESTED IN THIS CONCERT A FEW YEARS AGO FOR A VARIETY OF REASONS AND STARTEDTO LOOK TO THE HISTORICAL CONTEXT AND SO ON AND SO FORTH AND ONE THING THAT IS QUITE INTERESTING ABOUT PRIVACY IS WE ARE TALKING ABOUT IT A LOT TODAY BUT IN FACT IT IS A CONCEPT THAT'S BEEN AROUND FOR QUITE A LONG TIME SO IT WAS FIRST EMBODIED IN OUR LAWS IN THE LATE 1800S WHEN WARREN AND BRANDEIS WROTE THE FAMOUS LAW REVIEW ARTICLE ON THE RIGHTS TO PRIVACY FOCUSING ON INFORMATIONAL PRIVACY AND WHAT IS INTERESTING ABOUT THIS IS THAT IT CAME AT THE HEELS OF NEW INVENTIONS AT THE TIME. INCLUDING PHOTOGRAPHY AND NEWSPAPER AND THINGS THAT WERE CHALLENGING CONCEPTIONS OF PRIVACY. --- OF COURSE NOW WE HAVE A RANGE OF CONTEMPORARY INNOVATIONS THAT ENABLE A GENERATION OF VAST AMOUNT OF DATA THAT WE HEARD A LOT ABOUT THIS MORNING; VERY GRANULAR. IN MY CASE IT IS HEALTH DATA, CLEANING OUT THE PERSONAL HEALTH DATA FOOTPRINT AND A LOT OF THE DATA GENERATED BY MOLD DEVICES, BY COMPANY BASED DIRECTED TO THE CONSUMER GENETIC TESTING AND THE FLOWS OUTSIDE CURRENT HEALTH REGULATIONS WHICH MAKES IT CHALLENGING. SO I ASK AGAIN, WHAT IS THE MEANING OF PRIVACY IN THIS NEW CONTEXT THAT WE ARE FACED WITH? --- AND SO AGAIN, LOOKING AT THE LITERATURE IT IS INTERESTING BECAUSE PRIVACY IS ALSO A CONCEPT THAT IS TOUCHED ON IN A LOT OF DIFFERENT FIELDS; I WOULD NOT CLAIM TO BE AN EFFORT ON THE VAST LITERATURE RELATED TO PRIVACY BUT I WAS STRUCK BY WHAT VARIOUS SCHOLARS HAVE SAID. HERE'S ONE EXAMPLE. I'M REFERRING TO PRIVACY AS A CONCERT IN DISARRAY; NOBODY CAN A TICKET AT WHAT IT MEANS. AS A PSYCHOLOGIST I MIGHT PUT FORTH THE IDEA (DID NOT UNDERSTAND) -- BECAUSE IT MEANS DIFFERENT THINGS TO DIFFERENT PEOPLE AND SO THAT IS THE FOCUS OF A PROJECT THAT I KICKED OFF IN MY GROUP ABOUT FOUR YEARS AGO; THIS IS FUNDED BY THE NIH, THE GENOME INSTITUTE. --- THE GOALS OF THE PROJECT A PRETTY STRAIGHTFORWARD. FIRST TO DEVELOP A MODEL TO TRY TO UNDERSTAND INDIVIDUAL PRIVACY AND SECONDLY TO DEVELOP A VALID AND RELIABLE ASSESSMENT INSTRUMENTS THAT WE MIGHT BE ABLE TO USE TO MEASURE INDIVIDUAL PRIVACY. AND PERHAPS BETTER UNDERSTAND INDIVIDUAL'S PRIVACY RELATED BEHAVIORS AND OUTCOMES RELATED TO PRIVACY AND I'M GRATEFUL TO JAKE, HE GAVE A NICE OVERVIEW OF SOME OF THE PSYCHOMETRIC CONCEPT WHEN HE WENT OVER HIS CAMBRIDGE ANALYTICA CASE STUDY. I'LL SHOW YOU THAT OUR WORK IS IRB APPROVED IN ALL THE PARTICIPANTS PROVIDED INFORMED CONSENT. --- ONE REASON FOR GOING THIS ROUTE IS TO TRY TO UNDERSTAND HOW PEOPLE CAN THINK ABOUT PRIVACY ON A PERSONAL LEVEL AND ALSO I THINK PART OF THE BACKGROUND IS THAT ALTHOUGH OVER THE YEARS THERE HAVE BEEN SOME MEASURES AND TOOLS THAT HAVE BEEN DEVELOPED, THEY'VE HAD INCONSISTENT LIABILITY AND VALIDITY, GENERALLY POOR PREDICTORS OF OUTCOMES AND BEHAVIOR AND ALSO HAVEN'T BEEN FOCUSED ON THIS HEALTH-RELATED, HEALTH DATA RELATED LANDSCAPE SO THAT WAS A GAP WE WERE HOPING TO FILL. --- SO, THE WORK THAT WE'VE DONE HAD TWO COMPONENTS: A QUALITATIVE AND A QUANTITATIVE. TODAY I'M MOSTLY GOING TO FOCUS ON THE QUALITATIVE WORK THAT WE'VE DONE, AND WE WILL TALK ABOUT SOME BEST PRACTICES AND TAKE A RIGOROUS APPROACH. WE STARTED A FEW YEARS AGO CONDUCTING FAIRLY EXTENSIVE QUALITATIVE WORK, A COMBINATION OF DOING INDIVIDUAL INTERVIEWS AND FOCUS GROUPS ACROSS FIVE DIFFERENT PREVIOUSLY SELECTED PATIENT AND COMMUNITY COHORTS. --- AND THEN WHEN THAT WORK WAS DONE -- I'LL TALK ABOUT SOME OF THE FINDINGS FROM THAT WORK WE STARTED TO EMBARK ON A PROCESS TO DEVELOP A MEASURE WHERE TO GIVE A LITTLE BIT OF THAT INFORMATION, DEVELOPED ITEMS, TEST THEM, GO THROUGH A VERY ITERATIVE PROCESS OF FEELING THE ITEMS IN THE ULTIMATE GOAL IS TO GET TO AN INSTRUMENT THAT MIGHT BE ROUGHLY AROUND 20 OR SO ITEMS. YOU CAN SEE HEAR THE PROCESS THAT WE DO NOT. --- WE JUST FINISHED HIS THIRD ONLINE FIELDING. I DON'T HAVE RESULTS FOR THAT WORK TODAY. TODAY I WILL PRESENT WORK DONE WITH ABOUT 100 PEOPLE AND QUANTITATIVE WORK WITH ROUGHLY 700 INDIVIDUALS. --- SO THIS IS A SNAPSHOT OF OUR PARTICIPANTS GEOGRAPHIC PARTICIPATION, THIS IS A US-BASED STUDY, WHAT DID WE TALK TO PEOPLE? THESE WERE FAIRLY LONG INTERVIEWS AND FOCUS GROUPS THAT TOUCHED ON A WIDE RANGE OF ISSUES. WE WERE INTERESTED IN TALKING TO PEOPLE ABOUT THEIR EXPERIENCES RELATED TO PRIVACY AND EXPERIENCES THAT WERE FOCUSED ON THE HEALTH, HEALTHCARE; ALSO WE ASKED ABOUT OTHER FORMS OF DATA THAT PEOPLE SORT OF WORK CREATING AND THE EXTENT TO WHICH THEY THOUGHT OTHER ENTITIES WERE ACCESSING THOSE DATA. --- AND THE QUALITY WORK UNDERWENT A FAIRLY RIGOROUS ANALYSIS AND OF COURSE WE HAD MULTIPLE CODERS AND USED A GROUNDED THEORY-BASED APPROACH FOR THOSE OF YOU FAMILIAR WITH QUALITATIVE METHODS. --- I'M NOT GOING TO GO THROUGH ALL OF THESE BUT A NUMBER OF THEMES EMERGED FROM OUR WORK. THINGS LIKE ACCESS CONTROL, THE EXTENT TO WHICH -- THESE WERE DISCUSSIONS THAT PEOPLE'S PERCEPTIONS OF A CONTROL, ACCESS TO ELECTRONIC DATA, PEOPLE TALKING ABOUT SAFE VERSUS UNSAFE SITUATIONS WITH RESPECT TO THEIR FLOW OF PERSONAL INFORMATION. PEOPLE TALK A LOT ABOUT STIGMATIZING OR EMBARRASSING INFORMATION AND REALLY THE DISCUSSIONS WERE QUITE WIDE-RANGING. --- ONE THING I WANT TO EMPHASIZE IN TODAY'S PRESENTATION IS SOMETHING WE FOUND TO BE AN IMPORTANT OBSERVATION FROM THIS WORK. THAT IS, SOME OF THE THEMES OR CODES THAT EMERGED SEEM TO REFLECT CULTURAL FRAMING OF PRIVACY AND WAYS IN WHICH WE TALK ABOUT PRIVACY AS A SOCIETY OR CULTURE VERSUS OTHERS SEEM TO REFLECT MORE INDIVIDUAL VALUES AND ATTITUDES AND BEHAVIORS AND I'M GOING TO TALK ABOUT THIS A LITTLE BIT FURTHER. --- SO THESE CULTURAL PRIVACY VALUES -- WE STARTED TO REFER TO THEM ABOUT FRAMINGS OF PRIVACY. WHAT OUR QUANTITATIVE DATA HAS SHOWN AS WE TURNED THESE THINGS INTO SURVEY ITEMS IN THE DEVELOPMENT OF TESTING OF OUR INSTRUMENT IS THAT THESE ARE VERY IMPORTANT BUT THEY ARE ACTUALLY POOR PREDICTORS IN A SURVEY CONTEXT OF INDIVIDUAL BEHAVIOR BECAUSE THEY ARE EITHER TOO WIDELY ACCEPTED OR TOO WIDELY DISAGREED WITH. THEY DON'T DIFFERENTIATE. --- THE CODE THEY CAME OUT FOR PRIVACY IS A MORAL RIGHT, INDIVIDUAL SHOULD HAVE CONTROL OF INFORMATION BECAUSE THAT'S THE CORRECT MORAL ARRANGEMENT; THE IDEA OF PERSONAL RESPONSIBILITY, PRIVACY IS A PERSONAL RESPONSIBILITY IN SOMETHING PEOPLE HAVE TO WORK AT TO KEEP IT. --- ALSO THE IDEA OF A TRADE UP SO PRIVACY IS MORE TRANSACTIONAL, WE TREAT OUR PRIVACY FOR GOODS AND SERVICES AND THIS NOTION OF NOTHING TO HIDE WHICH HAS COME UP IN THE LITERATURE; THE ONLY REASON ANY PRIVACY IS IF YOU HAVE SOMETHING TO HIDE. I DON'T HAVE ANYTHING TO HIDE SO I DON'T NEED IT AND THE IDEA OF FATALISM, HARKING BACK TO THE DRAWING FROM ONE YOUNG PARTICIPANTS THAT PRIVACY IS ALREADY LOST OR DOES NOT EXIST. THESE ARE CULTURAL FRAMINGS OF PRIVACY. --- AGAIN, WE STARTED TO SEE THESE THINGS AS THINGS THAT EXIST IN THE WORLD -- I'D SAY THIS IS A SPECIFIC POPULATION IN THE US, NOT ACTUALLY IN THE WORLD BECAUSE WE HAVE A LOT OF CROSS-CULTURAL WORK TO DO IN THE AREA OF PRIVACY. BUT FOR THE PURPOSES OF COMPARING TO WHAT I'M GOING TO SHOW YOU NEXT, THINGS THAT EXIST IN THE WORLD THAT DON'T NECESSARILY DIFFERENTIATE, AND THE QUANTITATIVE DATA THAT WE COLLECT IT HAS SUPPORTED THIS IN TERMS TO THE EXTENT TO WHICH WE DON'T SEE CREATION OF THE ITEMS. --- WE STARTED TO REFER TO THESE INDIVIDUAL PRIVACY VALUES; WE CALL THEM IN SOME SENSES "PRIVACY DISPOSITION." ARE PEOPLE'S PRIVACY VIEWS REALLY TRAIT-LIKE? THAT'S AN OPEN QUESTION. WE BELIEVE THESE ARE WAYS IN WHICH PEOPLE RELATE TO PRIVACY AND WE BELIEVE THESE ARE GOOD PREDICTORS OF PEOPLE'S PRIVACY RELATED BEHAVIOR SO WE'RE STILL DOING WORK TO TEST THIS HYPOTHESIS. --- THIS AXIS IS HOW PEOPLE THINK ABOUT HOW PEOPLE THINK ABOUT SHARING PERSONNEL DATA IN INSTITUTION VERSUS PERSONAL DATA SETTING; ON THESE ADDITIONAL SIDE BEHAVIORS, PRACTICES AND PERSONAL CODES RELATED TO HOW PEOPLE MIGHT SHARE INFORMATION WITH, SAY, HEALTH INSURANCE COMPANY OR WEBSITE OR AN APP, VERSUS INTERPERSONAL RELATIONS, PEOPLE'S APPROACHES AND HOW THEY SHARE PERMISSION WITH THEIR FRIENDS AND COLLEAGUES AND FAMILIES. --- THE SECOND AXIS IS RECESS TO BE COMFORTABLE ABOUT SHARING INFORMATION VERSUS REASONS TO FEAR SHARING INFORMATION; ON THE REASONS TO SHARE, PEOPLE DISCUSSED ACTUALLY ARE PERCEIVED PERSONAL OR SOCIETAL -- ALTRUISTIC BENEFITS TO SHARE PERSONAL INFORMATION. THESE ARE THINGS THAT ARE IMPORTANT TO INDIVIDUALS AND MAY DIFFERENTIATE INDIVIDUAL'S BEHAVIOR AND OUR QUANTITATIVE DATA IS SUPPORTED THAT AND WE SEE A LOT OF AREAS IN TERMS OF THE QUANTITATIVE MEASURE ITEMS THAT REFLECT THESE THINGS. --- SO TO GIE A BIT MORE INFORMATION ABOUT THE TOOL AND WHERE WE ARE GOING, AND THEN I'M GOING TO MAKE HIM IS ABOUT HOW IT CAN CONTINUE TO THE FEEL AND HOW IT CAN BE USED, THESE ARE THE AXIS I MENTIONED; UP HERE YOU HAVE REASONS TO FEAR AND IF YOU HAVE SOMEONE WHO HAS MANY REASONS TO FEAR AND MANY REASONS TO SHARE THE MIGHT FEEL CONFLICTED ABOUT SHARING VERSUS FEW REASONS TO SHARE AND FEW REASONS TO FEAR. THEY MAY BE IN DIFFERENT AND SIMILARLY HERE WE MIGHT THINK OF PEOPLE WHO MAY BE OPEN IN BOTH HIS ADDITIONAL CONTEXTS, VERSUS CLOSED IN BOTH. AND THE CHARACTERISTICS OF PEOPLE IN THE MIDDLE. --- WE BELIEVE THAT PERHAPS PROFILES WITH RESPECT TO THESE SKILLS MAY RESULT IN PRIVACY TYPES SIMILAR TO PERSONALITY TYPES IF YOU WILL. THAT MIGHT REFLECT PEOPLE'S OPENNESS TO SHARING PERSONAL DATA ACROSS SPECIFIC CONTEXT. --- I'M NOT GOING TO GO THROUGH THIS WHOLE TABLE BUT WE TRIED TO MAP OUT SORT OF THE TYPOLOGIES FOR DIFFERENT COMBINATIONS OF THESE AXIS; AND WE DON'T KNOW YET IF THESE ARE THINGS THAT MAY EXIST IN THE WORLD. THAT'S TO BE DETERMINED AFTER WE COJMPLETE OUR WORK IN PROGRESS BUT YOU CAN SEE HERE THAT ONCE AGAIN PEOPLE WHO ARE MOTIVATED AND OPEN ME ALWAYS BE HAPPY TO SHARE THE INFORMATION AND THEY ARE NOT GOING TO WORRY ABOUT IT A WHOLE LOT VERSUS PEOPLE WHO ARE CONCERNED AND RESERVED; THEY WLL BE MUCH MORE CONCERNED ABOUT SHARING AND THEY FEEL IT IS DANGEROUS AND SO ON AND THERE'S MANY TYPES HYPOTHESIZED IN BETWEEN. --- AN IMPORTANT QUESTION IS WHY DO THIS? WHY SEEK TO UNDERSTAND AND MEASURE INDIVIDUAL PRIVACY? YOU KNOW, TO SOME EXTENT, ONE COULD ARGUE REASONABLY THAT, DOES IT REALLY MATTER? HOW STABLE ARE THE PUBLIC'S VIEWS ON THIS ANYWAY? AND TO WHAT EXTENT SHOULD POLICIES BE TIED TO THIS? I THINK THOSE ARE ALL REASONABLE QUESTIONS. --- SOME OF THE IDEAS I HAVE ABOUT WHY THIS IS IMPORTANT IS ONE, TO INCREASE OUR KNOWLEDGE ABOUT A VERY DIFFICULT TO UNDERSTAND TOPIC. AND START TO BETTER UNDERSTAND WHY PEOPLE MIGHT SAY ONE THING ABOUT PRIVACY BUT ACTUALLY DO ANOTHER. IS VERY INCONSISTENT WITH THEIR BEHAVIOR, A THING REFERRED TO AS THE PRIVACY PARADOX. POTENTIALLY HELP US UNDERSTAND THE EXTENT TO WHICH WE HAVE SEEN THE SELECTION BIAS IN RESEARCH. IN RESEARCH WE TEND TO SEE PEOPLE WHO ARE WILLING TO PARTICIPATE, WILLING TO PROVIDE THEIR INFORMATION; TO WHAT EXTENT ARE WE NOT SERVING IMPORTANT SEGMENTS OF THE POPULATION THAT MAY HAVE DIFFERENT PRIVACY RELATED USE? IN TODAY'S DAY AND AGE, THE CLINICAL SERVICES OF TELEHEALTH, YOU CAN IMAGINE CLINICAL OF OCCASIONS WHERE WE CAN TAILOR CLINICAL EXPERIENCES TO PEOPLE A BIT BETTER PERHAPS USING DIGITAL AND TELEHEALTH TECHNOLOGIES. I THINK WE MAY BE ABLE TO USE SUCH A TOOL TO ENHANCE THE RESEARCH PROCESS. YOU CAN IMAGINE SUCH A TOOL TO TAILOR INFORMED CONSENT; BEING ABLE TO DRILL DOWN ON CERTAIN ASPECTS OF THE INFORMED CONSENT THAT SOMEBODY MIGHT BE MORE CONCERNED ABOUT OR CARE ABOUT MORE. ALSO THE CREATION OF DECISION AIDS THAT MIGHT HELP PEOPLE MAKE BETTER DECISIONS THAT ARE MORE CONSISTENT WITH THEIR OWN VALUES WITH RESPECT TO THE EXTENT TO WHICH THEY WANT TO SHARE THEIR INFORMATION. AND FINALLY, IN THE AREA OF TECHNOLOGY DESIGN, I COULD IMAGINE SUCH A TOOL BEING USED TO JUST ENGAGED A USER-CENTERED DESIGN TECHNOLOGIES. --- I'M GOING TO END WITH A COUPLE OF REMARKS AND I LOOK FORWARD TO THE DISCUSSION BUT I THINK HIS WORK IS TO SOME EXTENT I'M GOING BUT WHAT WE'VE LEARNED AND GAINED IS REALLY SOME KNOWLEDGE AND EMPIRICAL SUPPORT FOR THIS DISCREPANCY THAT YOU CAN SEE WHAT PEOPLE. SOMETIMES THEY MAY SAY ONE THING INTO ANOTHER WHEN IT COMES TO PRIVACY RELATED ISSUES. AS SOMEBODY WHO WORKS IN THE FIELD OF PUBLIC HEALTH, I STARTED TO THINK ABOUT TRAVESTY AS A PUBLIC HEALTH ISSUE IN SOME CONTEXT AS WELL. ON THE INDIVIDUAL BEHAVIOR LEVEL, I'VE STARTED TO THINK -- ARE THERE ANALOGIES THAT CAN BE HELPFUL TO US? --- FOR EXAMPLE HE WE THINK ABOUT HEALTH BEHAVIOR LIKE DIET WE HAVE VERY ACCEPTED WAYS TO TALK ABOUT DIET AND WHAT CONSTITUTES A DIET IN OUR SOCIETY. YET, WHAT PEOPLE ACTUALLY DO WITH THEIR DIET VARIES WIDELY AND IN FACT IT IS VERY CONTEXT DEPENDENT. I MIGHT REALLY ENJOY A NICE 2000 CALORIE BOX OF POPCORN WHEN I'M AT THE MOVIES WITH MY KIDS BUT I'M NOT GOING TO DO THAT EVERYDAY. SOME OF THESE ANALOGOUS BEHAVIORS MIGHT GIVE US INSIGHT AS TO HOW WE MIGHT APPROACH POLICY AS WELL. --- AND I THINK THERE'S A LOT OF FUTURE AREAS FOR RESEARCH; THE EXTENT TO WHICH THESE THINGS ARE STABLE OVER TIME, THE EXTENT TO WHICH THERE ARE CULTURAL DIFFERENCES ARE COMING TO PLAY I THINK THAT THOSE ARE IMPORTANT THINGS TO LOOK AT AS WELL. --- I HAVE MANY CO-AUTHORS AND COLLABORATORS; THESE ARE SOME OF THE PAPERS THAT LEVERAGE SOME OF OUR PRIVACY RELATED WORK AND THESE PEOPLE HAVE CONTRIBUTED TO THE WORK I HAVE PRESENTED TODAY AND THANK YOU. >> [APPLAUSE] >> >> >> JODI DANIEL: SO WE CAN OPEN THIS TO A PANEL DISCUSSION FOR SESSION 1 AND AS MANY OF YOU REALIZE WE ARE OVER TIME; THIS SESSION NORMALLY STOPS AT 9:30 AND GOES TO 10:30, I'M GOING TO GIVE IT 5 MINUTES MORE, NOT TO GO LATER THAN 10:40. SO THAT WAS SUCH AN INTERESTED DISCUSSION AND COVERED SO MANY POINTS AND IT IS HARD TO THINK ABOUT WHERE TO START; I'M LOOKING FORWARD TO THE DISCUSSION WITH OUR PANELISTS AND OUR BROADER GROUP. --- I WANT TO START WITH WHERE WE ENDED; CINNAMON WAS TALKING ABOUT HOW TO USE DIFFERENT UNDERSTANDING ABOUT PEOPLE'S PREFERENCES AND PROFILES ABOUT PRIVACY AND THINK ABOUT A FRAMEWORK FOR PROTECTING AND THINKING ABOUT RESEARCH USING THEIR DATA; AND THEN WE ALSO HEARD JAKE TALKING ABOUT HOW FALSE DON'T UNDERSTAND NECESSARILY THE RISKS THAT MAY COME FROM DOWNSTREAM USES OF THE DATA. --- WE WERE TALKING ABOUT PERVASIVE DATA AND PERVASIVE DATA SCIENCE OH I WOULD LIKE TO BRING THOSE TWO TENSIONS TOGETHER AND HAVE A CONVENTION ABOUT THE. HOW DO WE THINK ABOUT PERSONAL PREFERENCES AND USING DATA AND RISK THAT PATIENTS MAY NOT UNDERSTAND WHEN THEY'RE THINKING WHETHER OR NOT TO SHARE DATA OR GIVE HER MISSION FOR YOU TO BE USED IN CERTAIN WAYS AND I WILL START WITH CINNAMOLN AND JAKE AND THEN OPEN IT UP TO OTHERS TO WEIGH IN AS WELL. >> CINNAMON BLOSS: THANKS FOR THE QUESTION. I THINK THAT'S A HUGE CHALLENGE; AND SOMETHING THAT WE THOUGHT A LOT ABOUT IT AS I ALLUDED TO IN MY REMARKS. I THINK THAT IT REALLY SPEAKS TO SOME OF THE LIMITATIONS WITH INFORMED CONSENT IN GENERAL, AS WE CURRENTLY ENGAGE IN IT. I DON'T REALLY HAVE AN EASY ANSWER; AS I MENTIONED, ONE THING WE'VE BEEN THINKING ABOUT IS POTENTIALLY THE USE OF DECISION AIDS IN THAT CAN A SITUATION, SOMETHING THAT HELPS PEOPLE WHO MAY NOT UNDERSTAND THE INTRICATE POLICIES AND THE INTRICATE MOVEMENT OF DATA AND WHAT THE IMPLICATIONS ARE. BUT IF YOU COULD SET UP A DECISION AID ON THE BACKEND THAT LINKS CERTAIN VALUES OR THE PROFILE THAT I MENTIONED THAT TIES IT TO CERTAIN KINDS OF DECISIONS AND OUTCOMES I CAN IMAGINE MAKING SOMETHING LIKE THAT WORK FOR PEOPLE. --- IT DEPENDS ON HOW YOU DEVELOP IT, I THINK. IT WOULD HAVE TO BE DEVELOPED BY TRUSTED ENTITIES AND REALLY WITH THE PARTICIPANTS' BEST INTEREST IN MIND. >> JAKE, ANY THOUGHTS ON THIS? >> JACOB METCALF: I THINK EMPHASIS NEEDS TO BE PLACED ON HAVING COLLECTED PROTECTIONS; WE SHOULD BE RELYING ON INDIVIDUAL DECISION-MAKING, COMING BACK TO THE HEALTHCARE ANALOGY; TO THE EXTENT POSSIBLE WE SHOULD BE SHAPING PRIVACY PROTECTIONS AS A MATTER OF -- IN TERMS OF SHARED HARMS AND SHARED RESPONSIBILITIES NOT JUST IN TERMS OF WHAT IS A PREFERENCE OF THE INDIVIDUAL WHICH OF COURSE IS VERY IMPORTANT. WE HAVE ROLES -- (DID NOT UNDERSTAND) IF YOU WANT ACCESS TO PUBLIC SCHOOLS. --- IF WE ARE GOING TO DEAL WITH THE COLLECTIVE HARMS OF PRIVACY INVADING TECHNOLOGIES THEN WE NEED TO BE THINKING ABOUT MORE COLLECTIVE SOLUTIONS THAT DON'T RELY ON THE PREFERENCES OF INDIVIDUALS PARTICULARLY WHEN IT IS SO HARD -- AND EVEN PROFESSIONALS CANNOT MAKE INFORMED CONSENT IF YOU TAKE "INFORMEDJ" SERIOUSLY. I THINK WE NEED A SHIFT IN TERMS OF HOW WE SEE WHAT THESE RISKS ARE. >> JODI DANIEL: DO YOU WANT TO JUMP IN ON THIS CONVERSATION? >> MARK BARNES: I THINK THIS ASL RELATES TO THE THIRD PANEL; WHEN ONE TALKS ABOUT CONTROLLING THE HARM THAT MIGHT COME FROM THE GREATER RESEARCH AND PROTECTING PRIVACY GENERAL BUT ESPECIALLY WITHIN THE CONTEXT OF DATA RESEARCH THERE IS ALSO JURISDICTIONAL QUESTION; THAT IS THAT WE HAVE A FIRST AMENDMENT AND UNDER THE COMMON RULE WE REGULAR THE RESEARCH NOT BECAUSE WE HAVE THE RIGHT TO RAISE ITS RESEARCH BUT BECAUSE THE FEDERAL GOVERNMENT FUNDS THE RESEARCH AND THAT IS A RESEARCH THAT IS REGULATED. IN THE FDA CONTEXT THE REASON WE REGULATE RESEARCH IS THAT THERE ARE MEDICAL DEVICES AND DRUGS THAT CROSS STATE LINES SO THERE IS JURISDICTION UNDER THE COMMERCE LAWS. SO I GUESS MY QUESTION IS, THERE'S A LIMIT -- AND EVEN IN THE CONTEXT OF REGULAR THE RESEARCH UNDER THE COMMON RULE IT IS NOT UNCOMMON IN CONTENTIOUS SITUATIONS FOR INVESTIGATORS TO SAY, HOW DARE YOU IRB TO REGULATE MY RESEARCH? THIS IS A FIRST RENOVATION. I SHOULD BE ABLE TO SAY AND DO WHAT I WANT WITH MY DATA FOR EXAMPLE. --- HOW DO WE SQUARE THE OPEN USE OF PUBLIC DATA WITH PROTECTING PEOPLE AGAINST HARMS IN A DIFFERENT KIND OF CONTEXT THAT THE THREE OF YOU HAVE SET FORTH? >> JACOB METCALF: OF COURSE -- YOU KNOW -- THE COMMON RULE MANDATES THE USE OF AN IRB WHEN USING FEDERAL FUNDS FOR THE RESEARCH BUT IT DOESN'T MEAN THAT YOU CAN'T USE IRBS IN DIFFERENT CONTEXTS. INSTITUTIONS CAN CONTROL THE USE OF THE RESOURCES; YEAH, A UNIVERSITY CAN'T TELL A RESEARCHER YOU CAN'T SPEAK THIS WAY OR YOU CAN'T DO THIS ON YOUR TIME BUT THEY CAN SAY YOU CAN'T USE THE LAB IN A WAY THAT VIOLATES THE PRIVACY OF INDIVIDUALS WHETHER USING FEDERAL FUNDS ARE NOT. WE NEED TO BE MORE EXPENSIVE IN TERMS OF THE PRIORITIES THAT WE ARE GOING TO EMPHASIZE IN SCIENTIFIC RESEARCH. --- THE BELMONT REPORT HAS THREE CORE PRINCIPLES: BENEFICENT, JUSTICE AND RESPECT FOR PERSON AND THE COMMON RULE IS VERY FOCUSED ON RESPECT FOR PERSONS. BUT BENEFICENT AND JUSTICE STILL MATTER. DOES THIS RESEARCH ADEQUATELY DISTRIBUTE THE -- ACROSS ABOUT THE POPULATION? THE JURISDICTIONAL QUESTIONS ARE REAL BUT I THINK THERE IS MORE (DID NOT UNDERSTAND). >> CINNAMON BLOSS: I WOULD ALSO SAY THAT IN THE CONTEST OF CONSUMER DATA USED FOR THIS RESEARCH -- NOW THAT THIS EVER OVERFLOWS INTO THAT BUT THE CONCEPT THAT A COMPANY CAN TAKE ITS DATA COLLECTED IN THE CONSUMER RELATIONSHIP AND THEN USE IT FOR SOME SECONDARY PURPOSE OR UNEXPECTED PURPOSE FALLS UNDER THE CONSUMER PROTECTION LAW THAT IS FAIRLY WELL-ESTABLISHED. IT'S NOT A HUGE STRETCH TO START TO APPLY IT IN THAT WAY AND SAY UNEXPECTED USES OR UNEXPECTED IMPACTS ON THE INDIVIDUAL WHEN THEY WERE PART OF THE AGREED-UPON EXCHANGE AS AN INITIAL COMMERCIAL RELATIONSHIP AND I DON'T KNOW THAT THERE IS REALLY A FIRST AMENDMENT RIGHT THAT WOULD BE INDICATED SINCE THERE IS NOT EVEN A GOVERNMENT ACT OTHER THAN THE RECORDS ARE AUTHORITY FOR CONSUMER PROTECTION BUT IT'S LIKE WE REGULAT OTHER THINGS TO PROTECT PEOPLE FROM EXPLOITATION INTERRELATIONSHIP, JUST BECAUSE IT IS MY DATA RATHER THAN MY LABOR OR MY SOMETHING ELSE THAT I PROVIDED. I DON'T KNOW THAT MEANS THAT WE CAN'T ACT THAT WAY. WHEN WE ARE TALKING ABOUT NOT IN A FEDERALLY FUNDED OR A STATE UNIVERSITY ACADEMIC SETTING >> JODI DANIEL: IN THE HIPAA CONTEXT WE HAVE SET RULES ABOUT DATA USE IN THEIR BASED ON COLLECTIVE -- SOME RELATIVE UNDERSTANDING OF COLLECTIVE HOMES AND BENEFITS SO WE ASSUMED THAT INFORMATION CAN'T BE SHARED UNLESS THE PATIENT AUTHORIZES IT BUT THERE IS AN IMPORTANT PURPOSE FOR TREATMENT AND RESEARCH, AND AN IMPORTANT PURPOSE FOR PUBLIC HEALTH SO WE HAVE COLLECTIVELY DECIDED THE REGULATION THAT CERTAIN USES OF DATA ARE ALLOWED. AND THE INDIVIDUAL DOES NOT GET TO SAY YES OR NO AND THEY ARE SHARING THE DATA FOR TREATMENT PURPOSE BUT MAYBE IT MIGHT BE USED FOR PUBLIC HEALTH PURPOSES SO WE DO HAVE A FRAMEWORK WHERE WE HAVE BASICALLY MADE A SET OF CHOICES AS A COUNTRY, ABOUT WHAT ARE PERMISSIBLE USES OF DATA AND NOT PERMISSIBLE; THAT REGULARS AND IS 20 YEARS OLD AND IT DID NOT CONTEMPLATE BID DATA AND PERVASIVE DATA WHEN IT WAS DEVELOPED AND THERE MAY BE SOME THINGS THAT ARE NOT AS WELL REPRESENTED IN THE CONCEPT, BUT THERE ARE EXAMPLES WHERE WE HAVE ME COLLECTIVE DECISIONS AS A COUNTRY ABOUT USES OF DATA AND WHAT IS PERMISSIBLE AND WHAT IS NOT AND WHERE THE INDIVIDUAL GETS TO MAKE A DECISION AND WHERE THEY DO NOT. >> MARK BARNES: THAT'S RIGHT, THE HIPAA LEGISLATION IS BASED ON THE ELECTRONIC SUBMISSION OF MEDICAL CLAIMS AND MAKING ONE OR TRANSACTIONS WITH MEDICAL REIMBURSEMENT IS WHAT TRIGGERS THIS AND IT IS BASED ON THE COMMERCE LAW ACROSS STATE LINES. HEALTHCARE IS A PARTICULARLY WELL REGULAR INDUSTRY AND I'M NOT ARGUING IN FAVOR OF THIS BUT I'M SAYING THAT THERE IS A FIRST AMENDMENT RIGHT FOR BUSINESSES; THERE IS SUPREME COURT JURISPRUDENCE IN THIS IN FACT THE FDA JURISDICTION EVEN TO TRY TO SUPPRESS SPEECH BY COMPANIES ABOUT OFF LABEL USES OF THE PRODUCT IS ITSELF UNDER ATTACK AND COULD WELL FALL UNDER THIS CONFIGURATION OF THE SUPREME COURT. SO MY POINT IS THAT -- YES IF THERE ARE CONTRACTS ESTABLISHED BETWEEN THE BUSINESS AND THE CONSUMER AND THE BUSINESS VIOLATES THAT CONTRACT THAT IS A PROBLEM, A CONSUMER FRAUD PROBLEM; BUT WHEN ONE TALKS ABOUT DATA THAT IS TRULY IN THE PUBLIC REALM AND ONE WANTS TO INTERDICT THE USE OF THAT DATA, YES, THERE ARE ETHICAL AND SUPPOSE AND INSTITUTIONS AND COMPANIES CAN VOLUNTARILY APPLY ETHICAL PRINCIPLES. --- I DON'T KNOW WHERE WE GET THE ABILITY TO CHOKE OFF IN ANY KIND OF CONFERENCE IN WAY THE USE OF THE DATA GOING FORWARD IN ALL CONTEXTS. THAT'S MY POINT. IT'S NOT THAT THEY SHOULD NOT BE ETHICAL RULES AND NORMS THAT WE ARE NOT SET UP AS A SOCIETY THAT WAY AND THAT IS MY POINT. >> JODI DANIEL: GO AHEAD PLEASE. I'D LIKE TO GO BACK TO THE LOCAL CONTEXT WITH CINNAMON. WE CONSIDERED THE LOCAL CONTEXT REVIEWS; YOU WOULD THINK ABOUT PRIVACY AND CINNAMON YOU REFERRED TO IT, LOCATION OR CULTURE SPECIFIC, CAN YOU TALK ABOUT WHAT THAT LOOKS LIKE FOR AN IRB? >> CINNAMON BLOSS: I WANT TO THINK ABOUT THAT FOR A MINUTE, IT'S A BROAD QUESTION. I THINK THAT -- WHAT DOES IT LOOK LIKE FOR AN IRB? MAYBE YOU CAN SAY A LITTLE BIT MORE ABOUT WHAT YOU'RE GOING FOR. >> JODI DANIEL: IF WE CONSIDER AS PART OF OUR REVIEW, WHAT IS THE LOCAL CONTEXT OF A RESEARCH SETTING? >> CINNAMON BLOSS: SO YOU'RE SAYING LIKE FOR DIFFERENT CULTURAL GROUPS AND THINGS LIKE THAT. >> JODI DANIEL: AND EVEN WITHIN THE UNITED STATES, DIFFERENT CULTURES HAVE DIFFERENT EXPECTATIONS OF PRIVACY; SO FROM THE IRB PERSPECTIVE LOCAL CONTEXT NOW INCLUDES PRIVACY IS A CONSIDERATION I WOULD AGUE. A LOT OF US A STRUGGLING WITH HOW DO WE KNOW AND UNDERSTAND WHAT THE PRIVACY PETITIONS ARE? GIVEN A SEGMENT OF CULTURE; GOES BACK TO BRENDA'S WORK IN TERMS OF WHAT ARE THE EXPECTATIONS OF WHETHER IT IS A CONSUMER OR A RESEARCH SUBJECT AND SO FORTH IN TERMS OF PRIVACY AND HOW DO WE ENSURE -- THIS IS A BIG QUESTION -- HOW DO WE ENSURE THAT IRBS ARE CONSIDERING THE PRIVACY PIECES ADEQUATELY. >> CINNAMON BLOSS: THAT QUESTION CAN BE APPLIED TO LIKE ALL SORTS OF ISSUES NOT JUST PRIVACY; HOW DO YOU ENSURE THAT RESEARCH IN GENERAL IS CULTURALLY SENSITIVE AND SPECIFIC AND SO ON AND SO FORTH? PERHAPS THIS IS NOT THE ANSWER THAT WE ARE GOING FOR AS A GROUP TODAY BUT PART OF THAT IS THE DIVERSITY AMONG OUR RESEARCH INVESTIGATORS; DIVERSITY AMONG OUR IRB MEMBERS. I MEAN DIVERSITY AROUND A LOT OF DIFFERENT DIMENSIONS. AND I THINK TO ME THAT'S ONE WAY OF ENSURING THAT THE WORK WE DO IS KIND OF CULTURALLY SENSITIVE. --- I THINK WITH PRIVACY IN PARTICULAR WE CAN ALSO PERHAPS AUGMENT THE WAYS IN WHICH WE MAY APPROACH THAT ISSUE THROUGH EMPIRICAL RESEARCH WITH THE UNDERSTANDING THAT MANY INVESTIGATORS HAVE SHOWN PRIVACY ATTITUDES HAVE WAXED AND WANED DEPENDING ON CURRENT EVENTS AND SO ON. BUT MY READ OF THE LITERATURE IS THAT THERE ARE LIKELY MORE GENERAL DIFFERENCES. BUT I THINK WE NEED MORE WORK IN THAT AREA, AND ALSO DIVERSITY AMONG OUR POLICYMAKERS AND IRB MEMBERS. >> JODI DANIEL: I'M GOING TO JUMP IN; I WANT TO PICK UP ON SOMETHING YOU SAID ABOUT PRIVACY IN SOMEBODY BROUGHT THIS UP IN A DISCUSSION, WE USE THE TERM "PRIVACY" A LOT, WHICH MEANS A LOT OF DIFFERENT THINGS TO DIFFERENT PEOPLE. AND I AM WONDERING HOW WE SHOULD BE THINKING, IS PRIVACY TRYING TO -- THE FACT THAT PEOPLE HAVE DIFFERENT VIEWS ABOUT WHAT PRIVACY MEANS CAUSES SOME CHALLENGES -- -- (DID NOT UNDERSTAND) -- I ALSO WANT TO GET THE CONVERSATION ABOUT FRAMEWORKS AND HOW WE SHOULD BE THINKING ABOUT THE ISSUES AND WHAT ARE THE RIGHT FRAMEWORKS WHICH WE STARTED TO TOUCH UPON IN THE PRIOR DISCUSSION ABOUT HOW DO WE ENSURE OR THINK ABOUT PRIVACY AS WELL OF THE BENEFITS THAT CAN COME FROM THE USE OF THE DATA? --- BUT STARTING WITH A CONVERSATION ABOUT THE FACT THAT PRIVACY MEANS DIFFERENT THINGS TO DIFFERENT PEOPLE DOES THAT MAKE IT DIFFICULT FOR US TO THINK ABOUT WHAT IS THE RIGHT ROLE OF PRIVACY AND RESEARCH IN A DATA-DRIVEN WORLD? AND SHOULD WE BE THINKING ABOUT THIS? SHOULD WE BE CHANGING OUR APPROACH OR TERMINOLOGY OR DISCUSSION? GO AHEAD MICHAEL. >> GREAT QUESTION. ONE OF THE CHALLENGES THAT WE FACE IS TO FRAME THE IDEA PRIVACY AROUND THE CONCEPT OF CONTROL, THE ABILITY TO CONTROL MY DATA AND THEN WE FALL INTO THIS CONSENT TRAP WHERE WE RELY TOO MUCH AND CONSENT AS THE LEVER TO ALLOW US TO DO OTHER THINGS WITH RESEARCH AND WHAT DATA. I THINK THAT PART OF WHAT JAKE WAS OUTLINING HAS MOVED DOWN THAT PATH AND THE DATA GETS REMOVED FROM ITS CONTEXT; THE VALUE OF ANY INFORMED CONSENT TO MEET WEEKENDS AS WE MOVE THROUGH THAT KIND OF CHAIN; WHEN WE START THINKING ABOUT THE DATA OR PERVASIVE DATA THAT IS PART OF OUR WHOLE POINT THAT ANY KIND OF CONTEXT IN WHICH CONSENT MAY ORIGINALLY HAVE EXISTED HAS BEEN VASTLY REMOVED. ONCE I HAVE THAT MAGICAL CONSENT I NOT HAVE DECADES OF DOING WHATEVER I WANT IN ANY WHICH WAY I WANT. THAT'S WHEN I STARTED THINKING ABOUT -- IF YOU LOOK AT THE EUROPEAN APPROACH THEY TALK ABOUT MUCH MORE ABOUT DATA PROTECTION AND THINKING MORE ABOUT THE BROADER LIFECYCLE OF DATA. THERE'S NOT JUST AN ETHICAL ISSUE AT THE MOMENT OF COLLECTION BUT THROUGHOUT ITS ENTIRE PROCESS AND I THINK THAT HE WE START BROADENING OUT HOW WE TALK ABOUT PRIVACY TO CAPTURE SOME OF THOSE FACTORS WE MIGHT BE MOVING MORE IN THE DIRECTION OF DEALING WITH SOME OF THE ETHICAL CHALLENGES. >> JODI >> >> IN TERMS OF THE WORDS WE USE, IS "INTENTION." IT LEADS YOU QUICKLY THAN THE PATH TO YOUR POINT MICHAEL; YOU NEED TO BE COGNIZANT THAT INTENTION CHANGES EVERY TIME AND IT LEADS YOU TO A PLACE OF MEETING TO SEQUENTIALLY CONSENTS AND ASK QUESTIONS REPEATEDLY ABOUT DATA USE BECAUSE I THINK THE COMPELLING CONTENT PRESENTED BY CINNAMON, THERE ARE PRIVACY SUBTYPES WHICH MEANS THE INTENTIONS ARE DIFFERENT. THAT IS THE CONTEXT IN WHICH WE FIND OURSELVES DEALING WITH PRIVACY POLICIES INTERNALLY IN OUR CLINICAL RESEARCH, MOST OF WHICH IS UNDER IRB BUT NOT ALWAYS BUT IT DOESN'T CHANGE THE WAY WE THINK ABOUT THIS WITH REGARDS TO WHETHER OR NOT IT IS AN IRB CONTESTER NOT BECAUSE IT IS AN INDIVIDUAL INTENTION WHICH IS A COMPLEX MATTER. --- I WAS VERY COMPELLED ABOUT THE WAY YOU PRESENTED THAT WORK THIS MORNING. >> THIS ASPECT OF THE CONVERSATION STILL PUTS A LOT OF THE FRAMING AND BURDEN ON THE INDIVIDUAL AND OF JAKE MENTIONED EARLIER WE ARE WAY BEYOND THAT POINT WHERE AN INDIVIDUAL CAN UNDERSTAND AND CONTROL THE WHOLE CHAIN OF EVENTS THAT HE WAS DESCRIBING IN THE IDEA THAT EVEN WELL-INTENTIONED COMPANIES. ALTHOUGH THERE ARE HONESTLY INDIVIDUAL ASPECTS AND BEHAVIORAL CHOICES THAT SHOULD BEEN ABLE TO THE EUROPEAN MODEL MENTIONED BEFORE ALSO TAKES A DIFFERENT TACT THAT CONSENT IS NOT EVEN ONE OF THE MORE LIKELY DEFENSES FOR THE USE OF DATA; IT HAS TO BE WITHIN THE LEGITIMATE BUSINESS PURPOSE OR THE PUBLIC INTEREST REQUIREMENT OR LEGISLATIVE GOAL OR WHATEVER. THOSE ARE THE SOCIALLY DEFINED AND AGREED UPON PURPOSES BECAUSE OF THE SCALE WE'RE TALKING ABOUT -- MACHINE LEARNING MODELS -- THERE IS NO INDIVIDUAL CONTROL. THERE ARE PLACES WHERE THAT CAN HAPPEN AND THERE ARE INDIVIDUAL BEHAVIORS THAT HE TO BE RESPECTED BUT AT THE OTHER END OF THE SPECTRUM WE HAVE TO HAVE OTHER CONTROLS AND LIMITS AND ASPECTS OF THAT SYSTEM THAT ARE WAY BEYOND ANY INDIVIDUAL'S ABILITY TO CHOOSE. >> JACOB METCALF: IT'S ALSO RELEVANT TO KEEP IN MIND THAT WHEN WE THINK ABOUT PRIVACY AS AN INDIVIDUAL RESPONSIBILITY TO CONTROL, THAT'S THINKING ABOUT RESEARCH AND SERVICES AND PRODUCTS IN TERMS OF THINGS THAT KNOW -- IN THE WORLD OF RESEARCH THESE ARE THINGS THAT PREDICT AND ACT. THE DIFFERENCE BETWEEN AI AND ML IS THAT ML ACTS IN THE WORLD. BUT THE DECISIONS BASICALLY STATISTICS ABOUT WHAT IT DOES IN THE WORLD. SO CONTROLLING WHO KNOWS WHAT IS VERY HARD BECAUSE OF THAT TECHNICAL STACK; CONTROLLING WHO CAN DO WHAT DO YOU OR WHO CAN DO WHAT TO US IS PERHAPS A MORE ATTRACTIVE SOLUTION. --- IT'S WHERE WE CAN HAVE MORE OF A SOCIAL CONTROL OR SOCIAL CONTRACT THAT IS NOT DEPENDENT ON RESTRICTING RESEARCH OR RESTRICTING SPEECH OR THINGS LIKE THAT. IT'S WHERE WE ARE ABLE TO SAY THESE ARE THE THINGS YOU CAN'T DO TO US BECAUSE THESE ARE THE KINDS OF THINGS WE HAVE DECIDED AGAINST OUR COMMON INTEREST. THAT DOES NOT RELY SO MUCH AN INDIVIDUAL CONTROL OF THE FRONT OF AN IMPENETRABLE, OPAQUE BLACK BOX SYSTEM; IT IS COLLECTIVE CONTROL OF THE OUTPUT. AND I THINK THAT IS ULTIMATELY MORE TRACTABLE. >> THE ONE THING I WANTED TO MAKE A POINT ON IS, MANY TIMES THE FOCUS IN INDIVIDUALS TO BE ABLE TO CONSENT BUT MANY PEOPLE DO NOT EVEN KNOW WHAT THEY ARE CONSENTING TO; THEY HAVE MOBILE PHONES AND SMART PADS AND EVERYWHERE THEY ARE GIVEN DATA AND EVERYBODY IS WATCHING LEARNING IN MANY HAVE -- PROJECT THAT DON'T COME INTO THE REALM OF RESEARCH AND DOES NOT PICK UP CONSENT BUT THEY ARE ALSO GIVING OUR DATA BUT THE MAIN FOCUS IS THAT IT IS INDIVIDUAL'S RESPONSE ABILITY TO CONSENT OR NOT, IS MORE TO THE POLICIES AND NOBODY CAN USE THE DATA AND THAT IS THE QUESTION BECAUSE JUST BECAUSE I CONSENTED OR NOT IS NOT A QUESTION BECAUSE I DON'T EVEN KNOW WHAT I CONSENTED TO. WE NEED TO HAVE RULES THAT CAN CONTROL HOW NOT TO USE THE DATA AND HOW TO USE THE DATA. >> MICHAEL: THESE ARE TWO IMPORTANT POINTS. MAKES ME THINK ABOUT HOW WE CONCEIVE OF HARM AND IN THE PREVIOUS WORLD WE COULD SAY THAT I'M CONSENTING TO THIS BECAUSE I KNOWN GOING TO GET THE SPECIFIC BENEFIT WHICH JAKE WAS SAYING IT'S A KNOWN FACTOR; EITHER THE BENEFIT DID NOT HAPPEN AND THEY BROKE THE CONTRACT SO WE CAN PURSUE SOMEBODY THAT WAY OR THERE WAS A MEANS THAT OF A BENEFIT. JAKE'S POINT IS IMPORTANT, NOW THAT WE ARE TALKING ABOUT PREDICTIVE SYSTEMS WE CANNOT PROTECT A METER, WE DO NOT KNOW WHAT THE SYSTEM CAN DO WITH MY DATA SO I CAN'T EVEN IMAGINE WHAT POSSIBLE HARM THERE MIGHT BE. A LOT OF THE WAY THAT I'VE BEEN THINKING MORE CONCEPTUALLY IS HOW TO THINK ABOUT THE NATURE OF HARM IN THIS NEW KIND OF CONTEXT AND THAT PUTS PRESSURE ON THE FRAMERS THAT IRB HAS BEEN USING; IT'S GOING TO BE AN INTERESTING CHALLENGE FOR US TO WORK THROUGH. >> JODI DANIEL: YOU'VE RAISED THIS POINT AGAIN ABOUT THE FRAMERS, WE'RE TALKING ABOUT CONSENT OF THE CHALLENGES TO CONSENT IN A DATA-DRIVEN WORLD BECAUSE WE DON'T KNOW THE BENEFITS OR THE HARMS THAT MIGHT COME DOWN THE ROAD FROM THE COLLECTION AND USE OF THE DATA MAKING IT VERY DIFFICULT TO HAVE A CONSENT MODEL. I WOULD LIKE TO KNOW FOLK'S THOUGHTS ON THEN WHAT? IS THAT THE RIGHT APPROACH? DO WE NEED TO TWEAK IT OR MODIFY IT TO WORK IN THIS CONTEXT OR SHOULD WE THINK ABOUT DATA STEWARDSHIP OR DATA FIDUCIARY MODEL? OR PROTECTING AGAINST PARTICULAR A HARMS. WHAT ARE THE DIFFERENT OPTIONS AND FRAMEWORKS THAT WE CAN TALK ABOUT AND HOW THOSE MAYOR MAY NOT WORK AS WE HAVE MORE PREDICTIVE SYSTEMS USING THAT DATA? WHO WOULD LIKE TO START? >> ONE OF THE CHALLENGES THAT WE WILL GET INTO IN TE NEXT SESSION IS THE ISSUE THAT NO DATA REALLY ARE DE-IDENTIFIED. WHEN THE HIPAA DEFINITION WAS DRAFTED IT WAS ALMOST OBSOLETE THE WAY WAS DRAFTED BECAUSE OF THE WORK OF VARIOUS FOLKS AT HARVARD AND OTHER PLACES; WE CAN TAKE A DATASET AND IDENTIFY MANY OF THE PEOPLE SO THEN THE QUESTION BECOMES DO WE DISPENSE WITH THE NOTION OF DE-IDENTIFIED DATA? OR THE EXCEPTION FROM THE COMMON RULE AND OTHER RELATIONS INCLUDING FROM THE GRP. WE SHOULD NOT BAN THE RESEARCH (DID NOT UNDERSTAND) --. >> THAT'S A VERY IMPORTANT POINT. DATA IS NOT NECESSARILY THAT. IT MAKES BUSINESSES FLOURISH AND FIND WHAT WORKS AND WHAT DOES NOT WORK. WHEN IT COMES TO PRIVACY AND PERVASIVENESS THAT IS WHAT NEED TO BE LIMITED. >> I KNOW WE'LL GET TO THIS IN THE NEXT SESSION. IT'S REALLY THROUGH THE LENS OF DIFFERENT GROUPS. I WORK WITH VARIOUS REGISTRIES FOUNDATIONS AND SOME OF THEM FEEL STRONGLY THAT THE DATA SHOULD BE SHARED; SOME DO NOT. SOME PEOPLE THINK THAT THE DATA CANNOT BE SHARED BUT MOST THINK THAT IT SHOULD BE THE SPEAKING ABOUT WHAT MARK IS SAYING, I DON'T THINK WE CAN TRULY SAY WHICH DATA CAUSES HARM OR NOT AND STARTING WITH THAT VANTAGE POINTE PERHAPS WOULD BE A MISTAKE AT LEAST FROM MY PERSPECTIVE OF WHERE WE'RE SITTING HERE TODAY. >> MARK I WAS SURPRISED BY WHAT YOU SAID. I DON'T WANT TO SKIP AHEAD TO THE NEXT SESSION; YOU'VE BEEN MAKING STRONG FIRST AMENDMENT CLAIMS AND ANY REGULATION THE BAN THE IDENTIFICATION OF PERSONAL DATA WOULD BE A FIRST AMENDMENT VIOLATION. >> MARK BARNES: I'M NOT SURE YOU COULD BAN THE IDENTIFICATION USING PUBLIC DATA BUT YOU COULD BAN THE DETRIMENTAL USE OF IT. BECAUSE THAT'S NOT A FIRST AMENDMENT ISSUE. >> RIGHT, PARTY A PUBLISHES RE-IDENTIFIED DATA AND THE HARM IS DONE BY OTHER PARTIES. >> JODI DANIEL: I'M STRUGGLING WITH THIS A LITTLE BIT; IS DIFFICULT AS IT IS TO SET ROLES ON THE USE OF DATA AND WHAT IS APPROPRIATE AND WHAT IS NOT, IT SEEMS TO ME THAT IT WOULD BE EQUALLY DIFFICULT TO FIGURE OUT UP FRONT WHAT THE POTENTIAL HARMS ARE AND REGULATE OR RESTRICT THOSE HARD SO I WOULD LOVE SOME THOUGHTS ON HOW -- IF TGHE THOUGHT IS, LET'S PREVENT FOLKS USING DATA IN A HARMFUL WAY AND WE KNOW THAT PEOPLE THINK ABOUT HARM DIFFERENTLY -- FROM CINNAMON'S RESEARCH -- SOME PEOPLE DON'T CARE IF THE DATA IS OUT THERE AND SOME PEOPLE DO. HOW DO WE KNOW WHAT THE HARM IS THAT WE'RE TRYING TO PREVENT SO WE CAN PREVENT THE IMPROPER USE? >> I WAS SURPRISED THAT NOBODY BROUGHT UP THE CONTEXTUAL INTEGRITY MODEL FOR PRIVACY; I THINK IT'S A USEFUL MODEL. IT DOESN'T JUST LOOK AT ISSUES OF CONSENT; IT ALSO LOOKS AT APPROPRIATENESS OF FLOWS OF INFORMATION. IT HAS A FRAMEWORK FOR PERFORMING AN ANALYSIS THAT LOOKS AT THE DATA SUBJECT, THE SENDER, THE RECIPIENT, THE KIND OF INFORMATION, THE INTENDED USES OF INFORMATION. IT'S WELL-DEVELOPED AND IT PROVIDES MORE FRAMEWORK FOR THINKING ABOUT THESE ISSUES. --- SO ONE OF THE PROBLEMS THAT WE ENCOUNTER IS THIS LACK OF VISIBILITY AND TRANSPARENCY AND WHAT THE DATA IS USED FOR. AS REGULATORS, THAT IS SOMETHING THAT YOU COULD ENFORCE AT A MUCH BROADER LEVEL; IF YOU SAY IT IS A DATA STEWARDSHIP MODEL, MAKING THE DATA SETS AVAILABLE -- META DATA -- SAYING WHERE IT WAS COLLECTED AND WHAT THE INTENDED USES WERE, GETTING CONCEPT OF THE ENTIRE RECORDS, THE NUMBER OF ROWS AND COLUMNS. IT'S A BIG DIFFERENCE IF SOMEONE SAYS WE COLLECTED DATA FROM INDIVIDUALS OR WE HAVE COLLECTED 100 RECORDS FROM EACH INDIVIDUAL AND EACH OF THOSE HAS 1000 DATA ELEMENTS. THAT KIND OF TACIT INFORMATION -- MAKING THAT INFORMATION VISIBLE WOULD BE A GOOD STEP. THERE IS A LACK OF TRANSPARENCY RIGHT NOW IN THESE DATA FLOWS; AS THE PUBLIC, WE DO NOT KNOW WHAT CORPORATIONS ARE SHARING WITH EACH OTHER EVEN WHEN IT IS HEALTH-RELATED DATA. MAKING THAT VISIBLE WOULD BE A GOOD FIRST STEP. >> JODI DANIEL: JAKE OR MICHALE, I'D LIKE YOUR THOUGHTS ABOUT MAKING THE DATA VISIBLE; YOU WERE SUGGESTING THAT AS WE GET FURTHER AWAY FROM THE DATA SOURCE, WE LOSE THAT CONTEXT. I'D LOVE YOUR THOUGHTS ON THIS TOPIC AS WELL. >> JACOB METCALF: TO SOME EXTENT THE LOSS OF CONTEXT -- IT'S A FEATURE OF THE TYPICAL SYSTEMS. THEY WERE NOT BUILT TO RETAIN THE METADATA. I TALK TO PEOPLE, THE PLOT FROM SEVERE USING DO NOT REPORT BACK TO THEM WHETHER THEY ARE USING PII OR NOT IN THEIR MODEL; IT'S JUST WHEN THE DATA SET WAS BUILT. WHEN THE DATA WAS COLLECTED FIVE YEARS AGO IT WAS NOT LABELED WITH METADATA; IT WENT TO THE DATA LAKE AND NOW THEY DON'T KNOW WHAT IT IS. A GDPR HAS HELPED SIGNIFICANTLY SIMPLY BECAUSE THE TECHNICAL TOOLS NECESSARY TO COMPLY WITH GDPR CAN BE PORTED OVER TO TOHER USES THAT AREN'T SPECIFICALLY GDPR-RELATED. --- WE ARE STARTING TO SEE MACHINE LEARNING PLATFORMS THAT HAVE THE CAPACITY TO PRESENT THE METADATA TO THE DATA SCIENTISTS IN ORDER T ENABLE A MORE INFORMED DECISION. IN SOME WAYS WE ARE AT A POINT WHERE WE SHOULD BE TAKING THE GOOD INTENT OF DATA SCIENTISTS SERIOUSLY AND MOVE TOWARDS TOOLS THAT ENABLE THAT GOOD INTENT TO SHINE THROUGH. IN A CERTAIN WAY, THE SYSTEMS THEY ARE OPERATING IN DOES NOT MAKE IT EASY TO MAKE A GOOD DECISION. THE EASE OF INFRASTRUCTURE IS AN IMPORTANT FEATURE THAT I'VE NEGLECTED, ALLOWING THESE INFRASTRUCTURES TO MAKE IT HARD TO MAKE DECISIONS TO SERVE THE CORE TOOLS OF THE INDUSTRY. >> JODI DANIEL: SO IF WE HAVE DIFFERENT RESPONSIBILITIES FOR COMMUNICATING THAT OR MAINTAINING THE METADATA OR HAVING SOME RESPONSIBILITY FOR UNDERSTANDING THE CONTEXT AND INTENT OF THE DATA USED WITH THAT HELP US TO PROGESS AS WE'RE THINKING ABOUT PRIVACY AND DATA AND USING DATA FOR RESEARCH? IF WE HAD THE ROLES IN PLACE, WOULD IT HELP TO CHANGE THE TECHNOLOGY TO ENABLE US TO COMPLY WITH THAT APPROACH? >> JODI -- B >> JACOB METCALF: WHEN WE TALK ABOUT THESE CONCEPTUAL APPROACHES IT IS THE MOST USEFUL FRAMEWORK FOR CONSIDERING THE INFORMATIONAL HARM PRESENT IN THIS. WE THINK ABOUT THESE FRAMEWORKS -- YOU'RE GOING TO HAVE TO IMPLEMENT THEM WITH INTEGRAL SYSTEMS IN SOME WAY. YOU ARE GOING TO HAVE TO FIGURE OUT HOW DO WE MAKE IT EASY TO MAKE A GOOD DECISION AND HARD TO MAKE A BAD DECISION; RIGHT NOW IT IS THE OPPOSITE OF THAT. AND WHILE WE MIGHT TALK IN TERMS OF BANNING, WE MIGHT ALSO TALK IN TERMS OF CARROTS AND STICKS WHERE WE FACILITATE THE BEST INTENSIVELY BY MAKING THE INFORMATION NECESSARY TO MAKE A REVERSE DECISION PRESENT. --- ONE WAY RULEMAKING COULD HELP IS TO SAY THAT ALL FEATURES MUST HAVE TROUBLE WITH METADATA THAT SAY PI OR NOT PII. (DID NOT UNDERSTAND). (DID NOT UNDERSTAND) >> JODI DANIEL: WHAT HAPPENS WHEN THE DATA IS RE-IDENTIFIED. PEOPLE REALIZE IT'S NOT PII. >> THAT REPRESENTS FAILURE TO UNDERSTAND THAT DE-IDENTIFICATION DOES NOT WORK. (LAUGHTER) -- AND WE KNOW THAT ANONIMIZATION IS A PROCESS THAT DOES NOT PRODUCE ANONIMIZED DATA. IT IS INSTEAD A "WISH" PROCESS. IF IT PRODUCED ANONIMIZED DATA IT WOULD BE A WISH FULFILLMENT. >> THIS IDEA OF HAVING METADATA IS CRITICAL AND HE WOULD COULD MARRY THE COMMENTS OF JACOB AND SIMSON, THAT WOULD INCLUDE THE CONDITIONAL FACTORS UNDER WHICH IT IS COLLECTED; AND ETHICAL CHECK ON THE DATA COLLECTION AND BUILD THAT PROVENANCE (SOUNDS LIKE) ON THESE DATASETS. AND HELP IDENTIFY HOW THESE DIFFERENT ISSUES COME INTO PLAY. >> CINNAMON BLOSS: WE'VE BEEN TALKING ABOUT THE DATA AND HOW THE DATA IS THE FOCUS OF POLICIES AND RULES; SOMETHING I'VE BEEN THINKING ABOUT, ONE RELATED TO THE IDEA THAT DE-IDENTIFICATION DOES NOT WORK IN PICKING UP ON JAKE'S THREAD, SOME OF THE REFERENCES TO DATA SCIENTISTS THEMSELVES. ONE THING I'VE BEEN THINKING ABOUT IS THAT WE CAN AUGMENT THE POLICIES FOCUS ON THE DATA WITH THINKING ABOUT HOW DO WE TRAIN DATA SCIENTISTS. I'M A CLINICAL PSYCHOLOGIST BY TRAINING; SOME OF YOU MAY HAVE TRAINED IN MEDICINE AND FROM DAY ONE IT'S A PROCESS OF HOW WE'RE SOCIALIZED TO FIRST DO NO HARM. RIGHT? IT'S A DIFFERENT CONTEXT, DATA SCIENCE, BUT I THINK WE WCOULD CONSIDER CREATIVE WAYS OF GUARDING AGAINST WHAT I SEE AS DE-PERSONALIZATION OF PERSONAL DATA. THE PEOPLE WHO WORK WITH IT TEND DO DE-PERSONALIZE IT. ONE STRATEGY IS, HOW DO WE TRAINED SCIENTISTS TO HAVE FIRST INFORMS THE IDEA THAT THERE IS A PERSON BEHIND A DATA POINT? --- WE CAN'T PERFECTLY PUT RULES AROUND THE DATA AND SOME LEVEL YOU HAVE TO KIND OF TRAIN THE PEOPLE WHO ARE WORKING WITH IT TO MAKE IT POSITIONS AND BE ABLE TO ACT ON THEIR INTENTION SO I WOULD ENCOURAGE US TO THINK ABOUT THAT. >> CINNAMOLN I CAN'T TELL YOU HOW MUCH I TOTALLY SUPPORT THAT; ONE OF THE AREAS WHERE WE HAVE A LACK OF RESEARCH -- DATA SCIENTISTS ARE NOT OUT NECESSARILY TO DO BAD THINGS BUT HOW DO WE TRANSLATE OUR STANDARD REQUIREMENTS FOR TRAINING AND EDUCATION AND PROTECTIONS FOR RESEARCHERS? AND WE HAVE MOVED INTO A NEW WORLD BUT WE HAVEN'T BUILT THE RESOURCES AROUND THAT TO ENABLE YOU TO DO IT IN A RESPONSIBLE WAY AND THAT INCLUDES MAKING SURE THAT WE HAVE THAT TRAINING AND EDUCATION ABOUT RESPONSIBLE AND ETHICAL USE, STARTING WITH THE INDIVIDUAL GATEKEEPER WHO IS RESPONSIBLE FOR USING THAT DATA AND NOT JUST EXPECTING THE KIND OF HARD HAND OF THE IRB TO COME DOWN AND START CONTROLS RUN CREATE A FRAMEWORK WHERE WE ARE ALL DOING THIS IN A RESPONSIBLE WAY. SO THINKING ABOUT THAT MOVES US FORWARD IN A MUCH BETTER DIRECTION TO ENABLE THIS RESEARCH TO HAPPEN IN AN ETHICAL MANNER. >> I THINK THIS SPEAKS -- I AGREE WITH YOU TOLO CINNAMON ENTIRELY -- IT SPEAKS TO THE DIFFERENT JURISDICTION OF LONG CULTURE AND JUST BECAUSE WE CAN'T DO SOMETHING LEGALLY DOESN'T MEAN THAT WE CAN'T DO SOMETHING CULTURALLY AND THAT IS A FUNDAMENTAL CHALLENGE. SOMETHING MAY BE MORE INDUCTIVE AND TAKES LONGER BUT IT IS MORE LONGER-LASTING THAN A LEGAL SOLUTION. AND THE OTHER THING I WANTED TO POINT OUT -- IN RESPONSE TO SOMETHING YOU WERE SAYING CINNAMON -- THE WEIRD THING ABOUT ALGORITHMS BEING APPLIED -- NOT JUST THE DATA -- SO I'VE OFTEN WONDERED AS I'VE HEARD THESE DEBATES. WHAT CONCERNS IS MOST? IS IT THAT EYE DO IS MAKE DECISION-MAKING HAS CORRECT THAT COMES? IS IT THAT IT HAS INCORRECT OUTCOMES? IS IT THAT ALGORITHMIC DECISION-MAKING IS A LOT AT ALL? THOSE ARE QUESTIONS THAT NEED TO BE ANSWERED AT SOME POINT. >> JODI DANIEL: WE HAVE A QUESTION FROM SOMEONE LISTENING ON THE WEBCAST AND I THINK THIS GOES TO THE QUESTION ABOUT CULTURE. THE QUESTION IS, COULD WE BE OVERESTIMATING THE RISK OF POTENTIAL HARM IN ANALYZING THE DATA IN A GENERATIONAL CULTURE WHERE PEOPLE PUT ANYTHING OUT IN THE OPEN? WHEN IS IT TOO MUCH INFORMATION? A WE OVERESTIMATING THE RISK AND IS THERE A CULTURAL SHIFT ABOUT THE CONCERNS OF PRIVACY FROM FOLKS WHO HAVE GROWN UP WITH DIGITAL TECHNOLOGY? >> I'D LIKE TO RESPOND TO THAT. IT INCORPORATES A COMMON FALLACY; PEOPLE WHO PUT THINGS ON THE INTERNET GENERALLY DO NOT PUT EVERYTHING ON THE INTERNET. THEY CONTROL WHAT THEY PUT IN WHAT THEY DON'T PUT; IT IS VERY RARE FOR PEOPLE TO PUT EVERYTHING ON THE INTERNET. SO RARE THAT THEY ARE PERSONALITIES BECAUSE OF THAT. --- ALSO, WHEN PEOPLE PUT THINGS "ON THE INTERNET," THEY'RE TAKING ADVANTAGE OF A SPECIFIC PLATFORM BECAUSE THEY USUALLY ARE NOT THEIR OWN WEB DEVELOPERS SO THEY ASSUME THE PLATFORM HAS CERTAIN CONTROLS IN TERMS OF WHO SEES THE CONTENT AND WHO DOESN'T SEE THE CONTENT. THEY MIGHT BE HAVING A DISCUSSION WITH A FRIEND AND NOT BE AWARE THAT THERE'S 200, OR 2000 PEOPLE VIEWING THE DISCUSSION OR THE DISCUSSION IS BEING ARCHIVED. --- IF YOU'RE HAVING A CHAT WITH SOMEBODY ON THE PHONE WITH TEXT MESSAGES AND THIS SCREENSHOT THE PHONE AND EMAIL THAT AROUND, YOU MIGHT THINK THAT THAT IS YOUR FAULT; ANYTHING PUT ON THE PHONE CAN BE SCREENSHOT. THE ISSUE AGAIN IS THAT THESE SYSTEMS ARE VERY COMPLEX AND WE CANNOT SEE OR CONTROL THE INFORMATION AND THEY WORK IN A WAY THAT IS NOT CONSISTENT WITH THE USER EXPERIENCE, WHERE THE USER SPIRITS AS PRESENTING THAT. AND THERE'S A ROLE THERE FOR EDUCATION IN REGULATION BUT ALSO THE IMPORTANCE OF DEVELOPING NEW SYSTEMS. >> BRENDA LEONG: THE QUESTION IS ALSO REFERRING ABOUT WHAT A PERSON MAY EXPERIENCE AND THE WHOLE POINT OF WHAT I WAS TALKING ABOUT IS ALL ABOUT THE LEFT SIDE OF THE DIAGRAM WITH THE DATA COLLECTION MODEL BUT NOT REALLY ADDRESSING THAT LEVERAGING OF THIS SYSTEM DEVELOP AND TO INFLUENCE THE FUTURE DEVELOPING WITH PEOPLE WHO WERE NOT PART OF THE PROCESS AND MOST PEOPLE WOULD AGREE THAT THEY WOULD PREFER NOT TO BE INFLUENCED WHETHER IT IS VOTING OR SHOPPING OR WHATEVER ELSE IT MIGHT BE IN A WAY THAT IS COMPLETELY OPAQUE TO THEM AND THAT IS IN FACT MANIPULATIVE OR EXPLOITIVE OF INFORMATION IN AN IMBALANCE THEY ARE NOT AWARE OF AND NOT REALLY PARTICIPANTS IT OR HAD AN OPPORTUNITY TO CONSENT ABOUT. SO I DON'T THINK WE'RE OVERESTIMATING THAT KIND OF HARM; THOSE ARE ASPECTS THAT WE'RE TALKING ABOUT IN SOME WAYS. >> I ALWAYS THOUGHT DIFFERENT THAN PUBLIC EDUCATION WHERE WE TEACH PEOPLE GRADES K-12 AND THEN LEARN TO TEACH THE PEOPLE BETTER SO WE MANIPULATE THE PEOPLE OF THAT SYSTEM K-12 SO THEY LEARN BETTER; IS AND THAT THE PRECISE PROCESS THAT YOU DESCRIBE WHERE WE COLLECT DATA FROM A LOT OF PEOPLE AND THEN WE BUILT MODELS AND USE THOSE MODELS TO MANIPULATE PEOPLE AND PUSH THEM AND NUDGE THEM IN WAYS THAT WE WANT TO AS A SOCIETY, AND RECALL THAT THE EDUCATIONAL SYSTEM? >> JACOB METCALF: IT TURNS OUT THAT THESE ADDITIONAL SYSTEM IS SUBJECT TO DMOCRATIC CONTROL WHERE THE POINT OF MANIPULATION -- WHAT IS THE PHILOSOPHY AND INTENT BEHIND THE? IS THAT TO AT LEAST SOME EXTENT BE ACCOUNTABLE TO THE PEOPLE WHO ARE BEING MANIPULATED? THAT'S THE DIFFERENCE BETWEEN INFLUENCE MANIPULATION. I WOULD NOT SAY SCHOOLS MANIPULATE -- UNLESS I'M CYNICAL -- BUT I'D SAY IT'S MORE ACCURATE THAT THE INFLUENCE. AT LEAST THERE'S A CHANCE FOR ARGUMENT AND OBJECTION AND A CHANCE -- IF YOU THINK THE PUBLIC SCHOOL SYSTEM IS MANIPULATING YOU, YOU CAN GO TO A PRIVATE SCHOOL OR HOMESCHOOL. --- IT'S A QUESTION OF VALUES. IT'S A QUESTION OF DO WE ACT AS IF WE ARE LIVING IN A SOCIETY WHERE INDIVIDUALS ARE FREE TO CHOOSE? AND IF YOU THINK PEOPLE SHOULD NOT BE FREE TO CHOOSE YOU MANIPULATE AND IF YOU THINK THEY ARE FREE TO CHOOSE YOU INFLUENCE. >> ONE THING I WAS THINKGING ABOUT, TAPPING BACK TO MARK'S COMMENT, SCIENCE RACES AHEAD OF THE FRAMEWORKS THAT MANIPULATE US. HERE WE ARE WITH AI AND MACHINE LEARNING AND WE'RE JUST CATCHING UP AND WE'RE NOT SURE ABOUT THE RAMIFICATIONS AND THIS IS THE PERFECT FORM TO START TO THINK ABOUT THE RAMIFICATIONS AND FALLOUT AT I WAS COMPELLED BY YOUR TALK JACOB; I HADN'T THOUGHT ABOUT THAT CAMBRIDGE ANALYTICA CASE TOO MUCH, BUT COMPELLED WITH THE THOUGHT THAT MAYBE FRIENDS CAN ACTUALLY IMPACT -- THEIR ACTIONS CAN IMPACT MY DATA WAS REALLY COMPELLING. SO MAYBE THIS IS THE RIGHT FORUM, AND OTHER FORMS LIKE THIS TO START TO TALK ABOUT FRAMEWORKS. >> FIRST OF ALL, THERE ARE PEOPLE WHO SHARE TOO MUCH; THERE ARE PEOPLE WHO DON'T EVEN KNOW WHAT THEY'RE SHARING AND CHECK INTO A LOCATION, SO WHEN YOU PUT AI BEHIND, YOU WILL KNOW A LOT MORE ABOUT THIS PATIENT EVEN THOUGH HE JUST POSTED A PICTURE SO IN TERMS OF SHARING INFORMATION IS OUT THERE AND NO ONE CAN CONTROL THAT BUT THE AI BEHIND IT, DO I NEED TO HAVE RELATIONS BEHIND THE? OR WHAT ARE THEY GOING TO DO WITH HIS ANALYSIS? LET'S SAY THEY DO IT A SEARCH FOR A NIGHT AND THEY WANT TO SHOP, AND THEY GET THESE KINDS OF ADS. IS THAT BAD? SO IT COMES TO AFTER THE ANALYSIS WHAT IS GOING TO HAPPEN THAT IS GOING TO DEFINE HOW PERVASIVE AND BAD THAT IS; THAT'S THE QUESTION WHICH COMES DOWN TO A MORE GRAY AREA FOR YOU TO FIND THE COMPONENT BUT IN TERMS OF DATA OUT THERE IT IS NOTABLE FOR COMPANIES TO DEFINE HOW THE WILL APPROACH YOUR BUSINESS AND WHAT THEY SHOULD FOCUS ON AND BUT THINGS SHOULD NOT SELL AND THAT KIND OF THING IS NOT BAD BUT PEOPLE WILL DIFFER IN THEIR OPINIONS; I DON'T WANT THE TO KNOW WHAT I LIKE. >> BRENDA LEONG: COLLECTING THE DATA AND KNOW THOSE STUDENTS -- WE HAVE COMPANIES COMING INTO THE EDUCATION SYSTEM WHO ARE IN FACT ELECTING THE SYSTEM ON THOUSANDS OF SIMILAR CHILDREN, CORRELATING AND IMPLEMENTING PATTERN RECOGNITION AND CREATING PREDICTIVE OUTPUTS TO SAY TO THE TEACHERS HERE IS NEW TOOL. SO RATHER THAN COLLECTING THE DATA YOURSELF YOU HAVE A TOOL THE COMMUNITY CHARACTERIZED PROFILE AND GET A READOUT ABOUT THE BEST STRATEGY TO TEACH THIS CHILD BASED ON THE DATA MODEL OUTPUT AND THERE ARE PEOPLE WHO ARE PUSHING BACK AGAINST THAT, SAY THAT IS A DANGEROUS THING. AND THERE ARE PEOPLE WHO THINK THAT COULD BE A LOT BETTER BECAUSE WE COULD VERY QUICKLY UNDERSTAND STRENGTHS AND WEAKNESSES AND WHAT TYPE OF EXERCISES OR INTERACTIVE OR SINGLE PERSON PROJECT THEY WOULD LEARN BEST FROM. AND CAPITALIZE ON THAT. THERE'S A GREY AREA, ARE WE MANIPULATED STUDENTS TO A ACHIEVE A PARTICULAR OUTCOME VERSUS TEACHING IN THE TRADITIONAL WAY? THERE IS A LOT OF CONTENTION ABOUT WHETHER THAT IS OKAY OR NOT. >> JODI DANIEL: THE SAME ANALOGY CAN BE USED IN HEALTHCARE; AGAIN THE DOCTOR HAS THE BENEFIT OF INFORMATION ABOUT POPULATIONS WAY BEYOND THEIR SCOPE OF VIEW WHICH IN SOME WAYS HAS A HUGE BENEFIT OF PROVIDING THEM MORE TOOLS AND INFORMATION TO HELP MAKE THE BEST DECISIONS FOR THEIR PATIENTS BUT IF YOU HAVE MODELS THAT ARE NOT REFLECTIVE OF THAT POPULATION OR INDIVIDUAL THEY HAVE IN FRONT OF THEM BECAUSE IT CAUSES HARM TO THE INDIVIDUAL AND IT LEADS TO CHALLENGES AND WORST RESULTS IN SOME CASES. WE ARE COMING TO THE AND OF OUR TIME BUT WE TEED UP A LOT OF & INTERESTING ISSUES WITH THE REST OF THE DAY AND I LOOK FORWARD TO HEAR FROM THE OTHER PANELISTS TO TAKE THE DISCUSSION FOR THIS MORNING AND BRING THAT INTO SOME OF THESE QUESTIONS AS WE DIVE MORE DEEPLY. WE HEARD ABOUT TRAINING AND CULTURAL NORMS; MAYBE CONSENT SHOULD NOT BE THE HOLY GRAIL, AND WE SHOULD BE THINKING ABOUT HARM AND CONTEXTUAL FRAMEWORK SHOULD WORK BETTER AND FOCUSING ON THE FACT THAT THERE ARE VERY DIFFERENT CULTURAL EXPECTATIONS AND PREFERENCES OF INDIVIDUALS WHEN WE TALKED ABOUT DATA THAT WE HAVE TO THINK ABOUT AS WE ARE TRYING TO ATTACK THIS VERY CHALLENGING AND INTERESTING ISSUE. SO THANK YOU VERY MUCH FOR ALL OF YOUR CONTRIBUTIONS AND ATTENTION. >> [APPLAUSE] >> WE'VE COME TO THE END OF THE FIRST SESSION. I'M GOING TO REALLY SOMEBODY FOR A COFFEE BREAK AND I'M TRYING TO GO BACK TO THE TIME FRAME SO WE WILL START AGAIN AT 10:55. THANK YOU EVERYBODY. MARK BARNES IS OUR MODERATOR FOR SESSION II. >> MARK BARNES: THANK YOU YVONNE AND EVERYONE IN THE PANEL SESSION AND MY NAME IS MARK BARNES WITH ROPES & GRAY, LLP. WE WILL TALK ABOUT DIFFERENT PRACTICAL APPROACHES TO PROTECTING OUR PRIVACY IN CONFIDENTIALITY AND BIG DATA RESEARCH ESPECIALLY INVOLVING DATA RESEARCH AND WE HAVE FOUR FOLKS, AT I WILL INTRODUCE THEM BRIEFLY. >> DEBORAH KILPATRICK, THE CEO OF EVIDATION HEALTH AND SHE'S HAD MULTIPLE ROLES IN HER CAREER IN R&D AND VARIOUS LIFE SCIENCES. ANDREW SHATTO IS THE DEPUTY DIRECTOR OF THE CENTERS FOR MEDICARE AND MEDICAID SERVICES AND HIS OFFICE INCLUDES A LOT OF THE CORE RESEARCH SERVICES FOR CMS, A VAST FOR FOSTER OF THINGS DATA AND OTHER DATA. --- REBECCA LI IS THE EXECUTIVE DIRECTOR OF VIVLI, A NOT-FOR-PROFIT INTERNATIONAL NGO, PROVIDING A DATA PUFFIN FOR DATA CONTRIBUTORS AND USERS TO SHARE MASSES OF DE-IDENTIFIED OR ANONYMIZED DATA, IN ORDER TO MAKE IT A RESEARCH BETTER AND FASTER AND MEDICAL TRIALS MORE FOCUSED. SHE'S A FORMER DIRECTOR OF TH MULTIREGIONAL TRIAL CENTER OF HARVARD. WE GO WAY BACK. SIMSON GARFINKEL IS A MASTER OF THESE ISSUES, MUCH BETTER THAN I DO. I ONLY UNDERSTAND THE DIM OUTLINES OF THE OPPORTUNIST PERFECTLY SO SIMSON THANK YOU FOR BEING HERE. I'M GOING TO BE VERSUS WITH REGARDS TO TIME; AND THAT'S BECAUSE SOMEONE HAS BEEN MERCILESS TO ME. >> THANK YOU MARK. THAN K YOU TO MISTY AND YVONNE FOR INVITING US AND AM GOING TO SPEAK THERE SPECIFICALLY ABOUT CLINICAL RESEARCH ACROSS THE US AND WE WERE PRIMARILY WITH LARGE GLOBAL IN PHARMACEUTICAL MANUFACTURERS AS WELL AS MED-TECH COMPANIES, AND WE ARE WORKING A LOT WITH THE UNIVERSE OF COMPANIES INVOLVED IN WEARABLES AND CONNECTED DEVICES AND THE PROBLEM WE WANT TO SOLVE -- HEALTH OUTCOMES USING LIMITED DATA SETS NOT THE FAULT OF ANYONE BUT THE FAULT OF THE SYSTEM IN THE SENSE THAT WE DID NOT HAVE ACCESS TO DATA FROM THE UBIQUITOUS REST OF OUR LIVES AND WE DO NOW. AND THAT DATA CAN BE ACCESSED UNDER INFORMED CONSENT AND BE USED IN CLINICAL RESEARCH FOR THE CONTEXT OF MEASURING HEALTH AND DISEASE VERY, VERY DIFFERENTLY. SO THE WAY I WOULD LIKE TO LOOK AT IT IN THE ONE SNAPSHOT, WE NEED TO SEE PATIENTS LIKE THIS IN THE DIGITAL AREA. YES, THE DHR DATA IS CRITICALLY IMPORTANT AND YES, THE CLAIMS DATA IS CRITICALLY IMPORTANT BUT THERE IS DATA FROM OUR DAILY LIVES THAT CHARACTERIZES WHO WE ARE IN THE WORLD AND HOW OUR STATE AND -- -- AND WITH INDIVIDUAL PERMISSION AND CONTROLS FOR THE PURPOSE OF USING IT IN CLINICAL RESEARCH SO I'VE BEEN ASKED TODAY TO TALK SPECIFICALLY ABOUT OUR APPROACH TODAY. --- BEFORE I GET INTO THE DIDACTIC ASPECTS OF THE WAY WE THINK ABOUT IT, I THOUGHT IT WOULD BE USEFUL TO TALK ABOUT WHAT WE ARE TALKING ABOUT HERE. THIS IS SORT OF WHAT IT LOOKS LIKE IF YOU PUT IT IN A GRAPHICAL FORM, WE CALL IT A "BEHAVIORGRAM EVIDATION." THIS IS A SNAPSHOT OF 24-HRS. OF THE STUDY OF AN INDIVIDUAL, LOOKING AT THE ABILITY FROM THE DATA STREAMS OF OUR DATA LIFE TO DISTINGUISH AND SEGMENTATIONS WITH DIAGNOSED ALZHEIMER'S VERSUS THOSE WITH MILD COGNITIVE IMPAIRMENT AND THIS IS VERY IMPORTANT TO POTENTIALLY STRATIFY PATIENTS WITH EXISTING DRUGS THAT HAVE OR HABS FAILED STAGE 3 BECAUSE OF THE HETEROGENEITY OF THE EXAMPLE. AND IT'S ABILITY TO HAVE AN OBJECTIVE SET OF MEASURABLE INFORMATION THAT IS READILY SCALABLE ACROSS POPULATIONS AND CAN BE ACCESSED USING INFORMED CONSENT AND INDIVIDUAL CONTROLS FOR THE PURPOSES OF IDENTIFYING WHO MIGHT BE A BETTER FIT FOR THE TRIAL IS QUITE IMPORTANT SPECIFICALLY IN ALZHEIMER'S AND IN OTHER CONDITIONS WHERE WE ARE STRUGGLING AS AN ECOSYSTEM TO FIND APPROPRIATE CLINICAL DEVELOPMENT PATHWAYS WITH CERTAIN THERAPEUTICS. --- I'D LIKE TO GIVE A VERY HIGH LEVEL EXAMPLE OF THE KINDS OF QUESTIONS THAT WE ARE LOOKING AT IN EVIDATION. I'M GIVING YOU THE EVERYDAY LANGUAGE BUT THERE ARE MANY WAYS TO CONSTRUCT A HYPOTHESIS GENERATED PROSPECTIVE STUDIES IN EACH OF THESE STUDIES; THESE ARE NOT HYPOTHETICAL STUDIES, WE ARE DOING WORK IN ALL OF THESE AREAS BUT MY POINT HERE IS THAT WE CAN CONSTRUCT VERY RIGOROUS CLINICAL TRIALS BUT BY USING DATA THAT CAN BE LIFTED UP IN A VERY PASSIVE AND CONTINUES WAY WITH INDIVIDUAL CONTROLS TO LOOK AT THINGS IN A VERY DIFFERENT WAY. WE THINK THIS IS QUITE COMPELLING BECAUSE IT ALLOWS YOU TO HAVE MORE DIVERSE PARTICIPANTS BECAUSE YOU ARE NOT REQUIRING THEM IN MANY CASES TO INTERSECT WITH BRICK-AND-MORTAR WALLS. YOU CAN ALLOW YOURSELF TO HAVE DIFFERENT TYPES OF POPULATIONS, UNDER LONGER PERIODS OF TIME BECAUSE IT IS LESS OF A BURDEN ON THE PATIENT SO IT OPENS UP ON THE WAY WE CAN THINK OF THE DESIGN SPACE OF CLINICAL RESEARCH. --- THIS IS SOMETHING RAPIDLY GROWING ACROSS THE US AND ON THE LOWER LEFT-HAND I'M SHOWING YOU A ZIP CODE DENSITY MAP OF THE INDIVIDUALS. IT REPRESENTS 3 OUT 4 OF THE ZIP GOES IN THE US; IF YOU LOOK AT HOW PEOPLE PARTICIPATED SINCE 2016, IT'S NOW OVER A MILLION. WE'RE TALKING ABOUT A SYSTEM WHERE WE CAN LOOK AT 100 PATIENTS AT A TIME OR WE CAN DO FULL RUN OF MY STUDIES OF HUNDREDS OF THOUSANDS; IT'S QUITE POWERFUL IN TERMS OF MOVING CLINICAL RESEARCH SORT OF TO THE NEXT LEVEL. --- GIVEN THAT CONTEXT, I WAS ASKED TO SPEAK ABOUT OUR PHILOSOPHY IS A COMPANY. I WAS ASKED TO BE AS DIDACTIC AS I CAN. I WANT TO BE AS SPECIFIC AS I CAN ABOUT HOW WE THINK ABOUT PROBLEMS AND SITUATIONS TODAY REALIZING THAT THIS IS A VERY RAPIDLY EVOLVING SPACE IN THE CONCEPT OF HOW PEOPLE THINK ABOUT PRIVACY AND CONTROL IS CHANGING AND THEREFORE THE WAY THAT WE DO PROVISIONING WILL ALSO INVOLVE BUT WITH THE STARTING POINT OF THE TRADITIONAL IRB CONSTRUCT WE DO HAVE A VERY STRONG STARTING POINT. SO I'M GOING TO TALK ABOUT EACH OF THESE TOPICS BEGINNING WITH OUR PHILOSOPHY ATT A HIGH LEVEL. --- THESE ARE INDIVIDUALS IN CONTROL OF HOW THE DATA CAN USE, AND THEY FORTIFY THEIR RIGHTS WITH APPROPRIATE LEGAL FRAMEWORKS THAT WILL GUIDE THAT USING AMUSING SPECIFICALLY APPROPRIATE USE VERSUS JUST COLLECTION. PREVENT AND MANAGE ATTEMPTS TO THIS CLINICIAN BASED ON THIS INFORMATION. A LOT OF THE DISCUSSION ON THE FIRST PANEL WAS BASED REALLY AROUND THESE TWO THINGS. I THINK THAT IT'S IMPORTANT TO ACKNOWLEDGE HOW WE THINK ABOUT HEALTH DATA BECAUSE IT IS EVOLVING VERY RAPIDLY. --- IT IS VERY CRITICAL TO GIVE EQUAL ATTENTION TO THE USE OF DATA AT THE POINT OF USE -- NOT JUST A COLLECTION OF DATA AT THE TIME OF COLLECTION AND HOW WE THINK ABOUT THIS IS REALLY GOING TO EVOLVE ESPECIALLY AS WE THINK ABOUT HOW WE GIVE INDIVIDUALS SOME ASPECT OF LONGITUDINAL CONTROL. OU--- XWE ALIGN WITH IPP DEFINITION PRIVACY; WE SUPPORT INDIVIDUAL'S RIGHT TO PRIVACY WHILE BALANCING A FEW THINGS. WE ALLOW THEM CONTROL OF DATA COLLECTION AND DATA USE FOR THE PURPOSES OF RESEARCH; MANY OF THE PEOPLE WE DEAL WITH IN A VERY LONG JOURNEY WITH A CHRONIC DISEASE SO THEY WANT THE EXPERIENCE TO BE VALUABLE TO THEM. WE LIKE TO THINK ABOUT COMPANIES AND RESEARCH INSTITUTIONS THAT NEED TO BE COGNIZANT OF USING THE MINIMUM AMOUNT OF DATA TO ANSWER THIS SPECIFIC QUESTION AND THAT IS DIFFICULT TO DO BUT WE HAVE TO FORCE OURSELVES TO THINK ABOUT THAT AND NEVER PROVIDING A SHARING AN INDIVIDUAL'S DATA WITH ANY OF OUR CLIENTS WITHOUT CONSENT OF THE INDIVIDUAL. --- WE WORK WITH MANY DIFFERENT COMPANIES AND THAT IS SOMETHING WE HAVE TO TAKE HER SERIOUSLY; BUT UNDERLINING THE PRIVACY ISSUE FOR US KEEP IN MIND THAT WE WOULD HAVE DATA ON OUR INDIVIDUALS THAT WAS DIRECTLY PERMISSIONED FOR COLLECTION OF DIRECTORY PROVIDED TO US. AND THEN WE TURN TO CONSENTING CONTROL. WE BELIEVE THAT INDIVIDUALS HAVE THE RIGHT TO USE AND SHARE THE DATA AND NEED TO CONSENT TO UTILIZE OR ENABLE WHETHER OR NOT THE DATA CAN BE USED TO SHARE. OUR CONSENTS ARE SPECIFIC AND INDIVIDUALS MUST MAKE AN ACTIVE CHOICE TO OPT IN; TO BE VERY SPECIFIC WE DO NOT BELIEVE THAT IT IS APPROPRIATE TO JUST HAVE A SINGLE POINT OF COLLECTION AND THEN YOU HAVE TO FIGURE OUT HOW TO OPT OUT OF THAT SYSTEM. THAT IS SOMETHING THAT I THINK IS NOT THE RIGHT WAY TO DO THIS AND WE HAVE TAKING A VERY DIFFERENT TACT. --- WE DO BELIEVE IN SEQUENTIAL OR SERIALIZED CONSENTING IN THE CONTEXT OF PEOPLE BEING IN MULTIPLE PROTOCOLS OVERTIME. IN OTHER WORDS THE SAME DATA SET MAY BE USED IN MULTIPLE DIFFERENT IRB APPROVED STUDIES OVERTIME AND THOSE SHOULD BE CONSENTED SEQUENTIALLY; NO PER-SINGLE USE ON THE FRONT END. OUR CONSENT ALWAYS DEFINE THE SPECIFIC USE AND DISCLOSES IF THE PROJECT IS SPONSORED SO PEOPLE UNDERSTAND AS MUCH AS POSSIBLE THAT THEY ARE IN THE MANUFACTURERS' SPONSOR PROTOCOL VERSUS SOME OF OUR RESEARCH THAT IS DONE FOR EXAMPLE WITH THE NIH OR HHS OR DARPA OR BARNA. ANY INDIVIDUAL CAN WITHDRAW THE CONSENT THAT ANY REASON. --- WE TAKE THIS VERY SERIOUSLY EVEN MORE SO IN THE CONTEXT OF RESEARCH; THERE IS A LOT OF TENDENCY TO BUCKET COMPANIES TOGETHER AND I THINK WE DO NOT NEED TO DO THAT. THERE'S A LOT OF TECHNOLOGY ENABLED COMPANIES WORKING IN THE HEALTHCARE SPACE, AND I HOPE THAT ALL OF US AS A SECTOR ARE HANDLING THIS VERY DIFFERENTLY THAN PERHAPS WHAT COULD'VE BEEN DONE IN THE PAST. OUR CLINICAL PROTOCOLS GO REVIEW BY EXTERNAL IRBS AND WE HAVE EXTERNAL CROSS FUNCTIONAL REVIEW FOR ETHICAL DATA AND CONSIDERATIONS IN OUR. SEVERAL OF YOU IN THE ROOM HAVE BEEN CONSULTED BY OUR IN-HOUSE COUNSEL ON MANY OF THESE ISSUES OVER THE LAST FEW YEARS AS WE MANAGE THROUGH THIS RAPIDLY EVOLVING DATA LANDSCAPE; WE TAKE A ROLE THERE SERIOUSLY TO PUBLICLY ADVOCATE. --- IN 2012 THE FIRST PERSON IN THE UNITED STATES ENACTED TO OUR TECHNOLOGY PLATFORM AND GAVE US PERMISSION TO COLLECT THEIR WEARABLES AND OTHER DIGITAL DATA AND THE REASON WE DID SO IS THAT WE BELIEVED VERY STRONGLY THAT THIS WOULD PROVIDE VERY USEFUL INFORMATION WITHIN A DECADE. THIS DATA IS NOT ENABLING NEW FORMS OF MEASUREMENT THAT WE COULD BARELY IMAGINE THAT THEY CAN GO AND IT CAN BE CONSTRUCTED PASSIVELY CONTINUOUSLY WITH INDIVIDUAL PERMISSION AND IT CAN BE DONE AT A SCALE ACROSS MILLIONS SPANNING MULTIPLE THERAPEUTIC AREAS. THIS RELEASE IS ON THE CONSTRAINTS THAT WE HAVE ON TRIAL DESIGN PARTICULARLY IN CONDITIONS WHERE WE HAVE SAMPLE SIZE ISSUES SO I THINK THIS IS VERY IMPORTANT. --- WE BELIEVE THAT THE PATIENT'S RIGHT TO HEALTH BETA DOES UNDERLIE -- AND DRIVES THE IMMEDIATE NEED TO FORTIFY INDIVIDUAL'S WRITES WITH FRAMEWORKS IN THE LEGAL AND RECKLESS OR MILIEUX THAT GUIDE APPROPRIATE USE OF THE TIME OF USE NOT JUST AT THE POINT OF COLLECTION. THANK YOU. >> [APPLAUSE] >> ANDREW SHATTO: HI, MY NAME IS ANDY SHATTO, THE DEPUTY DIRECTOR OF -- IT'S CALLED -- -- BUT IT'S BEEN REORGANIZE IS NOT WORTH MEMORIZING THAT. WHAT IS IMPORTANT IS WHAT WE DO. OUR OFFICE REGARDLESS OF ME IS RESPONSIBLE FOR RELEASING DATA FOR CMS FOR RESEARCH PURPOSES SO WE DO HAVE A PRIVACY OFFICE AND THE PRIVACY ACT OFFICER; WITHIN THE CISO (SOUNDS LIKE) OFFICE, BECAUSE THEY FOCUS ON PRIVACY OF INTERNAL SYSTEMS AND REVIEWING PIAS AND THAT SORT OF THING. WE WORK WITH THEM FOR SHARING CMS DATA; WE'RE CONSIDERED THE DATA NERDS AT CMS; WE KNOW THE EVOLUTION OF IT, ALL OF OUR DATA IS TO RUN AN INSURANCE PROGRAM AND NOT TO THE RESEARCH SO OUR JOB IS TO CLEAN THAT DATA AND MAKE IT USEFUL FOR RESEARCH OTHERWISE IT WOULD HAVE TO SPEND A LOT OF TIME CLEANING UP THE DATA. PHYSICIANS PROVIDE LOTS OF BILLS FORCING SERVICE IN THEYH MAKE ADJUSTMENTS AND CORRECTIONS; WE ALSO TRY TO BUILD FILES THAT AT A LOT OF VALUE; DRUG EVENTS FOR INSTANCE WHEN PART D STARTED, THE PHYSICIAN COULD PROVIDE A CLAIM WITH THEIR DEA NUMBER OR MPIS, AND THEY COULD USE EITHER. WE PROVIDE OUR OWN PHYSICIAN ID THAT LINKS BACK AND WE PROVIDE CHARACTERISTICS OF THOSE PHYSICIANS; WE PROVIDE FILES AND THAT IS IN A NUTSHELL WHAT OUR OFFICE DOES. FROM A PRIVACY PERSPECTIVE THERE ARE NUMBER OF RELATIONS THAT COME INTO PLAY. --- WEN WE TALK ABOUT RELEASE OF CMS DATA EVERY REGULATION THAT HAS CREATED A CMS PROGRAM COMES WITH ITS OWN RESTRICTIONS; THE PART D PROGRAM HAS ITS OWN RELATIONS AND THE MA HAS ITS OWN, TRADITIONAL A/B MEDICARE SO AS WE MERGE ALL THE DATA TOGETHER WE HAVE TO FIGURE OUT THE POLICIES AROUND ALL THAT SO IF YOU TO REQUEST CMS DATA YOU WILL SEE CERTAIN FILES YOU HAVE TO REQUEST AND YOU HAVE TO JUSTIFY EACH VARIABLE IN THAT DEPENDS ON THE LEGISLATION THAT CREATED THIS PROGRAM. --- NOW, THE OTHER THING I WANT TO MENTION, HIPAA, WE'RE A COVERED ENTITY UNDER HIPAA, NO OTHER ENTITY IS COVERED UNDER HHS, WHICH CREATES A LOT OF CONTENTION. HIPAA ALLOWS A PRIVACY BOARD TO GIVE YOUR WAIVER; WE INSISIT THAT PEOPLE GET THEIR WAIVERS FROM AN IRB. WE DON'T REVIEW AT TAT LEVEL BECAUSE OUR REVIEW STAFF IS CONCENTRATING ON MAKING SURE (DID NOT UNDERSTAND). I WANTED TO MAKE THAT DISTINCTION. --- NOW CMS MAKES TWO TYPES OF DATA. HONESTLY BECAUSE WE ARE COVERED ENTITY UNDER HIPPA, THE RULES ARE PRETTY PRESCRIPTIVE AND WE DEVELOP OUR OWN POLICIES AND PROCEDURES. WITH THIS AUDIENCE IS NEITHER PRODUCTIVE TO EXPLAIN THE RULES OF HIPPA. THERE'S A COPULE OF THINGS THAT PEOPLE GET CONFUSED WITH. LIMITED DATA SETS CANNOT BE CUSTOMIZED IT IS A RANDOM 5% SAMPLE OF THE MEDICARE POPULATION. SO YOU CAN GET DIABETICS; AND OF COURSE THEY DROP OFF THE IDENTIFIERS BUT THE RISKS DOESN'T NECESSARILY MEAN THOSE OF THE DEVICE DON'T EXIST SO A LOT OF THE RESEARCH PRODUCTS HAVE HICK OR MBI (SOUNDS LIKE); WE CREATE OUR OWN INTERNAL UNIQUE IDENTIFIERS AND YOU CAN LINK DIFFERENT PLANE TYPE SENIORS TOGETHER; THE ONLY WAY YOU CAN REQUEST SOMETHING LIKE A HICK IS SHOWING US THAT YOU ARE LINKING TO A DATA SOURCE THAT YOU ALREADY HAVE FOR YOUR RESEARCH. --- THE BIG THING WITH RISK IS THAT THEY ARE CUSTOMIZABLE SOLUTIONS. AND AT THE BOTTOM THE CMS PRIVACY BOARD HAS TO REVIEW ALL THE EDIT THE FILE DATA AND IT IS A QUICKER REVIEW PROCESS BECAUSE AGAIN THEY'RE JUST CANNED PRODUCTS; YOU CANNOT LINK THEM TO ANYTHING, IT IS A BIG PROVISION OF THE LIMITED DATA SET. IT IS IMPOSSIBLE TO LINK THEM TO STUFF YOU MAY WANT TO TRY OR WE WILL BE VERY ANGRY OR GO AFTER YOU. --- UNIVERSITY OF MINNESOTA IS A CONTRACTOR FOR US BECAUSE WE HAVE SO MANY REQUESTS; I CALL IT A "HELP DESK" NOT IN TERMS OF THE IP DESK WHERE YOU ARE TRYING TO RESET YOUR PASSWORD AND TAKES NINE HOURS. THERE'S LOTS OF TRAINING VIDEOS AND THEY ARE VERY GOOD AT KNOWING WHAT WE'RE WILLING TO APPROVE AND UNAPPROVED; SOME PEOPLE DON'T WANT TO LISTEN TO THEM, YOU SHOULD. THEY KNOW WHETHER WE WILL SAY NO. SOME PEOPLE DISAGREE WITH THAT, SENT IT BACK AND WE STILL SAY "NO." --- THESE ARE PRODUCTS THAT ARE AVAILABLE CURRENTLY; WE ADMINISTER THE AB PROGRAM, PHYSICIANS HAVE 12 MONTHS TO GIVE THE DATA. AND ALL THE DATA IN THE DATA CENTER GOES BACK TO 1999, ENDING UP WITH SOMETHING LIKE 850 BILLION DATA RECORDS, SOMETHING LIKE 220 MILLION PEOPLE. --- THIS IS OUR PROCESS WHICH IS EASY TO FOLLOW. RASDAC (SOUNDS LIKE) GIVES THE DATA TO US. THERE IS NO CONGRESSIONAL ALLOCATION TO PROVIDE IT FOR RESEARCH SO ALL THE COSTS WE INCUR TO MAKE THE DATA AVAILABLE IS FACTORED INTO THE CHARGE TO ACCESS OUR DATA. AND I KNOW THAT I DON'T HAVE MUCH TIME. WE HATE THE PHYSICAL PROVISION; WE DO NOT LIKE TO GIVE OUT OUR DATA, IT IS LEGACY. THE VIRTUAL ONE IS RELATIVELY NEW BUT YOU CAN REQUEST OUR DATA AND WE CAN PHYSICALLY SHIP IT TO YOU AND WE ARE ACTIVELY TAKEN STEPS TO MAKE THAT GO WAY AND ONE OF THE MOST CONVENIENT THINGS ABOUT THAT IS THAT AS POPULATION GROWS, PHYSICAL DATA FILES ARE GOING TO COST MORE AND MORE; IT'S GOOD TO BE SO EXPENSIVE PEOPLE WILL BEG US NOT TO SHIP THEM. --- THE PROBLEM WOULD ALWAYS HAVE WITH THIS IS, IN ACADEMIC YOUR I DEPARTMENT SOMEWHERE IS PAYING TO STORE THIS AND THE RESEARCHER DOES NOT SEE THE COST AND GENERAL WHEREAS IF YOU COME TO US THAT IS PART OF THE FEE. --- THE OTHER OPTION IS OUR VIRTUAL RESEARCH DATA CENTER SO THE CHRONIC CONDITION DATA WAREHOUSE -- AND THE -- IS THE SAME THING. THIS IS A VIRTUAL WHICH CONNECT TO OUR ENVIRONMENT USING YOUR OWN EQUIPMENT AND YOUR OWN COMPUTER, CONNECTING WITH A SECURE PORTAL AND YOU ANALYZE DATA IN OUR ENVIRONMENT; YOU ARE NOT ALLOWED TO TAKE IT OUT. YOU ARE ALLOWED TO TAKE OUT AGGREGATED DATA BUT YOU HAVE TO REQUEST IT AND WE ANALYZE IT. THIS IS A HIGH-VALUE CMS ASSET, SO IT IS A HOMELAND SECURITY ITEM SO THEY TRY TO HACK US EVERY COUPLE OF YEARS AS WELL. AND THE SAS, -- ANALYTIC PLATFORM -- THERE'S A NUMBER OF ADVANTAGE OF USING THE CRVC (SOUNDS LIKE); INSURANCE COMPANIES ARE STARTING TO PICK UP ON THE FACT THAT THE STATE IS VERY VULNERABLE AND A LOT OF CONTRACTORS EVEN FOR CMS ONE TO YOU COME INTO OUR ENVIRONMENT AND USE IT BECAUSE INSURANCE COMPANIES CHARGE THEM. --- WE DO MAKE A LOT OF PUBLIC FILES; WE TRIED TO MAKE (DID NOT UNDERSTAND) ACCORDING TO HIPPA, DOESN'T WORK WELL. WE HAVE A FIRE HR 7 DATABASE (SOUNDS LIKE) FOR APPLICATIONS; WE HAVE A CAN IS NOW WHERE APPLICATIONS -- 30 OF THEM ARE CLASSIFIED -- THEY CAN CONNECT DATA, YOU CAN DOWNLOAD THE APPLICATION AND AUTHORIZE THE SHARNG OF THE DATA IN THE HR7 FORMAT WILL MOVE TO THE APPLICATION CALLED BLUE BUTTON 2.0. LOTS OF PEOPLE ARE USING. WE HAVE SOME SYNTHETIC DATA SO PEOPLE CAN PLAY WITH IT, SUPER SYNTHETIC DATA. THAT'S ALL I HAVE. THAT'S AS FAST AS I CAN TALK. >> [APPLAUSE] >> MARK BARNES: WE HAVE HADAN INCREDIBLE ATTENTION AND DEVOTION TO MAKE CRITICAL TRIAL BETA-2 BE TRANSPARENT IN ORDER TO PROMOTE THE USE OF THIS DATA THAT ALSO A RECHECK ON ITS INTEGRITY; OF THE STATISTICIANS LOOK AT THE RESULT AND THEY CAN BE VERIFIED SO AS REBECCA IS GOING TO PRESENT, THERE IS TRANSPARENCY ON ONE HAND AND A HUGE DEMAND FOR CONCURRENCY ON THE OTHER; THERE IS A SOCIAL DEMAND FOR PRIVACY AND HOW THE TWO ARE RECONCILED. REBECCA? >> REBECCA LI: MARK IS ONE OF THE BOARD MEMBERS. HE KN OWS THIS WELL. TYPICALLY I SPEAK IN CONFERENCES ON HOW TO MAKE IT A MORE SHAREABLE AND MORE TRANSPARENT SOLAR HAPPY TO BE HERE TODAY. SO AT VIVLI, THE GLOBAL SHARING PLATFORM -- TODAY WE'LL TALK ABOUT HOW WE DO THAT, SOME OF THE MECHANISMS WE USE TO KEEP THE DATA CONFIDENTIAL AND PRIVATE AND HOW WE ENCOURAGE SHARING OF THAT DATA WITHIN SOME OF THOSE BOUNDARIES. SO ONE OF THE THINGS WE DO AS A NONPROFIT IS WE FOCUS ON POLICY, AND GOVERNANCE; MECHANISM AND RESOURCES. THE VIVLI SOLUTION IS WE PROVIDE STRATEGIC IN POLICY DEVELOPMENT AND HARMONIZE AGREEMENTS, WE PROVIDE THE PLATFORM AND WE USE MICROSOFT TO SHARE THE DATA AND WE HAVE A TEAM THAT MANAGES RESEARCH QUERIES SO THIS IS THE THREE OF YOUR PIECES FOR OUR SOLUTION. SO VIVLI AS MARK INTRODUCED IS A NONPROFIT ORGANIZATION AND WE OFFER A CONVENIENT FUNCTION THAT INCLUDES BIOTECH, ACADEMIC INSTITUTIONS AND NONPROFIT FUNDERS. WE HAVE BEEN HARMONIZING LANGUAGE AND AGREEMENTS; THIS IS NO SMALL FEAT AS A CAN IMAGINE AND LOWER THE BARRIERS FOR BRINGING TOGETHER DATA. --- WE ALSO OVERSEE THE IMPLEMENTATION OF THE ACTUAL PLATFORM ITSELF SO WE HAVE A VERY USER-FRIENDLY PLATFORM THAT IS SHARED BY THE INTERNATIONAL COMMUNITY. --- LET ME TALK ABOUT THE TYPE OF DATA THAT WE SHARE THROUGH THE VIVLI PLATFORM. FOR THOSE OF YOU THAT WORK IN THE CLINICAL TRIALS COMMUNITY YOU KNOW THAT THERE WAS VEDA (SOUNDS LIKE) THAT REQUIRED CLINICAL TRIALS TO BE REGISTERED IN THE 90S; AND THEN SUMMARY DATA WAS THEN SHARED A LITTLE BIT LATER AS VEDA PUBLISHED THE FINAL RULE. SO THEN WE HADSUMMARY DATA THAT WAS THEN REQUIRED TO BE SHARED SO WHEN WE THINK ABOUT SUMMARY DATA THAT SUMMARY DATA IS NOT THE SENSITIVE SO WHEN YOU THINK ABOUT CLINICAL TRIALS THAT MIGHT BE WHETHER A GROUP THAT HAS THE DRUG THAT WORKS, ONE GROUP FOR EXAMPLE HAS A PLACEBO AND THAT DRUG MAY NOT WORK IN THE GROUP. LATER ON WE HAVE CLINICAL STUDY REPORTS AND RAW DATA; THAT IS THE DATA STORED WITHIN VIVLI, AND THAT IS REALLY SENSITIVE DATA AS YOU CAN IMAGINE SO THAT IS DATA AT INDIVIDUAL PARTICIPANT LEVEL AND THAT IS WHAT WE STORE WITHIN OUR GLOBAL DATA PLATFORM. THAT IS REALLY WHERE I'LL FOCUS TODAY, THE THIRD DOT, INDIVIDUAL PARTICIPANT DATA AND THAT IS WHERE WE ARE TIPPING OVER TODAY SHARING THAT INDIVIDUAL PATIENT LEVEL DATA OR IPD. --- BY THE NUMBERS TODAY WE ARE THE LARGEST GLOBAL DATA SHARING PLATFORM FOR CLINICAL TRIALS; WE HAVE OVER FOR THOUSAND TRIALS BEING SHARED ON VIVLI, AND THAT REPRESENTS 2 MILLION PARTICIPANTS FROM OVER 100 COUNTRIES AND WE HAVE 20 MEMBERS THAT ARE DIVERSE, INCLUDING AN NUMBER OF PHARMACEUTICAL COMPANIES, MEMBERS OF NIH SUCH AS IMPORT AND OTHERS, AS WELL AS UNIVERSITIES. --- WHEN WE THINK ABOUT DATA BEING SHARED FROM SUCH A LARGE NUMBER OF DIVERSE STAKEHOLDERS -- THINK ABOUT THAT -- AT THE END OF A CLINICAL TRIAL THEY WOULD BE SHARING THEIR INDIVIDUAL PARTICIPANT DATA ON VIVLI AND YOU CAN ACCESS THROUGH EITHER DOWNLOAD OR MOST OFTEN A SECURE ENCLAVE LIKE IT WAS DESCRIBED, LIKE CRDC, WE USE MICROSOFT AZURE. I WILL BE TALKING ABOUT THE CONTROL THAT WE USED TO KEEP THE DATA CONFIDENTIAL AND SECURE. --- SO WE BALANCE THE RISKS AND BENEFITS; I FEEL THAT ONE OF THE BENEFITS OF COMING LATER IN THE PROGRAM IS THAT I CAN EDIT THE REMARKS. WE CAN TALK LESS ABOUT THE BENEFITS OF SHARING SINCE THIS IS ABOUT PRIVACY; JUST YESTERDAY AT OUR ANNUAL MEETING, A DOSHUN (SOUNDS LIKE) GROUP JOINED US. SOME OF THESE FOUNDATIONS THAT HAVE ORPHAN DISEASES WANT THE DATA SHARED. WHY WOULD PHARMA WANT TO SHARE THE DATA? THEY WANT TO MAXIMIZE THEIR TRIAL AND WE RESPECT THEIR CONTRIBUTIONS. LET'S NOT LOOK SIGHT OF THAT, AND THAT'S A BIT PROVOCATIVE. --- ESPECIALLY WHEN WE MEET WITH PATIENT GROUPS AT LEAST FOR CLINICAL TRIALS -- I'LL SPEAK TO THAT SEGMENT -- MOST DON'T WORK. AT THE END OF THE DAY, THAT TRIAL DATA GOES ANSWER ONE QUESTION: THE PRIMARY OBJECTIVE. BUT YOU DON'T USE THAT ANSWER INNUMERABLE OTHER QUESTIONS; WHEN THAT TRIAL DATA IS DEPOSITED INTO A REPOSITORY, DOES NOT HAVE TO BE VIVLI, THE DATA GOES TO ANSWER OTHER QUESTIONS AND WE HAVE TO REMEMBER THAT AND THINK ABOUT THE BENEFITS OF SHARING DATA, AND BALANCING THE BENEFITS AS WELL AS THE RISKS. --- THERE ARE OPEN ACCESS PLATFORMS AND KEY FEATURES -- THERE MIGHT BE NO CONTROLS OR SIMPLY CREATING AN ACCOUNT OR SIGNING A SIMPLE ONLINE DATA USE AGREEMENT; IN FACT, GOVERNMENTS HAVE THOSE. YOU CAN GO TO (INCOMPREHENSIBLE) AND THERE'S HEALTH AGENCIES AND DOWNLOAD THOSE AND THERE IS SOMETHING CALLED THE MANAGED ACCESS PLATFORM WHICH IS WHAT VIVLI IS; THERE IS AN INTERMEDIARY INTERPOSED TO IN THE DATA, WE KNOW WHO YOU ARE AND YOU PUT IN A PROPOSAL AND A SCIENTIFIC QUESTION AND YOU HAVE TO HAVE SOME SPECIALIZED EXPERTISE AND SIGN THE DATA USE AGREEMENT WHICH IS A LEGAL CONTRACT; THERE IS ALSO A MORE RESTRICTIVE PLATFORM WHICH IS BY INVITATION ONLY AND IT IS ACCESSIBLE GENERALLY TO THOSE WHO PROVIDE THE DATA SO THOSE ARE THE THREE LEVELS OF TRIAL DATA PLATFORMS. --- SO OUR GOVERNANCE PROCESSES ARE FLEXIBLE AND EFFICIENT; WE RESPECT THE REVIEW PROCESS OF EACH DATA CONTRIBUTOR SO THOSE MEMBERS SET THEIR OWN GROUND RULES. THEY HAVE TO BE TRANSPARENT ON EACH MEMBER PAGE; AND THEN WE ACCOMMODATE THOSE. BUT WE HAVE A HARMONIZED REVIEW PROCESS AND A HARMONIZED DATA USE AGREEMENT. THAT WAS NO SMALL FEAT BY THE WAY. --- SO THE DATA REQUEST PROCESS AND DATA ACCESS PROCESS IF SOMEONE SEARCHES FOR THAT INFORMATION THEY GO AHEAD AND REQUEST IT AND EACH DATA REQUEST IS REVIEWED THAN THE ACCESS THE DATA; MOST OF THEM ACCESS IT IN OUR SECURE RESEARCH ENVIRONMENT OR SOME OF THEM CAN'T DOWNLOAD IT PERMISSION. THEY ANALYZE IT USING ROBUST TOOLS AND WE FOLLOWED THAT SO THEY CAN MEET THE PUBLICATION REQUIREMENTS. --- VIVLI IS ONE OF THE PLATFORMS WHERE THEY CAN ANALYZE WITHIN A SECURE RESEARCH ENVIRONMENT ON YOUR OWN DESKTOP; YOU CAN ANALYZE DATA FROM PFIZER AND OTHERS. THAT'S AN EXAMPLE OF THAT. --- MARK ASKED TO SHOW SOME KEY PROVISIONS IN OUR DATA USE AGREEMENT SO WE MANAGE THE DUA PROCESS FOR THE MEMBERS THEY USE A HARMONIZED DUA. THE RESEARCHER AGREES TO ADHERE TO A RESEARCH PLAN. WE DO NOT ENDORSE FISHING OF POSITIONS; THEY MUST PUT IN A HYPOTHESIS TESTING AND ADHERE TO THE RESEARCH PLAN. --- SECONDLY THEY MUST MAKE REASONABLE EFFORTS TO PUBLISH WHAT THEY SAY THEY ARE GOING TO DO AND NOT TO IDENTIFY PARTICIPANTS. LASTLY WE THINK OF THE VIVLI PLATFORM IS HAVING MULTIPLE PILLARS OF SECURITY ONE OF THEM BEING SIGNING THE USE AGREEMENT AND SECONDED THE PLATFORM WHICH IS BUILT WITHIN MICROSOFT AZURE AND ANALYZING THE DATA WITHIN A SECURE RESEARCH ENVIRONMENT. THIRDLY, THE DATA HAS TO COME IN AS ANONIMIZED DATA. --- IF I WERE TO REQUEST DATA THROUGH ONE OF THESE PLATFORMS I WOULD NOT PUT THE REPETITION IN THE LINE AND TRY TO DO SOMETHING UNTOWARD; YOU CAN'T GET YOUR REPETITION BACK BUT THAT IS ONE OF THE OTHER PILLARS OF SECURITY THAT I THINK ABOUT AND WE DO PUBLISH NAMES OF PEOPLE WHO REQUEST DATA THROUGH THE PLATFORM. THANK YOU VERY MUCH. I'M SIMSON GARFINKEL, WITH THE US CENSUS BUREAU AND I WOULD LIKE TO THANK YVONNE FOR PUTTING THIS WORKSHOP TOGETHER. IT'S A HUGE HONOR TO BE HERE. THERE'S THE ABSTRACTS SO THAT GOOGLE CAN FIND THIS PRESENTATION; IT WORKS, I RECOMMEND IT. WE'RE HAVING A CENSUS IN 2020; IT'S VERY EXCITING, WE DO THIS EVERY 10 YEARS. CENSUS DAY IS APRIL FIRST AND HOPEFULLY ON APRIL FIRST EVERYONE WILL GO TO THE WEBSITE AND FILL OUT THE CENSUS FORM AND GET ALL YOUR FRIENDS TO DO THAT; IT IS VERY IMPORTANT FOR THE FUTURE OF THE REPUBLIC. --- OUR MOTIVATION IS THE CONSTITUTION, ARTICLE 1, SECTION 2, THE FOUNDERS SAID THAT THERE WOULD BE AN ACTUAL ENUMERATION MADE WITHIN THREE YEARS OF THE FIRST MEETING OF THE CONGRESS OF THE UNITED STATES AND EVERY 10 YEARS. FORTUNATELY THE FIRST CENSUS WAS IN 1790, ENDS WITH A "0," REALLY EASY, THE FIRST ONE WAS OVERSEEN BY THOMAS JEFFERSON. --- LATER THE CENSUS WAS ME CONFIDENTIAL; IN ORDER TO GET MORE CONFIDENT DATA WE HAVE TO PROVIDE CONFIDENTIALITY AND THAT IS CODIFIED IN TITLE 15, SECTION 9. THE CENSUS BUREAU IS NOT ALLOWED TO MAKE ANY POLITICIAN WHEREBY THE DATA FURNISHED TO ANY PARTICULAR ESTABLISHMENTS OR INDIVIDUAL CAN BE IDENTIFIED, THAT IS OUR BASIC ROLE. THIS IS NOT SAY THAT THE DATA WE PUBLISH CANNOT BE RE-IDENTIFIED. THIS IS A DIFFERENT STANDARD; IF YOU GIVE US A NUMBER NOBODY CAN FIGURE OUT THAT IF WE PUBLISH IT THAT IT CAME FROM YOUR. --- IT IS EASIER TO IDENTIFY DATA THAN YOU MIGHT IMAGINE AND I HAVE A CONSTRUCTIVE EXAMPLE OF A HYPOTHETICAL LAW; LET'S SEE IF I HAVE TOO MANY SLIDES. THE JOB OF AN OFFICIAL STATISTICS AGENCY AT ALL OF THIS WILL BE RELEVANT I PROMISE YOU-- IS TO COLLECT AND PUBLISH USEFUL STATISTICS. THIS SHOWS SEVEN PEOPLE AND EACH PERSON HAS A AGE AND RACE AND SEX AND MARITAL STATUS. THEY HAVE THESE VARIABLES; THERE'S A COUPLE, 66 AND 84, OTHERS SINGLE, 8, 12, 24. WE TAKE THAT DATA, PUT IT INTO OUR COMPUTER; WE HAD THE FIRST COMMERCIAL COMPUTER SOLD ON THE PLANET. AND WE TABULATE THAT AND PUBLISH STATISTICS. YOU CAN THINK IN YOUR HEALTH CONTACT THAT THIS IS LIKE DOING A SURVEY OR STUDY AND PUBLISHING HEALTH STATISTICS; OTICE THAT THE STATISTICS HAVE DS ON IT (SOUNDS LIKE), THEY HAVE SUCH SMALL NUMBER OF COMPUTERS THAT WE HAVE TO SUPPRESS THAT. WHY WOULD YOU HAVE TO SUPPRESS ONE CELL? OH YEAH, THERE'S ONE BLACK MALE ON THE BLOCK AND WE CAN GO HERE AND FIND OUT INFORMATION ABOUT THEM. THERE MIGHT BE TWO PEOPLE WHO MEET THAT CRITERIA AND ONE COULD SUBTRACT THEIR INFORMATION AND GET THE DATA ABOUT THE OTHER PERSON SO WE SUPPRESSED THAT, CALLED "DATA SUPPRESSION." NOTHING IN THE TABLE IS SUPPRESSED. YOU CAN BUILD OUT SOME OF THESE EQUATIONS AND BACK OUT THE ORIGINAL DATA AND AM GOING TO SHOW YOU HOW THAT IS DONE WITH BASIC ALGEBRA. YOU CAN THINK OF SYSTEMS OF LINEAR EQUATIONS WITH EXACT SOLUTIONS; ONE EQUATION AND ONE UNKNOWN AND I ASK YOU WHAT IS X, YOU SAY 10. TWO UNKNOWNS -- IF I GIVE YOU TWO EQUATIONS AND TWO UNKNOWNS, WHAT'S Y? 7, THIS IS BORING. SO THIS TABLE HAS 7 LINES WITH THREE STATISTICS. YOU CAN EXPAND THAT TO 164 EQUATIONS; BUT 7 PEOPLE, THERE'S ONLY 28 VARIABLES. WE CALL THAT DATABASE RECONSTRUCTIONS. THERE'S PROGRAM I HAVE, ON MY MAC IT TAKES -- SECONDS TO DO THAT, VERY FAST. ---Q IF WE SUPRESS THAT ONE ROW THERE ARE TWO COMPLETE SOLUTIONS; TO HAVE THIS COMMON ROW SO THAT ROW IS TRUE AND WE DON'T KNOW WHETHER (INCOMPREHENSIBLE) BUT THERE IS A WHITE SINGLE WOMAN WHO IS 24. THERE MIGHT BE EXTERNAL DATA; THE NEWSPAPER MIGHT SAY THERE IS NO ONE IN THE BLOCK OVER 85 YEARS OLD; YOU CANNOT CONTROL THAT EXTERNAL DATA SO WE KNOW THAT SOLUTION 2 IS NO LONGER CORRECT BECAUSE THERE IS EXTERNAL DATA THAT RULES IT OUT. THIS IS THE PROCESS BY WHICH DATA IS RE-IDENTIFIED; YOU DO LINKAGE WITH DATA SOURCES, BUT THIS IS DATABASE OR CONSTRUCTION. THIS IS SAYING SUMMARY STATISTICS HAVE BEEN PUBLISHED AND WE CAN STILL GET THE RAW DATA OUT OF THEM AND IF ENOUGH SUMMARY STATISTICS ARE PUBLISHED YOU CAN ALWAYS GET THE RAW DATA BACK. WHAT'S WORSE IS THAT THE STATISTICS MIGHT BE PUBLISHED BY 50-100 DIFFERENT RESEARCHERS DRAWING FROM OVERLAPPING DATA SETS AND YOU CAN FIND A CONSISTENT SET OF MICRO DATA THAT MAKES THIS IS THE SIX POSSIBLE. --- IF WE WANT TO PROTECT RESPONDENT DATA WE NEED TO USE SOMETHING THAT IS MORE MATHEMATICALLY RIGOROUS AND SIMPLE GENERALIZATION OF SUPPRESSION. THE ONLY GAME IN TOWN IS DIFFERENTIAL PRIVACY. --- WHAT DIFFERENTIAL PRIVACY DOES IS WE WILL ADD NOISE AND WHEN WE GO THROUGH THE NOISE BARRIER IT WILL NOT BE POSSIBLE FOR SOMEBODY TO BACK OUT THE ORIGINAL DATA TICKETS THEY WILL NOT ALWAYS KNOW IF WE ADDED +1 OR -2. IT IS SITTING ON TOP OF A BUNCH OF MATHEMATICAL THEORIES THAT GIVE US A MATHEMATICAL DEFINITION FOR PRIVACY LAWS CAUSED BY THE DATA RELEASE -- I'M NOT GOING TO SHOW IT -- GIVES NO DEPENDENCE ON THE ATTACKER'S EXTERNAL INFORMATION OR THE AMOUNT OF COMPUTING POWER THEY HAVE WHICH MEANS YOUR DATA RELEASES ARE FUTURE PROOF, NO MATTER WHAT DATA IS RELEASED IN THE FUTURE YOUR PRIVACY DEFINITION, YOUR PRIVACY GUARANTEES WILL CONTINUE TO HOLD. IT TURNS OUT THAT THEY ARE RELATIVE NOT ABSOLUTE PRIVACY GUARANTEES. --- YOUR PRIVACY GUARANTEE WITH DIFFERENTIAL PRIVACY IS HOW MUCH MORE PRIVACY LAWS YOU HAVE CAUSED WITH YOUR DATA RELEASE. AND WE DESCRIBE THAT IN TERMS OF THIS GRAPH WHICH IS SORT OF THE TRADE-OFF BETWEEN PRIVACY LAWS AND ACCURACY; AND AS WE INCREASE THE PRIVACY LAWS WE INCREASE THE ACCURACY. IF WE WANT NO PRIVACY LAWS WITH THE DATA RELEASE THEN THERE CAN BE NO ACCURACY SO THE ONLY WAY TO PREVENT PRIVACY LAWS WITH THE DATA RELEASE IS NOT TO RELEASE ANY DATA. AND I DON'T MEAN MICRO DATA, I MEAN ANY DATA. THE MORE DATA IS RELEASED, THE MORE PRIVACY LAWS THERE IS. IT'S A WAY OF BOUNDING THE MAXIMUM PRIVACY LAWS CAUSED BY DATA RELEASE. YOU HAVE A KNOW AND YOU CAN SET THAT WHERE YOU WANT. THIS IS AN IMPORTANT DATA RELEASE MAYBE THE SIGNAL IS SO STRONG --. --- THE BASIC IDEA IS TO TAKE THE STATISTICS YOU WANT TO PUBLISH AND YOU ADD NOISE OF HERE ON THE LEFT WE HAVE THE REAL STATISTICS AND ON THE RIGHT THEY HAVE NOISE ADDED AND IT IS MATHEMATICALLY IMPOSSIBLE TO UNDO THE ADDITION OF THE NOISY AND RELIABLY GET THE DATA. WE SAY WHAT'S THE MEDIAN AGE? WE CAN GET ONE BETWEEN 9 AND 73. AND IN THIS EXPERIMENT, I'M DOING 5000 RUNS, THE MEDIAN AGE IS BETWEEN 9 AND 73; AT THE BOTTOM I'M SAYING WITH 100 PEOPLE ON THE BLOCK THE MEDIAN AGE IS BETWEEN 21-22; THE AMOUNT OF NOISE ADDED IS ROUGHLY EQUAL TO THE CONTRIBUTION ONE PERSON SO IT GOES THROUGH ALL THE INFORMATION IN THE NOISE BARRIER SO IF YOU WANTED TO REPORT THE NUMBER OF PEOPLE IN YOUR TRIAL THAT WILL HAVE TO GO TO THE NOISE BARRIER ALSO AND IT RESULTS IN SOME PERPLEXING RESULTS SOMETIMES; OH, WE HAD ONE PERSON IN OUR TRIAL -- DON'T DO ANY TRIALS WITH ONE PERSON, HIGHLY IDENTIFYING. --- THERE ARE DIFFERENT WAYS THIS CAN BE USED YOU MIGHT PREFER TO THE LOCAL MODEL, THE NOISES APPLIED TO EACH RECORD, BEST USING AN INTERACTIVE QUERY SYSTEM WHERE THE RESEARCH IS DONE AND THE NOISE GETS ADDED TO THE PUBLISHED RESULTS SO THE REAL QUESTIONS THAT WE HAVE IS HOW MUCH NOISE SHOULD BE AT? WHAT IS THE CORRECT VALUE OF THE EPSILON? WHERE SHOULD THE ACCURACY BE ALLOCATED? SO THE 2020 CENSUS NEEDS AND DIFFERENTIAL PRIVACY AND WE HAVE SOME PUSHBACK FROM SOME OF THE DATA USERS THAT DON'T WANT US TO PUT MOISTEN THE DATA WE HAVE TO IF WE HAVE TO PROTECT THE PRIVACY AND CONFIDENTIALITY OF THE AMERICAN PEOPLE GIVE US IN THE REAL QUESTION IS NOT WHETHER WE SHOULD USE DIFFERENTIAL PRIVACY ARE NOT; THE REAL POLICY IS HOW MUCH NOISE DO WE ADD SO IT IS CONSISTENT SO WITH OUR RESEARCH AIMS AND THE OBLIGATION TO PROTECT CONFIDENTIALITY. THANK YOU. >> [APPLAUSE] >> MARK BARNES: WE'RE AHEAD OF TIME, SLIGHTLY, YVONNE. DON'T BEAT ME, PLEASE. ONE OF THE THINGS I'M CALLED TO SAY DEFENDING VIVLI BOTH FOR VERIFICATION RESULT IN INCREASING THE RISK FOR OF THE SEARCH RESULTS, IS THEY CALL ON ME TO DAY, YOU DON'T LIKE YOUR DATA BEING USED? GET OVER IT. YOU'VE ALL BENEFITTED FROM IT. THIS IS HELPED US IN EPIDEMIOLOGY AND SCIENCE AND MORE; THERE ARE THOSE THAT HAVE ESPOUSED THE IDEA THAT THE USE OF DATA FOR RESEARCH HAS LED TO TREMENDOUS BENEFITS, A DISTILLATION OF WHAT A COUPLE OF THE PANELIST HAVE INDICAT ESPECIALLY THOSE WHO HAVE USED THE CMS FOR RESEARCH. COMMENTS OR QUESTIONS? CAN BLOCK CHAIN BE USED TO SECURE DATA TO RESEARCHERS? >> BLOCK CHAIN IS THIS MAGIC OIL BUT TO COMPORT ANYTHING MAKES EVERYTHING GOOD AND FREE. SERIOUSLY BLOCK CHAIN -- THE TECHNOLOGY GOES BACK TO THE 1970S AND IT IS A WAY OF BASICALLY AUTHENTICATING ANY PUBLICATION WITHIN INTEGRITY CONTROL, AND YOU CAN ARGUE THAT ONCE THE DOCUMENTS PUBLISHED, ON OUR PEOPLE TO KNOW THAT THE DATA HAS NOT BEEN MODIFIED. BIT COIN SEEMS TO BE THE ONLY RELIABLE CLUTCHING SYSTEM ON THE PLANET; CAN YOU TAKE THE DATA AND PUT IT BIT COIN DATA? YES YOU CAN; ALSO PUBLISHING THE HASH IN THE NEAR TIMES. SO BLOCK CHAIN TECHNOLOGY IS AS LOVELY OR THAT YOU CAN FOR AND ANYTHING AND RAISED VENTURECAPITAL WITH BUT I DON'T THINK IT IS SOMETHING FOR US TO -- IT WOULD BE A MUCH DEEPER CONVERSATION. >> MARK BARNES: DOES BLOCK CHAIN HAVE THE ABILITY TO PRESERVE THE PROVENANCE OF DATA FOREVER? IN TERMS OF WORKING FROM? >> NO BETTER THAN PUBLISHING THAT PROVENANCE IN AN ACADEMIC PEER REVIEW LITERATURE. THEY HAVE TO BE SURE THAT PARTICULAR BLOCK CHAIN IS PRESERVED AND BIT COIN MAY OR MAY NOT BE PRESERVED FOREVER BUT MY BET IS WITH THE ACADEMIC LITERATURE. >> MARK BARNES: NEXT QUESTION IS FOR REBECCA LI. HOW IS THE VIVLI IPD DIFFERENT FROM THE NIH SHARING OF DATA AND ITS REQUREMENTS. >> REBECCA LI: THAT'S AN ESCELLENT QUESTION. MY GUESS IS IT THIS STEMS FROM CLINICAL TRIALS DOT GOV, SIGNIFICANTLY DIFFERENT BECAUSE IT TALKS ABOUT THEY DON'T HOLD ACTUAL DATA. >> MARK BARNES: THERE IS GUIDANCE WITH REGARDS TO THE ETHICAL PREREQUISITES FOR CONTRIBUTING DATA TO THE NIH DATABASES; YOU CAN FIND IT EASILY ON THE NIH WEBSITE THAT INCLUDES THE IDEA THAT EVEN WHEN DEIDENTIFIED DATA ARE CONTRIBUTED THE RESEARCH SUBJECT WAS NOTIFIED OF THE BEGINNING OF THE TRIAL THAT THE DATA WOULD BE CONTRIBUTED TO THE NIH DATABASE. IT'S NOT REQUIRED BY LAW -- NOT REQUIRED FOR NIH TO SAY THAT BUT THEY EMBEDDED THAT IN THE GRAND TERMS AND CONDITIONS FOR EXAMPLE FUNDING THAT WOULD SUPPORT ANY KIND OF GENOMICS RESEARCH AND IT WOULD BE CONTRIBUTE INTO DBGAP (SOUNDS LIKE) OR THE OTHER DATA BASES. THAT'S WHAT'S GOING ON NOW. --- THE OTHER THING THAT NIH HAS DONE IS THAT WITHIN NIH FOR THE INTRAMURAL RESEARCH DONE WITH THE IDENTIFIED DATA, NIH HAS ESTABLISHED AND YOU IRB TO HAVE JURISDICTION OVER USES OF DEIDENTIFIED DATA THAT WOULD BE TREATED BY THE IRB AS THOUGH IT IS AN APPLICATION ACTUALLY FOR THE USE OF IDENTIFIABLE DATA AND THIS IS BEEN CHARTED BY NHL LBI (SOUNDS LIKE) -- SUPPOSED TO BE A KIND OF POTENTIAL MODEL FOR THEIR OTHER VISITORS TO FOLLOW AND I'M A MEMBER OF IT. DR. BARBARA -- IS THE CHAIR OF THAT IRB, JUST STARTED MEETING THE MONTH AGO SO IT'S NOT THAT IT IS LEGALLY REQUIRED BUT IS THOUGHT TO BE AN ETHICAL GOOD TO HAVE THAT KIND OF REVIEW OF DATA AND IT'S BEEN INTERESTING; WE HAVE BEEN PULLING TOGETHER THE OPERATING PRINCIPLES FOR HOW ONE DOES A REVIEW WHICH IS NOT EXACTLY THE KIND OF REVIEW WITH IDENTIFIABLE DATA AND INCLUDES ISSUES FOR EXAMPLE IN THE RARE DISEASE AREA WHERE THERE THE DATA ARE DE-IDENTIFIED EVEN BY HIPAA STANDARDS. --- SIMSON YOU INDICATED THAT DP METHODS ARE BEST APPLIED TO STATISTICS RATHER THAN MICRO DATA SO IF AN AGENCY WANTED TO RELEASE MICRO DATA COLLECTED FROM A SURVEY WHAT WOULD YOU DO? >> SIMSON GARFINKEL: THIS IS A BIG ISSUE FOR US. THERE ARE SEVERAL APPROACHES. THE FIRST IS NOT TO RELEASE A MICRO DATA AND ALLOW IT TO BE USED IN THE CONTROLLED ENVIRONMENT LIKE THE VIRTUAL RDC OR THE CENSUS BUREAU RDC; THE SECOND IS TO USE THE DIFFERENTIAL MODEL AND USE THAT TO MAKE SYNTHETIC MICA DATA AND THEN TO COULD ALLOW PEOPLE TO DO RESEARCH WITH THE SYNTHETIC MICA DATA AND IF THEY NEED TO VALIDATE THAT IT WORKS ON THE REAL DATA THEY CAN SEND YOU THEIR CODE AND YOU CAN RUN THIS ON THE CONFIDENTIAL DATA. ANOTHER APPROACH IS TO -- SORRY, THOSE ARE MY TWOP APPROACHES. >> MARK BARNES: WE HAVE A QUESTION FROM THE PUBLIC THAT IS SOMETHING WE HAVE ALL DISCUSSED AT SOME POINT OR ANOTHER, WHETHER GENOMIC DATA OUGHT TO BE CONSIDERED AT THIS POINT IN TIME TO BE IDENTIFIABLE OR NOT IDENTIFIABLE; WHATEVER YOUR ANSWER YOU OUGHT TO GIVE A SOLUTION ABOUT HOW IT SHOULD BE DEALT WITH IN A RESEARCH DATABASE. >> MEGAN, FROM JOHNS HOPKINS? >> MEGAN: ESPECIALLY IN LIGHT OF THE REVISIONS TO THE COMMON RULE RELATED TO THE NEW DEFINITIONS OF IDENTIFIABLE, PRIVATE INFORMATION AND IDENTIFIABLE, PRIVATE BIO SPECIMENS, THIS CONCEPT IS ONE THAT WE SHOULD REALLY THINK ABOUT. I THINK THAT THERE ARE VARIOUS WAYS IN WHICH ONE MIGHT ARGUE THAT GENOMIC DATA MAY NOT BE CONSIDERED IDENTIFIABLE, DEPENDING ON EXACTLY WHAT YOU ARE DOING WITH IT BUT OTHERS I THINK THAT YOU WOULD HAVE A TIME ARGUING ARE NOT IDENTIFIABLE. I THINK THE CHALLENGE WE HAVE UNDER THE REVISED COMMON RULE IS NOT TO THINK ABOUT IDENTIFYABILITY IN TERMS OF GENOMIC DATA, FLAT, YES/NO, BUT IN TERMS OF ANALYSIS WHICH MAKES IT MORE COLLOCATED BUT THAT IS WHERE WE HAVE TO HEAD TO. HOW DO WE LOOK AT DEVELOPING A TOOL TO ASSESS IRBS, AND SAY BASED ON WHAT YOU ARE DOING THIS SPECIFIC GENOMICDATA IS NOT IDENTIFIABLE. WE WILL APPLY THIS RUBRIC AND SAY YES OR NO, LOOKING AT THOSE NEW DEFINITIONS IN THE NEW CONCEPTS AND THAT IS SOMETHING WE NEED AND IT WOULD BE PARTICULAR HELPFUL RIGHT NOW. >> MARK BARNES: ANYONE WANT TO ADD TO THAT? OKAY. ONE QUESTION THAT ALSO COMES UP IS PARTIALLY REFLECTED ON THE QUESTION WE GOT FROM THE PUBLIC. WHAT ARE THE ETHICS AND PRACTICALITIES OF HAVING ONE RESEARCH INSTITUTION THAT HAS THE IDENTIFIABLE DATA AND SOMEONE WHO IS NOT THE CUSTODIAN OF THE DATA WANTS TO USE THE DATA IN A DEIDENTIFIED LIMITED SET FORMAT? WHAT ARE THE ETHICS PRACTICALITIES AND LEGALITIES OF ALLOWING THAT USED TO HAPPEN WITHIN ONE INSTITUTION? MEGAN? >> GIVE ME AN EXAMPLE. >> MARK BARNES: LET'S TAKE CMS. THERE ARE MANY RESEARCHERS WHO WANT NOT FOR PUBLIC HEALTH PURPOSES BUT FOR RESEARCH PURPOSES WOULD LIKE ACROSS THE DATA AT CMS AND THAT PERSON IS WITHIN CMS, AND THEY HAVE A COMPLETELY IDENTIFIABLE DATA SET. WHAT WILL BE NECESSARY FOR THAT PERSON TO ASSUME THAT THE PERSON IS USING ANONIMIZED DATA FOR THE PURPOSES OF THE COMMON RULE AND THE DEIDENTIFIED FOR PURPOSES OF HIPAA? >> MEGAN: ONE OF THE WAYS WE ADDRESS THIS IS THROUGH DATA STEWARDS AND GATEKEEPERS WHO CAN SERVE AS A BARRIER BETWEEN THE RESEARCHER WHO IS A RECIPIENT OF THE DATA TO CRUNCH THE NUMBERS-- FOR EXAMPLE I'M A PHYSICIAN -- A CONTROLLING THE WAY THE PERSON HAS ACCESS TO THE DATA SET AND THERE IS VALIDATION OR VERIFICATION THAT WHAT YOU ARE RECEIVING FOR THE SIX BUT THE PURPOSE IS A LIMITED DATA SET. WE'RE ABLE TO HAVE SOME ASSURANCE THAT EVEN IN YOUR BROUGHT ROLE YOU MIGHT HAVE ACCESS TO A SCOPE OF DATA THAT IS LARGER WE ARE PROVISIONING THIS; AND WE HAVE SOME CONFIDENCE THAT THE DATA IS MORE RESTRICTIVE. WITH THAT COME RULES AND ASSURANCES THAT THIS IS ALL YOU WILL USE. YOU CONCERT WE HAVE YOUR CAPABILITY IN YOUR ROLE TO LOOK DEEPER BUT BY RECEIVING THAT DATA YOU WOULD HAVE TO ALSO HAVE SOME AGREEMENT THAT THIS IS THE ONLY WAY YOU WILL USE IT. >> I'M GOING TO MOVE ISN'T A DIFFERENT DIRECTION AND IT MAKES ME THINK ABOUT OUTSIDE OF THE MEDICAL CONTEXT WHERE WE TALK ABOUT OTHER DATA PLATFORMS, AND WE THINK ABOUT FACEBOOK AND THE EMOTIONAL EXPERIMENT AND ALL OF A SUDDEN THE INTERNAL USE OF DATA THE OTHER TRADITIONAL AB TESTING THAT MIGHT HAVE ETHICAL A PRIVACY IMPLICATIONS SO IN THAT CASE YOU HIRED THE FORMER HEAD OF STANFORD'S IRB, AND THE PUBLISHED PAPER DESCRIBING THAT; IT WILL BE INTERESTING TO SEE THAT AN I KNOW THAT JAKE AND I ARE ON A PROJECT TO SEE HOW OTHER PRIVATE ORGANIZATIONS WITHIN THE INDUSTRY ARE INTERNALIZING THESE KINDS OF ETHICAL REVIEW PROCESSES; IS AN IMPORTANT AREA FOR US TO EXPLORE. >> A SPECIFIC USE CASE THAT MIGHT BE HELPFUL ALL THE DIFFERENT GUARDRAILS THAT WE MIGHT BE THINKING ABOUT; I'LL USE THE BLUEBONNET 2.0, FOR PEOPLE WHO ARE NOT AWARE OF THE POWER OF THE 2.0 INTEGRATION OF BOOKS THE POWER INTO THE INDIVIDUAL'S MEDICARE BENEFICIARY HANDS, THAT IS ONE ELEMENT OF CONTROL. AND BEFORE THAT HAPPENS WE HAVE TO GO TO CMS AND ACTUALLY DISCUSS THE USE CASE IN GREAT DETAIL; THE CLINICAL BENEFIT AND THE CLINICAL QUESTION WE ARE ASKING, THE BENEFIT OF THE BENEFICIARY, THE BENEFIT TO THE MEDICARE PROGRAM ETCETERA IS ANOTHER ELEMENT OF CONTROL; THEN THE IRB HAS TO APPROVE THE PROTOCOL BEFORE WE CAN BEGIN TO THINK ABOUT HOW TO LAUNCH THIS AND THEN THERE IS INFORMED CONSENT APPROVED BY THE IRB AT THE INDIVIDUAL LEVEL AND ULTIMATELY INSIGHT EACH OF THE COMPANIES INVOLVED IN THIS TRIAL QUESTION, 160,000 PEOPLE RANDOMIZE LAUNCHING NOVEMBER USING BLUE BUTTON 2.0 WE ARE USING CONTROLS THAT ARE CRM20 COMPLIANT AND IN ADDITION TO THE REGULAR TOUR CONTROLS OF THE PHARMACEUTICAL COMPANIES THAT WE ARE WORKING WITH SO THERE ARE GUARDRAILS AND CONSTRUCTS, IN THIS QUESTION THAT YOU ARE DESCRIBING, THAT ARE IN PLACE IN EXISTENCE WITHIN THE LEGACY WAY WE HAVE DONE RESEARCH SO WE HAVE TO BE VERY COGNIZANT OF WHERE THEY ARE STILL APPLICABLE AND HOW THEY CAN BE USEFUL AND CONTINUE TO HAVE POWER IN THE SORT OF MAKE A EFFICIENT WORLD OF EVERYTHING BEING DIGITAL, AND BECAUSE WE ARE IN A MEGA EFFICIENT WORLD HOW CAN WE IDENTIFY THIS AND MAKE IT MORE EFFICIENT. >> MARK BARNES: FEEL FREE TO COMMENT ON WHAT REBECCA SAID. THIS IS A QUESTION FOR ALL OF US WHEN WE HAVE THESE DATA USE AGREEMENTS, AND WE LET PEOPLE DOWNSTREAM OUTSIDE OF INSTITUTION USE THIS ANONYMOUS DATA SET; FOR EXAMPLE THEY IDENTIFY INDIVIDUALS OR THEY VIOLATED THE TERMS OF USE ON THE DATA SET AND THEY GAVE IT TO ANOTHER PERSON FROM THE PERSON. YOU GUYS AT CMS, YOU DO SO MUCH OUT LICENSING, HOW DO YOU ENFORCE THIS AND HOW DO YOU PREVENT IT PRACTICALLY? >> -- ALLOWING SOME SORT OF DEIDENTIFIED ACCESS; WE DON'T ALLOW THAT. TO GIVES INFORMATION TO HAVE TO TELL US HOW THEY WILL BE USING IT AND HOW THEY ARE CONTROLLING IT BUT FOR THEM TO ALLOW THE CUSTODIAN IN CHARGE OF PROTECTING THE DATA TO ALLOW IT TO BE USED IN AN IDENTIFIED DATA THAT TAKES AWAY SOME LAYERS AND IT TAKES IS OUT OF THE PICTURE AND WE DON'T ALLOW THAT. PART OF THE AGREEMENT THAT YOU SIGN FROM US HAS A SELF SUPPRESSION POLICY WHICH IS TO GET TO WHAT WE TALKED ABOUT HOW DO YOU SUPPRESS CELLS WITHIN YOUR ANALYTICAL TABLE OR SOMETHING FOR YOUR RESEARCH RESULTS? PAR T OF THAT IS YOU CAN'T DEIDENTIFIED BECAUSE IT IS ALWAYS A LEVEL OF (DID NOT UNDERSTAND), AND THAT IS THE MAIN DRIVER FOR THE POLICY SO AGAIN YOU WOULD BE BREAKING EVERY TERM OF THE AGREEMENT WITH THE USE OF CMS DATA. >> MARK BARNES: BUT HAVE YOU HAD SOMEONE BREAK THE DATA AND WHAT HAVE YOU DONE? >> ANDREW SHATTO: I THINK THERE IS A HUGE RISK TO AN ACADEMIC ORGANIZATION TO USE OUR DATA BECAUSE WE WILL SHUT THEM DOWN AND A LOT OF PEOPLE DON'T WANT THAT HAPPEN TO THEM. THERE HAVE BEEN CASES WHERE WITHOUT BEING TOO SPECIFIC WE NOTICE THAT A UNIVERSITY PROFESSOR WHO RECEIVE DATA ALSO HAD A PRIVATE BUSINESS AND WE SAW THINGS HAPPENING IN THAT PRIVATE BUSINESS THAT COULD NOT HAPPEN WITHOUT THE CMS DATA AND WE CALLED UPON THE UNIVERSITY AND IT OUR OWN INVESTIGATION, THE DEPARTMENT OF JUSTICE WAS INVOLVED, AND WHEN THE DEPARTMENT OF JUSTICE CONTEXT THESE OF UNIVERSITIES THERE WILL BE AN INVESTIGATION AND ULTIMATELY THAT WAS RESOLVED BECAUSE THAT PROFESSOR LOST THEIR TENURE AND WAS REMOVED AND THEY ARE NO LONGER ALLOWED. HAVE CONTROLS IN PLACE IN OUR SYSTEM TO PERMANENTLY BAN ORGANIZATIONS AND PEOPLE SO WE DON'T HAVE TO TRY TO REMEMBER WHO DID THIS. WE DON'T HAVE A WALLET PHOTOS ON IT; BUT OUR DUA TRACKING SYSTEM HAS CONTROLS IN PLACE TO AUTOMATICALLY BLOCK PEOPLE FROM EVER REQUESTING THIS DATA AGAIN, EVEN IF IT IS SOMETHING LIKE YOUR DATA USE AGREEMENT GOT TO EXPIRATION DATE; WHEN THAT HAPPENS YOU NEED TO TELL US YOU DESTROY THE DATA AND WE DON'T HEAR FROM YOU YOUR ORGANIZATION IS OUT OF EVERY AGREEMENT THAT THEY HAVE ANY BLOCKS THE ENTIRE ORGANIZATION. THAT'S A MOTIVATOR FOR THE FOLKS TO CONSTANTLY BE ENGAGING WITH US WITH WHAT IS HAPPENING WITH OUR DATA. >> MARK BARNES: YOU'RE LUCKY BECAUSE YOU HAVE MARKET POWER; WHEN YOU EXCLUDE THEM THAT WILL HURT THEM. >> REBECCA LI: THERE ARE CONTROLS IN PLACE, AT MULTIPLE STEPS OF THE WAY AND IT'S A MATTER OF HOW WE AND FORCES CONTROLS AND THINKING ABOUT MAKING SURE THAT THEY ARE ROBUST; I'VE HEARD EXAMPLES WERE SOMEBODY SAYS AN INSTITUTION GAVE INFORMATION TO A TECH COMPANY WHO WILL DO SOME KIND OF QUALITY OR POPULATION MANAGEMENT ACTIVITIES AND NEVER GIVE BACK THE DATA. THAT'S NOT OKAY. HOW DO WE MAKE SURE THAT DOES NOT HAPPEN? THEY DID THE RIGHT THING UP FRONT BUT THEY DID NOT FOLLOW THROUGH LIKE CMS IS DOING. HOW DO WE MAKE SURE PEOPLE ADHERE TO THOSE? I THINK IN SOME WAYS HEALTHCARE IS DIFFERENT BECAUSE IT IS HEAVILY REGULATED AND PEOPLE THINK HOW THE DATA IS SENSITIVE BUT THIS ALSO RELATIONS IN PLACE; FOR EXAMPLE THERE ARE IRBS IN PLACE, BUT MAYBE THEY ARE NOT IN PLACE LIKE GENERAL FACEBOOK DATA BUT I THINK THE QUESTION COMES UP, WHAT YOU KNOW WHEN THE DATA IS IN A PROTECTED SPACE VERSUS DATA THAT IS NOT IN THE TYPICAL HEALTHCARE SPACE BUT IS IMPORTANT FOR HEALTHCARE RESEARCH AND HOW DO WE THINK ABOUT TAKING SOME OF THE BEST PRACTICES THAT WE MAY HAVE AN HEALTHCARE SPACE AND WHEN WE ARE DOING HEALTHCARE RESEARCH HOW DO WE USE THOSE EXISTING STRUCTURES THAT WE HAVE AND THE MECHANISMS WE HAVE TO THINK ABOUT THIS ADDED DATA THAT MIGHT BE USEFUL IN HEALTHCARE CONTEXT. >> IF I COULD ASK YOU A QUESTION. RECENTLY THERE WAS A PAPER PUBLISHED WHERE THE RESEARCHERS COULD INFER IF A PERSON WAS DEPRESSED OR NOT LOOKING AT INSTAGRAM FILTERS; IT WAS VALIDATED, JUST AS GOOD AS A CLINICAL DIAGNOSTIC. HOW WOULD WE PROTECT INSTAGRAM FILTER INFORMATION? >> REBECCA LI: I DON'T KNOW. THE DIFFERENCE IS, IN MY VIEW THERE ARE FOLKS THAT A THOUGHT ABOUT HOW DO WE GET CONSENT TO COLLECT? CONSENT TO USE? WE HAVE RESEARCH PROTOCOLS AND THEN THERE ARE THE PLAYERS THAT ARE PERHAPS DOING RESEARCH OR ENABLING IDENTIFICATION OF HEALTH INFORMATION OR DIAGNOSTIC INFORMATION, ETC., FROM OUTSIDE OF THAT WORLD WHERE THERE ARE NO PROTECTIONS AND THAT IS ONE OF THE BIGGEST ISSUES TO GRAPPLE WITH. IT IS THIS DICHOTOMY OF HOW WE PROTECT AND THINK ABOUT HEALTH DATA WHEN IT IS IN THIS BUBBLE OF HEALTH DATA; FOR THIS ALL OF THESE OTHER SOURCES OF DATA THAT CAN GET YOU TO THE EXACT SAME PLACE THEY DON'T HAVE THE SAME PROTECTIONS AND THAT IS ONE OF THE BIGGER ISSUES TO GRAPPLE WITH, HOW DO WE MAKE SURE THAT WE HAVE A LEVEL PLAYING FIELD WHEN WE ARE TALKING ABOUT HEALTH INFORMATION. >> IT WOULD BE VERY CHALLENGING TO PREVENT SOMEONE FROM READING THE PAPER. WHAT IS THE GUN COULD DO IS BLOCK THE TARGETING OF ADVERTISEMENT ON THE BASIS OF AUTOMATED FILTER PROTECTION. RIGHT? SO IT'S NOT SO MUCH A QUESTION ABOUT WHAT CAN WE KNOW ABOUT THE PERSON BUT HOW ARE WE ALLOWING THIRD PARTIES TO LEVERAGE THE INFORMATION IN A WAY THAT IS HARMFUL? WE PROBABLY DON'T WANT -- YOU KNOW -- THE SAME MECHANISM THAT ALLOWS YOU TO TARGET THEM FOR PUBLIC HEALTH INTERVENTION -- MAYBE PEOPLE USE THESE FILTERS, FROM THE SUICIDE PREVENTION LINE, A VERY PLAUSIBLE INTERVENTION; WE SEE STUFF LIKE THAT. BUT THAT EXACT SAME TOOL AND KNOWLEDGE CAN ALSO BE TARGETING THEM WITH ADVERTISEMENTS THAT ARE STICKY COMMERCIAL OR EVEN INTENTIONALLY HARMFUL. AND IT'S UP TO THE PLATFORM. >> SO YOU WOULD BE OKAY WITH INSTAGRAM TARGETING SO IS A PREVENTION ADS BUT NOT GUN SALES? >> MARK BARNES: AM I OKAY WITH IT? I WOULD PREFER THAT SYSTEM NOT EXIST WHATSOEVER. I WOULD RATHER HAVE NO TARGETED INFORMATION; BUT THERE'S ALSO A WAY TO WORK WITH A PLATFORM -- THEY'LL TALK TO YOU ABOUT THEY ARE PLENTY HAPPY TO SAY WE ARE USING TARGETED HEALTH INTERVENTIONS ESPECIALLY FACEBOOK. FACEBOOK HAS A SUICIDE PREVENTION ALGORITHM THEY RUN ON ALL OF US IN ORDER TO FIND HIS SMALL HANDFUL OF PEOPLE THAT ARE AT RISK. --- THEY ARE NOT WILLING TO RELEASE ANY STATISTICS ABOUT IT BUT WE KNOW IT IS THERE; WE HAVE CALLED LAW ENFORCEMENT TO DO A CHECKOUT, DON'T KNOW HOW MANY TIMES THEY HAVE DONE IT KNOW WHAT ACCURACY, THERE IS NO CONSPIRACY BECAUSE THEY SAY IT IS NOT RESEARCH AT IT IS NOT MEDICINE. I WOULD RATHER THAT SYSTEM WILL NOT EXIST BUT AM ALSO WILLING TO SAY THAT -- IT'S BETTER THAT WE HAVE THAT FORM OF TARGETING AND DISALLOW THE GUN TARGETING THAN TO ALLOW EVERYTHING. >> AS I MENTIONED EARLIER, WE NEED THE DATA; DATA IS USEFUL. FOR EXAMPLE AT THE APPLE EVENT, THEY SHOWED THAT THE APPLE WHAT'S HAS SAVED A LOT OF PEOPLE'S LIVES BECAUSE WHEN THEY FELL DOWN THEY COULD CALL 911. THAT IS GOOD USE OF THE DATA TO PROVIDE HELP FOR THE PERSON AND ALSO THERE IS AN APP WHEN PEOPLE HAVE DRUG ABUSE; THEY CAN TRACK AND START GETTING PHONE CALLS TO HELP THEM SO THERE ARE CERTAIN TOOLS THAT HELP PATIENTS IN OVERCOMING THEIR DISEASES AND THAT IS THE USEFUL THING. WE SHOULD NOT STOP THAT. THE QUESTION IS, MAKE SURE WE DON'T USE PERVASIVENESS OF THAT. AND MAYBE MORE FINDS SHOULD BE IMPOSED SO THE PATIENTS OR ENTITIES DO NOT ABUSE THAT INFORMATION. >> LET ME ASK -- ONE OF THE THINGS -- THIS RELATES TO SOME OF THE PREVIOUS PRESENTATIONS AS WELL, HISTORICALLY WHEN WE THOUGHT ABOUT -- WE'RE FOCUSED ON THE USE OF THE DATA IN HEALTH RESEARCH AND TRADITIONALLY WE TALKED ABOUT THIS DICHOTOMY BETWEEN ACADEMIC AND COMMERCIAL RESEARCH, WITH THE PRESUMPTION BY MANY OF US THAT ACADEMIC RESEARCH IS GOOD AND COMMERCIAL RESEARCH IS BAD. BUT IN FACT, IF YOU LOOK AT IT FROM THE 50,000 LEVEL, OR IF YOU LOOK AT THE US FROM THE MOON, HOW IS IT DIFFERENT? OF COURSE ACADEMICIANS WHEN THEY DO RESEARCH THEY AND THEIR UNIVERSITIES ARE COMMERCIALLY INTERESTED IN WHAT THEY DO AND PERHAPS MORE PERSONALLY INTERESTED THAN A RESEARCH SCIENTIST AT MERCK PFIZER. THERE'S AN ADDITIONAL PUSH FOR REVENUE WHICH EXISTS ON THE ACADEMIC SIDE JUST AS MUCH ON THE COMMERCIAL SIDE AND WHERE THE RUBBER MEETS THE ROAD-- I'M ASKED THIS A LOT -- UNDER THE WAY THE HIPAA PRIVACY RULE WORKS, YOU'RE ALLOWED TO GET OUT THE LIMITED DATA SET, BUT IT DOES NOT SAY ACADEMIC RESEARCH OR COMMERCIAL RESEARCH SO ONE ISSUE THAT BUBBLES UP PROBABLY ONCE OR TWICE A YEAR AT MOST OF THE SOPHISTICATED RESEARCH INSTITUTIONS IS ARE WE ALLOWED TO GIVE A DATA SET TO A COMMERCIAL ENTITY WHEN WE KNOW IT IS FOR COMMERCIAL RESEARCH AND DEVELOPMENT PURPOSES? THAT IS AN EXAMPLE OF WHY THE DISTINCTION MATTERS BUT IT CERTAINLY IS A CONCEPTUAL DIVIDE THAT ANIMATES A LOT OF DISCUSSIONS IN THIS AREA. >> REBECCA LI: SO THAT QUESTION COMES UP A LOT. WHAT I'VE OFTEN LOOKED THAT IS, WHAT IS THE PRIMARY PURPOSE OF THE RESEARCH OR THE ACTIVITY? SO THERE'S A DEFINITION OF RESEARCH IN HIPAA, SUPPOSED TO BE FOR GENERALIZABLE KNOWLEDGE. THERE ARE MANY CASES WHERE THERE'S BOTH A COMMERCIAL USE AND A GENERALIZABLE USE OF THE RESEARCH; WITH THE DATA-USE AGREEMENT MY SENSE IS THAT IF THERE IS A RESEARCH VALUE-- AS LONG AS THERE IS NO DEMINIMUS -- I THINK THE QUESTION OF WHAT IS RESEARCH, AND WHEN IT IS RESEARCH AND WHEN IT IS COMMERCIAL IS VERY AMBIGUOUS AND IT IS FACT SPECIFIC. I'VE SEEN PEOPLE MAKING IT "RESEARCH" TO PUT THE WRAPPING AROUND IT AND I CAUTION PEOPLE AGAINST THAT. YOU CAN USE THE DATA UNDER A LIMITED DATA SET IF AT LEAST THERE IS A COMPONENT OF THE RESEARCH OF THE ACTIVITY AND IT IS NOT JUST WINDOW DRESSING. >> I THINK NOT JUST ABOUIT WHAT YOU ARE LEGALLY PERMITTED TO DO BUT WHAT ARE YOU WANTING TO DO THE RESPONSIBLE STEWARD OF THAT DATA? MANY OTHER INSTITUTIONS AND IS REGARDED GATEKEEPERS FOR THE HEALTH DATA THAT THEY MAINTAIN AND THERE IS SOME EXPECTATION OF THE PATIENTS TO COME TO THE HEALTH SYSTEM AND ONE OF THE THINGS TO THINK ABOUT IS, IS IS THE BEST USE OF DATA? AND THAT BECOMES A QUESTION ABOUT THE SHARING; WHAT IS THE VALUE ADDED TO THE ORGANIZATION? WILL IT HELP THE PATIENTS IN THE FUTURE? THOSE ARE SOME OF THE TYPES OF QUESTIONS AND GATE POST THEY CAN BE SURROUNDING THE QUESTION OF IS IS LEGAL TO SHARE WITH A COMMERCIAL PARTNER? >> MARK BARNES: IT GOES BACK TO THE CULTURE ISSUE THAT WE WERE TALKING ABOUT, WHAT YOU WANT TO DO AS A POSITIVE, NORMATIVE GOOD. PART OF THE REASON FOR TRANSPARENCY AND CLINICAL TRIALS DATA AND SOME HEALTH DATA OF REQUIREMENTS FROM SOME QUARTERS AND IN SOME CASES ACADEMIC PUBLICATIONS LIKE THE ICMJE, THE CONSORTIUM OF THE LEADING MEDICAL JOURNALS AND THE EUROPEAN MEDICINE AGENCY, REQUIRING THAT IF THE COMPANY SUBMITS DATA TO SUPPORT THE MARKETING APPLICATION FOR APPROVAL TO MARKET MEDICINAL PRODUCTS WITHIN EUROPE, A CONDITION OF THAT WHETHER YOUR PRODUCT IS APPROVED OR DISAPPROVED WITHIN SIX MONTHS OF THAT DECISION YOU ARE REQUIRED AS A COMPANY TO MAKE YOUR PATIENT LEVEL DATA COMPLETELY AVAILABLE TO ANYONE STRIPPED OF IDENTIFIERS AND YOU CAN DO IT FOR YOU YOUR OWN PLATFORM OR THROUGH A DATA SHARING PLATFORM; YOU ARE REQUIRED TO DO THAT. AND THEN THE QUESTION IS, WHO USES THAT DATA WHEN IT IS PUT OUT LIKE THAT? ALTHOUGH FOR THE ICMJE IT IS DONE FOR ACADEMIC PURPOSES IN ORDER TO CREATE TRANSPARENCY WITHIN THE ACADEMIC COMMUNITY AND ALTHOUGH AT THE EMA IT IS DONE TO CREATE TRANSPARENCY -- WHAT OUR EXPERIENCE IS, WHO DO YOU THINK THE ENTITIES ARE THE TAKE ADVANTAGE OF THAT DATA TRANSPARENCY AND IT IS NOT ACADEMIC RESEARCHERS BUT OTHER INDUSTRIAL CONCERNS? BOTH HEALTH CONCERNS AND ALSO NON-HEALTH CONCERNS; THEY WANT TO LOOK AT THE DATA IN ORDER TO PLAN THEIR OWN TRIALS AND LOOK AT HOW PROMISING SOMETHING MIGHT HAVE BEEN OR WHAT THE SAFETY SIGNALS MIGHT'VE BEEN IF THEY ARE EMBEDDED IN THE DATA. THE OTHER THING ABOUT THE EMAS -- I'LL SHARE THIS VIGNETTE -- THE EMA IS NOT FULLY FACED THIS REQUIREMENT BUT THEY ARE REQUIRING THAT THERE BE THIS FULL TRANSPARENCY EVEN IN SMALL TRIALS AND THE DATA CANNOT BE TRULY DEIDENTIFIED BY GDPR STANDARDS WHICH MOST PEOPLE WOULD SAY HAS A TOUGHER THRESHOLD FOR REACHING THE DEIDENTIFIED ABILITY THAT HIPAA DOES THAT'S NOT BEEN RECONCILED -- A UNIT OF THE EUROPEAN GOVERNMENT ASHLEY TO TALK TO LAWYERS WHO ARE THE PRIMARY ATTORNEYS FOR INTERPRETING THE GDPR WITHIN THE EUROPEAN GOVERNMENT; HOW CAN WE HAVE COMPLETE TRANSPARENCY OF DATA IN SMALL DATA SETS AS REQUIRED BY THE EUROPEAN AGENCY AND AT THE SAME TIME HE THE REQUIREMENTS OF ANONYMITY OR DEIDENTIFIED FOR GPPR? AND THE ATTORNEYS ASKED WHAT HIS POLICY 70? THEY HAD BEEN NO CONTEMPLATION IN THE DRAFTING OF THE GDPR OF THE NECESSITY OF TRANSPARENCY IN DATA SETS FOR NON-PRIVACY PUBLIC HEALTH AND ACADEMIC PURPOSES; A VIGNETTE OF HOW THESE TWO WORDS EXIST IN PARALLEL UNIVERSES BUT THEY HAVE ACTUALLY QUITE A BIT TO DO WITH EACH OTHER. --- ANYBODY WANT TO RESPOND TO THAT? >> REBECCA LI: I WOULD SAY -- AS YOU ILLUSTRATED THERE IS OFTEN THIS TENSION BETWEEN TRYING TO PUSH FOR MORE OPENNESS AND PRIVACY AND LIKE YOU SAID IN THE REGULARTORY ARENA THEY ARE TALKING PAST EACH OTHER AND WE ARE TALKING HERE ABOUT WHAT FRAMEWORKS CAN WE PUT IN PLACE SO THIS IS A GOOD STARTING POINT. >> MARK BARNES: ANOTHER QUESTION FROM THE PUBLIC. IN VIEW OF THE POTENTIAL FOR DOWNSTREAM USES OF DATA, IN ONE RELIES AND CONSENT, WHAT KIND OF INFORMATION SHOULD BE GIVEN TO PEOPLE WHEN THEY DO CONSENT FOR THE DOWNSTREAM FOR A WIDE VARIETY OF DOWNSTREAM USES FOR THE DATA? HOW CAN ANYONE GET MEANINGFUL INFORMED CONSENT IN THAT SITUATION? >> I WAS TAUGHT THAT INFORMED CONSENT CAN BE WITHDRAWN AND LATER POINT IN TIME SO IN ORDER TO HAVE MEANINGFUL INFORMED CONSENT YOU HAVE TO HAVE A TRACK OF WHO IS USING YOUR DATA AND BE ABLE TO WITHDRAW CONSENT. THAT REQUIRES DATA PROVENANCE AND TRACKING; IT IS COMPLETELY SOLVABLE WITH TODAY'S TECHNOLOGY. THE CORPORATIONS BUILDING THESE PATTERNS DON'T WANT TO DO IT BECAUSE IT INCREASES THE LEVEL OF TRANSPARENCY THAT IS MORE THREATENING AND MAKES IT MORE EXPENSIVE. >> AND I THINK ONE ASPECT OF THAT TOO IS PATIENT EDUCATION. A LOT OF PEOPLE WILL CLICK THROUGH THE TERMS OF SERVICE FOR INSTAGRAM AND I DON'T KNOW THAT A LOT OF PEOPLE READ THAT OR MANY OF THE PEOPLE IN THE TERMS OF SERVICE -- I KNOW IN BLUE BUTTON WE PUT FLAGS TO READ THE TERMS OF SERVICE SO PART OF THAT IS THEY ARE AGREEING TO THINGS AND WE NEED TO MAKE SURE THEY KNOW WHAT THEY ARE AGREEING TO. I WISH I COULD REMEMBER WHO WAS BEHIND THE STUDY; THERE WAS A MACHINE LEARNING STUDY I SOUGHT SOME TIME AGO WHERE THE PLOT A GRAPH OF ALL THE TERMS OF SERVICE IN MAJOR WEBSITES; THE EDUCATION LEVEL YOU WOULD REQUIRE IN ORDER TO BE ABLE TO UNDERSTAND THOSE TERMS OF SERVICE AND THEY WERE PROGRESSIVELY GETTING HIGHER. BUT THEY STARTED VERY HIGH. --- ONE OF THE THINGS THAT NEEDS TO BE DISCUSSED IN ORDER TO REALLY HAVE PATIENTS IN CONTROL OF THE DATA IS TO MAKE SURE THEY UNDERSTAND THE TERMS OF SERVICE FOR THE DIFFERENT USERS AND THAT THOSE TERMS ARE WRITTEN IN A WAY THAT THEY ARE UNDERSTANDABLE. >> REBECCA >> I THINK TERMS AND CONDITIONS ARE TERRIBLE; EVEN IF YOU CAN READ THROUGH IT, UNDERSTANDING WHAT THE IMPLICATIONS ARE IS CHALLENGING. I WOULD THINK THAT THE POINT IS RIGHT ON THE MONEY; I WOULD ARGUE THAT THERE IS A BROADER OPPORTUNITY HERE TO DO EDUCATION AND EVEN THINK ABOUT HOW WE EDUCATE OUR CHILDREN AROUND APPROPRIATE ONLINE BEHAVIOR. AND HELPING THEM THINK THROUGH WHAT TYPES OF INFORMATION THEY WANT TO PUT OUT, FOR EXAMPLE IN SOCIAL MEDIA. I THINK THAT WITH THE INSTAGRAM EXAMPLE, YOU CAN IMAGINE A LOT OF GOOD THAT COULD COME FROM IDENTIFYING PEOPLE WITH MENTAL HEALTH CHALLENGES THAT NEED TO BE ADDRESSED; BUT I THINK ALSO GIVING PEOPLE THE TOOLS AND KNOWLEDGE INCLUDING KIDS TO BE ABLE TO MAKE DECISIONS ABOUT WHAT TYPE OF INFORMATION THEY WANT TO PUT OUT, FOR EXAMPLE IN SOCIAL MEDIA, AND AN UNDERSTANDING OF HOW THE INFORMATION CAN BE ANALYZED AND WHAT SORT OF ALGORITHMS CAN BE APPLIED; I THINK THAT'S SOMETHING REALLY UNDEVELOPED IN OUR SOCIETY AT THIS POINT AND I WOULD ALSO ADD TO THAT -- WE TALKED ABOUT THE DISTINCTION BETWEEN HEALTH DATA AND OTHER DATA AND I HAVE A COLLEAGUE WHO IS FOND OF SAYING ANY PERSONAL DATA IS HEALTH DATA. THAT'S WHAT I THINK ABOUT WHEN I HEAR THE CONVERSATIONS ABOUT SOCIAL MEDIA SO I WOULD BE AN ADVOCATE FOR THINKING SERIOUSLY ABOUT HOW WE APPROACH EDUCATION BROADLY. >> I WAS GOING TO SAY THAT LORI KRAMER AND ALYSSA MCDONNELL (SOUNDS LIKE) WROTE A NUMBER OF STUDIES IN THIS, 76 HOURS A YEAR OR 76 DAYS, RIDICULOUS. THEY DEVELOP MORE EFFECTIVE WAYS TO VISUALLY PRESENT THIS INFORMATION AND OVER THE COURSE OF 5-10 YEARS OF RESEARCH THEY DEVELOPED AND TESTED AND VALIDATED PRIVACY ATTRITION LABELS WHICH WERE NEVER ADOPTED BUT THAT RESEARCH COULD BE PICKED UP BY HHS OR HARP AND RECOMMEND A DISCLOSURE FOR WHAT WILL BE DONE WITH YOUR HEALTH DATA AND HAVING A STANDARDIZED GRAPHIC PORTRAYAL HAS WORKED VERY WELL, AND THE RESEARCH SHOWS THAT IT WORKS WELL FOR INFORMATION ABOUT HOW YOUR PERSONAL INFORMATION IS USED. >> I STRUGGLE A LITTLE BIT; I AGREE WITH AN INFORMED CONSENT MEANINGFUL; THEY'VE BALLOONED, 20-25 PAGES ON AVERAGE. I'M CONCERN ABOUT ADDING YET ANOTHER PAGE ABOUT HOW THE DATA IS SHARED, A LITTLE BIT ABOUT MY CONCERNS ALTHOUGH NOT THAT I DON'T AGREE THAT WE HAVE TO DESCRIBE IT TO THE PATIENT FOR NOW WE ARE IN THE REALM OF DESCRIBING TO PATIENTS HOW THEIR DATA IS SHARED MORE BROADLY AND THAT IS WHAT WE ARE STEPPING INTO. >> ONE THING I WAS GOING TO SAY IS THWE ARE SEE MORE COMPANIES USING USER DESIGN PRINCIPLES FOR COMMUNICATING CONSENT WHICH IS AN INTERESTING TREND; THERE IS A MODEL FOR PEOPLE TO LOOK TO, WHEN I WAS AT AT HHS THERE WAS A MODEL DESIGN FOR PERSONAL HEALTH RECORDS BUT MORE BROADLY FOR CONSUMER FACING AND HEALTH APPS. HOW DO WE MAKE IT SOMETHING THAT IS CONSISTENT AND COMPARABLE ACROSS DIFFERENT TYPES OF PLATFORMS OR USES OR COMPANIES THAT MIGHT BE TRYING TO ACCESS DATA? SO I WANTED TO CALL PEOPLE'S ATTENTION THAT THERE IS THIS PRIVACY NOTICE ON THE OIC (SOUNDS LIKE) WEBSITE, FOR INDIVIDUALS WHO MIGHT BE TRYING TO EVALUATE SOME OF THE PRODUCTS AND SERVICES TO BE ABLE TO SEE WHAT MIGHT BE THE DIFFERENCE IN THOSE DIFFERENT PRIVACY POLICY SO LIKE THE NUTRITION FACTS LABEL DOES NOT TELL YOU HOW MUCH SODIUM YOU CAN HAVE IN A CAN OF SOUP, YOU CAN SEE TWO DIFFERENT CANS AND YOU KNOW THAT THIS ONE HAS 50% MORE THAN THE OTHER. SO IT HELPS YOU WITH THE CONSISTENCY OF THAT COMMUNICATION. >> I THINK THAT TO TEACH EACH CONSUMER ONE INFORMATION WILL BE SHARED COULD BE A GREAT EXAMPLE; IF THEY CAN SEE A SIMPLE ONE-PAGE LABEL -- THERE ARE THINGS THAT ARE MORE CONCERNING THAT ARE GOING TO BE SHARED SO YOU SHOULD PUT THAT ON THE LABEL SO YOU CAN SEE WHAT IS COVERED AND NOT COVERED AND IT WILL SAVE TIME FOR PEOPLE SO INSTEAD OF READING 70 PAGES -- NOBODY HAS TIME TO READ ALL THAT. >> MARK BARNES: THERE IS A COMMENT FROM THE PUBLIC THAT THE NEW COMMON RULE IS A REQUIREMENT ABOUT KEY INFORMATION AND WHY THAT COULD NOT BE CARRIED OVER IS REALLY JUST A DIFFERENT DISTILLATION OF THE POINT YOU GUYS ARE MAKING. >> I WANT TO POINT OUT THE SPECIFIC POINTS OF AI LEARNING; THE VALUE OF THE TECHNOLOGY IS A MACHINE FINE'S CORRELATIONS THAT THE HUMAN WOULD NOT FIND, THE NEEDLE IN THE HAYSTACK. YOU CAN SAY I CONSENT TO SHAREING MY FILTERS, BUT THERE IS NO WAY YOU WOULD KNOW THAT THIS MEANT SHARING YOUR STATUS AS A DEPRESSED PERSON. IT COMES DOWN TO THE VALUES OF THE PEOPLE WHO ARE BUILDING AND MAINTAINING THESE PLATFORMS; DO THEY WANT TO ALLOW TARGETING OF PEOPLE BASED ON THEIR MENTAL HEALTH STATUS? THAT IS A VALUES QUESTION POSED ON INSTAGRAM. THE SOLUTION IS EASY. DISALLOW TARGETING OF PEOPLE ON THE BASIS OF THESE FILTERS IN THE BACK END OF THE ADVERTISING TOOLS. I DON'T THINK ANYBODY HAS EVER LOOKED AT THE ADVERTISING TOOLS BUT IT IS A VERY EASY INTERFACE AND IT CAN BE AUTOMATED. THE TARGETING OF ADS -- WHEN YOU'RE DOING 100,000 UNIQUE ADS OBVIOUSLY IT IS AUTOMATED, SCRIPTED; THEY CHANGE THE USER INTERFACE AND REMOVE THAT BOX. WE'RE IN A SITUATION WHERE YOU CANNOT KNOW WHAT YOU ARE DISCLOSING, IT'S IMPOSSIBLE. YOU CANNOT KNOW THE CORRELATIONS OF WHAT YOU ARE DISCLOSING, NOT LIKE MEDICAL DATA WHERE YOU KNOW ON SHARING THAT RESULTS OF MY BLOOD DRAW, THE MRI, MY BLOOD PRESSURE. THERE IS AN INTENT THERE ON THE INFORMED CONSENT THAT IS NOT PRESENT ON THESE VAST CORRELATION ACTIVITIES. >> THERE IS THAT ADDITIONAL DIMENSION, NOT JUST DATA COLLECTED OVER TIME BUT IT WILL BE REPURPOSED TO INTERFERE WITH YOUR LIFE POTENTIALLY BUT I DO WANT TO ASK. ONE ISSUE IS NOT LETTING SOMEONE, SOME FIRM ENTITY, EDUCATIONAL INSTITUTION AND THE GOVERNMENT IN THE SAME WAY -- PEOPLE DON'T WANT TO BE TARGETED NECESSARILY BY THE GOVERNMENT FOR CERTAIN THINGS LIKE FOR EXAMPLE FOR MEDICATION ADHERENCE FOR TUBERCULOSIS OR HIV MEDICATION OR ANYTHING ELSE SO IT'S NOT ONE THING NOT TO ALLOW THAT BUT WOULD YOU ALSO FOR BUT ARTIFICIAL INTELLIGENCE FROM USING IT TO LEARN SOMETHING NOT ABOUT THE INDIVIDUAL BUT SOMETHING THAT IS GENERALIZABLE? IT'S NOT ABOUT HEALTH INTERVENTION BUT ABOUT THE RESEARCH THAT WOULD NOT TARGET THE INDIVIDUAL FOR INTERVENTION. IT WOULD USE THE INFORMATION THAT WOULD ESSENTIALLY DEVELOP GENERALIZED KNOWLEDGE WHICH COULD BE LATER AT SOME OF THE POINT IN TIME COMPLETELY REMOVED AND BE APPLIED TO SOMEBODY ELSE TO APPLY TO THAT INDIVIDUAL. DO YOU SEE WHAT I'M ASKING? IT'S NOT ABOUT THE INTERVENTION OR THE RESEARCH. >> THE LINE BETWEEN INTERVENTION AND RESEARCH IS INCREDIBLY THIN IN THE SYSTEMS; THE MODEL THAT RESULTS FROM THE RESEARCH IS THE EXACT SAME MODEL THAT YOU WOULD USE FOR THE INTERVENTION WITHIN THE PLATFORM; AND WHEREAS YOU WOULD HAVE HAD TO DO TWO DIFFERENT STUDIES PREVIOUSLY, ONE WOULD BE INTERVENTIONAL, THE OTHER RESEARCH. YOU WOULD DO THE RESEARCH FIRST. THE WAY THESE PLATFORMS WORK IS THERE'S ZERO DISTINCTION BETWEEN THOSE TWO; IT'S THE EXACT SAME SET OF MATHEMATICAL WEIGHTS IN THE PLATFORMS ARE BUILT TO MAKE THAT HAPPEN ALMOST MAGICALLY. AND SO WE ARE WORKING WITH THIS CONCEPTUAL DISTANCE BETWEEN THOSE TWO, NOT SUBSTANTIATED INSIDE THE TECHNICAL SYSTEMS. >> MARK BARNES: LET'S CONTINUE THIS FOR A SECOND AND GIVE YOU AN EXAMPLE OF RESEARCH THAT IS GOING ON THAT IS AI-MOLTIVATED. THERE ARE A NUMBER OF FIRMS LOOKING AT RETINAL SCANS TO DEVELOP ARTIFICIAL INTELLIGENCE TO IDENTIFY DIABETIC RETINOPATHY; AND THAT RESEARCH AS FAR AS I KNOW -- IT'S A LITTLE OF BOTH COMMERCIAL AND RESEARCH AND IT WILL YIELD AT THE END OF THE DAY SOME VERY IMPORTANT INFORMATION EVEN THOUGH THE INDIVIDUALS WHOSE RETINAL SCANS ARE BEING USED ARE NOT GOING TO BE CONTACTED AS PART OF THE RESEARCH; THAT IS AN EXAMPLE OF WHERE THERE IS A DIVISION BETWEEN THE INTERVENTION BETWEEN INDIVIDUALS WHO ARE THE DATA SUBJECTS VERSUS THE DEVELOPMENT OF ULTIMATELY SOMETHING THAT IS A PUBLIC GOOD, I GUESS. >> WHETHER OR NOT IT IS A PUBLIC GOOD DEPENDS ON THE APPLICATION. AS I PASS BY A CAMERA AT THE MALL, IT TEXTS TO SEE WHETHER I HAVE DIABETIC RETINOPATHY. IT'S NOT THAT IMPOSSIBLE; MAYBE COCA-COLA WANTS TO TARGET PEOPLE WHO REALLY LIKE SUGAR. THAT'S A CORRELATION TOO. AND IT'S ULTIMATELY THE RESPONSIBILITY OF THE RESEARCHERS AND THOSE WHO HAVE THIS DISPARATE AMOUNT OF POWER TO MAKE GOOD DECISIONS BECAUSE WE CAN'T RELY ON THOSE SORT OF TEMP ORAL DISTINCTION BETWEEN RESEARCH AND INTERVENTION; WE CANNOT RELY ON THE FACT THAT RESEARCH HAPPENS IN UNIVERSITIES AND WE HAVE CONTROL OVER WHAT PEOPLE AT UNIVERSITIES DO, IT DOES NOT HAPPEN UNIVERSITIES ANYMORE. IT'S NOT THE CASE THE CORPORATIONS DON'T HAVE ACCESS TO THESE TOOLS JUST BECAUSE THEY'RE NOT ACADEMICS. --- THE IMPLICIT DISTINCTION SIMPLY RELY ON TO BE PRICES FOR OUR VALUES DON'T HOLD ANYMORE, SO WE NEED TO BE EXPOSING ABOUT OUR VALUES AND DEMAND THAT OTHER PEOPLE BE SPLIT ABOUT THEIR VALUES AND THEN WE CAN FIGHT ABOUT THE VALUES WHICH IS ANOTHER KIND OF ENGAGEMENT; WE RELIED ON THE EMPHASIS STRUCTURES TO PROTECT US BUT THEY ARE BEING TORN APART BY MACHINE LEARNING TECHNOLOGIES. >> ONE OF THE THINGS WE HAVE DONE WELL IS TO GET PEOPLE LANGUAGE ABOUT WHAT WE PERMIT THEM TO DO WITH THEIR DATA SO HERE IS WHAT, BUT WE HAVE FALLEN SHORT ON TELLING PEOPLE THE WHY, WHY WOULD LIKE YOU WANT TO SHARE THE DATA THIS WAY? OKAY, YOUR DATA IS GOING INTO THIS APP. BUT HERE IS WHY SOMEBODY WANTS TO LOOK AT THIS TO CREATE A NEW RESEARCH INTERVENTION, NOT JUST BECAUSE WE WANT YOU PERMISSION TO PUT YOUR DATA IN AN OPEN ACCESS REPOSITORY FOR PUBLICATION PURPOSES BUT ALSO TO EXPLAIN IT IS REALLY IMPORTANT THAT OTHER PEOPLE CAN VALIDATE IF OUR RESULTS ARE RIGHT AND THAT IS THE REASON WHY WE'RE SHARING THE DATA. THIS IS AN OBLIGATION TO CHECK ON HOW IT IS USED AND WE DON'T DO A GOOD JOB; AND REALLY FALL SHORT ON EDUCATING PEOPLE ABOUT THEIR USE AND WHY WE WANT IT; WHAT ISTHE NATURE OF THE DATA IN TERMS OF THE VALUE ADD OF ALLOWING THE DATA TO BE ACCESSIBLE BUT THAT WOULD TAKE US A LOT FURTHER IN AT LEAST HAVE A CONVERSATION WITH PARTICIPANTS ABOUT HOW WE WILL USE THE DATA IN A BIG WAY BUT HERE IS WHY, NOT JUST CAN YOU GIVE US YOUR PERMISSION TO SHARE THIS WITH OTHER PEOPLE. WE NEED TO PUSH HERSELF LITTLE BIT IN THAT REGARD. >> MARK BARNES: WHAT DO YOU THINK ABOUT -- THREE IS THIS DEBATE ABOUT THE ISSUE OF WHETHER THEY SHOULD BE AN ADAPTIVE INFORMED CONSENT SO THE INDIVIDUAL SHOULD ESSENTIALLY BE RE-CONSENTED PERIODICALLY ON AN ONGOING BASIS ABOUT THE USE OF THE DATA; AND HAVE THE SAME PRINCIPLE APPLIED TO DATA ANONIMIZED OR DEIDENTIFIED. THERE'S A LOT OF DEBATE ABOUT THIS. >> THERE IS SOME FASCINATING ACADEMIC WORK ABOUT THESE CONSENT AGENTS, SO YOU DON'T NEED TO BE BOTHERED -- SERIOUS -- IF YOU'RE ASKED 1000 TIMES A DAY IF YOU CONSENT TO THAT MICRO USE OF YOUR DATA -- SO MACHINE LEARNING IS THE FUTURE. I'M BEING VERY SERIOUS; THAT IS THE FUTURE. --- >> MARK BARNES: THERE'S A N UMBER OF COMMENTS ABOUT HTIS. IN A CLINICAL SETTING PATIENT'S HEALTH DATA IS ALREADY BEING SHARED WITH OUTSIDE ENTITIES; THAT IS NOT EXACTLY TRUE. HOW IS THAT DIFFERENT FROM RESEARCH USING BIG DATA? IN THAT I WILL ASK A SECOND QUESTION. FOR THE EXAMPLE OF INSTAGRAM AND DEPRESSION DOESN'T THE PERSON HAVE TO USE THE ANALYSIS (INCOMPREHENSIBLE) -- WHAT JURISDICTIONAL CATEGORY IT FALLS INTO? IS IT DIFFERENT THAT INSURANCE COMPANIES AND ACCESS ALL OF THE DATA AND THEIR OWN RESEARCH WITH ALL OF THE STUFF? HOW IS THIS ANY DIFFERENT THAN THE USE FOR ACADEMIC OR COMMERCIAL DEVELOPMENT PURPOSES? IN FACT, INSURANCE COMPANIES ARE REFINING THEIR INSURANCE PRODUCTS USING ALL THE CLAIM DATA THEY GET AND ALL THE DIAGNOSTIC DATA TO GET ALL THE TIME WHICH IS THE WAY TO GET MONEY. >> WE SEE THIS QUESTION A LOT ESPECIALLY IN THE COMMUNITY I WORK WITH. I THINK A LOT OF THE DISCUSSION IS TO CENTER BACK AROUND THE VALUE OF CONCEPTUAL INTEGRITY. WHAT IS THE PURPOSE OF THIS INTENDED USE? IS IT ALIGNED WITH THIS SORT OF BROADER SOCIAL POLITICAL VALUES TO WHY THE DATA IS COLLECTED OR SHARED IN THE FIRST PLACE? SO YOU CAN JUSTIFY AN INSURANCE COMPANY HAVING ACCESS BECAUSE YOU KNOW THAT THIS IS ENHANCED HEALTHCARE BUT IF IT IS SOME OTHER THIRD-PARTY THAT WANTS TO DO IT BECAUSE THEY WANT TO TARGET AN ADD TO YOU THAT YOU TRIGGER SOME ADDITIONAL REFLECTION ON THAT. >> MARK BARNES: LET ME THROW THIS OUT FOR ANY COMMENT. THIS IS ABOUT THE DISTINCTION BETWEEN THESE WORLDS IN COLLISION ABOUT ACADEMIA VERSUS COMMERCIAL AND GOVERNMENT; THERE'S A VERY EMINENT SOCIAL SCIENTIST AT HARVARD NAMED GARY KING WHO HAS DONE A LOT OF WORK IN BIG DATA RESEARCH ONE OF THE MOST EMINENT PEOPLE INTO THE RESEARCH. HE SAYS, TRADITIONALLY WE SOCIAL SCIENTISTS ARE LIGHT ON THE CENSUS STUFF AND RELY ON REUSING DATA COLLECTED BY OUR OWN COLLEAGUES AT OTHER ACADEMIC INSTITUTIONS THAT DATA IS NOT DWARFED BY THE MASSES OF DATA COLLECTED FOR COMMERCIAL PURPOSES AND THEY CROSS INTERNATIONAL BOUNDARIES ALL THE TIME AND THEY ARE NOT NECESSARILY EVEN CORPORATIONS HEADQUARTERED IN THE US, THEY ARE EVERYWHERE. HE SAYS HOW CAN WE CONTINUE TO THE RELEVANT RESEARCH WITHOUT HAVING ACCESS AS ACADEMIC RESEARCHERS TO THESE COMMERCIAL DATABASES? BUT THEN THE QUESTION BECOMES, THIS IS ME OPINING, HOW DO WE AS ACADEMIC RESEARCHERS INSINUATE OURSELVES INTO COMMERCIAL FIRMS TO ALLOW THEM FOR US TO USE THEIR COMMERCIAL DATABASES FOR OUR PURPOSE IS TO DERIVE GENERALIZABLE KNOWLEDGE FOR ACADEMIC PURPOSES AND THAT COMES WITH A LOT OF -- THOSE OF US WHO NEGOTIATE ON BEHALF OF UNIVERSITIES IN TRYING TO ACQUIRE DATA SETS FROM INDUSTRY WE END UP WITH SOME VERY DIFFICULT CHOICES BECAUSE INDUSTRY WANTS THINGS LIKE-- THEY'RE NOT REQUIRED TO GIVE OTHER DAY AT ALL. IT IS PURELY THEIR DECISION ABOUT WHETHER THEY WANT TO GIVE A DATA OR NOT SO THEY SAY THINGS LIKE -- WE WANT TO SEE EVERYTHING YOU PUBLISH IT FOR YOU PUBLISH IT AND WE WANT TO COMMENT ON IT AND THEN WE WANT YOU TO BE FORCED TO TAKE OUR COMMENTS OR WE WANT AN ARBITRATION MECHANISM ABOUT OUR COMMENTS OR YOU CAN USE OUR DATA BUT YOU CAN'T THEN REUSE THE DATA FOR ANYTHING EXCEPT THE NARROW PURPOSE FOR WHICH WE ALLOW YOU TO USE THE DATA. OR THEY WANT INTELLECTUAL PROPERTY AND WHATEVER ALGORITHMS YOU MIGHT DEVELOP AS A RESEARCHER BUT THE POINT IS WE ARE FACED AS ACADEMICS ASKING THEIR OWN UNIVERSITIES AND ACADEMIC MEDICAL CENTERS WERE NOT-FOR-PROFIT TO INTERFACE WITH INDUSTRIES FOR DATABASES THAT ARE MEANINGFUL AND SOMETIMES THESE INDUSTRIES ARE TRANSITIVE EXTRACT PRIVATE INFORMATION THAT WE WOULD NOT TOLERATE EVEN FROM A PLACE LIKE CMS. I WONDER IF YOU'RE SEEING THAT. >> SIMSON GARFINKEL: HAVING NEGOTIATED THE OTHER SIDE OF THAT ONCE OR TWICE THE CORPORATIONS ALSO AFRAID THAT THE RESEARCHER WILL EKE INFORMATION AND EMBARRASS THE CORPORATION AND THIS IS A REASON WHY SOME COMPANIES ARE REALLY EXCITED ABOUT DIFFERENTIAL PRIVACY BECAUSE IT GIVES THEM A MATHEMATICAL GUARANTEE THAT THEY ARE NOT GOING TO GET RAKED OVER THE COALS BY AN ACADEMIC RESEARCHER WHO IS CARELESS; THIS IS WHY WE NEED TO MOVE TO MATH-BASE PROTECTIONS OVER LEGAL BASE PROTECTIONS. WE NEED TO MOVE THE THINGS THAT ARE MATHEMATICALLY PROVABLE BECAUSE MATH DOES NOT MAKE MISTAKES THE WAY HUMANS DO. >> MARK BARNES: OFTEN CALLED "CONTROL BY DESIGN," RATHER THAN CONTROL BY AFTER RULES. WE WILL ADJOURN FOR LUNCH. YVONNE? >> YVONNE: THANK YOU VERY MUCH EVERYBODY FOR YOUR TWO GREAT SESSIONS. I'LL PASS THE MIC TO OUR MODERATOR THIS AFTERNOON, DR. ELIZABETH BUCHANAN. >> GOOD AFTERNOON, EVERYONE. HOPEFULLY YOU HAVE HAD A GOOD LUNCH AND YOU ARE READY FOR SESSION 3 WHICH IS ENTITLED, PROTECTING PRIVACY AND CONFIDENTIALITY, A SHARED RESPONSIBILITY. I'M ENDOWED CHAIR ETHICS AT UNIVERSITY OF WISCONSIN AND IT'S MY PLEASURE AND PRIVILEGE TO MODERATE THE SESSION TODAY. A NUMBER OF THE PANEL ARE FRIENDS AND COLLEAGUES AND I'M LOOKING FORWARD TO HEARING THEIR THOUGHTS TODAY. I WILL AS OTHERS HAVE DONE, I WILL READ A BIT FROM THE BIOS FROM THE BOOK BUT THEY ARE ALSO SO LONG AND DETAILED IT WILL TAKE ALL OF OUR TIME. SO OUR FIRST SPEAKER WILL BE MICHAEL ZIMMER AND MICHAEL IS A PRIVACY AND INTERNET ETHICS SCHOLAR WHOSE WORK FOCUSES ON DIGITAL PRIVACY, INTERNET RESEARCH ETHICS AND THE ETHICAL DIMENSIONS OF SOCIAL MOBILE AN INTERNET TECHNOLOGIES. MICHAEL IS CURRENTLY AN ASSOCIATE PROFESSOR AT MARKET UNIVERSITY, MY FIRST TIME SAYING THIS, HE IS AT MARQUETTE UNIVERSITY IN THE DEPARTMENT OF COMPUTER SCIENCE AND ALSO CO-DIRECTOR OF DATA SCIENCE PROGRAM THERE, WE WILL HEAR MORE ABOUT THAT LATER. MICHAEL IS EXTREMELY WELL PUBLISHED AND CURRENTLY ON THE PERVADE PROJECT WITH JAKE AS WE HEARD. MICHAEL HAS BEEN PUBLISHED AND CITED IN SUCH PUBLICATIONS AS THE NEW YORK TIMES, WALL STREET JOURNAL, THE ATLANTIC AND MY PERSONAL FAVORITE GQ. AND HIS RESEARCH HAS BEEN SUPPORTED BY THE NATIONAL SCIENCE FOUNDATION, THE INSTITUTE OF MUSEUM LIBRARY SERVICES AND AMERICAN LIBRARY ASSOCIATION. HE'S A REALLY NICE GUY. THE SECOND SPEAKER ALREADY ADARSH GUPTA FROM MY NECK OF THE WOODS, NEW JERSEY. DR. GUPTA IS AN AOA BOARD CERTIFIED PHYSICIAN IN SOUTHERN NEW JERSEY. HE IS CHAIRMAN OF THE IRB AT ROWN UNIVERSITY SCHOOL OF OSTEOPATHIC MEDICINE AND PROFESSOR OF FAMILY MEDICINE AT ROWEN SOM. HIS BIO IS LONG, A FEW POINTS. HE IS FOUNDING DIRECTOR OF THE CENTER FOR MEDICAL WEIGHT LOSS AND MET POLLIC CONTROL THAT FOCUSES ON IMPROVING METABOLIC DISORDER. AND FUNCTIONAL DISORDERS. OUR THIRD SPEAKER WILL BE MY DEAR FRIEND MEGAN KASIMATIS-SINGLETON. MEGAN IS ASSISTANT DEAN FOR HUMAN RESEARCH PROTECTIONS AND DIRECTOR OF THE HUMAN RESEARCH PROTECTION PROGRAM AT JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE. IN THIS ROLE SHE IS RESPONSIBLE FOR OVERSIGHT AN DIRECTION OF JOHNS HOPKINS SEVEN IRBs. MS. SINGLETON IS A LICENSED ATTORNEY IN PENNSYLVANIA AND SHE EARNED LAW DEGREES FROM TEMPLE AND MASTERS IN BIOETHICS FROM UNIVERSITY OF PENNSYLVANIA. SHE'S ALSO ACTIVE IN S IRB INITIATIVE IN ADDITION TO HER ROLE AS T IRBC CI RB LEAD, SHE IS HEAVILY ENGAGED IN NATIONAL EFFORTS TO STREAMLINE IMPLEMENTATION OF SI IRB REVIEW PROCESSES INCLUDING ADS MEMBER OF THE FEDERAL DEMONSTRATION PROJECT TASK FORCE ON S IRB AND AS MEMBER OF THE SMART IRB HARMONIZATION STEERING COMMITTEE. I DON'T KNOW IF I SHOULD READ YOUR BIOS AGAIN JAKE AND BRENDA. BUT TO REMIND IF YOU WEREN'T WITH US IN THE MORNING OUR FOURTH SPEAKER WILL BE BRENDA LEONG AND BRENDA IS SENIOR FUTURE OF PRIVACY FORUM. AND FINALLY OUR LAST SPEAKER IS JAKE METCALF, A Ph.D. AT THE DATA AND SOCIETY RESEARCH INSTITUTE. SO WITH THAT, I WILL TURN THE PODIUM OVER TO MICHAEL AND LOOK FORWARD TO HIS PRESENTATION. >> THANK YOU, ELIZABETH. THE GQ WAS ARTICLE ZUCKERBERG. I WAS JUST A MINOR PLAYER. I'M EXCITED TO BE HERE TODAY FRIENDING RESEARCH AS PART OF THE PERVADE PROJECT MENTIONED THIS MORN,, IT'S A MULTI-DISCIPLINARY TEAM TRYING TO UNDERSTAND AND ENGAGE IN THESE ISSUES OF BIG DATA ETHICS IN A VARIETY OF PERSPECTIVES. I WAS ASKED TO TALK ABOUT IRBs AND HOW IT FITS INTO THIS PICTURE. I HAVE SOME DATA TO PRESENT SHORTLY REGARDING THAT. TO RECAP A LITTLE BIT OF THIS MORNING FEEDS WELL INTO HOW I STARTED MY PRESENTATION. A LOT OF WHAT WE ARE SEEING HAPPENING TODAY IN THIS SPACE OF BIG DATA ETHICS AND BIG DATA RESEARCH IS CREATING CONFUSION, GAPS, SHIFTS IN UNDERSTANDING OF FUNDAMENTAL CORE PRINCIPLES AROUND RESEARCH ETHICS. THIS IS A CHALLENGE WE HAVE BEEN DISCUSSING AND FACING TODAY AND ALSO MENTION THIS MORNING WAS ALSO NEW CHALLENGES WITH BIG DATA RESEARCH ESPECIALLY BECAUSE THE TOOLS TO ENGAGE IN THESE PROJECTS ARE QUITE EASY TO GET YOUR HANDS ON. IF I ASK ONE OF MY UNDERGRADS TO GIVE ME FIVE MILLION TWEETS, THEY CAN HAVE THEM FOR ME BEFORE I GET HOME TONIGHT. IT'S A DIFFERENT RESEARCH ENVIRONMENT THAN 20 YEARS AGO WHEN DEALING WITH RESEARCH ETHICS WITH LARGE I DON'T THINKIC DATA SETS SO WE ARE OFTEN SEEING A LOT OF SCIENTISTS AND RESEARCHERS AND RESEARCH COMMUNITIES THAT DON'T HAVE A TRADITION OF DEALING WITH HUMAN SUBJECTS AS DEFINED BYING ARELATIONS. I'M NOW DEPARTMENT OF COMPUTER SCIENCE AND I'M WORKING WITH COMPUTER SCIENTISTS AND DATA SCIENTISTS WHO DON'T HAVE A LONG HISTORY OF DEALING WITH IRBs OR UNDERSTANDING THERE IS A HUMAN BEING ATTACHED TO THIS PIECE OF DATA THEY ARE ANALYZING. THAT'S WHAT'S MOTIVATING A LOT OF WORK AND PART OF THE REASON WE GOT TOGETHER WITH THIS PERVADE PROJECT TO TACKLE THESE CHALLENGES. THE SET OF RESEARCH THAT I WILL TALK ABOUT TODAY IS LOOKING SPECIFICALLY AT IRBs. WE ARE TRYING TO UNDERSTAND HOW AREIs THINKING ABOUT PERVASIVE DATA, HOW TO REVIEW PROTOCOLS THAT COME THROUGH THAT USE THIS DATA. ARE THEY ADEQUATELY PREPARED TO MANAGE THESE KINDS OF RESEARCH PROJECTS. SO WE ARE TRYING TO GET SOME BASIC UNDERSTANDING IT IS OF THOSE VARIABLES. NOW, THERE HAS BEEN EXISTING REALLY GOOD WORK TO BUILD ON. ELIZABETH BUCHANAN AND CHARLES DID ONE OF FIRST RESEARCH STUDIES TEN YEARS AGO TRYING TO UNDERSTAND HOWIs THOUGHT ABOUT INTERNET RESEARCH. THIS WAS A STAGE WHEN IT WAS STARTING TO EMERGE WITH PEOPLE STUDYING -- STARTING ONLINE FORUMS AND CHAT LOGS AND OTHER DIGITAL ENVIRONMENTS, AND THEY FOUND THAT LESS THAN HALF THE IRB MEMBERS THEY TALKED TO AT THAT TIME FELT INTERNET RESEARCH WAS AREA OF CONCERN OR IMPORTANCE. THIS WAS EARLY STAGE IN DEVELOPMENT OF THIS RESEARCH DOMAIN AND MANY IRBs WERE -- WERE NOT UP PHOTO SPEED YET, THERE'S INTERESTING ETHICAL DIMENSIONS IN THIS SPACE. SIMILARLY THEY FOUND ONLY 6% OF RESPONDENTS HAD GUIDELINES OR TOOL TO USE TO DEAL WITH THIS DOMAIN, SO THAT WAS VERY EARLY STAGE IN THIS NEW RICH RESEARCH ENVIRONMENT AROUND INTERNET BASED RESEARCH PROJECTS. MORE RECENTLY, SOME OF OUR COLLEAGUES STUDIED IRBs LOOKING AT ONLINE RESEARCH AND ASKING FOLLOW-UP QUESTIONS BUILDING ON WHAT BUCHANAN AND THEY FOUND HIGHER LEVEL OF RESPONSES WHO INDICATED ON LIEN RESEARCH IS SOMETHING TO WORRY ABOUT. 93% INDICATED THAT. YET ONLY HALF FELT COMFORTABLE& TO DEAL WITH THOSE CHALLENGES. SO THIS SHOWS THAT THERE STILL IS A GAP, THERE'S STILL PLACES FOR US TO BUILD OUT UNDERSTANDINGS AND COMFORT LEVELS AND KNOWLEDGE WHEN WE RELATE -- IT RELATES TO ONLINE RESEARCH. WHAT'S INTERESTING TO US AND JAKE DID A GOOD JOB OF OUTLINING THIS THIS MORNING, WE ARE CONCERNED NOT JUST WITH TRADITIONAL ONLINE RESEARCH, THE FACT THAT I CAN STUDY TRADITIONAL ONLINE RESEARCH IS KIND OF A NEW THING THAT I THINK JUST NOW WE CAN SAY THAT. WE ARE INTERESTED IN PERVASIVE DATA, INTERESTED IN THIS SHIFT TOWARD DIFFERENT TYPES OF DATA SETS COLLECTED IN DIFFERENT KINDS OF WAYS. AND SO WE ARE TRYING TO UNDERSTAND HOW AREIs THINK ABOUT THIS NEW DOMAIN OF RESEARCH THEY ARE INCREASINGLY FACED WITH. SO WE PUT OUT A SURVEY, JUST EARLIER THIS YEAR, MOST RESPONDENTS WERE FROM UNIVERSITY IRB AND MAJORITY WERE R 1 INSTITUTIONS TO GIVE A SENSE OF RESPONSE RATES. I HAVE HIGH LEVEL DATA TO SHARE, WE ARE STILL IN THE PROCESS RIGHT NOW OF GOING THROUGH THESE RESULTS BUT FINDING INTERESTING THINGS TO TALK ABOUT. I THINK IT HELPS FRAME THE REST OF OUR CONVERSATION ON THIS PANEL IN TERMS HOW ARE THESE IRBs FEELING PREPARED, HOW ARE THEY LACKING AT POSSIBLE RESEARCH PROTOCOLS. SO IN TERMS OF RESPONSES THAT WE HAVE, WE WILL SEE MAJORITY OF PEOPLE HOW OFTEN DO WE SEE PROTOCOLS THAT INVOLVE PERVASIVE DATA. ONLY HALF SAW ONE TO TEN PROTOCOLS A YEAR, MAYBE ONCE A MONTH GET A PROTOCOL THEY SAW USES PERVASIVE DATA AS WE DEFINED IT. BUT ABOUT A THIRD, 34% SAID THEY HAD ONE A WEEK. THEY HAD 50 OR MORE PROTOCOLS THEY WERE SEEING COMING THROUGH SO WE HAVE RANGE OF EXPOSURE TO THESE ISSUES. THAT'S A DIFFERENCE IN THE DATA THAT WE HAVEN'T PARSED OUT YET TO SEE HOW RESPONSES FUTURE SLIDES I'M GOING TO SHOW YOU MIGHT DIFFERENTIATE. IT'S INTERESTING TO SEE WE WILL SEE UNEVEN WAY AND WE SAW THIS WITH INTERNET RESEARCH, WE SAW THIS WITH OTHER DOMAINS DEPENDING ON TYPE OF INSTITUTION YOU HAVE, HOW BIG YOU ARE, OTHER FACTORS YOU MIGHT SEE MORE OR LESS OF THESE PROTOCOLS. WE ASKED IRB MEMBERS IF THEY FELT WELL VERSED IN TECHNICAL ASPECTS OF PERVASIVE DATA. THIS IS A SIMILAR QUESTION THAT BUCHANAN ASKED US BACK IN 2009. AND HERE WE WILL SEE ABOUT 25% AGREE OR SOMEWHAT AGREE THEIR IRB HAD SUFFICIENT TECHNICAL TO ENGAGE PERVASIVE DATA AND USE THESE KINDS OF PROTOCOLS. THAT IS A DROP ALREADY FROM JUST A YEAR OR SO AGO WHEN WE WERE ASKING IRBs ABOUT ONLINE DATA, ONLINE RESEARCH IN MORE GENERAL SENSE, HALF FELT LIKE THEY HAD A GOOD GRASP OF ONLINE RESEARCH BUT WHEN YOU START INTRODUCING PERVASIVE DATA AND NEW PROTOCOLS AND DATA SOURCE, THAT TECHNICAL UNDERSTANDING DROPS SO, THERE IS GAP THERE FOR US TO FILL TO MAKE SURE IRBs UNDERSTAND WHAT THESE PROTOCOLS MEAN. SIMILARLY WE ASK WHETHER OR NOT THEY UNDERSTOOD THE ETHICAL ASPECTS OF THESE KINDS OF DATA. SO SOME OF THE THINGS TALKING THIS MORNING SOME UNIQUE ETHICAL ISSUES AROUND PERVASIVE RESEARCH AND AGAIN AT ABOUT 50% FEEL THEY UNDERSTAND THE ETHICAL DIMENSIONS. SO THEY MAY NOT UNDERSTAND ALL THE TECHNICAL ASPECT BUS THEY DO UNDERSTAND SOMETHING ETHICALLY INTERESTING GOING ON HERE. SO AGAIN, THERE'S GAP THERE, WE NEED TO BRING -- ALIGN THESE TWO DIFFERENT CONCEPTS. FOLLOW THAT QUESTION ABOUT GUIDELINES. DO YOU HAVE SPECIFIC GUIDELINES. BUCHANAN ASKED IN THEIR SURVEY IN 2007, 2008, 6% RESPONSENESS INDICATED THEY HAD SOME KIND OF GUIDELINES TO DEAL WITH ONLINE RESEARCH. I THINK WE MOVED THAT SPORED IN THE ONLINE SPACE, BUT PERVASIVE DATA WE ARE BACK AT THAT STARTING POINT, WE ARE ONLY 6% OF OUR RESPONSES INDICATED THAT THEY HAD SOME KIND OF PROCESS, SOME KIND OF DOCUMENT THAT THEY& USE FOR LOOKING AT PERVASIVE DATA PROTOCOLS. THAT IS A GOOD OPPORTUNITY FOR US TO COME FILL THAT VOID. FOR THOSE THAT DID TALK ABOUT WHAT THEY USE WHEN LOOKING AT PERVASIVE DATA, IT SHOULD BE NO SURPRISE THEY ARE LOOKING AT THE COMMON RULE IN THE ONLINE REPORT. ALMOST HALF ARE LOOKING AT RECOMMENDATIONS THAT THE SACCARP ADVISORY COMMITTEE PUT OUT COUPLE OF YEARS AGO. I FOUND IT A LITTLE SURPRISING THAT SPECIFIC INTERNET RELATED GUIDELINES THE AMERICAN PSYCHOLOGICAL ASSOCIATION AND SHEESHIATION OF ENTERIN THE RESEARCHERS THAT ELIZABETH AND I BOTH HAVE BEEN HEAVILY INVOLVED IN ARE RELIED ON MUCH. WE ARE ASKING ABOUT PERVASIVE DATA SETS AND SOME GUIDELINES DON'T SPEAK TO THAT UNIQUE SPACE YET. BUT THERE'S CLEARLY SOME -- WE NEED TO FIND ADDITIONAL WAYS TO HELP THESE IRBs GRAPPLE WITH THESE ISSUES. WE FOLLOWED UP TWO OTHER QUESTIONS BEING ASKED BRIEFIUS SURVEYS TO SEE HOW THEY COMPARE, WE ASK ABOUT RESEARCHERS ON THEIR CAMPUS, ARE THEY RECEIVING TRAINING TO DEAL WITH PERVASIVE DATA AND DEAL WITH THESE KIND OF THINGS. WE HAVE 25% INDICATED EITHER ACQUIRE OPTIONAL SOME RESEARCHER TRAINING ON THEIR CAMPUS, WHETHER RCR TRAINING IN THE GENERAL SENSE OR SOMETHING SPECIFIC ABOUT DOING WITH DATA OR DATA MANAGEMENT PLANS, WE DIDN'T GET INTO THOSE FOLLOW-UPS YET. THAT IS NOD TOO BAD. WE HAVE 65% THAT INDICATE NOTHING IS THERE. SO THAT IS ALSO A GAP FOR US TO FILL. THEN WE ASKED ABOUT IRB MEMBERS THEMSELES AND DO IRB MEMBERS RECEIVE ANY SPECIFIC TRAINING ABOUT PERVASIVE DATA, THIS KIND OF COLLECTION, THIS KIND OF DATA USE. THIS IS ABOUT THE SAME AS BACK WITH THE BUCHANAN AND S RESEARCH FROM 2007. WHERE WE HAVE 37% WHO INDICATED THERE IS SOME KIND OF TRAINING IN PLACE FOR IRB MEMBERS BUT AGAIN, 70% SAYING THERE'S NOT SO AGAIN, A LARGE GAP FOR US TO TRY TO FILL AS THESE PROTOCOLS BECOME MORE FREQUENT. THEN WE DID THE FUN PART. STARTED CREATING SOME HYPOTHETICAL PROTOCOLS. OUR TEAM WENT THROUGH A BUNCH OF NSF ABSTRACTS AND WE STARTED GATHERING SOME DATA AMONG OURSELVES. INTERESTING EDGE CASES WE CAN PRESENT TO IRBs AND TRY TO GET THEIR OPINIONS ON. AND WE CREATED DIFFERENT VARIABLES ACROSS THESE PROTOCOLSMENT SOME OF THEM, THERE WAS HIGH LEVEL CONSENT PUT IN PLACE SO WE HAVE A LITTLE GREEN LIGHT FOR CONSENT AND OTHER PROTOCOLS, LITTLE CONSENT ATTEMPTED BY SUBJECTS. SOME OF THE DATA WAS CONSIDERED PUBLIC LIKE A TWITTER FEED, OTHER PIECES OF DATA WERE NOT PUBLIC, MAYBE IN SOME KIND OF HEALTH FORUM THAT REQUIRES A SIGN IN TO GET ACCESS TO. AND AGAIN, DIFFERENCE ANONYMITY AND ALSO DIFFERENCE INTRODUCED WHETHER OR NOT RESEARCHER WAS ADHERING TO TERMS OF SERVICE ON PLATFORM THEY ARE DOING. IS IT OKAY TO SCRAPE A LINKEDN PAGE TO GET RESEARCH DATA. SO WHEN WE PRESENTED A NUMBER OF THESE HYPOTHETICALS I WILL SHARE A FEW WITH YOU TODAY. FOR A LARGE NUMBER THERE WAS GENERAL AGREEMENT AMONGIRB MEMBER WHOSE RESPOND TO OUR SURVEY HOW THEIR IRB WOULD VIEW THIS PROTOCOL. IF YOU HAD THIS LITTLE ABSTRACT WOULD YOU THINK IT WAS KNOP HUMAN SUBJECTS RESEARCH, WAS IT EXEMPT, EXPEDITED, REQUIRE FULL REVIEW? AND IN MOTH CASES THEY FOCUS IN ON THE PUBLICNESS OF THE DATA BEING THE KEY INDICATOR. IF THE DATA WAS PUBLISH WE GENERALLY -- THEY WERE GENERALLY PUTTING THAT IN NON-HUMAN SUBJECTS RESEARCH OR EXEMPT STATUS. ONLY IF THE INFERENCE BEING MADE WAS RELATED TO WHAT I WOULD CHARACTERIZE AS SENSITIVE TOPIC ONE MENTAL STATE, DID THAT TRIGGER ADDITIONAL REVIEW IN THEIR MINDS. SO I'LL GIVE A COUPLE OF EXAMPLES WHAT THESE LOOK LIKE. THIS IS A SIMPLE EXAMPLE, WE ARE STUDYING A TWITTER ARCHIVE THAT SOMEBODY COLLECTED AND SHARING THAT WITH ANOTHER RESEARCHER TO UNDERSTAND PEOPLE'S -- I'M SORRY, THE UNDERSTAND TWEETS ABOUT POLITICAL EVENT. THEY ARE NOT GETTING CONSENT BECAUSE THE DATA WAS COLLECTED BY SOMEONE ELSE AND WE ARE USING THIS DATA, DATA IS PUBLIC, THE IT IS ANONYMOUS BECAUSE THEY TOOK MEASURES TO DEIDENTIFY. PERHAPS NOT SURPRISINGLY, MOST RESPONSES SAID THIS IS EITHER NOT HUMAN SUBJECTS RESEARCH OR WAS EXEMPT. I AM NOT TRYING TO SAY THAT'S THE CORRECT DETERMINATION. I'M JUST SHOWING THERE WAS ALIGNMENT HOW THE RESPONDENTS LOOK AD THESE PROTOCOLS. PUBLIC NEWSPAPER COMMENTS TO UNDERSTAND POLITICAL VIEWS. THEY ARE NOT GOING TO GET A CONSENT, COMMENTS ARE PUBLIC ON NEWSPAPER WEBSITE, NOT GOING TO AN NONMIZE, IF YOU HAVE NAME AND USER ACCOUNT IN PUBLIC COMMENT IT SHOWS UP ON DATA SET WITH USER ACCOUNT. AND ALIGNS WITH TERMS OF SERVICE. MOST SAID IT WAS NOT HUMAN SUBJECTS OR WAS EXEMPT. SOME MORE INTERESTING EXAMPLES, LET'S SAY I HAVE SOME MENTAL HEALTH RECORDS THAT THE UNIVERSITY KEEPS AND I HAVE THE SOCIAL MEDIA PROFILES OF SOME STUDENTS, I WANT TO COMBINE THIS DATA TO TRY TO UNDERSTAND THE WELL BEING OUR STUDENT BODY. WE ARE GOING TO CONSENT THOSE STUDENTS INTO THIS. THE DATA IS QUASI PUBLIC MAYBE SOCIAL MEDIA STUFF IS PUBLIC AND THEY WON'T ANONYMIZE BECAUSE THE POINT IS TO UNDERSTAND STUDENTS AND PERHAPS HAVE INTERVENTION. ALL THE RESPONDENTS SAID THIS WILL BE EXPEDITED OR FULL REVIEW. SO YOU CAN SEE WE CAME UP WITH INTERESTING HYPOTHETICALS AND PRESENTED DIFFERENT EXAMPLES. BUT I WANT TO GET TO SOME MESSY ONES IN A MINUTE. WE DID HAVE SOME, THEY WERE A LITTLE MORE MESSY. THIS OFTEN CAME DOWN TO THIS QUESTION OF SENSITIVITY. IN TERMS OF WHAT WE ARE DOING. THERE SEEMS TO BE CONFUSION ABOUT HOW IMPORTANT WHETHER THE DATA IS PUBLIC OR NOT, COMES INTO PLAY, THEN ACTUALLY THE TERMS OF SERVICE BECAME KEY BECAUSE PART OF THE SURVEY WE ASKED OUR RESPONDENTS WHEN THEY CHOSE FOUR LEVELS WE ASKED WHICH FACTOR CONTRIBUTED TO YOU MAKING THAT DECISION. SO HERE WE HAVE SCRAPING PUBLIC TWITTER FEEDS, SIMILAR TO AN EARLIER ONE LOOKING AT TWITTER FEEDS. AND WE ARE TRYING TO PREDICT RISKY DRUG USE BEHAVIOR. WE ARE NOT GOING TO GET CONSENT BUT IT IS PUBLIC, IT MIGHT BE -- WE ARE GOING TO DEIDENTIFY, AND WE ARE FOLLOWING THE TERMS OF SERVICE BUT WE NOW HAD A BIGGER SPREAD ON HOW PEOPLE RESPONDED TO THIS. PUBLIC TWITTER DATA TO LOOK AT POLITICAL VIEWS, EVERYONE SAID WAS NOT HUMAN SUBJECTS OR EXEMPT, NOW WE INTRODUCED RISKY DRUG BEHAVIOR PART OF WHAT WE WERE LOOKING FOR AND THAT CREATED A LITTLE BIT MORE CONFUSION AMONG OUR IRB RESPONSES ON WHERE THIS MIGHT FALL WITHIN IRB. SOME OTHER EXAMPLES, SCRAPING A PUBLIC HEALTH FORUM, WEBSITE, COMBINE WITH TWITTER DATA TO PREDICT ONE'S MENTAL HEALTH. THE DATA IS PUBLIC. AND THIS HYPOTHETICAL THAT DATA IS ACCESSIBLE AND IN THE OPEN. THEY ARE NOT GOING TO GET CONSENT, NOT GOING TO TRY TO DEIDENTIFY AT ALL AND THEN THEY ARE FOLLOWING THE TERMS OF SERVICE. WE HAD A LARGE DISAGREEMENT ON WHETHER OR NOT THIS WAS EVEN HUMAN SUBJECTS RESEARCH VERSUS NEEDING FULL BOARD REVIEW. SO WE ARE SEEING COMPLICATIONS COMING INTO THESE PROTOCOLS CAUSING SOME INTERESTK HICCUPS. I WILL SHOW ONE MORE, SCRAPING PUBLIC NEWSPAPER COMMENTS TO PREDICT ELECTIONS. PUBLIC NEWSPAPER COMMENTS, IT NECESSARY A FEED ONLINE, JUST A STREAM ON NEW YORK TIMES. BUT IN THIS WE ARE VIOLATING THE TERMS OF SERVICE. THIS IS A WEBSITE THAT SAID YOU CANNOT SCRAPE OUR COMMENT THREAD BUT THE RESEARCHERS WANT TO. WE SAW SOME PEOPLE SAY NOT HUMAN SUBJECTS RESEARCH, OTHERS SAY YOU ARE VIOLATING TERMS OF SERVICE, WE HAVE TOCAL CAMPUS ATTORNEY TO FIGURE OUT ARE WE PUTTING THE CAMPUS AT RISK. THE POINT BEING, THERE IS A LOT OF VARIANCE SHOWING UP ON SOME OF THESE COMPLEX INTERESTING BUT NOT OUTRAGEOUSLY UNLIKELY RESEARCH SCENARIOS THAT USE PERVASIVE DATA. SO COUPLE OF TAKE AWAYS, WE HAVE SOME GAPS THAT WE NEED TO FILL. WE HAVE SOME AWARENESS GAPS, EDUCATION GAPS, FOR US TO HELP GET IRBs UP TO SPEED, HELP GIVE THEM THE RIGHT KIND OF TOOLS TO DEAL WITH THIS AND WE WILL HAVE TO START THINKING ABOUT COMING UP WITH SOME BEST PRACTICES MAY NOT BE THE RIGHT APPROACH BUT WHAT ARE PRINCIPLES, WHAT ARE KEY THINGS FOR US TO BE THINKING ABOUT THAT IF YOU SEE THESE ALIGNING IN A PROTOCOL YOU MIGHT HAVE TO BE THINKING ABOUT THIS TYPE OF REVIEW VERSUS THAT TYPE OF REVIEW. THAT IS SOME OF THE WORK WE TRYING TO DO WITH OUR PROJECT. I WILL STOP HERE. WE ARE TRYING TO FIND GET MORE DATA GATHER MORE DATA BRING TOGETHER STAKEHOLDERS AND COME OUT WITH THESEKINE OF RECOMMENDATIONS IS. WE ARE HOPING THAT BRINGING IN DATA FROM MULTIPLE VIEWPOINTS NOT JUS IRBs BUT ACTUAL USERS WHOSE DATA ARE BEING COLLECTED, RESEARCHERS, THE COMPANIES THAT RUN THE PLATFORMS AND WE HOPEFULLY COME UP WITH A TOOL KIT WITH A SET OF GUIDELINES TO HELP MAKE MORE THE -- MAKE THE CONFUSION ON THESE SLICE DISAPPEAR. THANK YOU. >> GOOD AFTERNOON EVERYONE. SEEMS WE HAD A GOOD DISCUSSION ON BIG DATA WHERE WE ARE LOOKING AT THE DATA WE COLLECTED FROM MULTIPLE SOURCES AND USED TO UNDERSTAND BEHAVIOR. THE TOPIC I FOCUS ON IS DATA COLLECTED BY RESEARCHER. WHAT DIFFERENT THINGS THEY NEED TO KNOW OR THE IRB NEEDS TO KNOW SO THE DATA COLLECTED PROPERLY AND DOESN'T LEAK OUT BEFORE GETS COLLECTED. THOSE ARE THE KEY FEATURES WE WILL FOCUS ON IRBs DOING THE DATA WE UNDERSTAND THE RISK INVOLVED AND PEOPLE USING MOBILE AN WIRE NET TO COLLECT THE DATA. SO WE TALKED ABOUT DATA SECURITY ISSUES THAT HAPPENS WHICH IS MOBILE PHONE CLOUD PORTABLE DEVICES, NOW -- SO MANY OF THEM AND EVERYBODY COLLECTING SOME DATA AND WHAT THINGS TO WATCH OUT FOR, SECURITY RISKS ON THERE, WE WILL TALK ABOUT DIFFERENT STEPS THAT A RESEARCH ER INSTITUTION CAN PUT IN PLACE TO MINUTEMIZE THIS KIND OF RISK. SO WHY DO WE NEED TO SECURE DATA? THAT'S A VERY SIMPLE QUESTION BECAUSE WE WANT TO COMPLY WITH REGULATIONS, PROTECT SUBJECTS IDENTITY, KEY COMPONENT AND MAINTAIN INTEGRITY OF THE RESOURCE INSTITUTION OF THE STRUCTURE. WHEN WE TALK SECURITY, IT MEANS I SHOULD BE ABLE TO HAVE CONFIDENTIALITY WHICH MEANS DATA REMAINS ACCESSIBLE TO PARTS WHO SHOULD BE ACCESSING IT, NOT EVERYBODY HAS ACCESS. IT SHOULD HAVE INTEGRITY, NOBODY CHANGES THE DATA SO YOU DON'T GET -- THE WAY IT'S MEANT TO BE. AND LASTLY, SHOULD BE -- YOU HAVE COPY WHERE YOU KEEP IT, IT IS AVAILABLE SO COMPONENT OF DATA WE NEED IN ORDER TO HAVE DID RESEARCH AND RESEARCH DATA ANALYSIS. THERE ARE ALSO MANY REGULATIONS, I WILL GO THROUGH THEM QUICKLY BECAUSE SURE MOST OF YOU KNOW, SO MANY REGULATIONS CONCERNING THE DATA SECURITY COMMON WITH HIPAA PRIVACY AND SECURITY RULE, HIPAA NOTIFICATION RULE, PATIENT SAFETY RULES, THEN WE HAVE THE COMMON RULE TITLE 21 FDA COMES INTO PLAY STATE LAWS AND REGULATIONS SO EACH STATE HAS THEIR OWN LAWS AS WELL BUT PRIVACY RULE OVERRIDES ALL THE STATE LAWS AND SITUATIONS WHERE YOU HAVE TO DECIDE WHICH IS BETTER. PRIVACY RULE OVERRIDES THAT. ELECTRONIC PATIENT HEALTH INFORMATION. THIS INFORMATION COULD BE STORED IN MULTIPLE PLACES. YOU ARE USING CLOUD, DOESN'T MEAN ONLY IN THE CLOUD ALSO RELYING ON YOUR PHONE BECAUSE WHERE EVERYBODY ACCESSING THE CLOUD FROM, FROM YOUR PHONE, DESKTOP COMPUTER, A COPY OF THAT FILE IS ALSO STORED ON THE SAME LOCATION. SO IF YOU ARE USING CELL PHONE TO ACCESS DROP BOX, THAT IS ALSO SAVED ON YOUR MOBILE PHONE, THAT MEANS YOUR MOBILE PHONE IS STOLEN THAT DATA IS ALSO POTENTIALLY STOLEN HIPAA PRIVACY RULE USING EPHICS RESEARCH YOU MUST FOLLOW THE STANDARD, MAKE SURE THE INFORMATION IS NOT DISCLOSED TO ANYBODY WHO IS NOT AUTHORIZED TO GET IT. THAT'S CONFIDENTIALITY. INFORMATION COULDN'T BE ALTERED. THAT'S IMPORTANT TO MAKE SURE WHERE WE STORE THE DATA NOBODY MAKES CHANGES TO THAT OTHERWISE YOU LOSE INTEGRITY OF THE DATA AND ACCESSIBLE, USABLE ON DEMAND BY THE AUTHORIZED PERSON. ANOTHER REGULATION IMPORTANT TO UNDERSTAND IS EXPORT CONTROL. THAT MEANS SOME KIND OF DATA CANNOT BE STORED OUTSIDE U.S., CANNOT BE USED BY NON-U.S. CITIZEN, THAT MEANS WHEN USING CLOUD STORAGE YOU DON'T KNOW WHERE THE CLOUD SERVER IS. SO YOU NEED TO BE READY CAREFUL IF YOU HAVE ANY DATA THAT YOU FALL INTO THIS CATEGORY YOU GOT TO BE EVEN MORE SURE HOW YOU COLLECT DATA, WHERE ARE YOU STORING IT, YOUR STORAGE. >> THESE ARE ALL DIFFERNT REGULATIONS. SO MANY OF THEM, THAT TALKS ABOUT THE DATA SECURITY AND PRIVCY. WHAT HAPPENS WHEN THE BREACH DOES HAPPEN? CONFIDENTIALITY IS BREACHED, HAS MANY NOTIFICATIONSK ONE HAPPENS TO THE RISK TO HUMAN SUBJECTS ITSELF Z (INDISCERNIBLE) THOSE ARE THINGS WE WORRY ABOUT. WE TALK ABOUT WHEN WE WANT TO USE THE DATA, DO IT FOR GOOD CAUSE, FIND SOME USEFUL SOLUTION TO THE PATIENT FROM THE DATA ALGORITHM, THESE ARE BAD THINGS& THAT CAN HAPPEN FROM THE DATA. THAT'S SOMETHING WE HAVE -- SHOULD HAVE STRICT REGULATIONS AGAINST BUT THAT SHOULDN'T HAPPEN. RISK TO RESEARCH ITSELF, YOU LOST THE DATA YOU LOST THE INTEGRITY OF DATA. SOMEBODY INTERVENE AND CAPTURES DATA BEFORE YOU CAPTURE IT, IT COULD HAVE BEEN CHANGED. RISK TO RESEARCH INSTITUTION, THAT'S THE BIG ONE AS WELL AS TO THE INVESTIGATOR ITSELF. IF ANYTHING LIKE THAT HAPPENS THERE'S NO BACK UP, SOMETIME I WAS AT A CONFERENCE SOMEBODY SAID YOU DO ALL YOUR FINANCIAL TRANSACTIONS ON YOUR PHONE, EVERYBODY DOES, THIS IS SAFE TO DO THAT. YOU CAN PUT ANY PERSON ON YOUR PHONE NOW. REALITY IS -- HAPPENS ON YOUR FINANCIAL TRANSACTION BEING BREACHED THE COMPANY HUSBAND TO PAY. THEY ARE PAYING YOU FOR YOUR DATA SECURITY THERE'S NOBODY THERE TO BACK YOU UP, THAT'S WHY IT'S IMPORTANT FOR YOU AS RESEARCHER TO BE PROACTIVE AND MAKE SURE THOSE BREACH DON'T HAPPEN. HOW DO YOU PROTECT THIS ENVIRONMENT? YOU HAVE TO UNDERSTAND WHAT REALLY HAPPENS WHEN USING THE MOBILE CLOUD DATA. AND THAT'S SOMETHING I HAVE SEEN MANY RESEARCHERS WHO ARE HEALTH FIELD MAY NOT KNOW ANYTHING ABOUT TECHNOLOGY, I HAVE SEEN RESEARCHERS WHO HAVE NO PASSWORD ON THEIR PHONE, JUST FLIP, PHONE OPEN UP, BIGGEST SECURITY RISK FOR THE PHONE. IT SHOULDN'T HAPPEN. DON'T PUT ANY PHI ON THE CLOUD MOBILE ENVIRONMENT UNLESS IT'S REALLY NEEDED. SO THAT'S THE FIRST THING, MINIMIZE ANY IDENTIFY AND THAT NET IT'S THERE SOMEWHERE, LYING SOMEWHERE IN ANY FORM OR OTHER SOMEBODY WILL FIND IT AND ACCESS IT. THAT'S FIRST SIMPLE RULE, IF YOU REALLY NEED IT, -- IF YOU DON'T NEED IT DON'T GET IT. I HAVE SEEN SERVERS COLLECTING DATA, JUST COLLECTING WHAT ARE YOU GOING TO DO? NO, DON'T DO THAT. (INDISCERNIBLE) WE JUST DEFINITELY DON'T COLLECT ANY ACCESS NUMBER ANYWHERE, THAT'S TOTALLY A NO, NO. NO THROWS NO NEED. THAT'S THE RULES, THAT'S GONE. THERE ARE MANY OTHER THINGS LIKE ZIP CODE AN MOBILE PHONE, DEVICES LOCATION IS THERE. SO DON'T HAVE A WAY TO PROTECT THAT, THAT DATA COULD ALSO GO -- THAT'S ANOTHER FIELD TO EXPLORE. HOW DO YOU LIMIT THE DATA FROM DEVICES COMING TO YOU WHEN DOING RESEARCH BECAUSE YOU HAVE BEEN GETTING MORE INFORMATION ACTUALLY INTENDED FOR. MOBILE DEVICES BY USER, ANY RESEARCHERS WANTS TO USE MOBILE DEVICE NOWADAYS A PHONE IS MUCH MORE STRONGER THAN YOUR DESKTOP COMPUTER. YOU CAN DO WORK, XL POWERPOINT ADD PICTURE, EVERYTHING ON THERE, SO VERY USEFUL NO DOUBT BUT WE HAVE TO WORRY ABOUT WHAT AND HOW YOU USE THE PHONE. SO CAN DO ANYTHING, PICTURES WIFI, HARD TO LIMIT USE ON MOBILE DEVICES. CONCERN IS THIS IS CONCERN. DEVICES CAN GET STOLEN AND WIFI HOT POTTS SPOTS, IF YOU ARE IN A HOT SPOT, PEOPLE CAN CAPTURE AND PUT TROJAN VIRUS TO YOUR PHONE AND CAPTURE DATA. SO IF YOU HAVE DATA THAT'S STORED ON YOUR PHONE FROM CLOUD T COULD BE TAKEN AWAY. WITHOUT YOU KNOWING IT. IF YOU DOWNLOAD APPS AND DON'T KNOW THE OWNER OF THE APP BASICALLY IN THE ANDROID WORLD THERE'S SO MANY APPS WITH BUILT IN IN M ACTIONL WARE AND IN THERE, ONCE YOU DOWNLOAD THE APP,AL APP HAS ACCESS TO EVERYTHING ON YOUR PHONE. THEN WHATEVER YOU HAD IN FROM FROM YOUR RESEARCH THEY HAVE ACCESS TO IT. SO MANY TIMES IT'S IMPORTANT MANY INSTITUTION WHAT THEY DO, THEY KEEP WORK PHONE SEPARATE AND HOME PHONE SEPARATE BECAUSE DON'T WANT KIDS CLICK ON FACEBOOK AN CLICK SOMETHING, SOMETHING YOU DON'T WANT ANYBODY ON THE WEB SO THAT'S WHY -- ONE THING SO YOU CAN KEEP YOUR WORK DATA SEPARATELY. AND ALSO ENSUBSCRIPTION SOFTWARE SO LOG IN THE SITE AND ACCESS FILES SO YOU HAVE A SECURE TUNNEL THROUGH DATABASE ON THE COMPUTER, AND YOU WORK ON IT SO IT'S STILL REMAINS SECURE AND NOBODY OUTSIDE CAN ACCESS THOSE FILES. WHEN YOU MOVE FILES ON MOBILE DEVICE, TEMPORARY COPY OF THE FILES ALSO STORE ON THE PHONE. THAT FILE IS THERE. SO SOMEBODY WHO KNOWS HOW TO USE IT WANTS IT THEY CAN GET ACCESS TO IT. ALL OTHER -- IF YOU HAVE ENCRYPTION SOFTWARE OR TUNNEL GOING THROUGH LOGGING THE PORTAL AND ON THE ACCESS FILE THERE, THEN THOSE FILES ARE NOT STORED ON YOUR COMPUTER. THAT IS ONE WAY TO ACCESS ITS. SIMPLE THINGS, GO THROUGH FASTER BUT IT'S IMPORTANT AND ALL THE SLIDES WILL BE THERE. ALL WITH PASSWORD THAT ENSCRIPTSES THE PHONE AND THE DATA, SO VERY SIMPLE, ANDROID PHONE DIFFERENT WAY TO ENSCRIPT THE DATA BUT THAT ENCRYPTION IS NEEDED BY NOT PUTTING PASSWORD FOR SAKE OF EASE OPENING A PHONE FOR EASY ACCESS TO DATA. ONE THING EVERYBODY SHOULD DO. THEY FOUND LOOP HOLES WHERE THE HACKERS FOUND THE LOOP HOPES, THEY CAN GO INTO YOUR PHONE, FIX IT THAT'S WHY YOU NEED TO DO THAT. BE SELECTIVE, ACCESS THOSE INFORMATION ON DEVICES THAT YOU FIND ARE REALLY USEFUL. JUST DON'T GET EVERYTHING BASICALLY WHEN YOU ARE DEALING WITH RESEARCH DATA, ON THE PHONE BY ACCESSING THE RESEARCH DATA. IF DEVICE LOST AND STOLEN THERE ARE OPTIONS FINDS MY PHONE SO I CAN REMOTELY ERASE PHONE, THOSE FEATURES ARE THERE. CLOUD SECURITY. SO BE AWARE WHICH CLOUD YOU USE. SO THE ONE OF THE THINGS WE CAN DO IN OUR INSTITUTION WE HAVE A SECURE CLOUD WE PROVIDE TO ALL RESEARCHERS, THEY ARE ALLOWED TO USE ONLY THAT. THEY CANNOT GO DROP BOX GOOGLE, OTHER PLACES BECAUSE WE DON'T ALLOW, OTHER THINGS FROM THAT IS SURVEY SOFTWARE. GOOGLE FOR, SURVEY MONKEY, THE THINGS IS IF YOU ARE NOT SECURE PLATFORM THE DATA BEING COLLECTED BEFORE IT REACHES TO YOU CAN BE HACKED SO BEFORE YOU GET DATA SECURE SOMETHING CAN ALSO HAPPEN SO MAKE SURE YOU HAVE PROPERLY SECURED SURVEYS WE HAVE A SEPARATE SECURE VERSION OF THE SURVEY WE USE, THAT'S THE ONLY OUR INSTITUTION CAN USE. GO THROUGH QUICK SO TALK ABOUT OTHER IMPORTANT FEATURES FOR SAKE OF TIME. PORTABLE DEVICES. THIS IS MAKE SURE YOU KEEP THE DATA ENCRYPTED IN THE DEVICE AND DON'T USE ANY OTHER DEVICE THAT YOU FIND LYING AROUND. OR USB DEVICE WHEN YOU PLUG INTO YOUR LAPTOP YOU CAN INTRODUCE A TROJAN INTO YOUR LAPTOP AND DATA TAKEN FROM THERE, SO THOSE ARE SIMPLE SECURITY FEATURES FOR USING MOBILE DEVICES. DON'T USE PORTABLE DEVICE WITH STOLEN EPHI, LAPTOP, CELL PHONE, KEEP SUBJECT IDENTIFIES PHYSICALLY SEPARATE FROM THE DEIDENTIFIED DATA. ONCE YOU FINISH USING ON PORTABLE DEVICES, DELETE IT SO IT'S NOT THERE. DO NOT USE SS NUMBER OR TRANSMIT AND BY EMAIL, EMAIL IS NOT SECURITY. MANY THING THEY CAN IMAGE MY PORTION TO OTHER PERSON BUT EMAIL OPENS UP O THE WHOLE WORLD. YOU UNDERSTAND SOMETHING HAS TO BE SECURE. THAT'S MORE COMMON SENSE, LOG OFF, DO NOT SHARE WITH ANYONE AND DO NOT WRITE PASSWORD ANYWHERE. THAT'S A COMMON THING. WHAT WE CAN LEARN FROM THIS, AS COMMITTEE MEMBERS WHEN YOU DO RESEARCH PROTOCOLS, BE COGNIZANT ABOUT TYPE OF DATA OPTION USED BY RESEARCHER, CLOUD, USB DEVICES OTHER PORTABLE DEVICES AND WHAT SECURITY RISK COMES WITH THAT. ALSO SHOW THAT YOUR RESEARCHER KNOWS ABOUT THAT. IF NOT YOU CAN OFFER OPTIONS. SECOND, BE COGNIZANT ABOUT THE USE OF MOBILE DEVICES IN RESEARCH IN SECURITY RISK. SO WE NEED TO WHEN USING MOBILE DEVICE I HOPE THEY UNDERSTAND THAT YOU ARE ALSO ACCESSING THOSE DATA ON YOUR PHONE AND THAT MEANS IT IS SECURITY RISK AND LIABILITY AND MAKE SURE THEY UNDERSTAND IT. IF NOT, DON'T HAVE AN OPTION FOR THAT T. USE IT ON A SECURE COMPUTER AND ONLY USE IT THERE. ALSO COMES DOWN TO WHAT DATA THEY COLLECT. SO ARE THEY COLLECTING DATA ABOUT HOW MUCH PEOPLE LIKE ABOUT THE SODA. COLLECTING SOMETHING ELSE SENSITIVE, THAT BECOMES A QUESTION. OFFER OPTIONS IF POSSIBLE PROVIDE SECURE E ENVIRONMENT. IN YOU HAVE AN OPTION TO BE ABLE TO PROVIDE SECURE CLOUD STORAGE FOR YOUR INSTITUTION, THAT'S THE BEST, IF NOT THAT'S SOMETHING TO BE LOOKED AT WHAT THEY CAN DO. ENCRYPTED MOBILE DEVICE, THAT'S PRETTY MUCH -- INSTITUTION ALL THE MOBILE DEVICES THAT YOU CAN USE ON NETWORK HAS TO HAVE PASSWORD, AND ENCRYPTION OTHERWISE THERE IS NO -- YOU CAN USE THEM ON THAT NETWORK. LIMIT, STORING SENSITIVE INFORMATION ON E ENVIRONMENT AS MUCH AS POSSIBLE. IF YOU DON'T PUT IT, I HAVE'S SAFE, IF YOU PUT IT YOU ARE LIABLE. KEEP AN EYE ON THAT. SO THAT'S THE BASIC GIST ABOUT RESEARCH AND SECURITY BEHIND MOBILE AND CLOUD DEVICES, I DID COVER A LOT OF INFORMATION BUT THERE ARE ALSO SLIDES TO GO OVER THEM. THANK YOU. [APPLAUSE] >> HI, GOOD AFTERNOON. MICHAEL SUGGESTS TALK A LITTLE BIT ABOUT HOW IRBs AND ORGANIZATIONS CAN PRACTICALLY THINK ABOUT THE ACTUAL REVIEWS OF RESEARCH PROJECTS AND OPERATIONALIZE TOOLS TO HELP MAKE THIS PROCESS A LITTLE BIT EASIER. SO I WILL SHARE EXPERIENCES A T HOPKINS, BY MEANS NO SOLUTIONS BUT SOME WE HAVE IMPLEMENT TO TRY TO MAKE THIS PROCESS A LITTLE BIT EASIER. MY GOAL WILL BE TO OUTLINE SYSTEM OF THESE KEY OPERATIONAL CHALLENGES THAT WE HAVE BEEN TALKING ABOUT THROUGHOUT THE DAY SO FAR AND ALSO FOCUS ON SOME OF THESE PRACTICAL SOLUTIONS YOU MIGHT LEVERAGE AT YOUR OWN ORGANIZATION. SOME OF THESE CHALLENGES WE TALKED ABOUT REALLY START FROM THAT FIRST PRACTICAL PERSPECTIVE OF IS THE PROJECT EVEN SUBJECT TO IRB REVIEW? IF WE HAVE MOVED BEYOND THAT STAGE AND WE SAID YES, I THIS I THE PROJECT IS SUBJECT TO IRB REVUE AND WOULDN'T QUALIFY FOR EXEMPTION, ONE OF THE ITEMSIs ARE CHALLENGED WITH IS THIS ISSUE OF A WAIVER OF CONSENT. AND ALSO A WAIVER OF PRIVACY AUTHORIZATION. ONE OF THE MOST CHALLENGING ASPECTS OF THAT IS THIS CONSIDERATION AS TO WHETHER INFORMATIONAL RISK EQUALS MINIMAL RISK AND UNDER WHAT CONDITIONS THAT'S TRUE. SO THAT IS ONE AREA OF CHALLENGE. I WILL TALK ABOUT POTENTIALLY RELATED SOLUTIONS. ANOTHER IS THE IRB ISK AING AS PRIVACY BOARDS, OFTEN AS FUNDAMENTAL PRINCIPLE HAVE TO THINK ABOUT WHETHER THE DATA IS MINIMUM NECESSARY TO ACCOMPLISH THE RESEARCH OBJECTIVE. WHEN YOU PUT THAT INTO THE CONTEXT OF BIG DATA RESEARCH, THIS BECOMES PARTICULARLY CHALLENGING. BECAUSE OFTEN THE REQUEST IS I WANT ALL OF THE DATA. ANOTHER CHALLENGE WE HAVE TALKED ABOUT THIS A LITTLE BIT, SO FAR THAT TODAY, BUT I THINK WE ARE GOING TO PRESS ON THIS A LITTLE MORE THIS AFTERNOON IS THIS CONCEPT OF SUFFICIENT EXPERTISE TO REVIEW THESE PROJECTS. OFTEN IRBs DON'T EQUIP WITH A POOL OF DATA SCIENTISTS WHO ARE ALSO WILLING TO BE IRB MEMBERS. SO WE HAVE A CHALLENGE HAVING THE RIGHT EXPERTISE AROUND THE TABLE TO EVEN REVIEW THE PROJECT AND ASSESS MINIMUM NECESSARY IN THE CONCEPT OF RISK. THEN THE PROJECTS ARE DIFFERENT. OFTEN THEY DON'T FIT INTO THE FRAMEWORK OF STANDARD PROTOCOL THAT ONE MIGHT SEE. WITH OBJECTIVES AND HYPOTHESES AND PLAN THAT COULD BE EASILY FOLLOWED IN A STRUCTURED WAY. THEY'RE MUCH MORE DYNAMIC. DYNAMIC NATURE MAKES THEM COMPLEX TO REVIEW. SO ONE OF THE THINGS CHALLENGING IS HOW DO WE GET THE PROJECT PLAN IN. AND DESCRIBED IN WAY THAT WE UNDERSTAND IT. BUTLING ADAPT TO THE DINE MCNATURE OF THE PROTOCOL AND ISSUE APPROVAL GIVEN THE DYNAMIC NATURE OF THAT. I WILL TALK ABOUT A FEW OF THE SOLUTIONS WE HAVE IMPLEMENTED TO DATE WITH A HOPE THAT THAT CAN OPEN UP TO PRODDER AT THIS TIME CUSHION ABOUT POTENTIAL -- DISCUSSION ABOUT POTENTIAL SOLUTIONS IN OUR PANEL THIS AFTERNOON. SO ONE OF THE FIRST THINGS THAT WE DID TO ADDRESS THIS CONCEPT OF MINIMAL RISK GIVEN THE FACT PROJECTS INVOLVE BIG DATA DID REQUEST A WAIVER OF CONSENT AND WAIVER OF AUTHORIZATION. IS TOSS CREATE WHAT WE CALL A MINIMAL RISK ENVIRONMENT IF WE CAN OFFER A SOLUTION THAT CREATES A SAFE SPACE WE FEEL PRETTY CERTAIN IS A SPACE WHERE THE DATA IS ADEQUATELY PROTECTED WE MAY BE ABLE TO SAY IF YOU OPERATE WITHIN THIS SPACE WE ARE FAIRLY COMFORTABLE THAT THE RISKS HAVE BEEN MINIMIZED TO THINGS LIKE BREACH OF CONFIDENTIALITY AND INAPPROPRIATE ACCESS O THE DATA SO WE FOCUS ON OPERATIONAL SOLUTIONS FROM AN ORGANIZATIONAL PERSPECTIVE RATHER THAN PER PROCONTROL BASIS. ONE THING WE DEVELOPED AN OFFER IS SOMETHING CALLED SAFE DESKTOP, MANY HAVE SOMETHING SIMILAR IN THEIR OWN ORGANIZATIONS WHICH OFFERS ANALYTIC ENVIRONMENT THAT RESEARCHERS CAN USE AND ACCESS IN A CONTROLLED ENVIRONMENT THAT ALLOWS THEM AMONG COLLEAGUES TO SHARE AND ACCESS DATA. MORE RECENTLY WE HAVE LAUNCHED AND REALLY IN THIS ERA AND SPIRIT OF BIGGER DATA, PRECISION MEDICINE ANALYTICS PLATFORM OR PMAP. THE DESIGN IS TO POOL DATA TOGETHER FROM OUR ELECTRONIC MEDICAL RECORD AS WELL AS OTHER DATA SOURCES INTO SOMETHING WE CALL A DATA COMMONS. THAT DATA COMMONS ALSO WILL ALLOW FOR BASED ON IRB APPROVED PROJECTS YOU CAN GET A PROJECTION OR A PROVISIONING OF DATA TO A RESEARCHER THAT ALSO IS IN A SECURE ENVIRONMENT WHICH IS ENABLED WITH HIGH PROCESSING SOFTWARE, SO THAT YOU CAN DO THINGS LIKE MACHINE LEARNING ALGORITHM. S AND APPLY NATURAL LANGUAGE PROCESSING, ALL THE COOL TOOLS OUR RESEARCHERS WANT TO USE TODAY BUT WE WANT TO CREATE THE SPACE WHERE ONE THE PROJECTION, THE PROVISIONING OF THE DATA COULD BE CONTROLLED AND ALSO THE PLACE WHERE YOU GET TO PLAY WITH THAT DATA IS CONTROLLED AS WELL. ONE THING I REFERENCED EARLIER TODAY IS DATA STEWARDS OR GATEKEEPERS. WE HAVE WITHIN OUR INSTITUTE FOR CLINICAL TRANSLATIONAL RESEARCH A PROGRAM WHICH ACTUALLY PROVIDES ACCESS TO DATA THROUGH A -- THE RESEARCHER CAN QUERY OUR CLINICAL RESEARCH DATA ACQUISITION CORE. THEY WILL FULFILL THAT QUERY AFTER MATCHING WITH THE IRB APPROVAL. SO THESE ARE DATA STEWARDS TRAINED TO LOOK WHAT THE IS THE DATA ASK IS AND MATCH IT TO WHAT THE IRB APPROVED AND ENSURE WHAT IS PROVISIONED IS ACTUALLY THE DATA THAT WAS APPROVED IN THE PROTOCOL. SO THIS OFFERS AN ADDED LAYER OF SECURITY KNOWING THAT WE ARE TAKING AWAY THAT ACCESS OF RESEARCHERS GOING IN AND PULLING THAT YOU ARE OWN QUERIES OR TAPPING THE DATA THEMSELVES AND PROVIDING A TRAINED STEWARD WHO CAN HELP PROVISION THAT DATA. ANOTHER THING WE HAVE TRIED TO DO IS REALIZE PART OF THIS IS EDUCATIONAL PROCESS AND WE NEED TO ENFORCE THESE GOOD OPTIONS. SO WE HAD TO ASK SPECIFIC QUESTIONS TO SAY TELL US EXACTLY WHAT YOUR DATA STORAGE AND MANAGEMENT PLAN ARE. THEN REDIRECT TO GOOD OPTIONS SO WHEN THIS FIRST STARTED OUT, WE HAVE THESE CLUNKY PAPER FORMS AND CALL THEM OUR DATA SECURITY PROFILE AND CHECKLIST, THEY HAD TO BE UPLOADED IN OUR E IRB APPLICATION. IF YOU PICKED AN OPTION THAT WAS BASICALLY AND OTHER, UNAPPROVED OPTION YOU WERE ROUTED FOR A SOPHISTICATED REVIEW. THIS IS LIGHTLY CLUNKY AND HAD LOTS OF FORMS SO WHAT WE DECIDED TO DO WORKING IN PARTNERSHIP WITH DATA TROUGH WHICH I WILL TALK TO YOU ABOUT IN A MOMENT, WE CREATED WHAT WE CALL A RISK TIERS MATRIX. AND THE RISK TEASER MATRIX COMES WITH GRADES. BASE -- RISK TIERED MATRIX. SO THEY ARE MADE UP OF DIFFERENT PIECES OF INFORMATION LIKE WHERE ARE YOU STORING YOUR DATA, WHAT TYPE O DATA IS IT, IS IT IDENTIFIABLE OR NOT, IS IT SENSITIVE, IS IT DATA YOU PLAN TO SHARE WITH OTHERS. BASED ON THE TIER YOU GET ASSIGNED GRADE. A, B, C. NOBODY WANTS THE C. SO THROUGH THE MATRIX WE WERE ABLE TO CREATE A RANKING FOR BEST PRACTICES OF DATA SECURITY AND MANAGEMENT. WE FIRST DEVELOPED THIS AND SAID OKAY WE'LL USE IT TO TRIGGER CERTAIN REVIEWS SO IF YOU FALL INTO A LOWER CATEGORY, NOT THE GREATEST GRADE YOU WOULD BE ROUTED FOR A MORE SOPHISTICATE DATA REVIEW BY DATA GOVERNANCE ENTITIES. THEN MORE RECENTLY IN MAY WE LAUNCHED THIS INTO OUR E IRB APPLICATION. SO NOW THE LOGIC AND THE WAY PEOPLE ANSWER QUESTIONS WITHIN THE APPLICATION, GIVES THEM A -- BUILT INTO THE APPLICATION NOW HAT GIVES THEM THEIR GRADES. SO THE MATRIX NOW ACTUALLY HAS TRANSLATED INTO A GRADING SYSTEM BY WHERE THEY ARE AUTOMATICALLY ROUTED TO ADDITIONAL REVIEW IF THEY DON'T GET A GOOD GRADE, BUT IT ALSO TEACHES THEM HOW TO GET THAT GOOD GRADE. SO YOU CAN SEE HOW IF I PICK A MORE SECURE OPTION, OR IF I SELECTED TO USE LESS DATA THAT I DON'T NEED, SO I PICKED A LIMITED DATA SET RATHER THAN DIRECTLY IDENTIFIABLE DATA, I CAN SEE HOW THE RISK LEVEL IS ALTERED AND I LEARN ABOUT DATA PRACTICES THROUGH THIS PROCESS. SO WE DID THIS EDUCATIONAL PERSPECTIVE TO GET PEOPLE TO LEARN ABOUT BETTER DATA PRACTICES BUT ALSO BECAUSE IT HELPED FACILITATE OUR REVIEW PROCESS AND PUT THIS INTO OUR E IRB APPLICATION, NOW WE HAVE A DEDICATED ANCILLARY REVIEW AND OUR DATA TRUST. SO THAT PROJECTS THAT REQUIRE ADDITIONAL ORGANIZATIONAL REVIEW FOR DATA SECURITY AND MANAGEMENT ACTUALLY GET ROUTED WITHIN OUR E IRB SYSTEM TO THAT ENTITY FOR ADDITIONAL REVIEW. THROUGH THAT PROCESS WE EMBED THE ORGANIZATIONAL RESOURCES IN TO THE IRB DECISION MAKING. SO RATHER THAN RECREATE AND SAY NO, WE HAVE TO GO FIND FIVE DATA SCIENTISTS TO SIT ON IRBs WE INCORPORATED THE EXPERTISE THAT ALREADY EXISTED IN ORGANIZATIONS INTO THE IRB PROCESS, THROUGH THIS ANCILLARY REVIEW. AS I MENTIONED ONE THING THAT'S REALLY IMPORTANT IS WE DID IT THROUGH PARTNERS AND RESOURCES WITHIN THE INSTITUTIONAL TRANSLATIONAL RESEARCH, ALSO THE DATA TRUST COUNCIL. THE DATA TRUST COUNCIL HAS GOVERNANCE OVER JOHNS HOPKINS MEDICINE DATA AND IN ADDITION TO THE DATA TRUST COUNCIL WE HAVE GREAT RESOURCES THE UNIVERSITY LEVEL IN OUR INFORMATION SECURITY TEAM WHO PARTNERED WITH THE DATA TRUST AND IRB ON THIS INITIATIVE. WE WORKED TOGETHER WITH THEM TO ESTABLISH THE RISK TEASER AND MATRIX AND BUILD INTO OUR E IRB APPLICATION AND WE ALSO WANT TO KEEP THE DIALOGUE OPEN. WE CONSTANTLY MEET AND THINK HOW TO IMPROVE OUR PROCESSES. WE HAVE MEMBERS OF OUR IRB WHO SIT ON THE RESEARCH DATA TRUST SUBCOUNCIL, OVERALL COUNCIL MADE UP OF LITTLE SUBCOUNCILS THAT EACH FOCUS ON DIFFERENT AREAS LIKE QUALITY IMPROVEMENT, RESEARCH. SO FOR THE RESEARCH SUBCOUNCIL MEMBERS OF THE IRB SIT ON THAT SUBCOMMITTEE TO HELP LOOK AND REVIEW PROJECTS THAT HAPPEN TO REQUIRE BOTH IRB REVIEW AND ALSO MAY HAVE MET A TRIGGER FOR ADDITIONAL ORGANIZATIONAL REVIEW FOR DATA RISK OR SECURITY ISSUES. WHEN THE DATA IS NOT HOPKINS MEDICINE DATA OUR COLLEAGUES IN THE IT SECURITY TEAM DO A SIMILAR REVIEW. THAT FEEDS BACK INTO THE IRB REVIEW PROCESS TO CLOSE THE LOOP AND WE HAVE THIS NICE CIRCULAR PROCESS LEVERAGING THE EXPERTISE OF THOSE WHO REALLY ARE EQUIPPED WITH THIS DATA EXPERTISE. THEN THE LAST ITEM WAS THIS ISSUE OF WHAT DO THESE PROTOCOLS LOOK LIKE. WHEN IT COMES TO IRB IT LOOKS LIKE I WANT TO USE DATA FROM 10,000 PATIENTS, I DON'T KNOW WHAT DATA I NEED BUT I NEED A LOT OF IT. AND I'M GOING TO MAKE A TOOL OUT OF IT. NOT SURE WHAT THAT WILL LOOK LIKE EITHER BUT I PROBABLY WILL NEED MORE DATA ALONG THE WAY AS I REFINE. OUR STAN TOKER FORMS DON'T WORK FOR THAT. WE HAVE TO THINK ABOUT HOW TO WE ACTUALLY HAVE A RESEARCHERS TELL US WHAT THEY ARE GOING TO DO IN A WAY THAT EXPLAINS THE PROCESS. I AM GOING TO TAKE DATA FROM 10,000 PEOPLE AT FIRST AND WHITTLE IT DOWN TO DATA FROM A THOUSAND PEOPLE. AFTER I HAVE DONE SOME INITIAL VETTING. SO WE REALIZE WE HAD TO, THIS IS STILL A WORK IN PROCESS, CHANGE SOME TRADITIONAL FORMS BUT ALSO ASK DIFFERENT TYPES OF QUESTIONS INCLUDING WHO DO YOU HAVE ON YOUR TEAM WITH DATA EXPERTISE THAT WILL HELP SERVE AS DATA STEWARD FOR ALL OF US. SO ONE OF THE WAYS WE REALIZE THAT WE COULD MONITOR THESE PROJECTS AS THEY EVOLVE IS TO SAY YOU MAY NEED TO GIVE US A PROGRESS REPORT AFTER CERTAIN TIME POINTS THAT YOUR PROJECT HAS REACHED EVEN THOUGH WE KNOW YOUR PROJECT IS GOING TO LIVE AND BREATHE OVER TIME. WE HAVE MADE SOME STRIDES TRYING TO DEVELOP TEMPLATES BUT THIS IS SOMETHING TO COLLECTIVELY DO BETTER IN THE FUTURE, WHAT IS A MODEL FOR THAT PROJECT DESCRIPTION LOOK LIKE. TO HELP RESEARCHERS WRITE IT BETTER BUT ALSO HELP IRB AND REVIEW PROCESS SO THAT'S SOMETHING I HOPE WE CAN TALK ABOUT MORE TODAY. SO JUST WANT TO GET THE BALL ROLLING ON THINKING OF OUR OPERATIONAL SOLUTION. AND I HOPE THAT WE CAN TALK MORE ABOUT THESE LATER ON DURING OUR PANEL. [APPLAUSE] >> SO A LOT OF WHAT IS IN HERE WE DEFINITELY ALREADY COVERED SO I WILL GO QUICKLY AND FOCUS ON THINGS TOWARD THE END THAT MAYBE ARE A LITTLE BIT MORE THAN WHAT WE TALKED ABOUT SO FORTH OBVIOUSLY DON'T NEED TO COVER THE COMMON RULE HERE. PEOPLE FAMILIAR WITH IT AS WELL AS IT'S COVERED TODAY. THE BEYOND IRB PART OF THAT IS ETHICAL REVIEW PROCESS TO BE DEVELOPED FOR THE EXACT SITUATIONS THAT HAVE BEEN DISCUSSED THE LAST COUPLE OF PRESENTATIONS AND ALSO A COUPLE OF ADDITIONAL USE CASES THAT I WILL GET TO IN A MINUTE THAT ARE PART OF A PROJECT WE ARE WORKING ON AT FPF UNDER A GRANTS WE RECEIVED. SOME OF THE QUESTIONS THAT HAVE BEEN DISCUSSED THAT HAVE TO BE ANSWERED FOR THIS EXTERNAL REVIEW BOARD FOR PROJECTS INVOLVING DATA FROM COMPANIES THAT DON'T DIRECTLY FALL UNDER COMMON RULE OR FALL TO TRADITIONAL RESEARCH STRUCTURES IS WHETHER THEY SHOULD BE INTERNAL OR EXTERNAL. WHO COMPOSE THEM AND IN TERMS OF ARE THERE ASPECTS THAT DON'T FOLLOW MODEL OF EXPERTISE AND EVALUATE THE IMPACT THAT THE RESEARCH IS LIKELY TO HAVE, FRAMEWORKS THEY SHOULD BE BASED ON. WHETHER CONSENT PLAYS ROLE AND HOW THAT FAIRS INTO IT SINCE WITHOUT COMMON RULE UNDERLYING STRUCTURE FOR IDENTIFYING LEVELS OF APPROVAL BASED ON THOSE KIND OF THINGS, RECONSIDER FROM SCRATCH WHETHER THOSE WERE THE RIGHT STRUCTURES TO HAVE IN PLACE. AND SPECIFICALLY IN CONTEXT OF ARTIFICIAL INTELLIGENCE MACHINE LEARNING BASE SYSTEMS, THE ASPECT OF FAIRNESS DATA BIAS AND THING LIKE THAT, POTENTIALLY NEW OR AT LEAST DIFFERENT SCALES UNDER THESE -- THESE RESEARCH PROPOSALS. SO THE PART I WANT TO FOCUS ON THEN IS QUESTIONNAIRE FPF PUT TOGETHER TWO AND A HALF YEARS AGO FOR A SUMMIT THAT WE HAD, CALLED BEYOND IRB, ETHICAL RESEARCH FOR BIG DATA UNDER BIG DATA PURVIEW, THAT WAS ONE OF THE THINGS FOLKS HERE ASKED ME TO TWO INTO IN MORE DEPTH, IT PROVIDES A LITTLE BIT OF STRUCTURE AND FRAMEWORK HOW TO START THINKING ABOUT THIS IN WAYS THAT IT MIGHT DIFFERENCE FROM STRUCTURES AND PROTOCOLS THAT WE ARE USED TO. SO IT IS AVAILABLE IN THE WEBSITE IN SUMMIT CONFERENCE PROCEEDINGS, ATTACHED TO IT, MATRIX SET OF QUESTIONS THAT MIGHT HELP IN DECIDING WHEN AND HOW THEY PROJECTS NEED TO BE EVALUATED. OBVIOUSLY BACK TO THE QUESTION YOU TALKED THIS MORNING IS IT RESEARCH OR WHAT KIND OF RESEARCH IS IT HUMAN SUBJECTS RESEARCH ADVANCING THE GENERAL KNOWLEDGE OR ADDING PRODUCT AND SERVICES AND FEATURES TO EXISTING COMMERCIAL APPLICATIONS. IS THERE IDENTIFIABLE PRIVACY INFORMATION IN THE -- NEXT SLIDE, DATA COLLECTED AND USED, SENSITIVE, SOME OTHERWISE PROTECTED CATEGORY, IS IT DIRECTOR DIRECT COLLECTION AND WHAT ARE SAFEGUARDS OR CONTROLS AROUND IT. IN THE SAME CONTEXT WE CAN TALK ABOUT TODAY. ONCE THE DATA IS COLLECTED, OR IS IDENTIFIED THAT IS NECESSARY TO BE USED, WHAT KINDS OF PRIVACY ENHANCING TECHNOLOGIES CAN BE APPLIED TO WHAT WE OBVIOUS HUH TALKED A LOT ABOUT DIFFERENTIAL PRIVACY AS ONE OF THE OPTIONS AND DEIDENTIFICATION AS POTENTIALLY NOT A STRONG OPTION. D ON OTHER PRIVACY ENHANCING TECHNOLOGIES THAT MIGHT BE USED. THESE ARE SOME OF THE KINDS OF PROTECTIONS THAT CAN BE USED FOR THE DATA. THERE ARE OTHERS AS WELL, TO RETURN TO SUBJECT OF DEIDENTIFICATION, IT IS POTENTIALLY TRUE THAT CAN'T BE TRUE THROUGH IDENTIFIED OR ANYTHING THAT IS DEIDENTIFIED CAN BE REIDENTIFIED. NOT TO SAY THAT HAS NO VALUE OR PROTECTIVE ROLE OR PLACE IN IT, LIKE SAYING PASSWORDS ARE WEAK AND DON'T SERVE PURPOSE OF ABSOLUTELY SECURING THINGS. I STILL RATHER HAVE A PASSWORD ON MY ACCOUNT THAN NOT, STILL HAVE PASSWORD ON MY PHONE THAN& NOT. ENCRYPTION IS WEAK AND CAN BE BROKEN. HASHING IS SUBJECT TO BEING LESS PROTECTIVE BUT PROVIDES LEVEL OF PROTECTION MORE THAN NOT HAVING THOSE THINGS. SO CONTEXT MATTERS USE CASES MATTER AND WHERE THE DATA IS GOING OR WHO IS USING IT MIGHT IN FACT BE USEFUL PROTECTIVE MEASURES TO EMPLOY. HARMS, AGAIN WE COVER THIS FAIRLY WELL TODAY BUT IT'S I THINK GOING TO TAKE ON NEW IMPORTANCE AND COUPLE OF USE CASES I WILL TALK ABOUT IN A SECOND FOR EVIDENCECAL REVIEWS OF PRODUCT DEVELOPMENT OR PRODUCT DEVELOPMENT GENERALLY IN TERMS OF POTENTIAL IMPACTS ON INDIVIDUAL VERSUS GROUPS. MOSTLY BECAUSE THERE ARE A LOT OF CASES OR COMMON CASES ONE CAN ENVISION INDIVIDUAL WITH BENEFIT TO GROUP OR HARM TO GROUP BENEFIT TO INDIVIDUAL. THOSE ARE THE TRICKY CASES I THINK THAT PROPOSE REAL ETHICAL CHALLENGES HOW TO WEIGH THOSE RISKS, WEIGH THE LEVELS OF HARM. AND WEIGH THE VALUE OF THE PROPOSAL. THE TRIGGERS FOR THE REVIEW AS WE HAVE THEM IN THE CONTEXT OF THE CHECKLIST WE PROPOSE, ARE THOSE KINDS OF ISSUES WHETHER PI,, WHETHER RESULT RELATED TO HUMAN SUBJECTS, ARE THERE NEW UNEXPECTED USES TIES INTO THE CONVERSATION ABOUT CONSENT THAT WE HAVE HAD SO FAR TODAY. WHO THE EFFECTED POPULATIONS ARE, THIS GOES TO THE ISSUE OF RISK AND FARMS. WE TALKED ABOUT EDUCATION EARLIER WHICH OF COURSE IMMR. I CASE CHILDREN WHICH IS MORE VULNERABLE POPULATION THAT IS AWARDED EXTRA PROTECTIONS. SAME FOR DISABILITY COMMUNITIES, ELDERLY PRISONER POPULATIONS AND OTHER EXAMPLES OF PEOPLE WHO MIGHT BE MORE AT RISK UNDER CERTAIN MODELS. SO THE LAST THING IS, THE FROM APOLOGETIC WE ARE DOING IS LITERALLY DESIGNING AN ETHICAL PREVIEW BOARD FOR USE OF COMMERCIAL DATA. SO WE TALKED ABOUT SHARING OF COMMERCIAL DATA TO RESEARCHERS OR WHAT YOU CAN CALL TRADITIONAL RESEARCH PROPOSALS, THE PROJECTS THAT MIGHT BE REVIEWED BY IRB WE HAVE ALSO TALKED ABOUT CORPORATIONS DOING THEIR OWN RESEARCH AND WHEN IT IS RESEARCH RESEARCH AND WHEN IT IS PRODUCT DEVELOPMENT, WHERE THAT GRAY LINE MIGHT LIE AND WHETHER THE STANDARDS CHANGE FOR HOW TO DO THAT. THERE IS ALSO THE ISSUE THAT WE HAVE -- WE BEING PF TALKED ABOUT AND ARE CONSIDERING WHICH IS ACCEPTING PRODUCT DEVELOPMENT, IT STILL MATTER WHAT IS THE IMPACT AND HARMS ARE AND THERE'S STILL ETHICAL REVIEW TO BE HAD. SO IF A STEP FURTHER THAN TRADITIONAL RESEARCH PROJECTS THE IDEA THAT A COMPANY NEEDS TO HAVE AN ETHICAL REVIEW PROCESS FOR ITS PRODUCT DECISIONS AT DESIGN PHASE, AT THE SALE PHASE, AT THE MONITORING PHASE ONCE IN THE MARKET HOW IT CORRECTS FOR THINGS THAT GO WRONG, THIS TIES INTO THINGS IN THE MEDIA A LOT LATELY, NOT JUST LATELY BUT OVER THE LAST COUPLE OF YEARS, GOOGLE STATED POSITION OF NOT SELLING TECHNOLOGY FOR AUTONOMOUS WEAPONS TO THE DEPARTMENT OF DEFENSE. WHICH IS MAYBE REALLY ADMIRABLE STAND DEPENDING WHO YOU ARE AND WHERE YOU STAND BUT IT IS HARD TO DEFINE WHAT THAT MEANS AND WHAT THAT WOULD BE AND WHAT THAT MIGHT LIMIT AND WHETHER THAT WOULD IMPACT ANY SALES ANYWHERE. OR THE DEPARTMENT OF DEFENSE ABILITY TO GET THAT TECHNOLOGY IN SOME OTHER WAY. SO THOSE ARE COMPLICATED QUESTIONS I CANNOT FINISH IN TWO MINUTES THAT I HAVE JUST BEEN GIVEN. BUT THOSE KIND OF QUESTIONS ARE THE ONES WE THINK ARE REALLY IMPORTANT AND SORT OF NEW EDGE OF THE CONVERSATION, NOT THAT PEEP AREN'T ALREADY THINKING OF THEM -- CONSIDERING BUT THEY'RE THE NEXT LEVEL OF WHERE TO FOCUS THIS. IT TIES BACK A LITTLE BIT INTO ONE OF THE EXAMPLES THAT MICHAEL GAVE, ABOUT THE USES OF PUBLIC DATA AND THAT HOW PUBLIC THE DATA WAS WAS SORT OF ONE OF THE CRITICAL DECISION FACTORS FOR IRB. MAYBE THAT IS NOT NECESSARILY THE DECISION FACTOR THAT IT IS IN THAT CONTEXT WHEN TALKING PRODUCT DEVELOPMENT USE BECAUSE DATA TO USE OR DATA GIVEN TOBLIC THEM FOR ONE PURPOSE AND USE FOR ANOTHER PURPOSE OR USE FOR TRAINING DATA SETTOR MACHINE LEARNING BASE SYSTEMS DOESN'T NECESSARY HI MEAN IT SHOULD PASS ETHICAL REVIEW OR THRESHOLD. SO AN EXAMPLE OF THIS IS THE USE OF PUBLIC DATA SETS FOR TRAINING FACIAL RECOGNITION SYSTEMS. WE WANT FACIAL -- IF WE WANT TO USE FACIAL RECOGNITION SYSTEMS WHICH IS A QUESTION OF ITSELF, WE CERTAINLY WANT THEM TO BE ACCURATE. IF WE WANT THEM TO BE ACCURATE THEY NEED DATA BUT NOBODY WANTS TO GIVE DATA TO DO THAT OR THAT SEEMS A COMMON RESPONSE. HOW DOES THE SUBMISSION DATA TO TRAIN THOSE SYSTEMS UP TO A SUFFICIENT LEVEL OF ACCURACY? THAT WHERE THEY ARE USED THEY CAN BE USED RESPONSIBLY. THERE ARE COMPANIES THAT HAVE IN FACT TAKEN HEAT, THAT'S ONE COMPANY IN PARTICULAR THAT OWNS A PHOTO SHARING SITE AND HAS ANOTHER ASPECT OR DIVISION AN AI DEVELOPMENT COMPANY THAT DEVELOPS FACIAL RECOGNITION SOFTWARE, SO IT'S SOLVES EASY ANSWER OF USING ALL FACIAL PHOTO DATA IN ITS PHOTO SHARING OPERATIONS, AS TRAINING DATA FOR IMAGE RECOGNITION AND FACIAL RECOGNITION SYSTEMS UNDER AI DIVISION AND DIDN'T NECESSARILY SEE A PROBLEM WITH THAT, MAYBE THERE IS NOT A PROBLEM WITH THAT BUT THEY ARE USING -- MAKING USE OF THAT DATA WHICH IS PERHAPS UNEXPECTED FOR THE PEOPLE WHO POSTED TO THE PHOTO SHARING SITE. AND THERE'S THE QUESTION OF SHOULD THERE HAVE BEEN MORE NOTICE, OPT OUT CHOICE WHERE WAS IT WAS IMPLICITLY USED BECAUSE OF USING SHARING SITE. AND WHAT HAPPEN DEGREE THAT WAS ETHICAL OR UNETHICAL CONDITION AND HOW IT'S HANDLED DIFFERENTLY TO HAVE HIGHER ETHICAL STANDARD OF REVIEW. SO WITH THAT I WILL WIND UP AND LOOK FORWARD TO TO CONVERSATION. >> YOU'LL BE BACK, BRENDA. FIRST TALK LAST TALK. YVONNE SUGGESTED THIS PARTICULAR TITLE OR TOPIC BECAUSE A FEW YEARS AGO SOME COLLEAGUES AND I PUBLISHED IN THE PLOS SERIES TEN SIMPLE RULES, SO PLOS ACROSS MULTIPLE JOURNALS HAS A SERIES OF ARTICLES CALLED TEN SIMPLE RULES FOR X Y Z THAT ARE SORT OF METHOD LOGICAL GUIDES FOR AREA OF RESEARCH SO RULES FOR HANDLING GENOMIC DATA OR TEN SIMPLE RULES FOR ALL KINDS OF SCIENTIFIC PRACTICES. SO WE WERE INVITED TO SUBMIT ONE TEN SIMPLE RULES FOR RESPONSIBLE BIG DATA RESEARCH. THERE'S A LONG LIST OF AUTHORS WE WERE COUNCIL FORB BIG DATA ETHICS IN SOCIETY FUNDED BY NATIONAL SCIENCE FOUNDATION AND HOSTED BY DATA AND SOCIETY, THIS WAS ONE OUR OUTPUTS. SO WE ARE GOING TO TAKE A LOOK AT THOSE TEN SIMPLE RULES BUT ALSO DISCUSS HOW CAN IRBs HELP RESEARCHERS FULFILL THE SIMPLE RULES. SO THESE SIMPLE RULES WERE WRITTEN FOR RESEARCHERS BUT ONE OF THE ASPECTS OF ETHICS REGULAR RATION IS OF COURSE HELPING RESEARCHERS DO BETTER. NOT JUST LAYING DOWN RULES AND RENDERING JUDGMENTS BUT TO HOPEFULLY IN THE BEST SPIRIT OF RESEARCH ETHICS WE ARE ALL HELPING EACH OTHER DO BETTER. I'M ALSO OFFERING SHORT -- IRB ORIENTED VERSIONS OF WHAT THESE RULES MEAN SO NUMBER ONE, ACKNOWLEDGE THAT DATA OR PEOPLE CAN DO HARM. I THINK HOW IRBs HANDLE THIS IS TO START WITH THE ASSUMPTION THAT DATA ARE PEOPLE EVEN IF THE DATA SCIENTISTS NEVER INTERVENE WITH THOSE PEOPLE. SO AS WE DISCUSSED, DATA SCIENCE HAPPENS AT -- HAS A DIFFERENT TEMPALTY TO IT IN RELATION TO COLLECTION OF DATA, IT'S TYPICALLY IN A POSITION OF USING DATA THAT SOMEONE ELSE HAS COLLECTED. DATA SCIENTISTS ARE OFTEN NOT RESPONSIBLE FOR CONDITIONS OF THE COLLECTION. IRBs HELP DATA SCIENTISTS UNDERSTAND THAT. THE CATEGORIES ALONE ARE NOT SATISFY GUIDE TO ETHICAL RISK. TWO, RECOGNIZE THAT PRIVACY IS MORE THAN A BINARY VALUE. ADDRESS EXTENDED ECOSYSTEM PRIVACY RISK. START WITH THE ASSUMPTION DATA IS REPURPOSED RECOMBINED WITH THIRD PARTY DATA SETS NOT NECESSARILY KNOWING IN ADVANCE ASK RESEARCHERS TO CONSIDER THE CONTEXT OF COLLECTION, AND ADHERE TO THAT CONTEXT. SO AGAIN, THE INTEGRITY, COMES UP ONCE AGAIN, A VERY USEFUL FRAMEWORK. CONSIDER COLLECTION RATHER THAN SITTING WITH THE BINARY OF PUBLIC AND PRIVATE THREE, GUARD AGAINST THE REIDE IF ICATION OF YOUR DATA. FORB IRBs ASK RESEARCHERS TO CONSIDER ADVERSARIAL IRB DATA FROM RELEASE SETS. WE KNOW DATA IS SOME WAYS A MYTH. THERE'S MEN I HAVE TO REASONS TO WANT MATHEMATICALLY SECURE APPROACHES. IT IS USEFUL TO THO THROUGH ADVERSARIAL ATTEMPT AND ULTIMATELY WHETHER SOME DATA SETS MADE PUBLIC OR SOME MODEL. THE IRB VERSION REQUIRE RESEARCHERS -- FOUR PRACTICE ETHICAL DATA SHARING FOR IRB REQUIRE RESEARCHERS TO CONSIDER THE DOWNSTREAM CONSEQUENCES OF RELEASES MODELS TRAINED ON HUMAN SENSITIVE DATA EVEN IF DATA PUBLIC. SO ON THE ONE HAND THE COMMON RULE DOES NOT PERMIT YOU TO RENDER JUDGMENT ABOUT THE STATUS OF THE RESEARCH BASED ON DOWNSTREAM EFFECTS. ON THE OTHER HAND IRBs CAN SAY THING THAT ARE OUTSIDE OF THE COMMON RULE THEY CAN'T RENDER COMMON RULE JUDGMENT. YOU CAN SAY TO RESEARCHERS CONSIDER THIS. THE INSTITUTION IS FREE TO ASK WHAT IT WILL. OF RESEARCHERS. THERE'S SOME SPACE TO PLAY THERE AROUND WHETHER OR NOT ISSUES OF JUSTICE SHOULD BE PRESENT. THE IRB STILL AS A RESPONSIBILITY TO ASK THOSE EXTENDED QUESTIONS, EVEN IF THEY CAN'T SAY THAT IT E NOT EXEMPT. CONSIDER STRENGTHS AND LIMITATION OF DATA, BIG DOES NOT AUTOMATICALLY MEAN BETTER. IRB SHOULD CONSIDER WHETHER BIG DATA STUDIES PARTICULARLY THOSE WITH SENSITIVE DEVIATIONS OF PEOPLE OF VULNERABLE POPULATIONS. MEET NORMS OF JUSTICE. CONSIDER WHETHER SUCH STUDIES MAY TECHNICALLY QUALIFY EXEMPT YET FAIL TO SATISFY CORE PRINCIPLES OF BELMONT ORDER. BENNY FISHES COULD GO ALONG WAY TO ADDRESS THESE RISKS. DEBATE, TOUGH ETHICAL CHOICES T. WE REALLY WANT RESEARCHERS TO BE ENGAGED IN ETHICAL -- IN DEBATES ABOUT ETHICS. IT'S I CAN'T TELL YOU HOW MANY TIMES AS THE ETHICIST I GO LAST. TYPICALLY FIRST OR LAST ACTUALLY BUT USUALLY LAST BECAUSE ETHICS COMES AT THE END. WE WANT TO ENCOURAGE RESEARCHERS TO ENGAGE IN DEBATES, WE ALSO THINK IRB MEMBERS SHOULD BE ACTIVELY REACHING OUT TO DEPARTMENTS AT YOUR INSTITUTION T AT YOUR INSTITUTIONS THAT ARE INVOLVED IN DATA SCIENCE. THAT'S FOR GOOD HISTORICAL REASONS, MATHEMATICIANS, STATISTICIANS, COMPUTER SCIENTISTS THE PEOPLE WHO MAKE UP THE CORE OF THE DISCIPLINE OF DATA SCIENCE, METHOD DO LOGICAL CORE HAVEN'T BEEN TRAIN IN RESEARCH ETHICS THE WAY A PSYCHOLOGIST OR PHYSICIAN HAS BEEN. FOR GOOD REASON. LACK OF CAPACITY, IRBs AND OFFICES OF RESEARCH PROTECTIONS AND INSTITUTIONS SHOULD BE REACHING TOUT THESE DEPARTMENTS AN HELPING TRAIN THEM UP. AT THE LEAST BECAUSE YOU HAVE EXPERTS WHO CAN PARTICIPATE ON YOUR REVIEW PANELS. OFFER TRAINING AND GUIDANCE AND MAKE SURE EXEMPT DOESN'T EQUAL RESPONSIBLE OR ELSE I WILL CAPTURE THEM AND THEY WILL BE MY CASE STUDY. DEVELOP A CODE OF CONDUCT FOR YOUR ORGANIZATION, RESEARCH COMMUNITY OR INDUSTRY. IRB SHOULD WORK ON ESTABLISHING INTERNAL STRATEGY FOR ADDRESSING RISK TO SPECIFIC ANALYTICS AND AMI RESEARCH. YOU SAW GREAT EXAMPLES OF THAT. FROM JOHNS HOPKINS. THESE NORMS ARE EVOLVING, CONTRIBUTE TO EVOLUTION BY DEVELOPING A LOCAL STRATEGY AND POLICIES WITH SHARING THEM. DESIGN YOUR DATA AND ECOSYSTEMS FOR ARTIBILITY. ASK RESEARCHERS HANDLING SENSITIVE DATA HOW THEY WILL AUDIT AND AMELIORATE BIAS AND FAIRNESS RISK SPECIFC TO DATA ANALYTICS. AND MACHINE LEARNING. AND EMPHASIZE OBLIGATION TO ACT CONSISTENTLY WITH DUTIES BENEFICIAL TO JUSTICE. THE WAY THE REGULATORY LANDSCAPE IS MOVING, WILL BE TOWARDS AUDITING FAIRNESS AND BIAS AND AI AND ML. IMPACTS HUMAN LIVES SO IF YOU LOOK AT THE ALGORITHMIC ACCOUNTABILITY ACT PROPOSED BY SENATORS WIDEN AND BOOKER AND REPRESENTATIVE CLARK, IN THIS CONGRESS ESSENTIALLY IS MODELED ON ENVIRONMENTAL IMPACT REPORTS, WHICH IS SELF-REPORTING. CURRENTLY SYSTEMS OUR LANGUAGES CONCEPTS OR HABITS ALL RUN AGAINST AUDITTIBILITY AND MACHINE LEARNING. BUT THAT IS THE DIRECTION REGULATION IS GOING AND IT WILL BE HELPFUL IF IRBs ENCOURAGE RESEARCHERS TO THINK ABOUT THAT. NINE, ENGAGE WITH THE BROADER CONSEQUENCES OF DATA AND ANALYSIS PRACTICES. IRB VERSION. MANY IF NOT MOST HARMS OF DATA RESEARCH DON'T DIRECTLY EFFECT SUBJECTS OF RESEARCH. IT'S POSSIBLE TO ADDRESS DOWNSTREAM RISKS DURING COLLECTION. LOOK FOR SIGNALS OF ESSENTIALLY BAD BEHAVIOR ON THE PART OFFER O RESEARCHERS AROUND DECEPTIVE RECRUITMENT TACTICS. IF YOU ARE -- IF THERE IS VIRAL QUIZZES THAT'S PROBABLY ON THE OUTS NOW BUT AS YOU SAW WITH CAMBRIDGE ANALITICA. VIRAL QUIZZES ARE DECEPTIVE AND THEY LEVERAGE THE WEAKNESSES ON THE PLATFORM. IN ORDER TO GET PEOPLE TO PARTICIPATE. WE ALSO SEE A LOT OF MISTREATMENT OF WORKERS WHETHER UNDERPAYMENT OR ASKING THEM TO VIOLATE TERMS OF SERVICE AND WE SHOULD BE CAREFUL ABOUT THAT, IT ALSO TENDS TO LEAD TO BAD DATA. THEN VIOLATION OF TERMS OF SERVICE IS IMPORTANT DISCUSSION. NOT ALL TERMS OF SERVICES ARE IN VIOLET. THEY ARE WRITTEN IN PART TO COVER THE BUS IN THE LEGAL DEPARTMENT AT THE PLATFORM BUT IT CAN ALSO BE A SIGNAL OF LACK OF CARE OPT PART OF RESEARCH RESEARCHER AND DEFINITELY CONSIDER CONTEXTUAL INTEGRITY. TEN, KNOW WHEN TO BREAK THESE RULES. WE ASK RESEARCHERS TO UNDERSTAND THE RESPONSIBILITY BIG DATA RESEARCH DEPENDS ON MORE THAN MEETING CHECKLISTS, WE COULD PROBABLY USE MORE CHECKLISTS. WE NEED -- WE COULD USE MORE ACTUAL CONCRETE RESOURCES. ONCE WE HAVE THOSE RESOURCES THEN WE CAN MAKE MORE JUDGMENT ABOUT WHEN YOU SHOULD BREAK THEM. I THINK WE COULD USE MORE CHECKLISTS. PARTIALLY SUPPORTED BY NSF WITH TWO DIFFERENT AWARDS. THANK YOU OF COURSE TO MY CO-AUTHORS FROM THAT ORIGINAL PAPER. [APPLAUSE] >> JAKE, I HOPE YOU REALIZE THAT IT WAS PURPOSEFULLY DONE THAT WE STARTED WITH AN ETHICIST AND ENDED WITH AN ETHICIST. SOME OF THE COMMENTS AND QUESTIONS THAT WE HAVE COVERED TODAY, I'M ALMOST A LITTLE FRUSTRATED BECAUSE AS MICHAEL POINTED OUT THESE WACOS BUCHANAN AND S STARTED ASKS QUESTIONS 12 YEARS AGO SO WHAT DOES THAT SAY? YOU DID A GOOD JOB TALKING HOW THE SPACE IS SHIFTING SO FOR IRBs, THE CHALLENGE HAS -- IT'S BEEN ENORMOUS. BECAUSE WE HAVE TO KEEP RETOOLING AND RELEARNING. AND LEARNING THESE DIFFERENT DISCOURSES, AND TALKING TO COMPUTER SCIENTISTS AND PEOPLE THAT WE ON THE IRB SIDE ARE NOT USED TO BEING IN OUR SPACES. SO I'M THINKING AHEAD SO WE HAVE GONE THROUGH THESE VERY NICE EASY PHASES OF INTERNET RESEARCH AND SOCIAL MEDIA RESEARCH AND WE KIND OF KNOW WHAT YES DOING THERE. YOU GUYS COME ALONG AND YOU PRESENT PERVASIVE AI MACHINE LEARNING STUFF THAT,s THE NEXT PIECE FOR IRBs TO GRAPPLE WITH. THE QUESTION I WANT TO THROW OUT THERE AND SEE, I LOVE COMMENTS TOO FROM FOLKS ONLINE, HAS TO DO WITH SOMETHING THAT IS IN THE COMMON RULE. WITH THE FINAL RULE THERE ARE DEFINITIONS OF IDENTIFIABLE PRIVATE INFORMATION AND IDENTIFIABLE BIOSPECIMENS. THERE IS A STIPULATION IN THERE OR A REQUIREMENT IN THERE THAT THESE DEFINITIONS WILL BE REEXAMINED EVERY FOUR YEARS. SO I'M CURIOUS, I WOULD LOVE TO BE THE FLY ON THE WALL IN THOSE ROOMS BUT CURIOUS FROM THE PANEL, IF YOU WERE SITTING IN A CONFERENCE ROOM LIKE THIS AND YOU HAD REPRESENTATIVES FROM NIH OHPR, NIST, DOD, ALL ACROSS THE TABLE, WHAT WOULD THOSE CONFERENCES LOOK LIKE? WHAT WOULD THEY LOOK LIKE A YEAR FROM NOW, WHAT MIGHT THEY LOOK LIKE? I'M ASKING YOU TO PUT YOUR CRYSTAL BALL IN FRONT OF YOU. BUT I'M CURIOUS, HOW -- LOOKING BACK A WHAT WE HAVE SEEN WITH THE EVOLUTION OF INTERNET RESEARCH AND HOW WE THINK ABOUT PERSONAL INFORMATION AND SO FORTH TO WHERE WE ARE NOW, ANY THOUGHTS, MICHAEL? >> I COULD HOPE THAT THE FIRST MEETING TOMORROW WOULD BE THERE IS NOTHING THAT IS NOT PERSONALLY IDENTIFIABLE INFORMATION. EVERY FOUR YEARS WE ARE NEEDING TO SEE HAVE WE SOLVED THAT PROBLEM YET. BUT THERE IS DIFFERENTIAL PRIVACY OR OTHER HEAR MORE TALK TALKING ABOUT USING CLEAN DATA ROOMS WHICH I HAVE HAD COLLEAGUES PLAY AROUND WITH. GLAD TO SEE IT'S OPERATIONALIZED. SO NOW WE HAVE TO FIND NEW SOLUTIONS. TO LEVERAGE DATA IN WAYS THAT REDUCES HARM THAT NOTION IS GONE. JUST SWITCH THAT CONVERSATION A LITTLE. >> SO I WOULD RECOMMEND REWRITING THAT SECTION OF THE COMMON RULE. SERIOUSLY, THE GEOPOSITION OF A BUS OVER TIME SCHOOL BUS OVER TIME IT STOPS TO PICK UP THE KID AT SCHOOL, SUDDENLY THAT'S IDENTIFIABLE INFORMATION ABOUT WHETHER THE KID WENT TO SCHOOL OR NOT. SO ALL INFORMATION IS POTENTIALLY IDENTIFIABLE RECORDED ON PLANET EARTH BECAUSE THERE'S LOTS OF PEOPLE ON PLANET EARTH AND YOU CAN GET SIGNAL OUT. IT IS A FLAWED DEFINITION. >> FROM IF YOU ARE FOCUSING ON BIOSPECIMEN THAT WAS COLLECTED AND RESEARCH WORLD AND THERE IS NO OTHER DATA FOR THAT PERSON THAT'S THE DIFFERENCE BUT THINK ABOUT DATA COLLECTED THROUGH OTHER MEANS TO THIS DAY AND AGE YOU ARE REALLY INTO INTERTWINE TECHNOLOGY, SIMILAR TO ECHO. HEARING EVERYTHING WHETHER RECORDED DO ANYTHING TO YOU ABOUT AFTER THAT? NO ONE KNOWS. CELL PHONE, PEOPLE CAN TURN ON CELL PHONE FROM OUTSIDE WORLD AND YOU HAVE NO IDEA. SO TERMS OF TODAY DAY AND EDGE YOU CAN BE CAREFUL AS YOU CAN BEBUT YOU ARE LIVING IN THE DIGITAL WORLD AND IT IS EVERYWHERE T. IF GROW PROTECT DEFIED MID FACEBOOK DATA, GOOD LUCK, IT'S NOT POSSIBLE BECAUSE SO MANY THINGS YOU CAN CLICK, MANY VARIABLES THAT SOMEONE CAN FIND ALGORITHM AND GET TO IT SO THE MORE TOPIC FOCUS NOT TO MAKE IT DEIDENTIFY IS HOW TO MAKE USE OF THE DATA NOT PERVASIVE, THAT WILL BE THE MAIN CHALLENGE VERY DIFFICULT TO DEIDENTIFY EVERYTHING AT THIS POINT. >> FROM IF I LOOK AT THE DEFINITIONS AS WORDED, WHAT WOULD BE PARTICULARLY HELPFUL, WOULD BE TO SEE THAT PRAISE ABOUT -- PHRASE ABOUT MAYBE REALLY AS CERTAINED BECAUSE THIS IS REALLY USUALLY THE STICKING POINT WITH OUR RESEARCHERS. WHERE YOU SAY OKAY, TELL ME ABOUT WHAT YOU ARE DOING AND COULD THE IDENTITY OF AN INDIVIDUAL BE READILY AS CERTAINED BY WHAT YOU DOING AND THEY SAY NO. AND YOU SAY I DISAGREE BUT NOT QUITE SURE WHY. SO I THINK GIVING THE LITTLE BIT MORE DETAIL ABOUT HOW TO MAKE THAT ASSESSMENT AS OUR ABILITY TO USE INFORMATION CHANGES, SO THE DEFINITIONS WERE TO REMAIN AS THEY ARE, HOW DO WE MAKE THAT ASSESSMENT ABOUT COULD THE IDENTITY BE READILY ASCERTAINED WOULD BE EXTRAORDINARILY >> I WANT TO TALK ABOUT THE CONCEPT, MEGAN, THIS CAME UP IN YOUR TALK. THE CONCEPT OF MINIMAL RISK. WE HAVE BEEN THINKING ABOUT AND WHAT I TOOK FROM YOU IS YOUR RE-EVALUATING THAT CONCEPT IN A SENSE, WHAT IS MINIMAL RISK LOOK LIKE, WHAT IS IT IN THIS ENVIRONMENT. AS WE ALMOST EACH AND EVERY DAY HEAR ABOUT DIFFERENT DATA BREACHES, WHETHER IT'S TARGET OR EQUIFAX, ET CETERA, THAT IS BECOMING OUR NORMAL. THAT DATA AREN'T SAFE. SO TALK ABOUT THAT A LITTLE BIT MORE. I WOULD LOVE TO HEAR FROM YOU WHAT DOES THAT MEAN FOR US IN THE REGULATORY SPACE? IS THAT DEFINITION ALSO CHANGING? WHAT DO WE DO WITH MINIMAL RISK? I THINK THAT IS A GOOD CHALLENGING POINT WHICH IS THAT WE MAY ALL BE FACING MORE INFORMATIONAL RISK IN OUR DAY THE DAY LIFE WHEN WE LOOK AT THE CONCEPT OF WHAT MINIMAL RISK IS. BUT WE ALSO NEED TO THINK ABOUT WHETHER THE RESEARCH IS INTRODUCING A RISK THAT IS DIFFERENT THAN THAT. IS WHAT WE ARE DOING WITH THAT DATA INTRODUCING A DIFFERENT LEVEL OF RISK THAN THIS VERY SAME RISK THAT ALREADY EXISTS IN OUR DAY TO DAY LIFE? AND THE ANSWER MAY BE NO. BUT I THINK THAT THIS IS A VERY BIG CHALLENGE. ONE WAY WE ADDRESS IT IS TO SIMPLY SAY WE MIGHT TREAT IT AS IF IT COULD BE MORE THAN MINIMAL RISK. AND SO WE'LL OFFER AS MANY PROTECTIONS TO IT AS WE CAN TO REDUCE THAT, RATHER THAN MAKE A DEFINITIVE ASSESSMENT THAT SURE THIS DATA IS USE OF THIS DATA IS MINIMAL RISK VERSUS GREATER THAN MINIMAL RISK WE HAVE SAID WE HAVE TO TAKE THE HIGHER ROAD AND SAY WE HAVE TO AFFORD AS MANY PROTECTIONS AS POSSIBLE AND FOCUS ON RISK REDUCTION. AND I THINK THAT IS WHAT ORGANIZATION VERSUS DONE. >> SOME ARGUE AGAINST THAT RESEARCH EXCEPTIONALISM, THAT THAT PIECE THAT WE ARE MAKING THIS WORSE OURSELVES. >> THE PROBLEM WITH MINIMAL RISK FRAMEWORK IS IT DOESN'T COMPOSE THERE MIGHT BE THREE STUDIES WHICH DONE TOGETHER EACH IS MINIMAL RISK. BUT WHEN ALL RELEASED THEY CREATE SIGNIFICANT RISK FOR THE STUDY PARTICIPANTS. THOSE STUDIES HAPPEN AT DIFFERENT INSTITUTIONS. AND IT'S MORE LIKELY TO HAPPEN WITH LARGE DEATH SETS THAN SMALL DATA SETS BECAUSE MORE LIKELY TO BE OVERLAP BETWEEN THE SUBJECTS AND LARGE DATA SETS SO IT IS A FUNDAMENTAL PROBLEM NOT ADDRESSED BY THE CURRENT REGULATIONS. >> A LOT OF CONVERSATIONS HAVE WORKED AROUND THE ISSUES OF DATA SECURITY AND WE HEARD NUMBER OF COMMENTS AND TALKS TODAY ABOUT DATA SECURITY. SORRY TO POINT OUT MEGAN'S TALK BUT AS YOU WERE TALKING YOUR DATA SECURITY MATRIX AND NUMBER OF IRBs ACROSS THE COUNTRY HAVE GONE IN THAT ROUTE THEY WORKED WITH THEIR IT DEPARTMENTS ANDS ACROSS CAMPUS, TO HAVE MORE A COLLABORATIVE DATA SECURITY PACKAGE IF YOU WILL SO TO MICHAEL AND ADARSH AND JAKE SPECIFICALLY, DO YOU FEEL THAT THOSE EMERGING DATA SECURITY MATRIXES MATRIXES FRAMEWORKS, ARE THEY ENOUGH? ARE THEY SUFFICIENT IN A CRUDE PRACTICAL SENSE MAY BE BUT FROM A PHILOSOPHICAL SENSE DON'T GET TO NORMS AND VALUES THAT JAKE, YOU HAVE RAISED TODAY PARTICULARLY. >> YOU ARE RYE. AS OPERATIONAL TOOL THAT CAN BE HELPFUL TO HIGHLIGHT AND ELIMINATE WHEN WE NEED TO START FOCUSING ON SOMETHING MORE SPECIFICALLY. BUT THE WAY YOU JUST FRAME THAT QUESTION, ALSO REMINDED ME HEARING OUR SECOND SET OF TALKS AND GREAT WORK BEING DONE BY DEBRA AND REBECCA AND HAVING THESE INSTITUTIONS TAKING THESE IMPORTANT STEPS TO PROTECT DATA AND HEARING ABOUT HOW APPLE IS EMBRACING DIFFERENTIAL PRIVACY AND NOW WILLING TO PAY MORE FOR APPLE PRODUCTS BECAUSE THEY WILL PROTECT DATA. MY CREATING FALSE SENSE OF SECURITY AROUND DATA, I'M WILLING TO TAKE ON MORE MINIMAL RISK THINGS BECAUSE I KNOW THAT THERE IS SOME THINGS AND THINGS FALL THROUGH THE GAPS BECAUSE THINGS GET GYNOD AND THERE'S ALL KINDS OF DIFFERENT WORRIES. THAT'S WHERE WE START SEEING MORE COMPLICATED WAYS OF THINKING ABOUT PRIVACY, LIKE WHAT THEY WERE SAYING EARLIER, NOT JUST PRIVACY PARADOX, IT'S COMBINATION OF PRIVACY FATIGUE AND APATHY AND THE ORDER OF CONVERSATION THAT IT CAN HAPPEN. SO THAT'S -- YOUR POINT WE HAVE THIS CLEAN MATRIX DOES IT OVERSIMPLIFY THINGS, THAT'S A REASONABLE CONCERN. >> I THINK OTHER THING TO NOTE IF SOMEBODY WHO IS ID MAY NOT HAVE THE SAME KNOWLEDGE OF DATA IT COULD BE IN SCHOOL -- GET ANY PERSON THEY DON'T HAVE THE SIMILAR KNOWLEDGE OF DATA SECURITY THEY MAY KNOW ABOUT COMPUTER TELL YOU EVERYTHING ABOUT COMPUTER SECURITY BUT NOT HOW THE DATA IS USE SOD NOT REALLY DATA SCIENTISTS PER SE. WE CAN INVOLVE DATA BUT CANNOT GET SIMILAR EFFECT IN EVERY INSTITUTION BECAUSE IT COMES DOWN TO THAT PARTICULAR INDIVIDUAL. THAT'S ONE THING. THE ONE THING THAT WAS MENTIONED THIS MORNING, IN TERMS OF TEACHING THE DATA SCIENTISTS ABOUT THE RESEARCH THAT WOULD BE AREA, SO THEY CAN UNDERSTAND WHAT ARE THE CONCERNS, THEY ARE DOING THEIR DATA AND DOING THE DATA AND THAT'S COOL STUFF, I CAN FIND THIS OUT. BUT REALITY IS YOU KNOW WHAT THE IMPLICATION OF THAT IS GOING TO BE. IF THEY UPS THAT THAT'S WHERE WE HAVE BETTER SOLUTION. SO MAYBE PROPOSE A MANY -- FOR DATA SCIENTISTS. >> HER NEXT PROJECT SHOULD BE INTERVIEW DATA SCIENTISTS AND HAVE THEM TROUGH THEIR IDEA OF PRIVACY. MIGHT UNLEASH SOME INFORMATION. LET ME GO TO THAT IDEA THEN. SO WE TALKED DATA SCIENTISTS A LOT TODAY. AND FEW IN THE ROOM ARE SUCH. WHAT DO WE NEED TO DO TO TRAIN THEM, THEM, AND HOW DO WE BRING THEM INTO THIS COMMUNITY? HOW DO WE TRAIN DATA SCIENTISTS TO SERVE ON IRBs TO HELP AS RESEARCHERS IN THESE SPACE? >> YEAH. ONE OF OUR COLLEAGUES PROJECT CASEY HAS PUT TOGETHER ONLINE GOOGLE DOC TRACKING ALL COURSES THAT SOMEHOW ENGAGE WITH EFFECTS OF TECHNOLOGY AND WE ARE LEVERAGING THAT AND ONE PROJECT I'M WORKING ON NOW IS WE COLLECTED 650 DATA SCIENCE PROGRAMS DEGREE PROGRAM OR CONCENTRATION AND GOING THROUGH CURRICULA, LEARNING OUTCOMES TO TRY TO SEE THOSE ETHICS SHOW UP. AND FROM THAT, HOW CAN WE BUILD IT IN. THAT'S ONE OF OUR GOALS TO TRY TO UNDERSTAND THIS LANDSCAPE. IN MY OWN CASE, BEING COMPUTER SCIENTIST I CONVINCED -- TO BRING ME IN TO BASICALLY INTRODUCE ETHICS TO CURRICULUM AND ALSO CO-DIRECTOR OF DATA SCIENTIST PROGRAM, NOT A DATA SCIENTIST EITHER. SO IT'S WORKING SO FAR. WE ARE STARTING TO SEE GREATER REALIZATION WITHIN PART OF THE COMPUTER SCIENCE AND INFORMATION SCIENCE COMMUNITIES OF RECOGNIZING WE NEED TO BE ENGAGED IN THESE PRINCIPLES OR DEBATE NOW IS HOW TO DO THIS MAKING SURE IT'S NOT WEEK 15 OF CLASS INTEGRATING INTO THE CURRICULUM. WORKING ON IT BUT WE HAVE PLACES TO GROW. INDUSTRY DOES BETTER ACADEMIA BETTER OR WORSE IS PAYING ATTENTION TO CULTURE. THAT'S ACADEMICS WE ARE INDIVIDUAL PINNACLES OF EXCELLENCE. INFORMAL MOTTO OF CERTAIN WEST COAST UNIVERSITY. LOCATED NEAR SILICON VALLEY. FOR BETTER OR WORSE AND INDUSTRY YOU HAVE THE PAY MORE ATTENTION TO WHAT KIND OF CHURL ARE YOU CULTIVATING. AND ACTUALLY THERE IS MORE EXPERTISE THERE. THINKING THROUGH HOW DO WE BUILD RACE TO TOP IN OUR OWN ORGANIZATION? I THINK THAT'S SOMETHING THAT WE SHOULD PAY MORE ATTENTION TO. LET'S ACKNOWLEDGE IT'S A NASCENT FIELD. PEOPLE ARE INVENTING METHODS AND NEW TECHNOLOGIES, EPISTOMOLOGIES, DATA SCIENCE IS ONE F THE MOST INTERDISCIPLINARY PERSPECTIVES, AND HISTORY OF DISCIPLINES. DRAWING FROM SOCIAL SCIENCE, PSYCHOLOGICAL SCIENCE, MATHEMATICS, STATISTICS COMPUTER SCIENCE ENGINEERING, ALL THOSE PEOPLE HAVE A PLACE IN THIS NEW DISCIPLINE. SO IT IS GROWING BUT THERE NEEDS TO BE MORE ATTENTION TO HOW WE MODEL BEHAVIOR TO THE STUDENTS SO TECH COMPANIES ARE ON BOARDING, SOMETIMES THOUSANDS OF 22-YEAR-OLDS A WEEK T. MANY HAVEN'T HAD ANY ETHICS TRAINING. SO WHAT ARE THE THINGS WE NEED TO WE CAN FROM ACTIVELY DO TO BUILD FOOTHOLDS FOR A -- A CULTURE THAT STARTS WITH ETHICAL CONSIDERINGS, CONSIDERATIONS FOR THE COMMON GOOD? >> SO I LIKE TO JUST REMIND US THAT THE ASSOCIATION FOR COMPUTING MACHINERY HAS A CODE OF ETHICS WE SIGNIFICANTLY OWN WHAT WE HAVE ASSESSED TODAY, ACM IT CANNOLOGY COMMITTEE PUT OUTSET OF PRINCIPLES FOR AI AND TRANSPARENCY TWO YEARS AGO. TECHNOLOGY COMMITTEE. AND ALSO THERE'S THE NATIONAL RESEARCH STRATEGY ON PRIVACY THAT CAME OUT A FEW YEARS AGO THAT ARE ALSO SPEAKS TO THIS, SO THOSE ARE JUST ADDITIONAL REFERENCES. PEOPLE HAVE DONE THINKING ON THIS AND ONE OF THE PROBLEMS THAT WE HAVE IS THERE'S BEEN -- IT'S VERY DIFFICULT TO PULL THOSE DIFFERENT STRANDS TOGETHER SO THERE'S LOT OF DUPLICATIVE WORK GOING ON AND NEEDS TO BUILD ON OTHER WORK DONE. >> THERE IS ALSO IEEE WORK BEING DONE, MASSIVE EFFORT DEVELOPING ETHICS AND AI MACHINE LEARNING SYSTEMS GOING ON COUPLE OF YEARS NOW SO IT'S NOT THIS DICHOTOMY OF U US TEACHING THEM OFFER WHATEVER, THIS IS ORGANICALLY WITHIN COMMUNITIES AS WELL TRYING TO FIGURE HOW TO INCORPORATE, OPERATIONALIZED LEVEL. >> WITH RESPECT TO THE IEEE, THAT'S THE ETHICS AND AUTONOMOUS DESIGN. THEY HAVE PUT OUT A TEXTBOOK WHICH IS 200 PAGES LONG, UNLIKE MOST IEEE DOCUMENTS CAN BE FREELY DOWNLOADED AND IT'S LIKE READY TO USE IN A COURSE. THE PROBLEM WITH TEACHING AN ETHICS COURSE, I CURRENTLY TEACHING ONE IN MY PRIVATE CAPACITY, IS THAT THERE'S JUST TOO MUCH MATERIAL TO PUT INTO ONE COURSE, A TYPICAL DATA SCIENCE MASTERS DEGREE MIGHT HAVE 8 TO 12 COURSES NOT HAVE A THESIS, THERE'S MORE THAN TWO COURSES WORTH OF ETHICS MATERIAL THAT NEEDS TO BE PROVIDED TO COVER WHAT'S TAUGHT IN 12 COURSES. >> I WANT TO INTERJECT HERE THIS IS YVONNE. WE HAVE BEEN TALKING ABOUT OTHER RESOURCES AND WE REALLY INTERESTED FOR ALL OF YOU WHO ACTUALLY GIVE US REFERENCES TO THESE RESOURCES AND HOW TO ACCESS THEM. WE CAN APPROVE THEM TOGETHER COMPILE THEM AND PUT IT TOGETHER WITH OUR VIDEO RECORDING. AFTER THIS EVENT WE CAN ASK MORE INFORMATION. >> I'LL BE A SECOND. SINCE IT WAS JUST DESCRIBING I JUST DESCRIBED TO DO ETHICS CLASS FOR SCIENCE PROGRAM, WE JUST NEED TO HAVE 30 YEARS OF THIS WORK. IT'S ALMOST LIKE AFTER EVERY LECTURE ANY OTHER CLASS YOU CAN GO SPEND AN HOUR WITH ME, BECAUSE THERE'S SO MUCH FOR US TO COVER, WE NEED TO FIND A WAY TO COMPARTMENTALIZE THAT. A TEXTBOOK MIGHT START IT. >> I WANT TO SECOND THAT A SECOND COURSE IS NOT WHAT WE WOULD GO FOR. I THINK BACK TO MY TRAINING AND IN CLINICAL PSYCHOLOGY AND THINK ABOUT THE TRAINING OF MY COLLEAGUES IN MEDICINE AND I EVEN THINK A BIT BEYOND ETHICS IN MORE AROUND THE CONCEPT OF SOCIAL RESPONSIBILITY RAND PROCESS PROFESSIONALIZATION SOME COMPANIES TWO THROUGH. IT SOUNDS LIKE MANY ORGANIZATIONS HAVE STARTED DEVELOPING CODES AND I THINK IT WOULD BE GREAT TO THINK HOW TO THE VERY EARLY STAGE OF TRAINING AND THINK A BROADER VIEW PROFESSIONALIZATION. WHAT MIGHT BE DATA SCIENTIST THINK ABOUT HIPPOCRATIC OATH AND IF WE USE MEDICAL SCHOOL ANALOGOUS IDEA, WHAT DO OUR DATA SCIENCE PROGRAMS THEN LOOK LIKE, FEEL LIKE, WHAT DO THEY VALUE? GREAT WAY TO THINK ABOUT IT. THANK YOU. WE PICKED ON OHRP A LITTLE BIT IN TERMS OF PHRASING WE ARE NOT COMFORTABLE WITH, READILY AS CERTAINABLE, MONEY MALL RISK. I WANT TO THOUGH OUT ONE MORE. WE ALSO HAVE DANCED AROUND THIS THROUGHOUT THE DAY. WE TALKED ABOUT THIS IDEA OF HARM, RISK AND HARM, IN SOME INTANGIBLE WAYS. I WOULD LIKE TO MAKE THIS QUESTION, LIKE TO THROW OUT THIS QUESTION, WHEN AN IRB LOOKS A PROTOCOL AND HAS TO DETERMINE OR EVALUATE THE LIKELIHOOD AND PROBABILITY OF RISK OR HARM, THAT'S OFTEN A QUANTITATIVE MEASURE. HOW DOES THAT FIT? DOES IT FIT? WHAT OTHER LANGUAGE CAN WE USE TO THINK ABOUT LIKELIHOOD PROBABILITY OF HARM OR RISK, WHEN WE ARE EVALUATING A PROJECT THAT INVOLVES PERVASIVE DATA? A BENEFIT RISK ANALYSIS IS DESIGNED TO BE QUANTITATIVE MEASURE AND QUANTIFYING THINGS NOT NECESSARILY USUALLY QUANTIFIED AND SHOULDN'T BE THE FINAL SAY BUT THAT'S AN AREA IN MY PRESENTATION THIS MORNING I WAS TALKING ABOUT AND I STARTED TO ASK THIS QUESTION TALKING LOW RISK EARLIER, IS THE RISK NEVER EXISTS APART FROM THE HARM. SO IT'S A LOW RISK OF SOMETHING HAPPENING OR A LOW RISK OF THE HARM BEING LOW. TWO RISK COMBINATION. SO LOW RISK HAPPENING AND THAT'S A REASONABLE THING THE PURSUE WITHOUT SPECIAL PRECAUTION IS HIGHER RISK HAPPENING RANS HIGH HARM BY WHATEVER DEFINITION HAPPENING AND THIS IS A DANGEROUS THING TO BE DOING AND NEED ALL PROTECTIONS WE CAN GIVE IT OR NOT DO IT AND THAT MATRIX IN BETWEEN WITH OTHER CORNERS, JAKE HAD A SLIDE AS WELL WITH OPPOSITE CORNERS OF THAT HAPPENING. AS LONG AS WE UNDERSTAND QUANTITY OR MEASURE WE USE IS INHERENTLY SUBJECTIVE AND DEPENDS ON WHERE YOU STAND IN RELATION TO THE HARM, IT'S STILL -- THERE'S NO WAY AROUND USING THAT AS ONE OF THE TOOLS OR ONE OF THE MEASURES TO EVALUATE THOSE SITUATIONS EVEN HOPEFULLY THERE ARE OTHER MEASURES OF VALUES BASED OR FROM COMPANIES'S POINT OF VIEW, CORPORATE MARKET FORCES OR OTHER THINGS THAT ARE FACTOR Z AS WELL. THERE IS AN EXISTING CONSTRUCT ON THE MANUFACTURING SIDE OR INDUSTRY SIDE, FOR REGULATED PRODUCTS THAT ACTUALLY IS QUITE USEFUL I FIND, WE START TALKING RISK IN THIS CONTEXT AND IT'S DESIGN HISTORY AND FMEA FAILURE METHOD EFFECTIVE ANALYSIS. A METHODICAL WAY OF THINKING HOW YOUR PRODUCT GOES SOUTH. IN COMMERCIAL USE. WHEN IT DOES, HOW DOES IT IMPACT THE PEOPLE THAT THAT PRODUCT IS USED ON OR IN OR IN EPISODE OF CARE FOR. IT'S VERY LOGICAL, METHODICAL AND FITS INTO THE CONSTRUCT OF THE WAY WE THINK ABOUT PRODUCT DEVELOPMENT AND HEALTHCARE AND HAS A ROLE IN THE CONTEXT OF IMAGINING RISK ASSOCIATED WITH THE KINDS OF ALGORITHMS, THE SOFTWARE, THE KIND OF TOOLS WE ARE TALKING ABOUT IN THE CONTEXT OF AN IRB. SO WE THINK ABOUT OURISH SUBMISSIONS, IT OFTEN INTERSECTS WITH HOW WE THINK ABOUT PRODUCT DEVELOPMENT AND FMEA. A LOT OF INDUSTRY NEEDS TO GET THERE. IF THEY ARE GOING TO BE TECHNOLOGY COMPANIES WORKING IN HEALTHCARE. >> I THINK ONE OF THE WAYS I HAVE BEEN THINKING ABOUT THIS PROBLEM IS MOVING AWAY FROM HARM AND INTO THINGS LIKE DIGNITY. HARD TO OPERATIONALIZE. THAT'S WHAT'S CAUSING THE PROBLEM. BECAUSE YOU ARE IN A ROOM OF PEOPLE THAT NEED TO TRY TO OPERATIONALIZE THESE THINGS TO GET THINGS DONE. WE SPOKE ABOUT THIS OFFLINE EARLIER EVOLUTION OF ASSOCIATION OF RESEARCHERS ETHICAL GUIDELINES OVER THE YEARS MOVED FROM A VERY PRACTICAL IF THIS THEN THAT TO SECOND PHASE MUCH MORE FOCUSED ON LIKE ETHICS OF CARE AND SORT OF TAKING CONTEXTUAL APPROACH AND FEMINIST ETHICS APPROACH WE NEED TO BE -- CARE FOR OUR SUBJECTS AND LET THAT DRIVE WHAT WE DO, VERSUS LIKE IS THERE A TANGIBLE HARM THAT WE CAN PREDICT SO THAT IS A DIRECTION I WANT TO TRY TO SEE US SHIFT TOWARDS BUT I DON'T HAVE THE TOOLS YET. TO ACTUALLY TRY TO DO THAT. >> MUCH OF THE RESEARCH IN THE PERVASIVE DATA SPACE IS UNREGULATED FOR THE PURPOSES OF COMMON RULE. AND EARLIER SOMEONE MENTIONED JAKE OR MICHAEL, MENTIONED FACEBOOK VERSION OF THEIR IRB, RESEARCH ETHICS BOARD. WHAT MIGHT SOME MODELS BE FOR ETHICS REVIEW IN THIS UNREGULATED SPACE? >> DO YOU MEAN INDUSTRY OR IN -- IF IT'S INDUSTRY REGULATED NOT FEDERALLY FUNDED. >> FOR ACADEMIA. MOST WORK WITH THESE -- WITH DATA, IT'S UNFUNDED, UNREGULATED FOR PURPOSES OF THE COMMON RULE. >> ONE OBVIOUS SPACE WHICH ALREADY IS -- SOMEWHAT OCCUPIED IS CONFERENCE COMMITTEES SO COMPUTER SCIENCE, THE CURRENCY FOR ACADEMICS IS CONFERENCE PAPER NOT JOURNAL ARTICLE LIKE IT IS FOR HUMANITIES OR SOCIAL SCIENCE. AND THAT'S A PLACE FOR WHERE PIERCE CAN REVIEW EACH OTHER'S METHODOLOGY AND SAY I THINK WHAT THEY DID WAS HARMFUL AND WE SHOULDN'T ALLOW THIS PAPER TO GO FORWARD PROCEEDING. THE HARM IS DONE AT THAT POINT BUT IT IS A WAY ESTABLISHING PEER REVIEW PEERS CARE ABOUT SOMETHING OTHER THAN IRBs CARE ABOUT. SO THERE ARE -- THERE'S OPPORTUNITIES FOR THAT, THAT KIND OF ALREADY EXIST IN THE SYSTEM. AND A LOT OF COMPUTER SCIENCE AND HUMAN COMPUTER INTERACTION CONFERENCES NOW HAVE SOMETHING ALONG THOSE LINES SO I WOULD LOOK FOR IF THE COMMON RULE ISN'T GOING TO BE ABLE TO HANDLE KINDS OF CHALLENGES THAT DATA SCIENCE RAISES WE SHOULD BE LOOKING FOR OTHER -- I DON'T WANT TO SAY NECESSARILY SAY INFORMAL BECAUSE CONFERENCE COMMITTEE IS FORMAL BUT IF YOU LOOK FOR SECONDARY OR TERTIARY WAYS WHERE PEOPLE WITH RELATIVE EXPERTISE CAN HOLD EACH OTHER IN CHECK. >> SO I THINK CONFERENCE PROGRAM COMMITTEES ARE A TERRIBLE PLACE TO DO IT. HAVE BEEN ON PROGRAM COMMITTEES THAT HAVE DONE ETHIC REVIEWS AND AS YOU SAID, THE RESEARCH IS ALREADY BEING DONE. THERE'S ALSO A FEAR THAT IF IT'S NOT PUBLISHED AT YOUR CONFERENCE IT'S NOT PUBLISHED AT ANOTHER CONFERENCE SO MIGHT AS WELL GET THE GOOD PAPER. PRE-REGISTRATION IS ANOTHER ALTERNATIVE. THERE'S A GROWING INTEREST IN USING PRE-REGISTRATION AND THERE'S GROWING NUMBER OF JOURNALS THAT ARE REQUIRING PRE-REGISTRATION BEFORE INTERVENTION BASED RESEARCH IS DONE T. THAT WOULD BE AN OPPORTUNITY TO DO THE REVIEW. I ALSO THINK THAT THE PROCESS OF GOING TO AN IRB IS VERY DIFFICULT FOR COMPUTER SCIENTISTS BECAUSE THEY HAVE HAD PRACTICALLY NO TRAINING IN DOING EXPERIMENTATION. AND THEY DON'T KNOW WHAT THEY ARE DOING AND IT'S THEREFORE VERY DIFFICULT TO PUT TOGETHER THE IRB APPLICATION THAT IS A GOOD PROCESS FOR COMPUTER SCIENCE TO GO THROUGH. AND WHAT IS NEEDED TO FIX THE IRBs SO THEY HAVE COMPUTER SCIENTISTS ON THE IRBs SO THERE'S A BENEFIT TO BEING ON THE IRB FOR PRE-TENURE FACULTY AND TO HAVE FORMS THAT WILL HELP IRBs REVIEW THESE DATA SCIENCE PROJECTS. IT'S A WORKABLE SYSTEM. >> YOUR HONOR QUESTION IS ACTUALLY BOTH ABOUT PROCESS AND ALSO ABOUT PRINCIPLE. SO IT SEEMS TO BE RIGHT. SO THERE IS ONE KIND OF SUBSET OF IRB ACTIVITIES THAT I THINK ACTUALLY -- THAT HAVE BEEN REFORM SOMEWHAT OVER THE LAST FEW YEARS THAT DO PLAY INTO THIS AND ARE POTENTIALLY USEFUL MODEL. THAT IS THAT I THINK MOST PEOPLE IN THE ROOM MOST IN THE AUDIENCE ARE FAMILIAR WITH WHAT HAPPENED IN THE SUVI CASE, THERE WAS A SMALL TRIBE NATIVE AMERICAN TRIBE AT BOTTOM OF THE GRAND CANYON, HAD -- THERE WERE COMMUNITY CONSULTATIONS, INDIVIDUAL CONSENT THOUGH THE HISTORY I'M SURE ABBREVIATING THE FACTUAL HISTORY, IT'S COMPLICATED BUT THIS IS BASICALLY IT. THERE WAS INDIVIDUAL CONSENT AND A COMMUNITY AWARENESS OF THE FACT THAT THESE SPECIMENS AN DATA WERE BEING COLLECTED FOR RESEARCH ON DIABETES AND LATER SPECIMENS IN DATA AFTER THEY WERE USED FOR THE DIABETES RESEARCH FOR MENTAL HEALTH RESEARCH AND OTHER THINGS AND WHEN THEY WERE USED THEY WERE ON THE DATA WERE -- THE DATA WERE DEIDENTIFIED OR ANONYMIZED OR THE RESEARCHER ACTUALLY TOOK THE IDENTIFIABLE DATA, IDENTIFIED THOSE DATA, IDENTIFIED THE BIOSPECIMENS AND ASKED AN IRB TO WAIVE CONSENT SO IRBs MOST OF THE RESEARCH PERHAPS ALL OF IT BUT AT LEAST MUCH OF THE RESEARCH WAS ACTUALLY DONE UNDER COLOR OF LAW. BECAUSE IRBs DID WAVE CONSENT BUT THE QUESTION IN RETROSPECT WAS SHOULD THEY WAIVE CONSENT AND WHAT ARE THE STANDARDS APPLIED. WHAT'S HAPPENED, YOU KNOW WHAT THE DISCUSSION WAS, THIS WAS A SMALL IDENTIFIABLE GROUP OF PEOPLE, THOUGH DATA MAYBE DEIDENTIFIED OR THE DATA OR EVEN -- BUT THE GROUP IDENTITY IS SO CONSONANT WITH THE INDIVIDUAL IDENTITY THAT DOING SOMETHING LIKE THAT, EVEN IF DOING IDENTIFYING INDIVIDUALS COULD NEVERTHELESS HAVE PROFOUND INFLUENCE ON PER HE WAS O OF THEM AND SMALL GROUP AND UNDERMINE BASIC RELIGIOUS OR OTHER BELIEFS SO IRBs BEGAN TO DEVELOP -- HAVE BEGUN TO DEVELOP GREATER AWARENESS OF QUESTIONS ASKED ABOUT REUSE OF DATA FOR THE PURPOSES OF WAIVER OF CONSENT PROCESS REFINING IMPROVING THE WAIVER OF CONSENT PROCESS. THOSE ARE NOT EXACTLY THE SAME THAT WILL BE USED FOR ALL BIG DATA RESEARCH BUT THEY CERTAINLY ARE RELATED TO THE QUESTIONS USED FOR BIG DATA RESEARCH. >> THERE ARE SOME QUESTIONS ONLINE. THIS ONE IS FOR MEGAN AND DR. GUPTA. WITH INCREASING NUMBER OF CLINICAL TRIALS INCORPORATING ELECTRONIC PATIENT REPORTED OUTCOMES, CAN YOU DESCRIBE ANY SPECIAL CONCERNS FOR THOSE RESEARCH SUBJECTS? >> ONE THING WE FACED, I THINK DR. GUPTA CAN SPEAK TO THIS EVEN MORE SO THAN I. IS TECHNOLOGIES USED TO COLLECT THOSE OUTCOMES, SO ONE OF THE THINGS THAT IS A NEW CHALLENGE, IS RESOURCES THAT ARE INTENDING TO TAP INTO A SYSTEM. FOR EXAMPLE, I WANT TO EXTRACT DATA DIRECTLY FROM YOUR ELECTRON UK MEDICAL RECORD. AND TIE THAT INTO WHERE YOU WILL SELF-ENTER DATA TYPE THIS LOVELY APP. SO WE HAVE HAD TO SPEND MUCH MORE TIME EVALUATING THOSE NEWER TECHNOLOGIES IN ADDITION TO THE QUESTIONS THEMSELVES. BEFOREHAND WE MAY HAVE FOCUSED ON WHAT'S BEING ASKED, NOW WE ARE FOCUSING IN ADDITION TO THAT ON THE RISK OF TECHNOLOGY BEING INTRODUCED TO COLLECT THAT INFORMATION. I THINK THAT WE ARE SEEING MORE OF THAT IN THE COLLECTION OF THE PATIENT REPORTED OUTCOMES IS TRYING TO MAKE THIS MUCH EASIER AND MORE ACCESSIBLE FOR PATIENTS. SO THE DATA CAN BE COLLECTED JUST IN THE COURSE OF NORMAL LIFE BUT WITH THAT COMES ADVANCEMENTS IN TECHNOLOGIES THAT ALSO REQUIRE MORE SOPHISTICATED ANALYSIS BY THE IRB AS WELL. >> I DO -- MEGAN, I THINK THERE'S A COUPLE OF THINGS ONE FACTOR IS HOW ARE YOU COLLECTING DATA FOR THIS CLINICAL TRIAL? IF IF IT'S CHECKED BY BLOOD SPECIMEN, EXAMINATION PHYSICAL EXAM, DOCTORS OFFICES THAT'S MORE CONTROL THE DATA YOU ENTER IN THE DATABASE THAT IS SECURE AND FINE. WHEN YOU START CORRECTING DATA FROM AVAILABLE TEXT THE QUESTION COMES DOWN TO IS HOW MUCH AND HOW CAN YOU LIMIT WHAT ARE YOU GATHERING. YOU WANT TO CAPTURE HOW MANY STEPS THIS PATIENT TOOK OR HOW MANY TIMES HE WAS ACTIVE BUT GETTING ZERO LOCATION, ALSO GETTING WHERE HE IS AT AND HOW DO YOU REALLY LIMIT THAT DATA SO FOR YOUR RESEARCH YOU ONLY GET WHAT YOU INTEND TO GET AND THAT'S THE (INAUDIBLE) MANY TIMES WHEN OWE TRY TO CAPTURE DATA IT IS HARD TO LIMIT THAT, THAT'S WHERE MAYBE DATA STEWARD OR SOMEBODY IN BETWEEN WHO WILL MAKE SURE RESEARCHER WON'T GET EVERYTHING AND SOMETHING SOMEWHERE BLOCKED. >> ANOTHER QUESTION ONLINE. THIS ONE READS, I HAVE QUESTIONS ABOUT THE ROLE OF THE IRB WITH VARIOUS ONLINE PLATFORMS AND ENSURING THOSE WEB SIZE OR SOCIAL MEDIA SITES ARE APPROPRIATELY SECURE FOR THE TYPE OF RESEARCH BEING DONE. AND IN A WAY THAT FOLLOWS WHAT YOU ARE SAYING, DR. GUPTA, ABOUT THE APPLICABILITY AN APPROPRIATENESS OF TECHNOLOGY. ANYONE HAVE FURTHER COMMENTS OR RESPONSES TO THAT? >> SPEAKING IF I'M SHARING RESEARCH DATA ON GOOGLE DRIVE, IS THAT HIPAA COMPLIANT AND -- THOSE ARE IMPORTANT QUESTIOS TO BE ASKING. IT'S A -- ANOTHER SUBPROJECT WE ARE WORKING ON IS LOOKING AT DATA MANAGEMENT PLANS, NSF FUNDED RESEARCH PROTOCOLS THAT USE THIS DATA AND HOW THOSE RESEARCHERS TALK ABOUT HOW ARE THEY STORING AND SHARING DATA TRYING TO UNCOVER SOME OF THOSE MORE INFRASTRUCTUREIAL ISSUES TOO, NO T JUST MY PROCESSING RELEASE BUT WHERE IS MY DATA. IT'S REALLY INTERESTING QUESTION WE NEED TO MAKE SURE WE DON'T LOSE IN THE PROCESS. >> SO I HEARD SCOTT MCNEILY SAID THIS IN 1999 AND I HEARD MARK LAWRENCE SAY IN IN 2019 ABOUT GET OVER IT. IF YOU DON'T KNOW THE CONFECTION IN '99 CEO OF SUN MICROSYSTEMS SAID YOU HAVE ZERO PRIVACY. GET OVER IT. MARK HAD A DIFFERENT RIF ON THAT TODAY BUT NEVERTHELESS USE THE WORDS GET OVER IT SO WHAT DOES THIS MEAN? WE HAVE SPENT -- WE HAVE SPENT EIGHT HOURS TALKING ABOUT THESE TENSIONS, CHALLENGES, WE HAVE TALKED ABOUT THE DIFFERENT TECHNOLOGIES, DIFFERENT APPROACHES. I STILL THINK SOCIETALLY WE ARE SAYING WELL, IF BY 14-YEAR-OLD PUTS SOMETHING OUT THERE, I SHOULDN'T HAVE LET MY KID PUT SOMETHING OUT THERE. WE ARE STILL NOT THINKING ABOUT THIS IN A LARGER SOCIETAL CONTEXT. WHAT DOES THIS MEAN, GET OVER IT, WHAT DO WE NEED THE GET OVER? IS THERE SOMETHING TO GET OVER? >> I THINK THE GET AWAY COMES FROM THE FACT THAT SOME BELIEVE THEIR VERSION OF PRIVACY IS THERE. I DON'T CARE. SOME PERSON MAY FEEL -- NO, I CARE ABOUT IT BECAUSE WE HEARD FROM COMMENTS ABOUT THE PRIVACY, SOME PEOPLE HAVE JUST NO DIFFERENCE, I'M OPEN BOOK. SOME ARE REALLY CONCERNED SO I THINK THAT'S WHERE THAT COMES FROM, SOME FEEL LIKE OH, EVERYBODY KNOWS EVERYTHING ABOUT ME, MY -- EVERYBODY KNOWS I CARE T. BUT THERE'S THINGS THAT CARRY AN DON'T WANT TO MAKE IT PUBLIC SO I DON'T WANT ANYBODY TO COME INTERFERE IN -- SO THAT'S WHERE I THINK ONE OF THE ASPECT OF PRIVACY. >> ANYBODY TELLS YOU TO GET OVER PRIVACY IS SOMEBODY WHO FEELS VULNERABLE IN SOCIETY. AND THEY PROBABLY WANT TO SELL YOU SOMETHING T. BUT PRIVACY IS AN ABSOLUTELY ESSENTIAL PART OF PROTECTING YOURSELF AND YOUR COMMUNITY AND YOUR FAMILY. FROM UNJUST UNFAIR ASPECTS OF THE WORLD WE LIVE IN. IF YOU SAY YOU DON'T THINK PRIVACY IS DEAD, IF YOU ARE GOING TO GET FIRED FOR YOUR SEXUAL ORIENTATION, YOU DON'T THINK PRIVACY IS DEAD, IF YOU ARE SUBJECT TO TRACKING BECAUSE OF YOUR DISABILITY, YOU DON'T THINK PRIVACY IS DEAD, IF YOUR CHILD IS BEING BULLIED ONLINE, RIGHT? WE KNOW PRIVACY ISN'T DEAD, PRECISELY BECAUSE PEOPLE DON'T PUT WEB CAMS IN THEIR BEDROOM AND STREAM THEM 24 HOURS A DAY. THOSE WHO DO GET PAID TO. RIGHT? YOU WOULDN'T DO IT IF YOU WEREN'T GETTING PAID PROBABLY. EVEN IF YOU DON'T CARE ABOUT YOUR OWN INFORMATION, YOU SHOULD CARE ABOUT OTHER PEOPLE WHO ARE VULNERABLE, BEING ABLE TO MAKE THE CHOICINGS THAT THEY NEED TO MAKE FOR THEMSELVES AND THEIR FAMILY AN COMMUNITIES. INVASION OF PRIVACY IS A CORE PART OF THE HISTORY OF OPPRESSION. >> I WAS GOING TO SAY, I WAS STRUCK BY SOMETHING YOU SAID EARLIER, JAKE, ABOUT HOW THERE ARE A LOT OF THINGS WE CAN DO TECHNOLOGICALLY THAT COMPANIES CHOOSE NOT TO DO BECAUSE IT IS EXPENSIVE, NOT FINANCIAL OR COMMERCIAL INTEREST. AND I DO THINK SOMETHING WE HAVE BEEN TALKING ABOUT PRIVACY, BUT THINKING THIS CONCEPT CAME UP COUPLE OF TIMES ABOUT BALANCE AND THINKING ABOUT THE BENEFITS AND THE RISKS AND THINKING ABOUT HOW DO WE HAVE ENOUGH TRUST AND -- LIKE THE RIGHT AMOUNT OF PROTECTION TO PROTECT THOSE FOR WHOM IT MATTERS OR ALL OF US TO SOME DEGREE. BUT WHILE ALSO GETTING THAT BALANCE RIGHT SO WE ARE ENABLING DATA TO BE USED SO WE CAN LEARN FROM IT SO WE CAN HAVE MORE EFFICIENT EFFECTIVE HEALTHCARE DELIVERY AND OTHER SYSTEMS AS WELL. I DON'T THINK I AGREE, I DON'T THINK GET OVER IT IS THE RIGHT ANSWER BUT I THINK THAT THERE WHEN YOU HEAR GET OVER IT IN MY VIEW PEOPLE SORT OF DON'T HAVE A CHOICE ABOUT SERVICES THEY NEED ONLINE ABOUT THE WAY THE WORLD CONTROL OVER THEIR DATA AND THAT BUILDING SOME LEVEL OF TRUST AND PROTECTION INTO OUR ONLINE PRESENCE AND DATA BEING COLLECTED IS AN IMPORTANT NEXT STEP WHERE WE HAVE TO BE LIKE DATA AND TECHNOLOGY IS AHEAD OF CULTURE AND WHAT IS ACCEPTABLE AND WE NEED TO HAVE HARD COMFORTS WHAT THE DO TECHNOLOGICALLY AND PUT IN PLACE TO PROVIDE BASELINE OF PROTECTION AND TRUST SO WE DON'T HAVE THE FATALISTIC VIEW OF LIKE I HAVE TO DO THIS, I DON'T HAVE A CHOICE BUT I HATE IT AND I KNOW I'M GIVING UP PRIVACY AND COME BACK TO HER BEING DOWN THE ROAD NOT BECAUSE I WAS A 14-YEAR-OLD WHO DID SOMETHING, BUT BECAUSE I HAD TO ENGAGE BECAUSE OF MY JOB OR FAMILIAR -- FAMILY OR WHATEVER THE CASE MAY BE. >> ALSO COULD BE THE FACT THAT THEY ARE NOT AWARE OF WHAT THEY CAN DO TO PROTECT THEMSELVES. THEY FEEL LIKE I DON'T KNOW WHAT I CAN DO AND I GIVE UP BECAUSE THE FACT IS IT INVOLVES EDUCATION AND WHAT THEY CAN DO TO PROTECT THEMSELVES AND THEY MAY NOT HAVE THAT ATTITUDE AND FEEL DIFFERENTLY. ONE THING ALSO YOU MENTIONED AS JAKE MENTIONED PANT I THINK IN TERMS OF -- ABOUT I THINK IN TERMS OF CARRYING ANALYSIS AND ALGORITHM AND TRYING TO FIGURE OUT WHICH PERSON IS MORE RISK OF FEELING DEPRESSED, IT'S A GOOD THING TO KNOW FOR HELP ONE OF YOU BECAUSE YOU CAN PROVIDE RESOURCES IN FRONT OF THAT, REACH THEM SO IT COMES TO THEY KNOW WHO TO ASK AND WHAT TO DO. SO GOOD FOR THEM. IF THEY GET WRONG INFORMATION, POSTED TO THEM FROM NEGATIVE, THAT COULD BE A PROBLEM. SO WHAT GOOD OPTION IS, LIKE ALGORITHM WHERE IT ADDS UP THERE, THEY CAN BLOCK ALL THOSE NEGATIVE ADDS AND ONLY POSITIVE ADDS SHOW THEM. AND THAT'S THE WAY YOU CAN MAKE A CHANGE IN THE ALGORITHM IN BACK END CONTROL WHAT CAN SHOW UP. YOU CAN CONTROL THAT. THERE IS A BENEFIT OF THAT GATHERING OF INFORMATION. >> THE REASON THEY DON'T DO THAT IT TAKES HUMAN LABOR AND THEY DON'T WANT TO PAY FOR HUMAN LABOR. SO MUCH OF THE AGE OF BIG DATA IS GETTING RID OF EDITORIAL CONTROL. BECAUSE EDITORS COST MONEY, SEEING CONTEXT IS A SKILL. THE -- THERE'S A WEIRD THING HAPPENING WHERE THE SKILLS OF THE HUMAN SOCIAL SCIENTIST ARE BECOMING WAY MORE VALUABLE NOW THAT WE ARE IN THE AGE OF DATA SCIENCE. WHERE THAT REASSERTION AND MAINTENANCE OF CONTEXT IS A SKILL THAT IS NOW INCREDIBLY IMPORTANT IN ORDER TO MANAGE AUTOMATING THOSE SYSTEMS. >> WE HAVE TEN MINUTES LEFT. AND WE HAVE 11 SPEAKERS AROUND THE TABLE. SO JUST UNDER A MINUTE EACH, WHAT IS ONE TAKE AWAY? WHAT IS THE KEY TAKE A WAY FOR YOU TODAY? I DON'T KNOW WHO WANTS -- I WON'T PUT YOU ON THE SPOT TO START. I'LL THROW OUT A DART. WHAT WILL YOU TALK BACK? >> I WILL START. THAT WE ARE DEALING WITH A NEW FORM OF SYSTEMATIC SYSTEMIC BIAS HOW PEOPLE THINK ABOUT PRIVACY AND CINNAMON CONTEXTUALIZED THAT WELL AND WE NEED TO BE COGNIZANT OF IT ADS WE THINK ABOUT -- AS WE THINK ABOUT INDIVIDUAL CONSENT, INDIVIDUAL PERMISSIONING, INDIVIDUAL CONTROL OF DATA. I THINK THE ONE THING I WOULD SAY FOCUS MORE ON THE DATA PROTECTION AND USE OF DATA THAN SO MUCH MORE SO INFORM CONSENT BECAUSE PATIENT DON'T CARE PEOPLE DON'T CARE. >> I WOULD SAY THAT THE IMPORTANT PART TO KEEP IN MIND IS THERE'S NOT A SINGLE ANSWER, THERE'S NOT A FINANCE ANSWER. THIS IS GOING TO BE AN ONGOING CONVERSATION IN THE SAME WAY CIVIL RIGHTS, OR ENVIRONMENTAL PROE THES OR ALL THESE THINGS THAT ARE -- PROTECTIONS OR SOCIAL VALUE BASED SYSTEMS IS GOING TO BE -- GOING TO KEEP GOING AND CHANGING, GOING TO KEEP COMING UP WITH WAYS TO ADDRESS IT AS TECHNOLOGY CHANGES AND WE ADAPT TO THAT. >> I THINK HARM BOTH IN INDUSTRY CONTEXT AND SOME IN THE IRB CONTEXT THERE ARE GOOD EXAMPLES ON PEOPLE TRYING TO FIGURE OUT THIS PROBLEM. WHERE TO GO FROM AN AGGREGATE SCALE UP IT WILL BE HARD BUT THERE'S SOME PLACES TO LEARN AS WELCH >> I WOULD SAY SOMEONE ON THE END OF IMPLEMENTATION SIDE I DO AGREE A SYSTEMS FRAMEWORK WOULD BE EXTREMELY HELPFUL RATHER THAN RELYING ON INFORMED CONTENT OR INDIVIDUAL LEVEL. SO I REALLY APPRECIATE THIS CONVERSATION TODAY I LEARNED A GREAT DEAL FROM ALL OF YOU. >> GO AHEAD CINNAMON. >> I THINK I WOULD AGREE WITH THE POINTS HERE TODAY SOMETHING I WAS GOING TO GO OVER IN MY MIND WAS WELL, PERHAPS IT'S RATHER THAN FOCUSING ON DATA AND ON INFORMED CONSENT, MORE ON PEOPLE, AT POINT OF INTERVENTION, FOR EXAMPLE DATA SCIENTISTS, PERHAPS OTHER STAKEHOLDERS IN THE DATA ECOSYSTEM, I THINK ONE THING I WOULD ALSO -- HAS THIS UP BEFORE BUT IT WAS RAISED AGAIN TODAY'S SPECIALLY WHEN I HEARD BRENDA TALKING SIMILARITIES TO CIVIL RIGHTS AND ENVIRONMENTAL POLICIES AND GOING THINKING ABOUT THESE THINGS GOING FORWARD, IS JUST WE HAVE AN OPPORTUNITY TO RAISE AWARENESS AMONG YOUNG PEOPLE, NOT ABOUT NOT ONLY ABOUT HOW TO PROTECT YOURSELF AND PROTECT YOUR DATA BUT ALSO JUST MORE GENERALLY, WE ALL AROUND THE TABLE KNOW DATA CAN BE USED FOR GOOD THINGS, IT'S A COMPLEX LANDSCAPE WITH LOT OF TENSIONS. SO HELPING TO RAIDS AWARENESS IN IN OUR YOUNG PEOPLE, MY SON IS 8 HE LOVES WATCHING YOUTUBE VIDEOS I TELL HIM PLEASE DON'T LIKE THOSE VIDEOS YOUR DIGITAL INFLUENCE MIGHT INFLUENCE WHETHER YOU GET INTO COLLEGE. RATHER THAN ME BARKING AT HIM HAVING SOME LIKE MORE STRUCTURES IN PLACE, EVEN FOR KIDS THAT YOUNG I THINK IS REALLY CRITICAL. >> ONE OF THE THINGS I LEARN THAT I THOUGHT WAS INTERESTING ABOUT OUR CONVERSATION TODAY IS THAT TRADITIONALLY IRBs THINK IN THE LENS OF PROJECT SPECIFIC ANALYSIS. AND WE HAVE TO THINK ABOUT THIS IN TERMS OF COMPOUNDING RISK AND DATA AND WHAT THE IMPACT IS MORE GLOBALLY. OF ALL THOSE PROJECTS TOGETHER. THAT IS RESPONSIBILITY OF ORGANIZATIONS AND DATA SCIENTISTS AND EXTENDING THAT MESSAGE OUT REQUIRES US TO HAVE A DIFFERENT FRAMEWORK FOR THINKING ABOUT REVIEW OF THIS RESEARCH AS WELL. PRIVACY FRUST IN MULTIPLE CONTEXT SO WE FOCUS A LOT OF COLLECTION OF DATA AND CONSENTING TO THE COLLECTION OF DATA BUT ALSO THINKING ABOUT WHAT ARE THE RIGHT PROTECTIONS IN PLACE FOR USE OF DATA AS WELL AS MODELS CREATED FROM THE DATA THAT MIGHT BE USED IN OTHER WAYS. NOT AS ONE SIZE FITS ALL BUT THE DIFFERENT LIFE CYCLE OF DATA AND HOW WE MIGHT EACH HAVE DIFFERENT FRAMEWORKS IN DIFFERENT CONTEXTS OF HOW THAT DATA IS USED AS WELL AS HOW IT'S COLLECTED. IN A COUPLE OF YEARS WE MOVE FROM PANIC PERSUASION PHASE WHERE A FEW YEARS AGO THE PRESENTATION WOULD HAVE BEEN OH MY GOD WE DON KNOW WHAT'S GOING ON WE ALL NEED TO BE HORRIFIED AND NOW WE ARE ALL HORRIFIED BUT HAVE A SENSE OF WHERE TO GO NEXT. WE HAVE -- WE SAW GREAT EXAMPLES OF INFRASTRUCTURE FROM INDUSTRY AND FROM ACADEMIC IRBs, HANDS ON METHODS. DIFFERENTIAL PRIVACY AND MATHEMATICALLY SECURE PRIVACY. THERE ARE NOW TOOLS TO GET OUR HANDS ON. NOW THAT -- AND I THINK IT'S USEFUL TO THINK ABOUT IT AS LIKE NEW STAGE THESE CONVERSATIONS ARE MOVING INTO. WE DON'T HAVE TO PERSUADE EACH OTHER THIS WAS A PROBLEM RATHER THAN MOVING FORWARD WITH MORE CONCRETE SENSE OF WHAT'S HAPPENING. >> SO I THINK IT'S REALLY IMPORTANT TO REALIZE THE PRIVACY AND ABSTRACT DOESN'T WORK FOR THE KINDS OF ANALYSES WE ARE TALKING ABOUT, THERE'S THE PRIVACY RISK OR THE RISK OF THE DATA RELEASE VERSUS THE BENEFIT TO SOCIETY. NOT THE BENEFIT TO THE RESEARCHER. I THINK IRBs OR WHATEVER ORGANIZATIONS ARE REVIEWING RESEARCH PROPOSALS WHETHER IT BE FACEBOOK, IRB, NEEDS TO WEIGH THAT AND MAKE A DECISION. THAT'S SOMETHING TRADITIONALLY THE RIGHT SAY SUPPOSED TO DO IT BUT MOST IRBs DON'T DO IT, IF IT NEEDS THRESHOLDS THEY HAVE TO WEIGH RISK VERSUS BENEFIT. WE CAN DO THAT. BUT IT'S NOT SOMETHING REGULATORS AND REGULATORS REPRESENTATIVES ON THE -- AT THE UNIVERSITIES OR HAVE REALLY BEEN TAUGHT TO DO. THEY HAVE BEEN TAUGHT TO APPROVE OR REJECT. THIS BALANCING THEN CAN BE USED TO GRANT RESEARCHERS EXCEPTIONAL ACCESS TO THE DATA. THEY DON'T HAVE TO WORK WITH THEY CAN WORK WITH PRIVATE DATA WORK CLOSELY WITH THE DATA CUSTODIANS AND THEN MAKE SURE THE RESEARCH RESULTS ARE NOT DAMAGING TO INDIVIDUALS. SO WE REALLY NEED TO MOVE AWAY FROM THIS OPEN DATA IDEA TO AN IDEA OF TRUSTS CURATION OF DATA AND PRIVILEGED ACCESS AND ACCREDITED RESEARCERS. THAT'S GOES AGAINST THIS WHOLE IDEA OF OPEN DATA BUT IF WE WANT I THINK THAT'S WHERE WE ARE GOING TO HAVE TO BE. >> THAT BRINGS US TO GET OVER IT. >> MR. GET OVER IT. >> I THINK I HAVE LEARN AD FEW THINGS. ONE IS THAT THIS IS HIGHLIGHTED FOR ME THE FACT THAT THERE'S SO MUCH OF WHAT'S GOING ON IN REGARD TO THE USE OF DATA FOR HEALTH DATA FOR RESEARCH THAT IS ACTUALLY EVADES ALL OF THE LEGAL RESTRICTIONS THAT WE HAVE AND THE PARADIGMS AND REGULATORY STRUCTURES, ULTIMATELY WE HAVE TO WRITE -- IT'S LIKE BERNARD SAID AT THE END OF THE DAY WE HAVE TO WRITE ETHICS. IT'S THE ETHICS ARE THE MOST IMPORTANT THINGS AND EVERYBODY ELSE BUILT ON THAT BUT WITH NO CULTURE WITH RIGHT ETHICS IT DOESN'T WORK WHICH GOES BACK TO TRAINING OF THE INVESTIGATORS. THE SECOND THING WE DIDN'T TALK ABOUT TODAY IT'S KIND OF A HIGHER LEVEL, WE TALK ABOUT RESEARCHERS USING DATA BUT RESEARCHERS CAN ONLY GET -- CAN ONLY USE DATA THEY GET FOR SOMEBODY AND SO WHAT WE DIDN'T TALK ABOUT IS DATA GOVERNANCE IN A DEEP WAY AND ABOUT INSTITUTIONS, COMPANIES, INDIVIDUALS MAKING THEIR DATA SETS AVAILABLE TO SOMEBODY TO USE EVEN IF THEY ARE NOT THE PERSON WHO HAS THE AUTHORITY TO APPROVE THE RESEARCH OR NOT. THAT'S A DIFFERENT -- IT'S QUITE RELATED BUT WE GIN GET INTO IT TODAY BUT SOMETHING MANY INSTITUTIONS ESPECIALLY UNIVERSITIES AND MEDICAL CENTERS ARE CAREFULLY CONSIDERING TO WHOM SHOULD THEY GET ACCESS WHAT WHOLE SET OF ISSUES.THE -- JUST- AND FINALLY I DO THINK THAT WE HAVE TO THINK ABOUT THE BENEFITS OF BIG DATA RESEARCH. I WAS THE NUMBER TWO AT THE NEW YORK CITY HEALTH DEPARTMENT MANY YEARS AGO BEFORE SOME OF YOU WERE BORN IN THIS ROOM. WE HAD A TERRIBLE TUBERCULOSIS PROBLEM. THANK GOD WE HAD MANDATORY SURVEILLANCE WITH NAMES ADDRESSES OF DOCTORS LABS PEOPLE WHO HAD TUBERCULOSIS, I WROTE REGULATIONS THAT SAID YOU TAKE YOUR MEDICINE OR YOU ARE LOCKED UP AND WE WILL GIVE YOU DRUG TREATMENT AND DETOX AND MENTAL HEALTH TREATMENT AND GIVE YOU HOME BUT YOU HAVE TO TAKE YOUR ED MANY SIN BECAUSE THAT IS YOUR PUBLIC RESPONSIBILITY. THAT'S BASED ON BIG DATA RESEARCH. THOUGH IT WAS AN EMBRYONIC FORM OF BIG DATA RESEARCH AND THAT MANDATORY DETENTION PROGRAM BROKE THE BACK OF THE MULTI-DRUG READIESSANT -- RESISTANT TUBERCULOSIS IN 91, 92, 93. SOME CAN BE NEFARIOUS, MAL INTENTIONED, SOME CAN BE USEFUL. >> WITH THAT, I WILL TURN IT BACK TO YVONNE. THANK YOU. [APPLAUSE] THIS BRINGS US TO THE END OF INTELLECTUALLY STIMULATING WORKSHOP. I HOPE THAT EVERYBODY HAS FOUND IT -- THE WORKSHOP TO BE USEFUL, MEANINGFUL AND INTERESTING. I HOPE THAT YOU HAVE -- IN VIEW OF THE WORK WE ARE DOING AND BRINGING THIS WORKSHOPS THROUGH OFFICE OF HUMAN RESEARCH PROTECTION DIVISION EDUCATION DEVELOPMENT IS SOMETHING THAT IS WORTHWHILE. THIS IS OUR SECOND YEAR DOING IT. AND WE HOPE THAT WE HAVE THE OPPORTUNITY TO CONTINUE TO BRING SOMETHING LIKE THIS EVERY YEAR. THIS -- WE ONLY GOT ONE DAY. AND WE ARE TACKLING A REALLY BIG TOPIC. BUT I THINK THIS STARTS A CONVERSATION AND AS I WAS SAYING THROUGHOUT THE DAY, THAT WE REALLY WOULD LIKE TO COLLECT REFERENCES AND RESOURCES PEOPLE HAVE TALKED ABOUT, MENTIONED. WE WANT TO PUT THEM TOGETHER ON OUR WEBSITE SO THIS IS SOMETHING THAT WE CAN ARCHIVE AND BE AVAILABLE TO ANYBODY ESPECIALLY IN A WORLD WE LIVE IN, THAT WE CAN GO INTO THIS PLACE ON OUR WEBSITE TO GET THESE RESOURCES IN A MORE CENTRALIZED WAY. GOING FORWARD IN THE FUTURE WE MIGHT HAVE THE OPPORTUNITY TO REVISIT THIS TOPIC AGAIN, MAYBE AT A DIFFERENT LEVEL AND SLIGHTLY DIFFERENT TOPIC MOVING FURTHER ALONG THIS EVOLUTION OF THIS INTELLECTUAL CHALLENGE. SO I WOULD LIKE TO THANK EVERYBODY HERE, ESPECIALLY OUR DEAR GUESTS, IT -- WE ARE SO APPRECIATIVE OF YOU. AGREEING TO SPEND TIME WITH US AND TO TALK TO OUR AUDIENCE AND PROVIDE YOUR EXPERTISE AND YOUR PERSPECTIVES. EVERYBODY WHO IS ONLINE WATCHING IT, WE HAVE HAD HIGHEST NUMBER CERTAIN POINTS WAS ALMOST 900 PEOPLE WATCHING AND THROUGHOUT THE DAY WE NEVER DROP BELOW 650. OR MORE OR LESS. SO I THINK THAT'S REALLY GREAT ATTENDANCE. AND GOING FORWARD WITH THE ARCHIVE VIDEOS MORE PEOPLE WILL BENEFIT. THANK YOU, VERY MUCH FOR PARTICIPATING IN THIS. [APPLAUSE]