>> GOOD MORNING EVERYONE. I WANT TO WELCOME THOSE IN THE ROOM WITH US TODAY AND THOSE ON THE WEBCAST TO OUR SECOND DAY OF THE NIH BD2K WORKSHOP ON COMMUNITY BASED DATA AND METADATA STANDARDS DEVELOPMENT, BEST PRACTICES TO SUPPORT HEALTHY DEVELOPMENT AND MAXIMIZE IMPACT. WE HAD A TERRIFIC DAY YESTERDAY. SOME VERY ROBUST AND ARRANGING DISCUSSIONS, I HOPE THOSE WILL CONTINUE TODAY. I WANT TO START OUT BY REMINDING ALL OF US OF THE OVER ALL GOAL FOR THIS WORKSHOP AND THAT IS REALLY TO FOCUS ON THE PROCESSES BY WHICH DATA STANDADS ARE DEVELOPED AND IDENTIFY AND WHAT ARE THE SOME OF THE COMMON PAIN POINTS AND WE HEARD A LOT ABOUT DIFFERENT PAIN POINTS WHETHER THEY'RE SOCIAL, FINANCIAL, AND TECHNICAL AND I AM GOING TO CHALLENGE EVERYONE TO AS THOSE PAIN POINTS ARE SURFACED TO REALLY BE BOLD AND GIVE US YOUR IDEAS FOR HOW NIH COULD HELP ASSIST OR INTERVENE OR ADDRESS A PARTICULAR PAIN POINT WHETHER IT'S A SHORT-TERM STAFF OR MAYBE A LONG-TERM ACTIVITY, WE WANT TO HEAR ALL YOUR IDEAS. SO THIS IS THE PLACE TO MAKE YOUR VOICES HEARD AND WE WILL BE CAPTURING AND COLLECTING THOSE IDEAS FOR AND ACTING ON THEM. SO I WAS GOING TO START--OUR DAY TWO STARTS ANOTHER PANEL AND WE HAVE A SERIES OF BREAK OUT SESSIONS FOR THOSE JOINING US FOR THE BREAK OUT SESSIONS AND DURING THE BREAK OUT SESSIONS THERE WILL BE REPORTS SO IF YOU COME BACK PERIODICALLY, YOU WILL HEAR THE BREAK OUT SESSION REPORTS ALTHOUGH THE INDIVIDUAL BREAK OUT DISCUSSIONS AREN'T GOING TO BE AVAILABLE FOR YOU TO LISTEN TO SINCE THERE'S GOING TO BE MULTIPLE ONES HAPPENING SIMULTANEOUSLY. SO ERIC? ERIC NEUMANN WILL BE THE MODERATOR FOR OUR NEXT PANEL SO I WILL TURN IT OVER TO HIM. >> THANK YOU. COULD I HAVE THE PANELISTS COME TO THE FRONT. SO THIS SESSION AS FAR AS I UNDERSTAND IT IS TO THINK ABOUT IF WE'RE GOING TO REALLY APPLY DATA, METADATA, STANDARDS AND APPROACHES. YOU KNOW THERE'S GOT TO BE SOME SET OF TOOLS, SOFTWARE, FRAMEWORKS, WHATEVER YOU WANT TO CALL IT THAT CAN WORK IN THIS SPACE AND YOU KNOW THE QUESTION THAT REALLY PUT FORWARD HERE TO THE PANEL IS WHAT IS THERE TODAY, WHAT ARE THE ISSUES AND SEPARATE PROBLEMS AND TOOLS AND WHAT THERE NEEDS TO BE THERE SO SOME OF THE DISCUSSION WAS ALSO LOOKING AT VISION OF HOW WE CAN MAKE NOT JUST THE STANDARDS WORK NICELY TOGETHER, BUT TO BE ABLE TO DO MORE WITH THE DATA AND INCREASING VOLUME AND COMPLEXITY OF THE DATA AND SO, WHAT IS OUT THERE TODAY THAT CAN'T HANDLE THAT, WHAT NEEDS TO BE STILL CONSIDERED AND IS IT GOING TO BE SOMETHING THAT DEVELOPED BY RESEARCHERS? IS IT COMING FROM VENDORS? IS IT A COMBINATION SO THERE'S A LOT OF HIGH LEVEL COMBINATIONS AROUND THIS AND THERE WILL BE SOME PRESENTATIONS THAT WILL SPEAK TO TOOLS AROUND CLINICAL INFORMATION WHICH IS ITS OWN GIGANTIC AREA OF WHAT IS THERE THAT CAN BE APPLIED TO THIS VISION OR WHAT NEEDS TO CHANGE. DO WE NEED TO GET VENDORS TO ADOPT THESE APPROACHES AND USES OF METADAT AND AND ONE OF THE STAR QUICKLY WITH A BIT OF SETTING THE GROUND WORK HERE THAT WHEN WE DISCUSS THIS SET OF TOOLS, WE DISCUS METADATA WE'RE REALLY TRYING TO HAVE A BROAD DEFINITION. WE LOOK AT METADATA AS REPRESENTING A LOT OF INFORMATION THAT IS USUALLY, SOMEONE WILL VIEW FOR A PARTICULAR DATA SETOT SIDE, THEY'RE AN ILLEGALSARY BUT ONE PERSON'S METADATA IS ANOTHER PERSON'S DATA SO WE WANT TO BE INCLUSIVE AND WE'RE TALKING ABOUT INFORMATION PROVIDENCE, CONTROL VOCABULARIES, ONTOLOGIES THAT DESCRIBE SOME SORT OF MODEL OR STRUCTURE AND EVEN SOME ELEMENTS THAT PROBABLY AREN'T AS IMPORTANT TODAY BUT WERE DISCUSSED A LOT AND EVERY TIME WE SEE A VALUE, WE SEE THERE'S PRESSURE, WHICH UNITS ARE YOU TALK BECOME AND SOME PLACE THIS IS IS IMPORTANT IN ININSTRUMENTATION AND STUFF LIKE THAT AND REALLY IDENTIFY LIMITATIONS AROUND ACCESS TO THESE TOOLS OR THE FEW THAT ARE OUT THERE ARE REALLY SORT OF PROTOTYPES, YOU REALLY CAN'T PUT THEM INTO LARGE SCALE UTILIZATION. IT'S ALSO ONCE YOU HAVE METADATA, WHAT DO YOU DO WITH IT. IF YOU'VE GOTTEN ASSOCIATION, HAVE GO G ON STUFF, IS IT UP TO YOUR OWN TEAM TO WRITE AN ALEGORITHMS TO MINE THAT STUFF, WHAT ARE THE THINGS THAT BUILT INTO AHEAD OF TIME THAT MAKE IT EASIER FOR RESEARCHERS AND CLINICIANS TO TAKE ADVANTAGE OF THAT. SO RIGHT NOW WE HAVE INTERESTING PLACES TO HANG THINGS BUT IT REQUIRES AN ARMY OF SOFTWARE DEVELOPERS AND BIOMAITITIONS TO TAKE ADVANTAGE OF IT. MORE COULD BE LEVERAGES IF TOOLS COULD BE VISUALIZATION OR THEY JUST TOOK ADVANTAGE OF VOCABULARY IN THEM, CLUSTERS WHATEVER YOU WANT TO CALL IT AND THE OTHER THING IS TO HAVE THE FRAMEWORKS AND THE DATA SORT SYSTEMS THAT ARE NECESSARY FOR THIS TO BE ABLE TO WORK TOGETHER, TO BE ABLE TO LINE DIFFERENT KINDS OF DATA THAT HAVE OVERLAPPING ELEMENTS AND METADAT ACONE OF THE PRESENTATIONS WOULD SPEAK TO IS THAT WE SHOULD TRY TO ASSOCIATE THE METADATA AT THE POINT OF DATA CAPTURE, AS IT MIGHT COME OFF OF A SEQUENCING MACHINE OR SOMEWHERE IN THE DOCTORS OFFICE SO THAT IT'S NOT HAVING TO BE ADJUSTED MORE OR LESS ADHOC IN THE LATER STAGE. THAT ARGUMENT HAS BEEN APPLIED INTERESTINGLY TO PAPERS WHEN THEY'RE SUBMITTED WHICH IS RATHER THAN TEXT MINING AND TRYING TO FIGURE OUT WHAT TAGS OR MESH ITEMS YOU NEED TO PUT ON IT, WHY NOT HAVE AUTHORS ASSOCIATE THAT WITH THE HELP OR TOOL OR SOMETHING FROM THE BEGINNING. SO ANYWAYS, I THINK THAT SORT OF SETS THE GROUND WORK OF THE THINGS THAT ARE SORT OF CHALLENGES, THERE'S PROBABLY MANY MORE AND I WELCOME EVERYONE TO BRING UP WHAT THEY SEE AS SERIOUS GAPS. AND THIS REALLY JUST REFLECTS THE KIND OF APPROACHES THAT MIGHT ADDRESS THIS, BUT AGAIN THIS IS A PARTIAL LIST AND REALLY THE PANEL HERE IS TO DISCUSS ALL THE DIFFERENT PERSPECTIVES OF WHAT IS THERE AND WHAT NEEDS TO HAPPEN. YOU KNOW IT'S NOT JUST ABOUT A DATA STANDARD IN XML OR RDF, IT IT'S HOW DOES IT FIT TOGETHER IN THE LARGER PICTURE, WHEN WE SAY TOOLS WE WANT TO INCLUDE THE NOTION OF FRAMEWORKS OR HOW DIFFERENT SETS OF PIECES OF SOFTWARE AND EVEN DIFFERENT LABS CAN COLLABORATE TOGETHER AND CAN WORK BETTER TOGETHER, MAP THE INFORMATION IN A WAY SO THAT WHAT THEY'RE REALLY TRYING TO SORT OF ALLOW IS EXTRACTING SOME TYPE OF--AND I LEAVE THE DEFINITION FOR THIS OPEN KNOWLEDGE FROM THE DATA, EACH FROM LARGE AMOUNTS OF DATA. SO THAT'S AN EXPRESSION I CAN REMEMBER GOING BACK TO THE EARLY 90S, TURNING INFORMATION OR TURNING DATA INTO KNOWLEDGE. YOU KNOW WE STILL SAY THAT TODAY AND I DON'T THINK WE'VE ACHIEVED IT BUT IT IS A VISION AND ONE OF THE QUESTIONS IS A TOOL TO SOMETHING THAT'S INSIDE KNOWLEDGE THAT SOMETHING YOU CAN SURE SO CAN YOU CLINICAL DOCTORS IN OUR CASE LIKE ONCOLOGISTS CAN MAKE INFORMED DECISIONS WITH, SO THEY'RE TRAINING WILL BE IN GMES AS DEEM AS THE OTHER FOLKS HERE BUT AGAIN HOW DO WE RELATE THAT INFORMATION OR KNOWLEDGE APPROPRIATELY. SO I KNOW THESE ARE THE KIND OF CHALLENGES MANY ORGANIZATIONS FACE, WE FACE THEM TOO, WE HAVE THESE STANDARDS AND WE HAVE TO THINK ABOUT HOW ON BRIDGE AND MAP THAT F. THERE THERE WERE SYSTEMS IN PLACE, IT WOULD MAKE OUR JOB A LOT MORE EASIER, IT WOULD ALSO INCREASE THE QUALITY OF INFORMATION THAT EVERYBODY CAN CONTRIBUTE BECAUSE YOU WOULD HAVE THEN, SOME KNOWLEDGE OF WHAT THE APPROACH OR THE QUALITY LINE FOR THE STANDARD IS RATHER THAN HAVING SOMEONE, MAYBE AN INTERN RYE TO MAP THIS STUFF FOR YOU. SO I LEAVE IT OPEN TO THE PANEL AND I THINK THE FIRST PRESENTER WILL BE JAMES OVERTON. >> THIS IS MY SLIDE. THIS IS ALL I CAN DO, I WILL TALK ABOUT IN THIS TERMS OF MY EXPERIENCE, AS SOMEONE WHO WORKS WITH BIOMEDICAL ONTOLOGIES, WS WITH CURATORS AND AS I DO THIS I THINK ABOUT THESE TWO LOWER LAYERS SO I PUT THEM UP THERE FOR YOU TO SEE. BUT I MEAN THE TAKE HOME MESSAGE FOR ME IS THAT THE STANDARDS WE'RE DISCUSSING ARE ULTIMATELY ABOUT COMMUNICATING CLEARLY AND THINKING CLEARLY AND THOSE ARE NOT THE KINDS OF THINGS THAT YOU HAVE ONE SPECIALIST ON YOUR TEAM THAT EVERYBODY DOES, EVERYBODY ON YOUR TEAM HAS TO BE ABLE TO COMMUNICATE CLEARLY AND THINK CLEARLY. SO EVERYBODY ON THE TEAM HAS TO KNOW SOMETHING ABOUT THE STANDARDS. NOW YOU DON'T WANT EVERYBODY TO BE AN EXPERT IN THE STANDARDS, THAT WOULD JUST BE A WASTE. EVERYBODY HAS THEIR SPECIALIZATION, SO YOU DON'T WANT EVERYONE TO BE A SPECIALIST AND THAT MEANS NOT EVERYBODY SHOULD SEE ALL THE COMPLEXITY OF THESE DIFFERENT LAYERS. SO, THE WAY TO DO IT JUST TO GIVE THEM NOT THE FULL COMPLEXITY, NOT A BOX OF BUILDING BLOCKS OR LEGO BRICKS OR SCREWS OR WHATEVER THESE THICKS ARE, BUT GIVE PEOPLE GOOD TOOLS, SPECIALIZED GOOD TOOLS THAT SOLVE THEIR PROBLEM IN SPECIFIC CASES AND SO, THE THING ABOUT TOOLS IS GOOD TOOLS ARE EXPENSIVE TO BUILD. SO GOOD TOOLS CAN MAKE YOU MORE EFFECTIVE, MORE EFFICIENT, MULTIPLY PRODUCT IVIT BUT THEY ALSO HELP YOU THINK MORE CLEARLY BY MAKING THE RIGHT ASSUMPTIONS AND BUILDING THEM INTO THE SYSTEM AND NOT MAKING THE WRONG ASSUMPTIONS AND NOT MAKING TOOLS YOU HAVE TO FIGHT IN ORDER TO GET WHAT YOU WANT DONE. SO GOOD TOOLS, HIGH COMPLEXITY AND THEY HIDE THESE LOWER LEVELS HERE AND AS WE'VE BEEN WORKING, AS THING PROGRESSED, I PUT OBO FORMAT AND OBO EDIT WITH STRIKE THROUGHS BECAUSE IN THE EARLY DAYS WE HAD THESE CUSTOM TOOLS THAT WERE DESIGNED FOR OUR COMMUNITY AND THEN IN ORDER TO TAKE ADVANNEDDAGE OF THESE BIGGER TOOLS AND RDF AND SPARKLE WE ABANDON OUR SPECIALIZED TOOLS IN THESE GENERALIZED TOOLS AND THIS INTRODUCES A LOT OF COMPLEXITY SO THE ONTOLOGY DEVELOPERS I NEED TO WORK WITH, I NEED TO KNOW PROTEGE AND THEN A LOT OF THEM NEED TO KNOW PROTEGE IN ORDER TO WORK WITH THE DATA THAT AMOUNT OF EXPERIENCE GIVING THEM, THE ONTOLOGIES I'M GIVING THEM AND THAT'S JUST NOT THE RIGHT THING, THAT'S JUST THE WRONG LEVEL, JUST THE WRONG AMOUNT OF COMPLEXITY FYOU EVER OPEN UP PROTEGE, YOU WILL AGREE WITH ME, SO WE HAVE SMALL NUMBER OF COMMUNITY WIDE TOOLS, LIKE ONTOBEE, OH OORT, THE ISATOOLS, WE HAD THIS SMALL NUMBER OF TOOLS THAT KIND OF THE RIGHT LEVEL OF ABSTRACTION IN MY MIND, COMMUNITY WIDE TOOLS, NOT JUST PROJECT SPECIFIC TOOLS AND THEY ARE HIDING THE COMPLEXITY IN THE RIGHT WAYS AND THEY LET SCIENTISTS GET IN AND DO THE WORK THAT THEY NEED TO DO, CURATORS GET IN AND DO THE WORK THEY NEED TO DO WITHOUT HAVING TO LEARN ABOUT ALL THE LOWER LEVEL LINK DATA ONTOLOGY STUFF AND WE HAVE A LOT OF USERS IN OUR FIELD, POTENTIAL USERS BUT AT THE END OF THE DAY, WE HAVE VERY FEW EXPERT WHO IS CAN DEVELOP THESE SYSTEMS AND PUT TOGETHER--LEAVE THESE BUILDING BLOCKS, BUILD ON THESE LAYERS TO GET THE KIND OF SYSTEMS WE ACTUALLY NEED. AND I MEAN, BIG GRANTS LIKE THE GENE ONTOLOGY CAN HAVE TOOLS GROUP, PEOPLE THAT CAN BUILD THESE TOOLS SPECIFICALLY FOR THEM, BUT I WOULD LIKE TO SEE THE KIND OF COMMUNITY WIDE FREELYY AVAILABLE TOOLS THAT LET YOUR GRAD STUDENTS DIVE IN AND LEARN ABOUT ONTOLOGIES AND STANDARDS AND WORK WITH DATA WITHOUT HAVING TO HIRE AN INFORMATICS PERSON, THAT KIND OF THING. SO THE THING ABOUT GOOD TOOL SYSTEM THAT THEY'RE HARD TO BUILD AND THEY'RE EXPENSIVE SO I USE THE WORD ERGNATIONAL LIBRARY OF MEDICINICS, I MEAN GOOD HUMAN INTERACTION, IT'S EASY TO SLAP TOGETHER A TOOL THAT LIKE I SAID UNDERSTAND AS A PROGRAMMER, IT'S HARDER TO SLAP TOGETHER THAT BUILDS ON ALL THESE ASSUMPTION, THE RIGHT ASUNKS CONCHESES AND NOT THE WRONG ASSUMPTIONS AND HAS THE INTERFACE AND THE ERROR CODES AND ALL THESE GET NEEDED FOR PEOPLE TO DO THEIR WORK PROPERLY AND EFFICIENTLY. AND THE THEN THE OTHER THING IS THAT GOOD TOOLS NEED TO BE MAINTAINED LONG-TERM. THEY NEED TO BE UPDATED, SOFTWARE, IF YOU DON'T KEEP THE PORTIONS AT BAY, AND LIKE STANDARDS GOOD TOOL DEVELOPMENT DOESN'T FIT THE GRANT CYCLE AS CLEANLY AS WE WOULD LIKE SO LONG-TERM MAINTENANCE MEANS NOT JUST THREE YEARS AND NOT JUST FIVE YEARS, IF IT'S A GOOD TOOL AND PEOPLE ARE USING IT IT NEEDS TO BE MAINTAINED EVEN IF IT'S OPEN SOURCE OR OUT THERE FOR THE COMMUNITY TO USE, YOU NEED EXPERTS OR CONTINUITY OF DEVELOPMENT SO THOSE ARE KIND OF THE TAKE HOME MESSAGES THAT I WANT TO LEAVE WITH YOU THAT WE HAVE THIS BIG DOMAIN, WE HAVE POTENTIAL USERS OF OUR STANDARDS, BUT RIGHT NOW THE STANDARDS ARE TOO COMPLEX, BECAUSE WE DON'T HAVE THE TOOLS TO HIDE THE RIGHT COMPLEXITY AND PRESENT IT AT THE RIGHT LEVEL OF ABSTRACTION, WE ONLY HAVE A HUL OF DEVELOPERS AND WE HAVE A SMALL NUMBER OF COMMUNITY WIDE TOOLS, BUT WE NEED THESE SPECIALIZED TO NOT JUST A BUILDING BLOCK AND WE NEED GOOD TOOLS WITH GOOD ERGNATIONAL LIBRARY OF MEDICINICS AND ERGNATIONAL LIBRARY OF MEDICINIC SUPPORT SO WE CAN CONTINUE TO HAVE THESE TOOLS AND HAVE SUCCESS WITH THESE STANDARDS THAT WE SPENT SO MUCH TIME DEVELOPING. >> NEXT PANELIST IS TOM ONIKI, THAT'S YOUR SLIDE THERE. >> ALL THE THINGS THAT JAMES TALKED ABOUT REALLY RESONATED WITH ME. I HOPE I'M NOT STRETCHING THE BOUNDARIES OF DEFINITION OF STANDARDS DEVELOPMENT HERE BUT WHAT I'D LIKE TO TALK ABOUT IS ADOPTION OF THOSE STANDARDS AND GETTING THEM IMPLEMENTED. THAT'S WHERE I SEE A REAL NEED FOR SOME GOOD TOOLS AND --I DRAW MY COMMENTS FROM NEW EXPERIENCES AND THE SHOP NORMALIZATION PROJECT, THE O. N. C. PROJECT, WE PARTICIPATEDOT NORMALIZATION PART OF THAT PROJECT CHRIS SCHUTE WAS THE ARCHITECT OF THAT PROG EXPECT WE TRY TO CREATE THESE MODELS AND GET EXPERIENCES FROM THEM. THE OTHER EXPERIENCES ARE FROM INTERMOUNTAIN HEALTHCARE WHERE WE HAVE THESE MODELS IN PRODUCTION USE AND WE'VE HAD THEM FOR A COUPLE DECADES NOW AND SO SOME OF THE OBSERVATIONS ARE FROM THAT EXPERIENCE GETTING THOSE INTO USE AND THE OTHERS ARE FROM--THE OTHER EXPERIENCE IS TRYING TO GET OUR NEW GENERATION OF THESE MODELS INTO USE. WE WORKED WITH A VENDOR TO TRY TO CREATE A SYSTEM THAT WOULD WOULD DO THAT. SO ANYWAY, THOSE ARE THE EXPERIENCES I'M DRAWING ON AND THE OBSERVATIONS I MAKE, FIRST OF ALL, WHERE THE--WE'RE THE EXPERTS, THIS IS ALL KIND OF COMPLEX, THESE MODELS AND TERMINOLOGIES THAT THEY REFERENCED. AND THE OBSERVATION WAS WE NEED SOMEBODY LIKE JAMES WAS SAYING, EVERYBODY THAT'S ADOPTING THESE STANDARDS AND IMP ELEMENTING THEM IN THEIR SYSTEM EITHER NEEDS A CONSULTANT OR SOMEBODY IN-HOUSE THAT KNOWS ABOUT THESE TERMINOLOGIES AND THESE MODELS OR YOU NEED REALLY SMART TOOLS, TOOLS THAT HIDE THAT COMPLEXITY AND BECAUSE WHAT WE'VE FOUND IS AN APPLICATION DEVELOPER DOESN'T KNOW ABOUT RXNORM AND DOESN'T KNOW ABOUT THESE DETAILED CLINICAL MODELS AND AND THEY'RE NOT INFREQUENTLY CHOOSING THE MODELS TO PUT SOMETHING OR THE WRONG CODE AND WE NEED TO HELP THEM, WE NEED TO HELP THEM WITH SOME KIND OF TOOLS THAT HIDE ALL THAT KNOWLEDGE. SOME MIGHT ARGUE, WELL, IT'S DANGEROUS, SOMEBODY THAT REALLY KNOWS THE DOMAIN HAS TO BE INVOLVED IN DEVELOPING THESE APPLICATIONS IN IMPLEMENTING THESE MODELS AND I AGREE WITH THAT BUT CAN WE--CAN WE BUILD TOOLS THAT ISOLATE THAT KNOWLEDGE, THIS ISOLATE THE NEED FOR THAT KNOWLEDGE I GUESS AND MOST OF THE WORK CAN BE DONE BY AN APPLICATION DEVELOPER OR AN INTERFACE DEVELOPER THAT DOESN'T KNOW ALL THE DETAILS OF THOSE MODELS AND TERMINOLOGY AND WE ISOLATE ONLY TO A CERTAIN POINT IN THE PROCESS THE NEED FOR THAT KNOWLEDGE. THE NEXT THING IS, YES BUT I NEED SOMETHING SLIGHTLY DIFFERENT BECAUSE WE ALL KNOW THAT YOU CREATE THESE STANDARDS FOR MODELS AND TERMINOLOGY AND YOU PUT IT OUT THERE, SOMEBODY TRIES TO IMPLEMENT IT AND THEY NEED JUST A LITTLE DIFFERENT THING, A LITTLE--ONE MORE CODE IN THIS VALUE SET, ONE MORE SLOT THIS MODEL. THEY ALWAYS NEED SOME KIND OF TAILORING AND SO CAN WE CREATE TOOLS THAT HELP THEM CUSTOMIZE THE MODEL THAT DON'T ALLOW THEM TO DO SOMETHING THAT BREAKS THE STANDARD THAT'SA AGAINST THE STANDARD BUT ALLOWS THEM TO DO THINGS THAT ARE WITHIN THE BOUNDS OF THE STANDARD BUT ALLOW THEM TO MAKE THE CUSTOMIZATION THEY NEED AND THEN DO IT COLLABORATIVELY SO THEY SHARE BACK THOSE MODIFICATIONS THEY MAKE AND THEY'RE FED BACK INTO THE STANDARD. THE NEXT THING IS THAT THIS CALLS FOR A WHOLE NEW APPROACH TO EXCEPTION HANDLING. WHAT I MEAN BY THAT IS THAT IN THESE MODELS THAT WE'RE DEVELOPING WE'RE KIND OF LOCKING DOWN DO A GREATER EXTENT THAN HAS BEEN DONE BEFORE WHAT VALID DATA MEANS, WHAT A VALID HEART RATE MEASUREMENT IS STRUCTURED LIKE AND WITH THE CODES, THE UNIT OF MEASURE LIKE ERIC WAS TALKING ABOUT BEFORE WHERE BEING VERY MUCH MORE SPECIFIC THAN TRADITIONAL SYSTEMS ARE. WELL, IF WE ARE THEN AND PEOPLE IMPLEMENT THOSE MODELS THAT MEANS A WHOLE LOT MORE DATA THAT DON'T COMPLY, THAT ARE EXCEPTIONAL DATA, I GUESS YOU WOULD CALL IT AND SO WHAT DO WE DO ABOUT THAT? IT CALLS FOR A NEW VIEW OF EXCEPTION HANDLING AND NEW TOOLS THAT ALLOW YOU TO CATEGORIZE AND LOG THOSE EXCEPTIONS SO THEY CAN BE QUERIED AND SO THAT PEOPLE CAN LEARN FROM OR SYSTEMS CAN LEARN FROM THOSE EXCEPTIONS THAT ARE--THAT ARE COMING THROUGH AS THE PATIENT DATA IS PROCESSED. IT CALLS FOR A LEARNING FRAMEWORK SO THAT WE CAN LEARN FROM THE EXCEPTIONS THAT ARE BEING MADE AND SAY, OH, WELL, MAYBE WE DIDN'T THINK OF THAT, MAYBE IT NEEDS TO BE SET BACK INTO THE STANDARD? OR CONVERSELY, WELL, NO, THAT'S JUST WRONG AND THERE HAS TO BE SOME EDUCATION HERE, SO THOSE ARE THE THREE CATEGORIES THAT I CAN THINK OF WELL TOOLS WILL BE REALLY USEFUL. I LOVED WHAT WE TALKED ABOUT IN OUR BREAK OUT YESTERDAY THAT MAYBE NIH, IT'S GREAT FOR NIH TO ENCOURAGE PEOPLE TO USE STANDARDS AS THEY FUND RESEARCH, BUT, IN ADDITION WOULDN'T IT BE GREAT IF NIH FUNDED PROJECTS TO BUILD SOME OF THESE TOOLS AND TO HELP PEOPLE ADOPT THE STANDARDS. SO, THANK YOU. >> THANK YOU TOM, OUR NEXT SPEAKER WAS ABLE--WASN'T ABLE TO ATTEND DUE TO THE SNOW, I HAD A SLIDE IN THERE ANYWAYS YOU CAN SEE HIS VIEW OF SOME OF THE TOOLS ARE REGISTRIES, METADATA REGISTRYS THAT CAN BE USED I GUESS BY A BROAD SET OF USERS AND AND I THINK SOME OF THE WORK THAT'S BEEN PIONEERED BY THE NCBO, BIOPORTAL HAS BEEN VERY, VERY FRUITFUL, A LOT OF US USE THIS AND ALLOWS US TO AT LEAST ALIGN A LOT OF OUR METADATA WE HAVE IN-HOUSE TO OTHER SYSTEMS. I THINK BUILDING THESE THINGS OF COMPONENTS OF OTHER TOOLS WILL BE VERY VALUABLE AND AGAIN IT'S NOT TRIVIAL BECAUSE OF MAP, BOTH VOCABULARIES ARE THE SAME CONCEPTS AND THAT MIGHT NECESSARILY BE TRUE. YOU MAY USE THE SAME WORD FOR A DISEASE BUT YOU'RE LOOKING AT IT FROM A DIFFERENT PERSPECTIVE SO THAT'S A WHOLE OTHER AREA, BUT I THINK TRYING TO DO SOME AMOUNT OF ALIGNMENT WOULD BE GOOD TO CONTINUE IN PARALLEL IN THE DEVELOPMENT OF METADATA SYSTEMS AND TOOLS HAVING A WAY TO PROVIDE TOOLS TO THOSE ORGANIZATIONS, MAYBE IT'S A SELECT SET THAT OVERSEE THE MAPPING OF THESE THINGS, MAYBE IT'S PART OF NIH, MAYBE IT'S THROUGH SPECIAL CENTERS, I CAN ONLY IMAGINE GETTING MORE AND MORE IMPORTANT ESPECIALLY WITH CLINICAL INFORMATION AND ITS ASSOCIATION TO GENOMICS. GOING ON, OUR NEXT SPEAKER I DO BELIEVE IS MIKE FEOLO, YES? >> YES, HI, ERIC, I'M A STAFF SCIENTIST AND I RUN THE GROUP THAT PROCESSES AND ARCHIVES THE DATA THAT COMES INTO THE DB GAP. DB GAP IS AN ARCHIVE OF PHENOTYPE INFORMATION OBTAINED FROM LARGE SCALE GENOMIC SCALE STUDIES. THERE'S 800,000 PEOPLE REPRESENTED IN DBGAP, SEVERAL VARIABLES, 10S OF TRILLIONS OF GENOTYPES. SO I'M NOT A STANDARD EXPERT, I KNOW SOMETHING ABOUT STANDARDS, I DID TAKE A LOT OF MEDICAL INFORMATICS CLASSES AT UTAH AND--BUT I I'M GOING TO GO THROUGH THREE OBSERVATIONS FROM DB GAP THAT MIGHT BE TOUCHING ON WHAT YOU'RE TRYING TO GET TO AT THIS MEETING. THE DATA THAT COME INTO'S THE DBGAP, WE KIND OF BROADLY CATEGORIZE IT PHENOTIP AND I CAN GENO TYPIC DAT A. THE PHENOTYPIC DATA COMES IN AND THESE ARE DEMOGRAPHIC EXPOSURE OR TREATMENT VARIABLES ON PEOPLE AND THEY CAN BE LONG STEWEDINAL OR NOT, OR CROSS SECTIONAL AND RLY FOR RESEARCH STUDIES ARE THEY STANDARDIZED TO A CONTROL VOCABULARY, NOW I KNOW YESTERDAY, I OBSERVED THAT THERE WAS KIND OF TWO BROAD USES OF STANDARDS AND SOME PEOPLE WE'RE TALKING ABOUT CONTROL VOCABULARIES AS STAN ARDS AND THEY ARE, THEY ARE STANDARD AND THERE'S THIS IDEA OF SPECIFIC FILE DATA FORMAT AND THOSE ARE ALSO STANDARDS AND I THINK I'M GOING TO TOUCH ON BOTH OF THOSE TO SOME DEGREE. NOW, WHEN THE DATA COMES INTO DBGAP AND LIKE I SAID WE HAVE A LOT OF DATA, IT'S NOT--IT'S NOT USUALLY STANDARDIZED, WITHIN THE STUDY, WITHIN ANY INDIVIDUAL STUDY IT'S USUALLY STANDARDIZED TO SOMETHING THAT THEY HAVE DEVELOPED AND SO WHAT WE DID AND WE SAW THIS ON THE FIRST, YOU KNOW SEVERAL STUDIES IS WE DESIGNED OUR INTAKE TO MODEL THE DATA THAT THEY--VERY GENERAL WAY OF MODELING THE DATA SUCH THAT IT WAS INDEPENDENT OF ANY--ANY CONTROL VOCABULARY OR THEIR REPRESENTATION OF WHAT A VARIABLE WAS SO WE TAKE TABULAR FILES OF THESE MEASURED PHENOTYPIC VALUES AND DATA DICTIONARY AND WE CAN TAKE ANY NUMBER OF TABLES AND THEY CAN BE LONGITUDINAL AND THOSE--THE DATA DICTIONARY CAN POINT OUT TO CONTROL VOCABULARIES IN FACT, IT WOULD BE GREAT, ONLY ONE PROJECT EMERGED, HAD LINKED BACK TO--BACK TO A CONTROL VOCABULARY, WE DID POST TALK OF LINKING OF STUDIES TO PHOENIX WHICH IS A KIND OF A CONTROLLED MEASURE SITE BUT ANYWAY. POST TALK APPLICON IS STANDARD AND IT'S REALLY PROBLEMATIC BECAUSE I CAN ARGUE WITH ANYBODY ANYONE IN THIS ROOM, IT REQUIRES CURATION, REGARDLESS AND IT'S ALWAYS GOING TO BE LIMITED BY THE GRANULARITYS AND OF THE VOCABULARY VERSES THAT OF THE DATA AND IT'S THE--SO, I DESCRIBED THAT WE TOOK A VERY PRAGMATIC APPROACH AND WE TRY TO GATHER THE INFORMATION AND WE DID IT IN VERY SIMPLE FORMATTED WAY THAT FAITHFULLY REPRODUCE DATA COME NOTHING AND LINK THAT BACK TO ANY DOCUMENTATION THEY HAVE COLLECTION FORMS OR PROTOCOLS. THE PROBLEM WITH THIS, IS THAT IT FORCES EACH ANALYST THAT GETS THE DATA FROM DBGAP TO HARMONIZE THE DATA. ALTHOUGH YOU CAN ARGUE THAT IT PROBABLY WOULD ANYWAY, BUT THAT'S ALSO A PRO, IS IT ALLOWS THE ANALYSTS TO HARMONIZE AT THE GRANULARITY THAT HE WANTS TO OR SHE. THERE ARE TOOLS THAT CAN AID THE HARMONIZATION OF PHENOTYPIC DATA, THEY EXIST. I THINK THE TOOLS IN THESE EFFORTS WOULD BE GOOD, IT'S A TOOL WHO AID THE HUMAN CURATION IN MY OPINION BUT IDEALLY THE STANDARDS WOULD BE PLAN AND CAPTURED STANDARD FORMAT PRIOR TO THE DATA CREATION AND SO THIS WOULD BE SOMETHING EARLY ON, IT IS STUDY SAYS WE ARE GOING TO COLLECT THIS TERM IN THIS FASHION BECAUSE IT MATCHES HOW PHOENIX RECOMMENDS THAT WE--OR WHATEVER, YOU KNOW. SO THE OTHER OBSERVATION WHICH IS KIND OF THE OTHER REALM IS FROM SEQUENCING DATA AND GENO TYPE DATA PRODUCED BY ARRAYS SO NG S STANDS FOR NEXT GENERATED SEQUENCING SO THESE REEDS ARE PRODUCED IN RAW DATA IF A VERY STANDARD FORMAT, JUST A FEW VENDORS ARE CREATING THESE MACHINES THAT ARE PRODUCING THESE VERY, VERY LARGE AMOUNT OF DATA AND WE'RE TALKING ABOUT BIG DATA. THIS IS WHAT WE'RE TALKING ABOUT, THE PHENOTYPIC DAT IN DBGAP AND TINY COMPARED TO SEQUENCE DATA AND I KNOW NEXT YEAR WE'RE GOING TO HAVE, YOU KNOW THE 10th--WELL, WELL 10S OF THOUSANDS OF WHOLE GENOMES WILL BE IN DBGAP NEXT YEAR JUST FROM THE STUDIES I KNOW ABOUT AND THERE'S OTHERS I DON'T KNOW ABOUT, SO THE DATA IS ONLY GETTING BIGGER, SO THE DATA IS PRODUCED OFF THESE MACHINES IN A VERY STANDARD FORMAT FROM THE VENDOR AND IT'S IMMEDIATELY UNSTANDARDIZED INTO A--WELL, A LESS STANDARD FORM WHICH IS MOST COMMONLY BANNED WHICH ARE A LINE REEDS SO IT'S A LINE TO A REFERENCE GENOME, SO THESE ARE ALINED ACROSS THE STUDIES AND YOU CAN CALL THESE VARIANTS AND THE GENO TYPES ACROSS THE ULTIMATE GOAL SO IN THESE STUDIES. SO THESE BAND PILES ARE VERY LARGE AND THEY'RE QUITE REDUNDANT AND THE OTHER PROBLEM WITH THEM, I SINGLE THAT THEY'RE NOT INDEXED BY POSITION. BUT IT'S A DEFACT OR STANDARD AND THEY'RE SUBMITTING BANDS SO IT'S DE FACTO. I THINK IT'S REALLY THE ONLY WAY THAT STANDARD BECOMES STANDARD TO BE HONEST. YOU KNOW, SONY--CAN YOU GO BACK TO VHS BETA MAX, BATTA MAX IS PROBABLY THE BETTER PRODUCT BUT VHS IS ONE, SO THE FACT OF STANDARDS, WHY THEY COME ABOUT IS ANOTHER ISSUE, BUT YOU KNOW.BAM FILE SYSTEM FURTHER PROCESSED IN THESE STUDIES TO CALL THE VARIANTS. AND SO THE CALL VARIANCE THERE'S ANOTHER REALLY COMMON FORMAT AND THIS IS BECAUSE SOME OF THE VERY EARLY LARGE PROJECTS DEVELOPED AND USED IT IS THE.VCF FORMAT, AND SO THESE ARE THE CALL GENOTYPES CREATE FRIDAY THE ALIGNMENT OF THE SYSTEM STUDIES. .VCF IS NOT PERFECT BY ANY MEANS BUT IT'S THE DE FACTO STANDARD AGAIN AND THE BIG PROBLEM WITH.VCF THAT WE SEE IS THAT IT DOESN'T CAPTURE THE PRIOR PROCESSING FROM A QUITE STANDARD FORM COMING OFF THE MACHINES THERE'S A LOT OF PROCESS TAG'S DONE AND THEN IT DOES ALLOW YOU TO PUT IN THE PROCESSING, KIND OF IN THE HEADER OF THE FILE IN A PRETEXT FORMAT, IT ALSO HAD--IT'S A STANDARD AND IT HAS WAYS TO PUT IN FLAGS BUT THESE, IT ALLOWS NONSTANDARD FLAGS AND THESE PROGRAMS THAT PROCESS THE D TO DO--ADD WHATEVER FLAGS THEY THINK ARE NECESSARY BUT THIS IS STANDARDIZED ACROSS ANY OTHER TOOLS. SO I THINK MAYBE STANDARDIZING THOSE WOULD BE HELPFUL. NCBI'S APPROACH HERE IS WE CREATE AND WE TAKE IN THE DATA AND THEN WE NORMALIZE IT TO A FORMAT WE CALL.SRA, WHICH IS REDUCES REDUNDANCY A LOT AND WE WANT TO REDUCE THE FOOTPRINT BECAUSE THE TAXPAYERS ARE PAYING FOR THAT SPACE ULTIMATELY SO WE WANT THESE THINGS TO BE AS SMALL AS POSSIBLE WITHOUT LOSS AND SO THE SRA FORMAT DOES THAT, IT ALSO DOES ANOTHER VERY IMPORTANT THING FOR THESE FILE SYSTEM THAT IT PROVIDES THE GENOMIC POSITION INDEX BUT AROUND THE SRA FORMAT WE BUILT THE SRA TOOL KIT AND THAT ALLOWS TO YOU GO IN AND PULL IT OUT AND ONE OF THE THINGS A LOT OF PEOPLE ARE USING IT FOR NOW BECAUSE ALL OF PEOPLE WERE BUILT ON BAM WAS TO PULL THE SRA OUT AND IMMEDIATELY CREATE A BAM FILE ON IT AND THAT'S FINE IF THEY WANT TO CREATE A MACHINE ON THEIR SPACE, BUT STORAGE WISE, ARCHIVE WISE, WE WANT TO STORE IT IN A COMPACT FORMAT. SO, I THINK THOSE ARE, YOU KNOW KIND OF OUR EXPERIENCES, I'M NOT SURE WHAT--WHAT'S AT THE BOTTOM HERE, IT LOOKS LIKE THERE'S SOME CUT OFF. SO I THINK I WILL LEAVE IT AT THAT. >> THANK YOU, MIKE. AND THE LAST PANELIST IS RYAN BRINKMAN: >> HI, I'M RYAN BRINKMAN, AND WE'VE BEEN INVOLVE WIDE FLOW CYTOMETRY, THINGS LIKE THE EQUIVALENT OF THE BAND FILE AND ACCIDENT SRA FILE AND BAM FILE AND THESE ARE HIGHLY TECHNICAL DOCUMENTS THAT EXPLAIN HOW SOFTWARE, EXCHANGES AND DATA FORMATS. --FOR THE SOFTWARE THAT'S OUT THERE THAT PARSES THE XML FILE BUT TO UNDERSTAND AND DESCRIBE WHAT THOSE THINGS MEAN, THE TEXTURING OF LOGICAL TRANSFORMATION THAT'S AN EQUATION LIKE THIS AND YOU WANT TO EXPLAIN IN THE DOCUMENT, HERE'S ALL THE PARAMETERS AND THE EXAMPLE OF THE USE AND TAKE THE DATA IN AND GET IT OUT. THAT'S 128 PAGES WHILE THE STANDARD ITSELF YOU CAN PUT ON ONE SO TO WRITE THAT DOCUMENT THAT ALLOWS PEOPLE TO UNDERSTAND WHAT THOSE STANDARDS ACTUALLY MEAN, THE ALL YOU NEED IS A TYPE WRITER, IT'S NOT THAT COMPLEX, WE DON'T NEED NEW TOOLS TO DEVELOP AND INTERCHANGE STANDARDS, I DON'T WANT TO GET UP HERE AND TELL NIH THE PROBLEM IS SOLVED BUT AS FAR AS WRITING TECHNISTANDARDS WE DON'T NEED TO DO THAT. PARCEL DOESN'T NEED TO KNOW ANYTHING ABOUT FLOW CYTOMETRY TO BE ABLE TO USE THAT. SO WHAT IS A PROBLEM IS ALL THE STUFF THAT'S AROUND THERE, TRYING TO GET PEOPLE TO AGROWOT 128 PAGE DOCUMENT TO DESCRIBE THE DATA INTERCHANGE STANDARD IS. ALL RIGHT, SO, BEFORE WE OPEN UP QUESTIONS FROM THE AUDIENCE WE WANTED SORT OF POSE A FEW BASIC QUESTIONS, BASED ON THESE PRESENTATIONS WHICH ONE OF THEM COMES TO MIND IS SO, THIS IS A VERY BROAD AREA. I MEAN TOOLS CAN BE IN A LOT OF DIFFERENT FORMS AND CERTAINLY DISCUSSIONS OF USE AND CLINICAL SPACE AND GENOMIC SPACE, OCCUPIES A LARGE FRACTION OF IT, WHAT DO YOU SEE AS SOME OF THE GAPS BETWEEN THIS FULL SET OF TOOLS ARE THERE AREAS THAT ARE AMISS, IS THERE A NEED TO BRIDGE THINGS EFFECTIVELY, I THINK ONE AREA THAT I WAS WONDERING IF ANYONE'S GOING TO PRESENT WAS IS THE VISUALIZATION OF THIS TYPE OF INFORMATION VERY DOABLE TODAY WITH THE GIVEN SET OF OFTWARE TOOLS OR IS THAT AN AREA THAT MIGHT NEED TO BE DEVELOPED FURTHER, WHAT ARE THE GAPS WHEN YOU LOOK AT THE BIG PICTURE? >> IF WE'RE TALKING ABOUT DATA STANDARDS I DON'T SEE HOW DIGITIZATION GETS INTO THAT REALM BECAUSE I SEE THAT AS A SEPARATE ISSUE. >> SO I THINK MOST OF US WILL USE-- >> RIGHT, BUT I DON'T THINK ANY TOOL OR ANY USER PROCESSING DATA AND WANTS TO PROCESS SOME ANALYSIS THAT'S BEEN DONE, FOR VISUALIZATION, WILL TOOL HAS TO BE ABLE TO SLURP IN THAT DATA TO PRESENT IT TO THE USER AND YOU DON'T WANT TO, YOU KNOW, LOTES OF WAYS CAN YOU DO THAT AND THAT'S AN UNSOLVED PROBLEM, YOU KNOW? LOTS OF, I THINK THERE ARE A LOT OF GROUPS WORKING ON VISUALIZATION, AND THAT--THEY DO AND IF YOU WANT TO DEVELOP THOSE VISUALIZATIONS FROM A STANDARD FORM, THEN EVERYBODY HAS TO AGREE LIKE YOU WERE SAYING LIKE ON WHAT IS CAPTURED AND HOW IT'S CAPTURED AND IN THE DATA FORMAT PART AND IT ALSO GOES DOWN TO THE CONTROL VOCABULARIES, WHAT THE VOCABULARIES ARE USEDDED AND WHAT THOSE MEAN, THERE'S A GRANULARITY PROBLEM WHEN YOU DO IT POST TALK. TO ME THE BIGGEST--FROM MY PERSPECTIVE, THE BIGGEST THING THAT NIH COULD DO WOULD BE TO ADD AWARD TIME, FORCE THE IDEA THAT THEY'RE CHECKING THE DATA MANAGEMENT AND CHECKING THE STANDARDS, YOU HAVE CERTAIN STANDARD THEY'S WANT TO BE USED. DB GAP IS AGNOSTIC TO WHAT STANDARD IS USED. WE WOULD LIKE IT VERY MUCH FOR INDEXING AND SEARCHING PURPOSES THIS STANDARD WAS USED. THE PROBLEM WITH THE STUDIES IN DBGAP AND I'M NOT THE FIRST TO SAY THIS, IS THAT IF YOU'VE SEEN ONE DBGAP STUDY, YOU'VE SEEN ONE DBGAP STUDY. THEY ALL ARE DIFFERENT BECAUSE THEY WERE DEVELOPED DIFFERENTLY AND IT WOULD BE NICE TO HARMONIZE THEM BUT IN REALITY, IT'S NOT NECESSARILY SOMETHING THAT YOU WANT TO DO, MAKE IT TOO EASY TO DO BECAUSE THEY WERE ACTUALLY ASCERTAINED IN DIFFERENT WAYS AND YOU KNOW THERE'S A LOT OF--YOU DON'T JUST ADD THESE STUDIES TOGETHER AND GET ALL THE POWER THAT YOU WOULD, YOU HAVE TO CONTROL FOR THAT IN YOUR ANALYSIS AND THE POWER DOESN'T NECESSARILY SHOW UP, YOU KNOW IN AN ADAATIVE WAY, SO THAT'S MY PERSPECTIVE. >> I WANT TO FOLLOW UP ON THIS POINT. SO, THIS IS THE QUESTION I WAS GOING TO ASK YOU, MIKE, IS IF YOU HAD A DECISION--[INDISCERNIBLE]--CERTA IN REQUIREMENTS THAT THOSE REQUIREMENTS--[AUDIO CUTS OUT ] BUT ACTUALLY ARE LOOKING FOR DOCUMENTATION THAT ONE CAN COME BACK AND UNDERSTAND THAT PARTICULAR STUDY, ONE OF THE KD--SALLY WAIS EXACTLY WHAT YOU JUST SAID WHICH IS WHEN YOU SEE ONE STUDY ON DBGAP, YOU SEE ONE STUDY, HARD TO LOOK AT STUDIES ACROSS DBGAP AND IT SEEMS TO ME, IT'S NOT [INDISCERNIBLE]. AFTER INQUIRING FROM THE [INDISCERNIBLE]. >> WELL,--YEAH, NO. I THINK--SO I THINK THE PROBLEM WITH THAT IS THE GRANULARITY PROBLEM. YOU MUST USE SNOMED CT, I MEASURE BLOOD PRESSURE IN A VERY SPECIFIC WAY, I HAD THE PERSON COME IN, SIT DOWN, WE MEASURED IT THREE DIFFERENT TIMES WE AVERAGE THAT,A WHAT IS THAT, SNOMED CT, WHAT IS THAT, THERE ISN'T ONE. MAYBE THERE'S ONE IN HPO FOR THAT, BUT WE DID IT IN AN AGNOSTIC WAY AND SUCH THAT YOU COULD SAY THIS IS WHAT WE'RE TALKING ABOUT, IN THE DATA DICTIONARY YOU COULD SAY THIS IS HPO TEXT AND YOU COULD ALSO IN OUR FORMAT SAY THIS IS SNOMED CT TERM, WHY? BUT WE DON'T WANT TO FORCE SPECIFIC TERMS OR EVEN TO DO THIS, ASK OUR USERS TO DO THIS BECAUSE WE WOULD RATHER FAITHFULLY CAPTURE WHAT THEY HAVE AND ALLOW EXPERTS AND UNFORTUNATELY, IT'S USUALLY THE ANALYSTS THAT GET THE DATA TO THIS BECAUSE AND THERE'S SOME MODERATION THAT CAN BE DONE BUT FOR MY OWN-- >> YEAH. >> WELL IT'S A YOU HAD PLAN CURATION PROBLEM EITHER AT THE POINT OF THE REPOSITORY OR THE POINT OF THE STUDY, I THINK, YOU KNOW OBVIOUSLY IT'S JUST NOT POSSIBLE IN DB GAP, YOU KNOW, THE FUNDING, YOU KNOW WHAT IT'S LIKE. >> THAT LET'S GO. >> THAT'S WHY WE'RE HERE. >> THE AUDIENCE HERE AND I THINK [INDISCE. >> MY COMMENT IS ABOUT DBGAP BUT I THINK CAN IT CAN [INDISCERNIBLE]--A LITTLE BIT LOUDER SO FIRST OF ALL I THINK THERE'S A DIFFERENCE BETWEEN INDEXING SOMETHING FOR RESEARCH THAN THE METADATA STRATEGIES THAT ARE USED FOR REUSE, RIGHT, SO ONE OF THOSE IS THAT WE KIND FIND THERE SO WE ALL PROCESS IT AND IN ALL OF OUR FAVORITE WAYS AND WE CAN ACTUALLY USE IT IN DIFFERENT WE CAN USE IT FOR DIFFERENT PURPOSES BUT IF WE CAN THINK ABOUT TOOLS THAT CAN HELP YOU FIND THE RIGHT METADATA FOR RESEARCH, THEN DBGAP CAN IMPLEMENT RESEARCH AND LESS TOOLS THAT HANG OUT AND BASICALLY WHAT ENDS UP HAPPEN SUGGEST MULTIPLE PEOPLE ARE BUILDING DIFFERENT TOOLS THAT ARE DBGAP SO WE NEED MORE COLLABORATION AND THE POINT OF SOURCE THAT THAT HAPPENED SO THAT'S THE SECOND POINT IS REALLY HOW DO WE--HAVE THESE REQUIREMENTS AND NEEDS AND THESE BETTER COLLABORATE WITH SOURCES LIKE NCBI, AND-- >> I CAN SAY STAY TUNED. WE JUST GOT A CRITICAL MASS AND AND WE WILL HAVE SEARCH NOTHING THE NEAR FUTURE. YOU NEED A CRITICAL MASS REALLY TO DO THAT WELL AND WE HAVE THAT NOW. THERE'S ABOUT 520 STUDIES AND I THINK YOU'RE RIGHT. THERE'S A SEARCH PROBLEM, WE'RE AWARE OF IT, HAD THINGS COME IN STANDARDIZED IT WOULD BE EASY TO BUILD SEARCH STUFF. SO THAT'S MY POINT. >> YEAH AND POST--WE'RE KIND OF WARY DOING POST TALK STANDARDIZATION BECAUSE OF THE GRANULARITY PROBLEM. IF WE CALL IT TERM EXPECTATIONS, IT'S NOT QUITE LIKE TERM EX, YOU WILL SEE BROAD TERM RESEARCH VERY SOON. RESPIRATORY >> WARREN ON BEHALF OF THE NCBI, BUT I'M NOT SPEAKING ON BEHALF OF THE NCBI, SO SOMETHING I SEE WITH GAO AND DBGAP IS THAT THERE ARE DIFFERENT LEVELS OF SPECIFICATION OF METADATA AND IT'S IMPORTANT TO REALIZE--REALLY, LAWYER? >> THERE, NOW? >> OKAY, GOOD, SO I THINK THERE'S DIFFERENT LEVELS OF GRANULARITY AND SPECIFICATION THAT WE ALL AGREE WE WANT AND WE DON'T WANT TO BE TOO SPECIFIC. IT'S REALLY NOT GOOD TO SAY, WELL YOU HAVE TO USE A SNOMED CT CODE OR THIS CODE OR THAT CODE, BUT YOU HAVE TO USE SOME KIND OF CODE. SO THATTA'S WHERE WHEN YOU LOOK AT THE DIFFERENCE BETWEEN GAO AND EXPRESS, IT'S GONE TO EFFORT TO MAKE SURE WHEN PEOPLE SUBMIT, THEY SUBMIT SOMETHING WITH THEIR METADATA AND WE CAN ALL ARGUE IS IT GOOD ENOUGH BUT AT LEAST IT'S SOMETHING. WHEN WE LOOK AT VERY SIMPLE THINGS IN DBGAP, LIKE GENDER OR SEX, BECAUSE IT GETS REFERRED TO BOTH WAYS, IT'S REALLY HARD TO GO ACROSS DIFFERENT DATA SETS BECAUSE PEOPLE WILL USE ZEROS AND ONES AND THEY MEAN DIFFERENT THINGS BY ZEROS AND ONES Ms AND Fs, THERE'S VARIATION, I THINK VERY SIMPLE--VERY SIMPLE ITEMS LIKE THAT SHOULD BE STANDARDIZED BUT I THINK AGAIN TO YOUR POINT, YOU DON'T WANT TO BE SO SPECIFIC ABOUT IT THAT IT GETS IN PEOPLE'S WAY OF DOING THAT SUBMISSION, SO IT'S--I THINK THAT'S THE LINE I WE WANT TO BE ABLE TO WALK IS BETWEEN HOW MUCH BURDEN TO WE PUT ON SUBMITTERS, HOW EASY TO WE MAKE IT TO SEARCH AND THEN HOW EASY DO WE MAKE IT FOR PEOPLE TO BE ABLE TO LOOK ACROSS THOSE DATA SETS, NOT JUST FOR DISCOVERY BUT NOW FOR ANALYSIS. SO I'D LOVE TO SEE WHERE YOU'RE HEAD WIDE ALL THAT BECAUSE I THINK THERE'S LOTS OF ROOM TO MAKE BIG MISTAKES AND THERE'S LOTS OF ROOM TO HELP PEOPLE GET SRE BETTER. >> YEAH, I THINK WE'RE WORKING ON IT. WE SPEND MOST OF OUR TIME FIXING AND RESOLVING INCONSISTENCIES WITHIN IT IS DATA ITSELF, SO THIS--THIS FILE SAYS THIS PERSON IS A WOMAN AND THIS OTHER FILE SAYS THAT THAT PERSON IS THE FILE. SO THERE'S SOMETHING WRONG TO FIX THEM WE NEVER DO KNOW, THE THING WITH THE MALE-FEMALE, WE HAD ONE, ONE THAT CAME IN WAS M AND F AND IT WAS TRIO STUDY AND WHEN WE LOOK AT THE GENETICS TO CHECK IT WAS COMPLETELY OPPOSITE, WE'RE LIKE WHAT'S GOING ON AND THEY WERE LIKE, M MEANS MOTHER AND F-MEANS FATHER AND WE'RE LIKE ... OKAY. CAN YOU SWITCH IT? THOSE KIND OF THINGS WE DO ALL THE TIME THERE IS A AREA ASSOCIATEDDED WITH EVERY LEVEL IN DBGAP, THEN YOU SHOULD BE ABLE TO USE THE DATA. YEAH, WELL, YEAH. YEAH, YEAH. SO,--SO I'M GLAD YOU POINTED OUT THAT MESSINESS BECAUSE IT LENDS A LITTLE NONSANITY TO WHAT I WILL SUGGEST BECAUSE WHEN YOU DESCRIBE THE BLOOD PRESSURE THING, THESE THINGS DON'T NEED TO BE MUTUALLY EXCLUSIVE, WE WILL WANT TO CAPTURE THE WAY IN WHICH YOU COLLECTED BLOOD PRESSURE, BUT COULD WE ALSO, THERE'S GOT TO BE SOME SNOMED CT FOR BLOOD PRESSURE JUST TO HAVE THE CONCEPT TO KNOW IT WAS THERE AND IT DOES INTRODUCE SOME PROBLEM, IF YOU TALK ABOUT THESE AND THE NEAT, SO IT'S SCRUFFY, WE MAY BE GETTING MISLEADING DAT OR INCOMPLETE DATA ESPECIALLY WHEN COMPARED TO THE REALLY ALL-OUT WRONG DATA WE CAN GET, MAYBE THAT'S BETTER THAN NOT GETTING IT AT ALL. YEAH, I THINK WE'RE RELUCK ANT TO DO THAT BUT IF NIH PUT IN THEIR FUNDING YOU MUST DO THIS. IT WOULD BE GREAT IF THE SUBMITTER DID IT AND NOT SOMEBODY AT M. I.T. DOING THAT, I THINK THAT'S WHERE WE'RE AT AT THIS MOMENT. >> SO TO TRY TO BRING IN--[INDISCERNIBLE] >> I WANTED TO FOLLOW UP WITH POINTS THAT I THINK TOMMING [INDISCERNIBLE] [INDISCERNIBLE] WITH THE HANDLING AND I THINK [INDISCERNIBLE] VERY STRONGLY WITH ME YOU CAN'T HAVE DATA THAT COMPLIES 95% WITH A STANDARD, SOMETHING THAT MAY NOT QUITE FIT, AND THE QUESTION IS WHAT DO YOU DO THEN? AND I THINK THE MECHANICS RIGHT NOW IN MOST SYSTEMS SO HEARING MY VOICE, WHAT WE DO TAKE ADVANTAGE OF AND RELY ON TOO HEAVILY IS THERE'S INTEGRITY CHECKING AND DATABASE TECHNOLOGY AND WHAT WE'RE TALKING ABOUT IS THE TOOLS THEMSELVES, ANYTHING THAT'S GOING TO SHAKE HANDS AND GET DATA, IT HAS THE ABILITY TO DO SOME SORT OF, IT'S THERE, IT NEEDS TO BE THERE, THIS WAS MISSING, THERE ARE, I THINK SYSTEMS ALL RIGHT IN PLACE THAT WILL RESPOND BACK AS THEY'RE TRYING TO UPLOAD PIECES OF INFORMATION, THEY WILL SAY THIS CHUNK WAS NOTES PRESENT OR NOT IN THE RIGHT FORMAT, YOU CAN EVEN HAVE EXCEPTIONS HANDLED THERE AND WILL TAKE RULES THAT ARE INCORPORATED AS IS LIKE A IMPLEMENTATION RATHER THAN SHOE HORN IT, BUT WHEN WE TALK ABOUT LOTS OF DIFFERENT TYPES OF DATA MODELS AND LOOKING AT DIFFERENT STRUCTURES, I DO THINK THIS ISSUE OF HAVING THESE NOT PERFECT FIT, THESE EXCEPTION SYSTEM A VERY COMPLEX PROBLEM AND IT NEEDS TO BE ADDRESSED FOR MANY REASONS. IT'S GOING TO BE, YOU KNOW IS IT THE RIGHT WAY OF DESCRIBING THE DATA ABOUT A PATIENCE. IT WILL BE A LOT OF LIABILITY AROUND THESE ISSUES, TOO, SO I WOULD LIKE TO OPEN IT UP TO THE PL, TOM OF COURSE POINT TODAY OUT, BUT IF OTHERS WANT TO SPEAK TO IT, MAYBE EVEN REGARD TO SOME OF DATABASES LIKE DB GAP, LIKE IF SOMEONE PUTS IN SOMETHING THAT YOU'RE NOT QUITE THERE TO HANDLE BUT YOU DON'T WANT TO THROW IT AWAY EITHER. >> SO ANYONE CAN TAKE THIS, IT'S OPEN TO ANYONE. >> I CAN SAY WE EXPAND OUR MODEL. PEOPLE SAY WE HAVE THESE SCANS ON PATIENTS THAT HAVE GENETIC AND PHENOTYPIC DAT AND IT'S LIKE SURE, LET'S TAKE THEM. >> I MEAN IMAGES ARE GREAT BECAUSE THE STANDARDS ARE THERE, THERE'S NO PROBLEM, THE FORMATS OR THE READERS. >> IN THE CONTEXT OF THIS DISCUSSION ABOUT TOOLS WHAT I WAS SAYING IS THAT WE NEED TOOLS TO BE ABLE TO CONFIGURE ALL THAT SO THAT WE CAN KEEP TRACK OF WHERE THINGS ARE FAILING AND WHAT DO WE WANT TO DO AND THEY BOUGHT INTO THE WHOLE NOTION THAT WE'RE TRYING TO WIN VISION OF MODEL BASED DATA AND ACCEPTED THE FACT THAT WE HAVE A MODEL FOR THE PATIENT AND THIS IS THE DEFINITION OF VALID DATA FOR THE PATIENT AND THEY ACTUALLY WENT TO THE EXTREME BECAUSE WE WERE--WHEN REGGISTERRING A PATIENT, WE GOT PATIENTS THAT HAVE INVALID MARITAL STATUS CODES, CODES THAT WERE NOT WITHIN THE VALUE SET WE DEFINED SO THEY WERE THROWING AWAY THE WHOLE PATIENT BECAUSE THE MARITAL STATUS WASN'T VALID. BECAUSE WE DIDN'T SPECIFY ENOUGH WHEN THIS IS INVALID OR AND WE NEEDED THE TOOLS TO CONFIGURURE ALL THAT TO SAY WHAT'S VALID BUT WHAT DO YOU DO WHEN DATA AREN'T VALID. AND JUST CONTINUE ON THAT POINT, I'M IN THE MIDDLE OF WRITING A VALIDATION SYSTEM, SO CURATORS ARE GOING TO SUBMIT SOMETHING AND SOMETHING MIGHT GO WRONG AND THE SUESERS ARE GOING TO SUBMIT SOMETHING AND SOMETHING MIGHT GO WRONG. IT'S EASY TO THROW IN AN ERROR AND THIS IS WHAT WENT WRONG THIS IS HOW TO FIX IT WITHOUT HAVING TO SPECIALIZE IT AND SAY, THIS IS OUR DATABASE SCHEMA AND HOW THEY LINK TOGETHER JUST TO SAY, YOU KNOW, THIS WENT WRONG, THIS IS USUALLY THE PROBLEM AND IT COULD MEAN A LITTLE BIT DEEPER, THIS IS HOW YOU GO TO FIX IT, THAT TAKES EASILY 10 TIMES AS MUCH EFFORT TO DEVELOP THE TOOL, BUT IT'S--IT'S SO MUCH MORE VALUABLE BECAUSE YOUR CURATORS WON'T WASTE TIME ON OBSCURE CODES AND THEY'RE NOT GOING TO BUDGET UNTIL THEY VALIDATE AND THEY WILL FIX IT AND YOU WILL GET BETTER DATA BUT IT'S EXPENSIVE IT AND TAKES TIME. >> SO WE HAVE THESE HIGH-TECH NIICAL DOCUMENTS, YOU SHOULD HAD THE MACHINE AND THE LAST BIT IS SUPPOSED TO BE A CERTAIN CODE. SOME VENDORS PUT IN A DIFFERENT CODE AND THEY'RE NOT FOLLOWING THE STANDARD AND THEN ALL THE IMPLEMENTORS HAVE CHECKS AND BALANCES AND THEY FIND OUT ABOUT IT BECAUSE IT DOESN'T WORK UNTIL THEY CHANGE THAT. BUT IT'S ONLY ONE BECAUSE THEY CHANGE SOMETHING AND THEN 50 VENDORS ARE DOING SOFTWARE LIMITATIONS HAVE TO FIX THAT. BUT WHAT DO YOU DO? AND EVERYBODY JUST DOES THE FIX. ANY OTHER QUESTIONS OUT THERE? SO BEFORE WE WRAP UP, I WANT TO TRY ONE MORE POINT THAT CAME UP A LOT LESS YESTERDAY, WHICH WAS, T'S A CONCEPT THAT A LOT OF US USED WHICH IS USING LEGO BLOCKS AND WE WANT DATA TO SORT OF SNAP TOGETHER THE RIGHT WAY, OF COURSE ANY OF YOU WHO PLAYS WITH LEGOS KNOWS THERE'S NOT A LOT OF VERY COMPLICATED SHAPES CAN YOU MAKE OUT OF THE BASIC BRICKS, THE STUFF WE GREW UP ON THAT THEY'RE LIMITING SO TO REALLY EXPRESS SOMETHING INTERESTING, YOU HAVE TO HAVE SOME SMARTER PIECES THAN JUST THE BLOCKS AND THE WAY THEY SNAP TOGETHER BUT IN THE TOOLS WE'VE BEEN DISCUSSING THERE ARE--IS IT POSSIBLE WITH A CURRENT SET TO DO THIS TYPE OF LEGO BLOCK BUILDING IF I WANTED TO GRAB SOMETHING FROM DB GAP, WHAT IT WOULD TAKE TO BE ABLE TO ALIGN OR CLICK TO SOME OTHER DATA BESIDES SEARCHING FOR THE TEXT THAT'S IN THERE, IS THERE ANOTHER WAY WE COULD MAKE THIS THING OR THESE SETS OF TOOLS PLAY TOGETHER NICELY. >> WELL, I THINK COMBINING DATA SETS REQUIRES SOME--MORE THAN JUST A TOOL THAT JUST COMBINES BECAUSE THERE'S OTHER ISSUES THAT THE ANALYST, ANALYZING THE DATA NEEDS TO TAKE INTO ACCOUNT AND CONTROL FOR. LIKE ASCERTAINMENT AND OTHER THINGS AND I THINK JUST IT'S PROBABLY NOT, YOU KNOWOOSE NOT OPTIMAL, CLEARLY, IF IT CAME IN STANDARDIZED IT WOULD BE MUCH BETTER, THAT'S WHAT I CAN SAY. >> JUST SPEAK FREE RADICALS GENERATED MY EXPERIENCE, WORKING IN THE OBO FOUNDRY GROUP, WE'VE DEVELOPED A LOT OF BEST PRACTICES THAT I THINK HELP MAKES THESE THINGS WORK, SO WE HAVE THE LINK DATA LAYERS AND THOSE ARE STANDARDIZED AND THEN THEY HAVE A SERIES OF COMMUNITY BEST PRACTICES FOR HOW AN ONTOLOGY HAS TO WORK AND HOW IT HAS TO RESOLVE IN WHAT TEXTURAL DEFINITION HAS TO BE AND HOW THESE COME TOGETHER SO THOSE GO A FAIR WAYS TO SOLVING SOME OF THESE INTEGRATION PROBLEMS AT THE LOWER LEVEL, BUT AGAIN, YOU DON'T SHOW THAT CURATORS AND YOU NEED LIKE AN INTERFACE CONSTRICTS THE RANGE OF INFINITE POSSIBILITIES OF THE THINGS YOU BUILD DOWN TO THE SHAPE THEY'RE EXPECTING SO, IT'S--IT'S NOT THE ENTIRE SOLUTION BUT IT DOES GO A LONG WAY, I THINK TO SOLVING THOSE KIND OF LOW LEVEL PROBLEMS. >> I MIGHT JUST SAY THERE'S PROBABLY MORE SHAPES AND LEGO BRICKS THAN ATOMS AND MAYBE THE GNULARITY OF THESE MODULAR THING SYSTEM TOO BIG, I DON'T KNOW. >> I THINK PART OF THIS WOULD BE DRIVEN WITH WHAT PEOPLE WANT TO BE ABLE TO DO AND IF THEY CAN IMAGINE LIKE FIVE, 10 YEARS WITH GENOMICS AND CLINICAL INFORMATION THAT HAS TO BE SOME EXPECTATION OF WHAT YOU WANT, BUT I THINK THE OTHER ADVANTAGE OF THE BRICKS BEING DONE THE RIGHT WAY IS YOU'RE NOT CONSTRAINED TO ONLY THOSE USE CASES, RIGHT? WE WANT TO ENVISION SOMETHING THAT IS--CAN GO BEYOND TWO YEARS AND THAT'S AWFULLY HARD TO FILL IN THOSE TOOLS. ANY OTHER QUESTIONS? >> I HAVE A QUESTION IN THE SENSE THAT FROM THE CLINICIAN, RESEARCHER PERSPECTIVE, WE'RE NEVER GOING TO HAVE CLEAN PROOF, BECAUSE SCIENCE CHANGES AND SO I THINK THAT IF WE COULD SOMEHOW GET THE TOOL TOOL DEVELOPER COMMUNITY TO UNDERSTAND, THERE WILL NEVER BE TRUTHS SO IF WE CAN HAVE SOME LAYER OF WHAT IS ACCEPTABLE MOVE IN ACROSS THE AREA AND THAT WE STILL HAVE THE FLEXIBILITY TO MOVE AND DISCOVER BACTERIA ACTUALLY CAUSE THAT AND STOP TRYING TO TOP THAT CLEAN, REAL PROOF THAT NEVER WILL BE AND WE HAVE TO LIVE WITH THAT AND YOU HAVE TO BE ABLE TO [INDISCERNIBLE] AND I'M NOT A TOOLER, I'M NOT A COMPUTER [INDISCERNIBLE]. SOMEHOW WE HAVE TO FIGURE OUT-- >> I COMPLETELY AGREE, I THINK SOME OF US WHEN WE TALK ABOUT PROVIDENCE IN THE DATA, WE ALSO MEAN THAN OITATION AS PROVIDENCE BUT WHAT WAS SAID AT ONE POINT DOESN'T, YOU KNOW INHIBIT WHAT YOU CAN SAY ABOUT IT IN A LATER POINT, TOO, BUT YOU HAVE TO BUILD IT IN FROM THE BEGINNING, YOU KNOW? >> PHIL, I WANT TO THROW THIS OUT THERE AND TELL ME IF YOU THINK IT'S A GOOD IDEA OR A BAD IDEA AND I CAN'T DECIDE MYSELF BUT IT'S THE BASIC NOTION THAT WE RUN SOME KIND OF CHALLENGE. THIS IS AN NIH THING TO DO, WE RUN SOME KIND OF CHALLENGE, AUTHENTICTION CHALLENGE OR VALIDATION CHALLENGE WHERE WE ENCOURAGE PEOPLE TO COME ALONG WITH A PARTICULAR AUTHENTICATION VALIDATION TOOL FOR CERTAIN TYPES OF DATA AND YOU KNOW THERE'S KIND OF--THEY RUN THAT AND THERE'S A SORT OF EVALUATION OF HOW MUCH THAT EACH OF THOSE TLS HAS FOUND THAT'S NORMALLY IN THOSE PARTICULAR DATA SOURCES. I THINK THAT WOULD HAVE SOME EFFECT AS CHALLENGES DO IN BRINGING ATTENTION TO THESE ISSUES BUT AT THE SAME TIME IT ALSO POTENTIALLY IMPACTS MORAL AND IT'S NOT DONE CAREFULLY, IT CAN MAKE CERTAIN RESOURCES WHICH MAY NOT BE THEIR FAULT WHICH THEIR DATA THEY'RE DEALING WITH IS MUCH MORE HETEROGENEOUS THAN OTHER KINDS. SO I'M JUST TRYING TO THINK OF WAY--A WAY TO SORT OF STIMULATE THIS KIND OF ACTIVITY. >> ONE THING I GOT FROM YESTERDAY'S PRESENTATION FROM PETER FOX IS THAT THERE MAY BE A NICE SET OF MECHANISMANISMS AND PROGRESSIONS WHERE MAYBE EVEN PEOPLE ARE IMPORTANT TO FORM A CLUT OR THIS, BUT YOU CAN SEE HOW THAT CAN PROGRESS UP THE LADDER AS MORE PEOPLE WANT TO SORT OF THINK ABOUT IT AND CONTRIBUTE, MAYBE IT DOES TURN INTO A CHALLENGE AT SOME POINT WHERE IT'S MAYBE A USE FOR STANDARDS AND YOU SEE WHY THIS IS DIFFICULT, MAYBE THAT'S--THAT'S SUFFICIENT TO SPECIFY A WELL FOUNDED CHALLENGE, YOU KNOW WITH SOME OBVIOUS METRICS AND MAYBE A HANDFUL OF RESEARCHERS OR VENDORS WANTING TO CONTRIBUTE TO THAT. SO ABSOLUTELY, I THINK PUTTING A SPOTLIGHT ON THIS WOULD HELP MAKE OTHER FOLKS AWARE BECAUSE OTHERWISE THE TOOLS, I MEAN WHAT ONE OF THE QUESTIONS, HOW ARE THEY GOING TO EVOLVE OVERTIME FWE SPECIFY THERE'S SOME BETTER WAY TO WORK WITH DATA, ASSOCIATED METADATA BUT THE TOOLS ARE AS THEY ARE AND THEY CHANGE GRADUALLY WHAT'S GOING TO HAPPEN. IS THAT GOING TO GET IN THE WAY OF US OR NOT, SO HAVING SOMETHING PUTS A SPOTLIGHTOT ISSUE SYSTEM ABSOLUTELY ONE OF THE THINGS THAT WOULD BE GOOD TO COME OUT OF HERE. >> YES? >> I JUST HAVE A COUPLE OF COMMENTS, SO SO THE FIRST ONE DEALS WITH THE FACT THAT PEOPLE TAKE PEOPLE'S DATA WITH OTHERS AND TRANSFORM THEM INTO SOMETHING THAT IS POTENTIALLY MORE USEFUL FOR CERTAIN KINDS OF USE. WE HAVE NO ABILITY TO DISCOVER THIS ENORMOUS AMOUNT OF WORK WE ARE DOING FROM DATA PROVIDERS, THERE'S NO SENSE OF COMMUNITY [INDISCERNIBLE] SO THAT USERS LEVERAGE THE WORK OF OTHERS, THIS IS ONE SIMPLE THING THAT CAN HAPPEN THAT VASTLY ALLOW PEOPLE OR ARE HIGHLY INNOIVATIVE IN METHODS BY WHICH THEY GENERATE THESE NEW DATA PROFILE AND HAVE THEM AVAILABLE FOR OTHERS TO REUSE THAT WOULD HELP CREATE A CULTURE OF DATA IMPROVEMENT. >> IS ISN'T ONLY IN THE LAB THAT PEOPLE ARE DOING IT THAT BECOME ARTIFACTS THAT OTHER PEOPLE CAN USE. >> THE OTHER THING I WANTED TO COMMENT ON WAS AN IDEA OF JUST IN TIME SEMANTICS. SOMETHING WHERE WHERE EXPRESS THERE'S A USER WHO HAS A DATA SET HAS A TERM IN MIND TO DESCRIBE DESCRIPTORS IN MIND CAN AND WHERE DOES IT GO AND HOW DOES THAT PROCESS OF LOOK, I COULDN'T FIND ANYTHING AND WHAT DO I DO NEXT. SO I FEEL LIKE THERE'S SO MUCH OVERHEAD, SO MUCH YOU NEED TO KNOW FOR A CONTRIBUTION TO EXISTENCE THAT THAT WE'RE IMPEDED FROM IMMEDIATELY IMPROVING. SO I WILL ARGUE THAT WE NEED A FACILITY OF SOME KIND THAT LOWERS THAT BARRIER, WORK THAT THEY NEED TO GET DONE WHICH IS PROVIDE A SEMANTIC DESCRIPTION OF THEIR DATA SO THAT DB GAP DEPOSITORS, THOSE DATA ARE ACTUALLY--THEY HAVE TERMS BUT NOW THE QUESTION IS THE HARD WORK OF THE ONTOLOGYST IS MAYBE TAKE THOSE TERMS AND SAY, OKAY, HOW DO WE FIT THAT INTO OUR TERMINOLOGY, INTO OUR ONTOLOGY, WHICH VOCABULARY AND WHO'S GOING TO TAKE CARE OF THAT. SO I WANT TO PUT OUT THERE THAT MAYBE WE NEED TO THINK ABOUT METHODS SORT OF HOW DO WE PROMOTE ASK ENCOURAGE AND ENVIRONMENT WHERE WE HAVE AND CREATE THE MECH MECHANISMS BY THE CURATOR ULTIMATELY IS A TOOL THAT WOULD ASSIGN THESE VOCABULARY CABULARS TO WHAT WE CALL SPECIFIC VARIABLES IN DB GAP BUT AT THE END OFIDATE, YOU HAVE TWO CHOICES AND YOU CAN PICK ONE. YOU GO MORE BROAD OR I'M GOING TO--IT'S MORE GRANULARITY AND THERE'S NO COMPUTER PROGRAM THAT WILL DO THAT ACROSS THE BOARD. >> I DON'T THINK IT'S GOING TO BE A FULLY AUTOMATICKED THING, IT REQUIRES PEOPLE WITH EXPERTS. >> YEAH, THAT TOOL--THAT TOOL THOUGH TO DO THE JUST IN TIME SEMANTIC ASSIGNMENT WOULD BE GREAT. TO ADD OBJECTS SO OTHER PEOPLING CAN SEE WHAT THEY'VE DONE. >> AND WE'RE ALSO. >> YEAH, SO-- >> LAST ONE, SO JUST BECAUSE MY JOB IS TO MAKE SURE WE DOWN ACTIONABLE ITEMS FOR THE WORKSHOP, I WANTED TO KIND OF--MICHELLE TALKED ABOUT THIS ALL THE TIME IS WE SPEND THIS DATA AND WE ARE KIND OF [INDISCERNIBLE] AND IF WE'RE GOOD CITIZEN WE FIND THAT AND GO BACK, MAYBE WE'LL----YES, I HAVE ALL KINDS OF STUFF I WOULD LOVE TO TALK ABOUT. BUT WE DON'T--IT IS SOCIAL PROBLEMS AND TECHNICAL PROBLEMS THERE, WHAT SORT OF TOOLS WE BUILD SO THAT THOSE DOWN STREAM PEOPLE ARE MINING THE DATA THAT ENHANCE THE SOURCE SO THAT THOSE ENHANCEMENTS, SO MAYBE THERE'S A GAP THERE IN TERMS OF [INDISCERNIBLE]. SO RIGHT NOW WE DO IT IN E-MAIL AND WE DO IT UNDER CONTRACT BUT WE SHOULD GET BETTER FOR THE PROCESS, I MEAN, RIGHT? MAYBE THERE'S A WAY THAT WE CAN THINK ABOUT, ESPECIALLY FOR THE BIG RESOURCES ANOTHER WAY OF FEEDING THIS CONTENT BACK AND DOING THAT WHEN YOU GO BECAUSE NOT EVERYBODY KNOWS WHAT EVERYBODY'S DOING WITH DBGAP, BAUSE DBGAP DIDN'T KNOW WHAT I'M DOING SO HOW DO WE MAKE THAT EASIER TO HAVE THOSE ENHANCEMENTS. >> WE DO HAVE SOME PRECEDENCE ALONG THOSE LINES THAT WAS YOUR FIRST POINT IS THAT WHERE RESEARCHER TOOK THE DATA AND THEY CREATED A NOVEL INTERPRETATION OF THOSE DATA AND THEIR METHOD AND QUITE IS QUITE IMPRESSIVE ACTUALLY AND HE RECONFIGURED IT AND THAT TUMOR NORMAL CALLING AND THINGS LIKE THAT, BUT WHEN HE WENT TO PUBLISH, THE PUBLISHER SAID WE WANT SOMETHING LIKE THAT IN THE PUBLIC REPOSITOR SCHETCJ SAID THAT'S NOT OUR DATA SO WE MADE A NEW STUDY FOR IT. SO WE MADE A SEPARATE ONE AND IT'S ATTRIBUTED TO HIM AND IT'S ALSO ATTRIBUTED TO DATA, SO WE DO HAVE PRECEDENCE FOR THAT IDEA. IS THAT VALUE ENHANCED DATA IS THAT SOMEBODY CAN GO THROUGH DB GAP AND DO UNIFORM--THIS IS JUST AN EXAMPLE, IF THE NIH COULD PAY FOR SOMEBODY TO PAY FOR DBGAP, ALL THE GENOME SEQUENCE OR GWAS DATA AND IMPUTE, ALL THE SNPs ACROSS THAT IF A UNIFORM WAY ACROSS ALL DATA SETS THAT WOULD BE GREAT, THAT WOULD BE A GREAT MONEY IN MY OPINION, YOU COULD DO THE SAME FOR PHENOTYPES I SUPPOSE BUT IT'S GOING TO TAKE PEOPLE, SO THOSE ARE OTHER ISSUES, THAT COULD BE FUNDED I SUPPOSE. >> OKAY, SO THAT'S ALL THE TIME WE HAVE FOR THIS PANEL SESSION SO LET'S THANK THE SPEAKERS. AND A FIVE OR 10 MINUTES BREA. >> OKAY, WE'RE GETTING STARTED. OKAY, SO THE FIRST BREAK OUT GROUP, WE TOOK AT A LOOK AT THE QUESTIONS, NOT WHAT ARE THE GAPS IN STANDARDS DEVELOPMENT PROCESSES, BUT WHAT ARE IT IS GAPS IN--WHAT ARE THE GAPS IN SAY NIH'S SUPPORT OF THE STANDARDS DEVELOPMENT PROCESSES, WHICH TURNED INTO A VERY INTERESTING CONVERSATION. ONE OF THE THINGS THAT WE FOCUSED A LOT OF ON WAS A QUESTION OF INFRASTRUCTURE AND HOW WE--HOW NIH SUPPORTS THE DEVELOPMENT OF THAT INFRASTRUCTURE AND THAT INCLUDES NOT ONLY STANDARDS AND METADATA BUT ALSO THE TOOLS, ET CETERA THAT GO TO SUPPORT THAT. AND UNFORTUNATELY, THAT'S THE THING THAT PEOPLE DON'T GET A LOT OF RECOGNITION FOR. THE PROGRAMMATIC AREAS AT NIH, AREN'T SPECIFICALLY FOCUSED ON IT, THERE'S A LOT OF AMBIGUITY THERE IN TERMS OF WHO'S RESPONSIBLE. SO THERE WAS TALK ABOUT SECTIONS FOCUSED ON NIH'S INFRASTRUCTURE SUPPORT. WE TALKED A LITTLE BIT ABOUT THE WORK THAT WAS DONE BY THE NSF IN SUPPORT OF THE CREATION OF THE DATA NET PROJECT BACK IN, I JUST WANT TO SAY 2009, THEY FUNDED SUPPORT FOR--IT HAD ORIGINALLY BEEN PLANNED FOR FIVE NETWORK NODES BUT IT ENDED UP JUST BEING TWO. THEY FUNDED 25 YEAR PROJECTS AT NEW MEXICO, UNIVERSITY OF NEW MEXICO AND HOPKINS, THAT MIGHT BE A MODEL FOR SOMETHING NIH COULD LOOK AT IN TERMS OF SUPPORTING THE INFRASTRUCTURE THAT INCLUDES STANDARDS BUT ALSO INCLUDES THE TOOLS AND THE EXCHANGE METHODS FOR MOVING INFORMATION AROUND. WE LOOKED AT TARGETED RECOMMENDATIONS FOR NIH IN TERMS OF REFERRING TO THIS WORK AS INFORMATION ENGINEERING OR INFORMATICS AND TRY AND GIVE IT A HOME WITHIN NIH. THE FACT THAT THE DEVELOPMENT OF THESE STANDARDS AND THE DEVELOPMENT OF THIS INFRASTRUCTURE IS NOT SOMETHING THAT IS SIMPLY A QUICK FIT INTO A GRANT FUNDED SHORT-TERM PROJECT. THAT IT NEEDS AND REQUIRES THIS ONGOING MAINTENANCE, IT NEEDS AN ONCATEGORY SUPPORT STRUCTURE AND HOW DOES NIH SUPPORT THAT? WAS AN AREA OF CONVERSATION. SO PRACTICAL THINGS ALONG THE LINES OF DATA GUIDES AND IMPLEMENTATION GUIDES FOR RESEARCHERS BOTHOT SUBMISSION SIDE BUT ALSO ON THE--ONCE YOU'RE DONE WITH YOUR PROJECT SIDE. AND THEN SOME FOCUS ON HOW TO ENHANCE AND EXPAND AND IMPROVE EXISTING STANDARDS RATHER THAN CREATING SOMETHING NEW. PEOPLE GET RECOGNIZED FOR CREATING THE NEXT NEW GREAT THING BUT HOW CAN NIH SUPPORT THE ENHANCEMENT OF EXISTING STANDARDS AND THE INTEROPERABILITY OF THOSE STANDARDS AND FOCUSING ATTENTION ON THAT AREA AS WELL. SO ANYONE IN THE GROUP WANT TO ADD TO WHAT I JUST SUMMARIZED? YES? RIGHT BEHIND YOU? >> I WAS ACTUALLY THINKING THIS AFTERNOON WE WOULD TALK ABOUT, YOU KNOW THERE ARE THESE STANDARDS OF ACTIVITIES THAT ARE RICH AND LONG STANDING BUT THERE'S PROBLEMS CAN'T GET THERE, MEMBERSHIP AND ALL THAT AND WHETHER NIH COULDN'T FUND THAT SO FOR RESEARCHERS IN THE SPACE IT HAS SOME STANDARDS NEEDS, THEY COULD GO TO HLSEVEN OR DIACOM, AND THEY COULD PUT THAT IN THE GRANT, BECAUSE THAT WOULD ACTUALLY--THAT WOULD HAPPEN TO PARTICIPATION AND IT WOULD ALSO--ACTUALLY HELP FUND THE STANDARDS A LITTLE BIT BECAUSE YOU WOULD HAVE TO PAY TO GO TO THE MEETINGS, BUT YOU'RE RIGHT, THERE'S JUST DISCONNECT. IN FACT, I THINK I SHOULD BE CAREFUL BUT I'VE ONLY BEEN HERE EIGHT YEARS SO I'M NOT SURE I KNOW ALL THE RULES YET BUT I DON'T THINK THEY GET MEDICAL INFORMATICS OR ALL THAT STUFF IN GENERAL. YOU KNOW THEY GET A LITTLE--OKAY, GENETIC STANDARDS THAT'S PRETTY GOOD BUT BEYOND THAT I DON'T THINK--THEY DON'T HAVE RECEPTORS THAT PICK UP ON THAT STUFF. >> YEAH, THAT WOULD--YEAH, THAT WAS SOMETHING THAT WAS BROUGHT UP YESTERDAY IN THE CONVERSATION ABOUT PARTICIPATING IN THE PROCESS, YOU KNOW, IT'S A SORT OF SECONDARY TERTIARY RESPONSIBILITY AND THAT YOU KNOW PEOPLE WILL SHOW UP ON THE CALL IN THEIR FREE TIME AND WHO HAS FREE TIME ANYMORE, BUT IF YOU COULD FUND PEOPLE TO PARTICIPATE-- >> WELL, YEAH YOU HAVE A TRAVEL SECTION IN GRANTS BUT YOU DON'T HAVE A STANDARD ACTIVITY SECTION OR SOMETHING LIKE THAT AND TRAVEL IS PART OF IT. >> SURE, TRAVEL IS PART OF IT AND PARTICIPATION IN STANDARDS ORGANIZATIONS ISN'T NECESSARILY INEXPENSIVE AND DOES IT MAKE SENSE FOR PARTICULAR RESEARCH INSTITUTIONS TO BE MEMBERS OF W-THREE-C OR ANY OTHER STANDARDS BODY BECAUSE IT MIGHT ONLY APPLY TO ONE SMALL DEPARTMENT OR ONE SMALL SEGMENT IS THAT INSTITUTION AND THEN YOU LOOK AT A RELATIVELY LARGE BILL AND WHO WILL PICK UP THAT TAB BECAUSE STANDARDS AS A PERSON WHO RUNS THE STANDARDS ORGANIZATION, THESE PROCESSES ARE NOT INTEX PENSIVE AND THE INFRASTRUCTURE TO SUPPORT THEIR CREATION IS NOT INEXPENSI. >> IT'S SOMEHOW TO FIGURE OUT A WAY TO HAVE A SECTION IN THE GRANT APPLICATIONS THAT WOULD COVER SOME. >> YEAH. ANY OTHER THOUGHTS OR COMMENTS ON? OKAY. >> A LOT OF OPINIONATED PEOPLE IN THE ROOM, I'M SURPRISED NO ONE HAS ANY? >> GREAT JOB. OKAY. >> ALL RIGHT THEN. >> RIGHT? AND ACTUALLY THERE WAS A REALLY INTERESTING THREAD IN THE CONVERSATION THAT WE HAD ABOUT INTERMURAL I. E. THINGS THAT NIH COULD DO INTERNALLY VERSES WORK THAT COULD OR SHOULD TAKE PLACE OUTSIDE OF NIH AND I POINTED TO THE WORK DONE BY NCBI, AND PUBMED, AS INTERNAL TO THE COMMUNITY HERE AND INTERNAL RESOURCE THAT NIH CREATED AND NLM CREATED BUT IT'S IMPACT ON THE COMMUNICATIONS COMMITTEE. I COOPERATED MEAN INTRAMUL IN TERMS OF ONLY FOR NIH BUT WHETHER NIH COULD DO IT IN-HOUSE. COULD N'RE IH CREATE A STANDARD DATA REPOSITORY FOR ALL THE DATA IN ALL OF MEDICAL SCIENCE? SURE, MAYBE, BUT THERE'S--COULD IT DO IT INTERNALLY OR EXTERNALLY IS REALLY THE CONVERSATION THREAT, AND SO, THERE'S SIGNIFICANT CHALLENGES WITH BOTH AND WHAT'S THE ROLE OF NIH IN DOING SO? ACTUALLY THIS IS BEING RECORDED AND FOR THOSE WHO AREN'T IN THE ROOM, BEING A PROJECT THAT IS FUBBEDDED INTERNALLY WOULD SOLVE SOME OF THE LONG-TERM STABILITY ISSUES, I THINK IS WHAT YOUR POINT IS. SO I'M KIND OF TWO MINDS, SO THE INTRAMURAL WORK, MAYBE THEY CREATED A NEW WORKING GROUP FOR AN INTEREST IN ACTIVITY, THEY HAVE ANSI APPROVAL, THEY HAVE ALL THE PROCESS WHICH IS A PAIN BUT MAKE SURE IT'S A CONSENSUS, SO THAT'S ONE THE ONE SIDE AND I THINK THE FEDERAL GOVERNMENT SHOULD DO MORE WITHOUT INSERTING THEIR OWN PECULIAR INTERESTS INTEREST IT. I MEAN THEY SAME SOMETIMES--I COULD TALK ABOUT THIS BUT I WON'T. I DON'T KNOW HOW YOU STOP THAT BUT ON THE OTHER SIDE, THE WAY IT WORKED, THE REASON NCBI WORKED SO WELL IS THEY JUST DO IT AND THEY DO RECEIVE AND THEY DO LISTEN TO EVERYBODY BUT IT ISN'T A CONSENSUS PROCESS SO YOU MAYBE NEED BOTH KINDS. >> YEAH AND IT'S SOMETHING I WAS THINKING ABOUT YESTERDAY IN THE CONVERSATION ABOUT THE NEED FOR INTEROPERABILITY BETWEEN STANDARDS AND THE NEED FOR TOOLS 245 ORGANIZATIONS LIKE NISO, AND THEY'RE NOT STANDARD ORGANIZATIONS ARE NOT DESIGNED TO DEVELOP TOOLINGS WE'RE ORGANIZED TO DEVELOP STANDARDS. THE COMMUNITY NEEDS TO DEVELOP THOSE TOOLS AND SO, IN PARTNERSHIP, THAT MIGHT BE A WAY IN WHICH NIH CAN SUPPORT THE EMPLOYMENT AND ADOPTION OF THOSE STANDARDS BY SUPPORTING THE TOOL AND INFRASTRUCTURES THAT ARE DEVELOPED AROUND THE STANDARDS. >> I WANT TO ADD TO A FEW THINGS. WITH HL-SEVEN THEY HAVE SO MUCH INFRASTRUCTURE THERE,OT ONE HAND IT'S GOOD BECAUSE IT'S RIGOROUS AND VALID AND ALL THAT AND IT'S ALSO OVERWHELMING AND IKE ANOTHER UNIVERSE AND I WENT TO A CONFERENCE, JUST ONE AND I HAVEN'T BEEN BACK BECAUSE I WAS SO SCARED. I MEAN IT REALLY FELT LIKE I WAS ENTERING A DUNGEONS AND DRAGONS UNIVERSE WHERE THEY USE THESE DIFFERENT TERMS AND I DIDN'T RECOGNIZE ANYTHING AND IT WAS VERY SCARY AND IT'S IMPORTANT WHAT THEY'RE DOING, I LOOKOT E-MAIL THREAD JUST SO I CAN KIND OF KEEP TABS BUT I FIND IT VERY INTEMIDATING, BUT IT'S INTERESTING THAT YOU SAY THE GOVERNMENT MIGHT HAVE ITS OWN PECULIAR INTEREST BECAUSE IN OUR GROUP WE TALK ABOUT THE NEED TO BRING MONEY AND FUND NOTHING AS WELL, AND IT'S A GREAT SOURCE, AND I DON'T THINK THIS IS THE SOURCE BUT THIS IS A SENSE, I US TO WORK AT MICROSOFT AND I HAVE BEEN IN INDUSTRY THAT IF YOU BRING IN INDUSTRY THEYVILLE THEIR OWN AGENDA AND I LIKE THE IDEA OF THE GOVERNMENT BEING LIKE THE STANDARDS OR THE HIGHWAYS OF OUR BIOMEDICAL RESEARCH AND I WANT THE GPT TO PAY FOR THAT AND KEEP IT NEUTRAL AND NOT TELOMERE SHORTENING -TOLL ROADS AND ALL THAT. >> THEY WEREN'T NEUTRAL IF I WASN'T WORKING, I WOULD HAVE COME TO DINNER LAST NIGHT AND TOLD YOU SOME OF THIS STORY. SO IT'S TRUE AND I THINK HLSEVEN IN SOME SPACES AS GOTTEN SO ARCANE, YOU WANT TO HATE IT AND I DID FOR A WHILE. BUT I THINK THE FIRE THING IS BRINGING THE AIR BACK IN WHERE YOU CAN READ THE DOCUMENTS, THERE IS BIZARRE STUFF AND COMPLEXITIES THAT ARE EMBEDDED THAT SHOULDN'T HAVE TO BE THERE. VERSION TWO WHICH I WAS INVOLVED WITH IS READABLE AND PEOPLE MAY NOT LIKE IT BECAUSE THERE'S AN XML VERSION OF IT BUT WHATEVER IT IS, IT'S KIND OF EASY AND IT'S OUT THERE, I THINK FIRE IS GOING TO BE MORE READABLE, ONE OF THE PROBLEMS WAS IN THE MIDDLE, THEY WANT THIS VERSION THREE, WHICH HAD THESE ABSTRACTION LAYERS, WHERE, YOU NEVER KNEW WHAT THEY WERE REALLY TALKING ABOUT. I MEAN THIS, IS ACTUALLY THE PATIENT I.D. BUT IT SAYS, YOU KNOW GOBBLE-DEE-GOOP OR SOMETHING, AND HOW DO I THEN SO THERE'S ATTENTION BETWEEN THIS CONCRETENESS BUT FOR IMPLEMENTATION, YOU HAVE TO HAVE LABELS THAT PEOPLE--ORDINARY PEOPLE WOULD UNDERSTAND. FIRE IS GETTING MUCH CLOSER TO THAT, AND I THINK--FHIR IS GETTING MUCH CLOSER, AND THEN WE HAVE ALL A LITTLE BIT OF GEEK IN US, BUT THE DEEP GEEKS NEED TO HAVE COUNTER FORCES FROM THE SHALLOW GEEKS THAT WOULD STOP THEM FROM EVEN GOING FORWARD IN THOSE DIRECTIONS SO THAT'S THE PARTICIPATION. BUT THE PEOPLE--PEOPLE ARGUE ON LITTLE STUFF AND THAT'S THE CHALLENGE OF STANDARDS. WE HEARD FROM MANY OF THE DIMENSIONS TODAY AND THERE'S THE BIGGEST PROBLEM IS THE PERSON WHO HAS THE ABSOLUTE RIGHT WAY TO DO IT AND IT'S GOING TO TAKE 400 YEARS TO GET IT ALL DOWN. I MEAN IF THE DOCUMENT GETS BIGGER THAN 200 PAGINGS, I THINK WE HAVE TO START WORRYING THAT WE NEED--WE NEED TO STOP AND GET--SEE WHERE WE CAN GET THAT WORKING SOPHISTICATED TILL WE GO ON TO HAVE THE PERFECTION. >> ALSO TOUCHES ON THE STANDARD YOU WERE WORKING ON WHERE YOU KNOW IT GOES AND FITS AND STARTS AND IT GETS BUSES SCHEPEOPLE ARE NOT DOING THEIR THINGS AND IF WE PAID FOR THE DEVELOPMENT OF STANDARDS EXPLICITLY, YOU WOULD HAVE LESS OF THAT. THE FLIP SIDE IT YOU DON'T WANT TO PAY PROFESSIONAL STANDARDS DEVELOPMENT PEOPLE WHO ARE DISASSOCIATED FROM THE NEED FOR THOSE STANDARDS SO YOU DO WANT INTEGRATION BUT AT LEAST IN A WORLD OF PARTIAL EFFORT ON DIFFERENT GRANTS SO IF THEY CAN HAVE SUBSTANTIAL PART OF THEIR TIME COVERED BUT NOT ALL. IF--WELL THERE'S LOTS OF WEIRD THINGS IF YOU GET THE GRANT, YOU GET THE MONEY, BUT THERE SHOULD BE EXPLICIT OVERHEAD TIME ALLOWED FOR PEOPLE INTERESTED NOW THE SIDE OF EUROPE PAID THE STANDARD DEVELOPER AND I WAS KIND OF READY FOR ONE TERM OF ONE OF THOSE THINGS IT'S REALLY QUITE INTERESTING. SO WHAT HAPPENS IN THE EXPANSIONIST WORLD, EVERYBODY WANTS TO COVER EVERYTHING IN THEIR STANDARD, THERE THEY KEPT SHRINKING IT BECAUSE YOU GET PAID WHEN YOU GOT DONE AND SO IT REALLY WAS A PERVERSE PHENOMON AND I DON'T THINK IT CAME OUT NEARLY AS WELL AS THEY DO IN THE U.S. AND I THINK THE BIGGEST--MOST IMPORTANT PARTICIPANT IN THE U.S. ARE THE BUILDERS AND THE VENDORS IN THIS STUFF AND I KNOW THEY CAN--IF THEY ARE--IF THERE'S ONE DOMINANT, TELL BE THAT BUT THEY BRING IN SENSIBILITY AND THEY GO, YOU KNOW WE COULD NEVER MARKET THIS OR IT'S GOING TO BE TOO COMPLICATED AND THAT'S--THAT'S WHAT'S NEEDED MORE THAN ANYTHING ELSE, I THINK? SOME SENSE. NOW THEY MAY WANT TO SAY, I GOT THIS NEW THING, I WANT TO DO IT AND THEN THEY HAVE TO DUKE IT OUT A BIT BUT YOU WANT VENDORS, YOU THE PRACTITIONERS OF THE FIELD, I MEAN IT REQUIRES FOUR DIFFERENT KINDS OF GROUPS TO HAVE BALANCE. YES, BALANCE OF INTERESTED PARTIES. >> I'M NOT SURE REALLY, WHETHER IT HAPPENS THAT LBUT YOU KNOW YOU WOULD LIKE PEOPLE WHO ARE OF THE FIELD BECAUSE OF KIND OF WHAT THEY LIKE BUT THERE HAS TO BE THIS TRADE OFF BETWEEN MAGICAL THINKING AND EXPECTATIONS AND GETTING SOMETHING DONE THAT PEOPLE WOULD USE. I REALLY THINK THE VENDORS, THEY'RE OFTEN VERY USEFUL, USUALLY SENSIBLE, THEY USUALLY PULL IN THE EXTREMES YOU KNOW TO SAY, YOU CAN'T BUILD THAT, IT WON'T BE RELIABLE OR WHATEVER, OKAY. ALL RIGHT. >> RYAN? , I WOULD LIKE TO THANK EVERYBODY IN GROUP TWO. ARE I'M JUST A SPOKES MODEL. WE'RE GOOD IMPLEMENTERS BECAUSE WE FOLLOWED WHAT WE WERE SUPPOSED TO DO SO FIRST THING ASSIGNED IS ASSESSING THE DATA LAND CAPE SO THINGS LIKE THERE MAY BE EFFORTS ON ONTOLOGY FIRE PORTAL, AND IT MAKES IT REALLY HANDY BUT THE PROBLEM IS THAT THE STANDARDS COME IN DIFFERENT FORMATS AND DISPARATE PLACES AND USERS DON'T KNOW HOW TO GO OUT AND SEARCH FOR THE CONTENT THAT THESE HIGHER LEVEL STANDARD CONTEXT NEEDS TO BE FOUND AND ONE IDEA THAT WAS BROUGHT UP IS THAT MAYBE PUBLISHERS CAN HAVE A STANDARD RESOURCE ISSUE AND NOW N. E. R. HAS I DATABASE ISSUE WHERE ONCE A YEAR THEY CAN ALL BE MADE AVAILABLE FOR USERS TO BECOME AWARE OF AND UTILIZE AND KEEP GOING THROUGH AND MAYBE COME BACK AT THE BACK AND MAYBE AT THE END OF THE QUESTIONS SO WHAT FACTOR SHOULD BE CONSIDER INDEED PRIORITIZING NEEDS OF DEVELOPMENT OF METADATA STANDARDS SO MEWATOLOGY WAS ONE THAT MAY SPUR THE NEED FOR THE NEW STANDARD AND ALSO THE AMOUNT OF DATA AND PRESENCE OF EXISTING STANDARDS. ONE THING PEOPLE THOUGHT OF WAS TO ENGAGE THE SOFTWARE TECHNOLOGY AND THE VENDORS EARLY, AND ONE IDEA IS THAT HAVE A STANDARDS BODY THAT THEY CAN WORK WITH, FOR THE FDA, SO YOU KNOW ABOUT THE STUDENTS, ONE OF THOSE THOUGHTS WAS SO FOCUSED ON GETTING THE INFORMATION OUT DOOR AND THE DETAILS WORKED OUT THAT A HISTORY OF THEM MAKING SURE THAT DATA STANDARD VS BEEN WELL THOUGHT OUT FOR MORE GENERAL NEEDS THAN WHAT THEY THOUGHT OF SO MAYBE SHAH THE N. A. H. COULD HELP FACILITATE THAT, HAND IT OFF TO OPEN STANDARDS DEVELOPMENT BODIES AND HOW DO THEY HAND IT TO THE VENDORS DURING THAT IMP ELEMENTING CYCLE WHEN THEY'RE COMING OUT SO WHAT HAPPEN SYSTEM WHEN THESE COME OUT, YOU'RE LOCKED IN AT THAT POINT IN TERMS OF WHAT'S GOING TO BE SPIT OUT AND IN OUR EXPERIENCE, THAT DOESN'T CHANGE IN THE INSTRUMENT CYCLE SO THAT'S ANOTHER FIVE YEARS AWAY AND THEN YOU'RE STUCK WITH SOMETHING THAT A SOFTWARE TOOL AND IT'S IMPLEMENTING AND BY THEN IT'S TOO LATE TO CHANGE. GRANTEDEES NEED TO TELL THE NIH WHY THEY CAN'T USE THE STANDARD FIST THEY'RE NOT GOING TO USE ONE. AND IF THEY'RE NOT GOING IT USE ONE AND HAVE THE INFORMATION AND THE NAH TO HIT THE CYCLE TO TRACK THE POTENTIAL AREAS WHERE WORK CAN BE DONE AND--NO, I SAID THAT, TRY TO GET THAT BACK, SO PRETTY MUCH, I THINK WE SUGGESTED IN THE SAME WAY FOR DATA RESOURCES, HAVE THAT DATA STANDARDS SECTION IN THERE AS WELL. SO LOOKING AT THE SIZE, THAT WILL IMPLEMENT THE STANDARD AND HOW SUSTAINABLE IT WILL BE. THAT'S HOW MANY PEOPLE WILL BE SIGNED OFF AS CO-INVESTIGATORS OR COLLABORATORS ON THIS EFFORT AND THEN USE THAT PERCENTAGE OF THE OVERALL PEOPLE IN THE AREA THAT ARE ACTUALLY DEVELOPING THE STANDARD AT THE PERCENTAGE OF THE OVERALL USERS WITH A REPRESENTATION IS LIKE ON THAT GROUP, IT GIVES THE IDEA THAT METRIC HOW GOOD IT'S GOING TO BE. NEEDS TO BE A MEMBER OF THE COMMUNITY TO REPRESENT THE STANDARD OR IN NEED OF THE COMMUNITY REPRESENTING THE STANDARD SO THAT WAY, THEY SORT OF SAVE THIS STANDARD AND RATHER THAN JUST PISAYING TO DEVELOP A STANDARD OR SOCIETIES TO COME TOGETHER AND THEY WOULD REQUIRE SOCIETIES TO SEEN OFF OR OTHER AUGUST BODY TO SAY YES ON THAT NEED RATHER THAN JUST A P. I. SAY SO MIGHT HELP ON THAT. IT THERE'S MORE. >> AWKWARD PAUSE, PREDICTION OF SUCCESS IS HOW MUCH SUPPORT WILL BE AVAILABLE FOR THE STANDARD, WHO ARE EFFECTED AND INVOLVED AND RESOURCES SO WHAT THEY FELT IS HAVING TWO DIFFERENT PILOT FORMATION, IS ACCESSORY REQUIREMENT FOR WTHREE C AND THEY HOPE THAT'S A GENERAL REQUIREMENT FOR DATA STANDARDS OVER ALL. SOMETIMES POTENTIALS FOR THAT NEED BY THE STANDARD. AND THEN WE NEED TO SUPPORT AND LOOK AT WHO ALL THE POSSIBLE STAKEHOLDERS AND ARE THEM WEIGH IN AND SO THINGS LIKE PUBLISHERS AND VENDORS. THE FIRST QUESTION IS URGENT AND UNMET NEEDS NEAR DATA STANDARDS TODAY. SO USE CASES,--THE USE CASE FOR CENER OF EXCELLENCE--I'M NOT SURE WHAT THAT MEANS. OKAY, SO OH SO THERE'S A USE CASE THAT WAS GIVEN FROM THE XR OF EXCELLENCE AND--CENTER OF EXCEL EXPENSE IT MADE DATA MINING DIFFICULT, IF YOU LOOK AT ANY DATABASE OUT THERE TODAY, MAYBE WITH A COUPLE OF DIFFERENT EXAMPLES THAT ARE NOT TRUE, GENERALLY THE CASE NEED TO BE THAT DATAAN OITATION HAS GENERALLY BEEN DIFFICULT ACROSS DOMAINS, THERE'S A LOT OF HETEROGENEITY IN THE WAY PEOPLE ARE IMPLYING IMPLICATIONS SO IT'S NOT A QUESTION OF DATA STANDARD BUT HELPING PEOPLE ANNOTATE ACCORDING TO DATA STANDARDS WOULD SEEM TO BE A PROBLEM AND PEOPLE BROUGHT UP YESTERDAY THE ZERO VERSES ONE BEING A BEG DIFFERENCE AND HAVINGAN OITATION A LOT BETTER THAN NOTHING SO HAVING TEMPLATES FOR COMMON YOUTH CASES IN THESE REASONS IS TO HELP ANNOTATION WOULD BE WORTH THROWING RESOURCES AT TO HELP GUIDE THESE AND OS THIS IS ONLY AS GOOD TO GET THE DATA OUT OF THESE RESOURCES AND THANKS ARE ANNOTATED AND PEOPLE CAN'T SEARCH THEM AND FIND THEM AND THEN YOU CAN'T LOOK AT THE NOTES AND THEN HAVE YOU THIS GLOB OF DATA THAT ANYBODY WILL KNOW EXISTS. SO YOU CAN SAY HOW TO USE BLAH, BUT THAT GIVES NO INFORMATION ON THE REPRODUCIBILITY, OR HOW TO REPRODUCE THAT ANALYSIS AND USE THAT ALEGORITHMS BLAH, IS NO INFORMATION ON HOW THAT IS PATHWAY GIVES RAMITIZED AND WHAT VERSION AND ALL THE OTHER STUFF BECAUSE QUITE OFTEN IT'S NOT ENOUGH ALEGORITHMS BLAH BUT THERE'S A WHOLE SERIES OF PREPROCESSING STEPS AND THE THERE'S ONE CLUTTENER THE WHOLE PROCESS SO THERE'S THE IDEA OF REPRODUCIBILITY IS AN IMPORTANT ONE IN TERMS OF ANNOTATION OF THE METADATA OF THE RESULTS THAT YOU'VE GOTTEN. SO THERE'S RESULTS IN AND HOW IT'S RELATED TO THE TOOLS. I HOPE I CAPTIONED THAT WELL BUT I WELCOME ANYBODY FROM MY GROUP OR ANYBODY ELSE IN THE GROUP TO STEP IN WITH QUESTIONS. >> ANOTHER CRITERIA ARE MOSTLY TO GET STANDARDS TO GET THINGS FROM PLACE A TO PLACE B. THEY ARE SOME PLACE AND THEY NEED A--NYOU TALK ABOUT FILE FORMAT STANDARDS? >> NO I'M JUST TALKING ABOUT THE REALITY OF THE WORLD WHEN YOU TALK ABOUT STANDARDS. IT'S OFTEN--THE STUFF IS SITTING HERE AND I WANT TO GET IT OVER THERE. I'M A DIFFERENT PERSON OR WHATEVER, AND SO THAT'S RECOGNIZING THE STANDARDS ARE USUALLY A BI PARTIDE THING, NOT ALWAYS BUT THE COMMONEST, SO IT'S THE ONES YOU SEE THE MOST OF SO IF THE HUMAN VS TO DO STUFF ON EITHER SIDE, IT'S NOT AS GOOD A BET AND YOU TALK ABOUT SOME THAT IS HUMANS SO WHEN YOU GET A MACHINE THAT'S MAKING THIS STUFF, THAT'S THE HIGHEST PAY OFF YOU KNOW GET UP AND DO IT THE SAME THE MACHINE OR A FLOW SIGNIFY TOMETER MACHINE BECAUSE THERE'S NO HUMANS POETIC PERSUADE TO CHANGE THEIR BEHAVIOR ON THE WAY. THOSE ARE REALLY TOUGH AND A LOT OF STUFF HAPPENING IN HEALTHCARE, BUT THEN THAT REQUIRES THE POSITION HAS 10 MINUTES WITH ANOTHER 15 ISN'T HAPPEN, YOU KNOW THERE'S NO REALIZE THAT HUMAN PART AND AND SO IT'S SIT NOTHING A MACHINE AND PARTICULARLY DOING A LOT EASIER AND BET TORE NAIL WHAT'S NEEDED THAN WHAT YOU'RE STARTING WITH PEOPLE ON BOTH SIDES KIND OF TRYING TO DISCUSS, WHAT IS WHAT, YOU DON'T HAVE THE [INDISCERNIBLE] AND EVEN LOGICALLY, IN TERMS OF, I THINK OF THAT AS SPECIFIC LAB TEST, YOU ARE TALKING ABOUT LIKE WESTERN PLOTS OF RAW DATA AND STUFF LIKE THAT WHEN YOU SAY THE DATA IS SBAN OITATED. >> FOR THIS EXAMPLE, MAYBE, BECAUSE WHEN IT COMES OUT OF LABORATORIES, YOU KNOW YOU JUST PRETTY WELL NAILED BUT IT'S SOMETHING THEY DON'T SAY THAT YOU WISH THEY SAID. SO I DIDN'T GET THAT ON OR TEXT NARRATIVE DATABASES WHERE THEY TALK ABOUT SOME NUTRIFILL ANTIBODIES OR SOMETHING LIKE THAT. SO I DIDN'T GET THAT AUGHT ALL. >> MAYBE WE COULD GIVE THEM MORE EXAMPLES OF THE--WAS IT YOU? YOUR CASE WAS--WAS IT HERE? >> SO IT ALSO COVERS REPERTOIRE SEQUENCING, A LOT OF SEQUENCING DATA BEING PRODUCED BY THESE GROUPS. SO STATISTIC THERE'S A GUY AT VANDERBILT WHO HAS HIS GELLERT IS THAT EVERYBODY WOULD HAVE EVERYTHING THEY DID AND A BUNCH ARE, THERE WOULD BE A STACK OF R ROUTINES THAT DESCRIBE, THAT DEFINE AND YOU COULD REEXECUTE EVERYTHING. >> THAT'S CALLED SWE, VE. >> WHAT'S THAT. >> I THINK IT'S SWE, VE, THAT WE TALK DOCUMENT, THE PDF DOCUMENT AND THE CODE TO GENERATE THAT DOCUMENT, BEHIND--YOU ARE LOOKING AT A PAPER, AND THAT PAPER IS GENERATED PROGRAMMATICALLY SO ALL THE FIGURES ARE GENERATED BY THE DATA. >> I DON'T KNOW IF THEY WENT THAT FAR BUT THAT'S CERTAINLY THE EXTREME END AND HE'S DOING IT AND HE CAN DESCRIBE HOW TO DO IT AND I DON'T KNOW IF ANYBODY CAN INVEST THE LABOR IN IT. >> A TOOL IN R HAS BEEN DEVELOPED THAT WILL ALLOW THAT HAPPEN, ALREADY. >> SO THAT WOULD--IT MIGHT BE AN ACCEPTABLE BECAUSE IT'S TOO MUCH. >> BUT THAT'S NOT REALLY ACCESSIBLE AND USER AT A BENCH. >> SO TWO POINTS TO CHALLENGE THE GROUP, ONE, I QUESTION WHHING'S ABOUT FILES? I THINK FILES BECAUSE I LIKE TO SEE THE LITTLE ICON ON THE DESKTOP, AND HOW THE WORLD WORKS OF COURSE, MOST WEBPAGES AREN'T FILES ANYMORE, THEY'RE DYNAMIC RECONSTRUCTED SO ARE WE ALSO CONSIDERING IF IT DOESN'T ABOUT A FORMAT THAT'S SENT, BUT YEAH, IT'S TRANSMITTED AND IT'S CONSITUTED SORT OF LIKE A STAR TREK TRANSPORTER BUT THE NOTION OF FILES IS, I THINK IT'S HOLDING US BACK SOMETIMES, LIKE I HAVE TO GO BACK TO THAT. SOMETIMES YOU WANT TO PRINT THINGS OUT, ONE IS IICAL THEANCH NOTATION AND THERE MAY BE SYSTEMS THAT DON'T WORK WITH FILES ANYMORE, YOU GET THE MESSAGE, A CLUSTER OF THINGS BUT IT'S TRANSIENT AND YOU CAN--LISTEN A FILE MIGHT NOT BE APPROPRIATE FOR SOME AND THE WHOLE NOTION OF FILE FORMATS DOESN'T FIT INTO THAT, ANOTHER ONE IS, I THINK IN THIS POINT IN THAN OITATION, THAT'S BI BIG ONE, THERE ARE A HANDFUL OF US HERE WHO HAVE BEEN IN THAN O TAGS WARS FROM LIKE THE [INDISCERNIBLE] AND I THINK IT'S ABOUT TIME WE SOMEHOW EITHER MODIFY THAT TERM'S MEANING OR FIND A DIFFERENT WORD BECAUSE IF YOU THINK ANNOTATION IS A GLOB OR SEARCH POINT, SOMEWHERE, INDEX THOSE AND FIND THOSE SO THERE ARE OTHER APPROACHES THAT ARE MORE VALUABLE AS HAS BEEN A BIT OF TAGGING BUT IF IT'S TAGGED SOME SORT OF MORE STRUCTURE THAT COULD BE SUBSTANTIAL AND IT COULD BE FOUND OR CROSS CONNECTED SO THIS GOES BACK TO A PROPOSAL THAT DOUG LAUGHEN BURGER AND DAVID [AUDIO CUTS OUT ] RATHER THAN DIG AN ONTOLOGY, WHY NOT IF YOU HAVE DATA ABOUT SOMETHING, YOUR ELEMENTS IF YOU TALK ABOUT PROTEINS OR MECHANISMS YOU COULD POINT TO A REFERENCE BUT THE STUFF YOU SAY ABOUT IT IS OPENLY LIKE A PUBLIC DIRECTORY CAN BE MINED OR TROLLED OR GONE THROUGH BY AUTOMATED ROUGH ATOM BUT BOTS THAT THAT THE ANNOTATIONS COULD DO WHAT YOU WOULD LIKE TO DO, I THINK GOOGLE TO FIND OR SOMETHING LIKE GOING TOLL FIND A PARTICULAR STATEMENT ABOUT THIS OR THAT GENE. IT'S NOW EVERYWHERE, SO THE ANNOTATION SHOULD BE UNIVERSALLY AND BE ABLE TO FIND IT ANYWHERE, SO IN THAT CASE, I COULD SAY SOMETHING ABOUT A PARTICULAR THING OR TWO THINGS TOGETHER BUT IT DOESN'T HAVE TO BE ATTACHED TO A PIECE OF TEXT IN A FILE. THAT FILE I JUST SORT OF [AUDIO CUT OUT ] >> ALONG THOSE LINES AND ALONG THE LINES OF SWEEVE, SPA WAS MENTIONED I DON'T KNOW IF PEOPLE ARE FAMILIAR WITH STAGE BIONETWORKS AND THEIR SYNAPSE PLATFORM AND I'VE ONLY BEEN PERIODICALLY INVOLVED BUT I'M BEHIND, BUT ERIC QUONK SHOWS A PAPER THAT THEY PUBLISHED AND IN REALTIME GENERATES THE GRAPH, THE FIGURES THAT WERE USED IN THE PAPER AND THEY HAVE THE LINKED UP WITH THE DATA SOURCES AND CODE SOURCES WHICH ARE FILES, OR WHATEVER AND IT PULL ITS TOGETHER AND HAVE YOU THE DATA PROVIDENCE THERE, SO THATY A COOL MODEL FOR THESE THINGS, I ALSO WANT TO THROW OUT A FEW THOUGHTS I'VE HAD THAT HAS SORT OF THOUGHT OF, I'VE HEARD CERTAIN COMMENTS, WE HAVEN'T TALKED ABOUT THE NOT INVENTED HERE ISSUE PEOPLE HAVE BUT THAT'S A HUGE STANDARD IN ISSUES, HLSEVEN SEEMS TO HAVE GOTTEN AROUND THAT BY BEING BIG ENOUGH OR VIABLE AND FOR THAT MATTER, WELCOMING AND I WAS SCARED WHEN I WENT TO THE MEETING BUT PEOPLE WERE VERY WELCOMING, THEY ENCOUR PEOPLE TO PARTICIPATE. SO AS WE TALK ABOUT STANDARDS AND NOT DUPLICATING STANDARDS AND ALL OF THAT TRYING TO FIGURE OUT HOW TO EMBRACE PEOPLE, SO THEY DON'T FEEL THE SENSE OF SOMEONE ELSE'S DEVELOPING THAT AND I WILL DO MY OWN THING, AND THEN THE BOTTOM UP, TOP DOWN. WE TALKED ABOUT WHEN DOES THE STANDARD DIE, HOW DID WE MAKE SURE TO NOT PARALLELIZE THESE STANDARDS EFFORTS. WE DO ALL SORTS OF REASONS WHY IT DOESN'T WORK TO DEVELOP THE STANDARD FROM THE TOP DOWN BECAUSE PEOPLE ARE USING IT, ET CETERA, ET CETERA, BUT AT THE SAME TIMES PROBLEMS ARE DEVELOPING IT FROM THE BOTTOM UP AND THE FLOWERS ARE BLOOMING SO I DON'T REALLY HAVE A RECOMMENDATION BUT I WANT TO AT LEAST RAISE THE ISSUE OF THINK BEING TOP DOWN AND HOW TO ENCOURAGE, HOW TO GARDEN AND SORT OF ENCOURAGE THE RIGHT FLOWERS TO BLOOM AND DISCOURAGE THE--I DON'T WANT TO SAY WEEDS BUT THE LESS ATTRACTIVE FLOWERS. >> I THINK WHAT WE WILL DO NOW IS LET'S TAKE FIVE MINUTES JUST TO BIOBREAK, SIGN UP FOR TAXIS OR WHATEVER, BUT WHEN WE COME BACK WE WILL DO THE NEXT BREAK OUT SESSION IS PROBABLY ABOUT 30 MINUTES AND THEN WE'LL HAVE A REPORT OUT SO I WOULD SAY 11:15 OR SO, WE WILL START AGAIN, 45 FOR THOSE WHO ARE WATCHING THE WEB CAB, WE WILL PROBABLY START THE REPORT OUT AND MIGHT BE A FEW MINUTES AFTER THAT. BUT WE WILL HAVE EVA, WITH YAFFA ON THAT SIDE AND WE WILL HAVE HIRSCH MICHEL ON THIS SIDE. SO PEOPLE CAN JUST KIND OF PICK. >> SO I--BEING DIFFERENT VIEW FOR WHAT WE WANTED TO ASSESS BUT ESSENTIALLY WHAT WE DISCUSSED HOW TO IDENTIFY THE STAKEHOLDER, WHAT DO YOU USE TO CHOOSE THE STAKEHOLDER? S WHAT ARE THE INCEPTIVE FOR THE STAKEHOLDERS TO INVOLVE IN THE IN THIS THE PROJECT AND HOW HO YOU DO YOU--MAKE THEM TO CONTINUE BE INVOLVED AND TO SUSTAIN THE PROGRAM OR THE PROJECT AND WHAT ARE ONE OF THE THINGS THAT WE KIND OF DISCUSSED THAT IF EACH ONE OF THE STAKEHOLDERS WE KNOW WHAT VALUE THAT THEY'RE GETTING OUT OF THIS, BEING PART OF THAT, MAYBE THAT WILL BE KIND OF IN [INDISCERNIBLE] BEHIND THEM TO CONTINUE TO BE INVOLVED. WELL I MUST SAY FOR MY OWN POINT OF VIEW, THAT THERE ARE TWO MAIN PARTS I SEE, ONE IS WHAT DO I GET TO IT MYSELF OUT OF IT? AND THE OTHER THING IS THE BIG PICTURE, HOW DO I CONTRIBUTE TO A BIGGER PICTURE TO THE GOODS OF THE COMMUNITY? SOMETIME ITS DOESN'T COME OUT IN THE BENEFIT TO THE INDIVIDUAL GROUPS BUT YOU FEEL LIKE YOU REALLY CONTRIBUTED TO THE SOCIETY AND TO MAKING THINGS BETTER FOR THE PEOPLE THAT ARE LIVING ON THIS EARTH. BUT ANYWAY, SO, WE TALKED ABOUT WHAT KIND OF PROCESS IS NEEDED TO IDENTIFY THE RELEVANT STAKEHOLDER FOR THE DIFFERENT PARTS BECAUSE YOU DEVELOP THE STANDARD TO SELL DIFFERENT--DIFFERENT PURPOSES, SO WE HAVE TO DEVELOP FOR THAT, AND WE ALSO DISCUSS THAT WE HAVE TO CREATE A SENSE OF DATA EXCHANGE WHERE PEOPLE PUT THE DATA TOGETHER WHERE I CAN BE AVAILABLE TO QUERY OUR ACCESS BY THE OTHER COMMUNITY. WE TALKED ABOUT WHAT ARE THE INCENTIVES OF THE PARTICIPATION IN THE DIFFERENT STAKEHOLDER, CITATION FOR USER PUBLICATION, PUBLISHING FROM THE DATA AND NEW MODELS THAT WE NEEDED TO IDENTIFY. ONE OF THE THINGS THAT WE ALSO TALKED FOR THE PUBLIC PRIVATE PARTNERSHIP IS ONE GOOD EXAMPLE THAT HOW YOU CAN BRING THE DIFFERENT STAKEHOLDER FROM THE COMMUNITY AND--THAT INCLUDES GOVERNMENT AGENCIES AND MAYBE GREAT MODELS FOR CONSIDERING THE INCENTIVES AS WELL AS STAKEHOLDER ENGAGEMENT. WE SEE THAT IN THE PRIVATE SECTOR OF ACADEMIA AND THE PHARMACEUTICAL COMPANY INCLUDED IN THIS PUBLIC PRIVATE PARTNERSHIP, ONE OF THE QUESTIONS WAS HOW DO WE PROMOTE THE CULTURE THAT ENCOURAGE THE PUBLIC PRIVATE PARTNERSHIP AND LOOK TO OTHER MODELS OF PCORI, MAYBE THAT WOULD BE A MOTHER TO FOLLOW AND MAKE--HOW THEY MADE IT SUCCESSFULLY, THIS IS SOMETHING THAT STILL NEEDS TO BE SEEN. CAN NIH FACILITY SUCCESSFUL PARTNERSHIP IN DATA CENTER, SPACE AND ALSO IN THE DATA REPOSITORY AS WELL. THE OTHER ISSUE THAT WE TOUCHED ON IS THE I. P. AND PROPRIETARY AND OWNERSHIP OF SOFTWARE STANDARDS OR AND I THINK THAT WE HAVE TO BE CAREFUL OF THAT BECAUSE THAT CAN CREATE AN OBSTACLE IN TERMS OF THE DATA SHARING AND THEN WE NEED TO GIVE SOME SAFE GUARD OF WHAT IS ON THE OWNERSHIP AND WHAT IS UP AND AVAILABLE TO THE COMMUNITY. AND AS I MENTIONED BEFORE, WHAT CREATES A VALUE IN THE AREA OF STANDARD DEVELOPMENT AND MONEY, SPECIAL CONSIDERATION FOR GRANT REVIEW AND CITATION AND THE LAST THING THAT WE TALKED ABOUT, AMONG THE LAST THING IS HOW TO GOVERN A MODEST APPROACH OR ENCOURAGE A COMMUNITY TYPE OF ENGAGEMENT, HAVING ENOUGH BALANCE IN PLACE IS THE KEY SO THAT WE CAN ENCOURAGE IN THE COMMUNITIES TO DO WHAT WE WANT TO DO TO TOGETHER AND ENGAGEMENT TO MAINTAINING AND THE PROJECT, IF COMMUNITY IS THE VALUE WE TALK ABOUT BEFORE TO THE STAKEHOLDERS, TELL BE EASIER TO SUSTAIN AND THE COMMUNITY WILL BUY IN AND BE INVOLVED, I OPEN UP HERE TWO COMMENTS FROM OUR GROUP TO CONTINUE HERE IN CASE I MISSED ANYTHING OR IF THERE IS ANYTHING? >> THIS IS THE NEXT GROUP SO I JUST NOTICE THAT WHAT CREATES VALUE IS FOCUSED ON ACADEMEC INVESTIGATORS, MONEY, GRANTS AND PAPERS AND THERE'S A LOT OF OTHER PLAYERS THAT I KNOW Y'ALL RECOGNIZE AND SO, YOU KNOW I THINK WE NEED TO THINK ABOUT WHAT THE VALUES ARE FOR ALL OF THE STAKEHOLDERS, HOW TO GET THEM INTO OUR MIND. >> YOU'RE ABNORMALITIESLY RIGHT BECAUSE VALUE CAN BE FOR ONE PROFIT, ONE CAN BE PUBLICATION, ONE CAN BE CREDIT, SO, I THINK THAT THE VALUE FOR EACH STAKEHOLDER IS DIFFERENT. >> AND I WAS JUST REASSURING ELAINE THAT WE HAD A LONG CONVERSATION ABOUT, YOU KNOW PUBLIC PRIVATE PARTNERSHIPS AND THE VALUE SYSTEMS THAT THEY MIGHT HAVE, YOU KNOW IN THAT CONTEXT AND THROUGH HOW YOU MANAGE THE IP AND HOW YOU NEED ALMOST A--HOW TO FORM A PUBLIC, PRIVATE PARTNERSHIP FOR DUMMIES BOOK TO HELP US FIGURE OUT HOW DO WE ACTUALLY DO THAT SO WE CAN UNDERSTAND WHAT THOSE VALUE SYSTEMS ARE AND HOW TO NEGOTIATE, YOU KNOW SOME OF THOSE HURDLES SO I THINK THAT, YOU KNOW AS WE GO THROUGH AND CLEAN UP THIS DOCUMENT, EVERYONE CAN HELP US UNDERSTAND WHAT ARE THE VALUE SYSTEMS IN PLACE FOR EACH OF THOSE STAKEHOLDER GROUPS THAT PEOPLE HAVE DONE A GOOD JOB OF LISTING OUT. >> MONEY IS A BROAD TERM BUT IT COULD INCLUDE THE SAVINGS AND THE PHARMACEUTICAL COMPANY THAT THEY WOULD HAVE, THAT THEY WOULD GO FORWARD AND DEVELOP BY OTHER GROUPS. THEY WOULD RECOGNIZE THAT AS A VERY USEFUL ALREADY IN THE PAST AND THAT SAVINGS TRANSLATES TO SO WE WILL PUT AN EFFORT OR CONTRIBUTE, MONEY COULD ALSO BE IN TERMS OF HOW YOU SAVE IT. >> SO I SEE THAT IS FACILITATE INDEED LIMITATION OF THAT, BUT I MAY BE NAIVE BUT I WILL--AS I SAID BEFORE I WISH THAT PEOPLE WILL COME INTO IT, WANTING TO BE PART OF A MOVEMENT TO IMPROVE SOMETHING FOR THE GOOD OF EVERYBODY AND NOT TO LOOK FOR WHAT DO I GET OUT OF IT OTHER THAN ASK WHAT CAN I DO TO CONTRIBUTE THAT AND I THINK THAT IF YOU COME INTO A PROGRAM OR TO SOME PROJECT TO THAT IN A SENSE THAT YOU WANT TO CONTRIBUTE TO THAT, THE INCENTIVE FOR YOU IS HIGHER THAN WHAT CAN I GET OUT OF IT? BUT THIS IS MY OWN POINT OF VIEW. >> ON THE POINT ABOUT SAVINGS, IN THAT STANDARDS ABOUT EFFICIENCIES SO THAT COULD BE MAKING ANYTHING YOU DO MORE EFFICIENT. IF IT MAKES YOUR JOB AS A RESEARCHER MORE EFFICIENT, IF IT--SO THAT YOU'RE NOT HAVING TO CLEAN UP THE DATA TO DO YOUR ANALYTICS BECAUSE--YOU KNOW, NO ONE IN THIS ROOM IS REALLY INTERESTED IN CLEANING DATA, YOU'RE INTERESTED IN DOING, TRYING TO DRIVE KNOWLEDGE FROM THAT DATA, OR YOU KNOW CORPORATIONS ARE NOT INTERESTED IN HAVING LOTS OF DATA TO HAVE LOTS OF DATA, THEY WANT TO DO SOMETHING WITH THAT. AND SO, THE VALUE IS IN MONEY OR MONEY NOT SPENT, BUT ALLOWING US TO SORT OF RAISE THE BAR IN TERMS OF THE--WHAT WE'RE TRYING TO DO AND ACHIEVE AND THAT'S THE VALUE THAT SHOULD BE APPLIED TO ALL OF THE STAKEHOLDERS HOW IS THE METADATA, STANDARDS FOR METADATA MAKE THEM MORE EFFICIENT IN PROCESSING INFORMATION TO LEAD TO USEFUL KNOWLEDGE. MANY YEARS AGO THE OMG WAS WORKING LIFE SCIENCES, THERE WERE ABOUT EIGHT, AT LEAST EIGHT DIFFERENT FORM SUITICAL COMPANIES PUTTING IN A LOT OF TIME AND MONEY INTO THIS, THE REASON WAS THEY THOUGHT THERE REALLY SHOULD BE SOME LESSONS LEARNED IN EFFICIENCIES THAT WE COULD ALL SHARE, THE WORD BACK THERE IS PRECOMPETITIVE. RESPIRATORY >> OKAY, WE HAVE--WE HAVE VERY GOOD DISCUSSION, I THINK WE HAD A LOT OF THINGS SO I'LL TRY TO SORT OF SUMMARIZE WHAT WE--WE DISCUSSED WHAT ARE THE BEST PRACTICES FOR IDENTIFYING RELEVANT STAKEHOLDERS, I THINK WE'RE IN AGREEMENT THAT THE DEVELOPMENT OF A BROAD SET OF USE CASE SYSTEM KIND OF QUINTSENTIAL TO THE DEVELOPMENT OR STANDARD IN THE FIRST PLACE TDOES TWO THINGS, ONE OF COURSE IS IDENTIFIES THE PEOPLE WHO WOULD BENEFIT FROM SUCH CASES AND IT ALSO HELPS ESTABLISH A WAY TO EVALUATE THE STANDARD ONCE IT'S DEVELOPED. DOES IT REALLY DO WHAT YOU PURPORTED THAT IT WOULD BE ABLE TO DO ONCE YOU DEVELOP THE STANDARD? SO THAT'S--THAT CAME OUT AND WE ALSO TALK ABOUT THE POTENTIAL TO EXTEND THE PARTICIPATION TO A VASTLY BROADER COMMUNITY OF EVEN CITIZEN SCIENTISTS AND THAT'S NOT SOMETHING WE TRIED TO DO BUT YOU COULD HAVE PATIENT ADVOCACY GROUP OR WHATEVER PARTICIPATING AND PROVIDING INPUT IN TERMS OF USE CASES AND UNEXPECTED APPLICATIONS WHICH COULD BE INFORMATIVE FOR THE DEVELOPMENT STANDARD. I THOUGHT THAT WAS PRETTY ENLIGHTENING, WE TALK ABOUT ENGAGEMENT OF ACADEMIC SOCIETY AS A WAY TO REACH TO OTHER PEOPLE WHO WERE AFFECTED BY THIS STANDARD OF DEVELOPMENT AND AT LEAST INFORM THEM THIS IS HAPPENING EVEN IF THEY DON'T WANT A SPECIFIC PARTICIPATORY ROLE AND WE TALKED ABOUT ENGAGING PUBLISHERS WHO POTENTIALLY COULD PROVIDE THE STICK/CARROT, YOU WANT TO PUBLISH HERE, THEN HERE'S THE STANDARDS THAT OF COURSE WE RECOMMEND AND JOURNALS ARE DOING THAT SO THEY'RE NATURAL VEHICLES FOR ADOPTION BY SCIENTISTS AT THE END OF THE DAY. WE TALKED ABOUT SIMPLIFYING CALLS FOR PARTICIPATION SO WE NOTED THAT THE LANGUAGE THAT IS CURRENTLY USED BY THE NIH IS USED TO REFOND A PARTICULAR REQUEST IS NOT ACCESSIBLE TO PEOPLE OUTSIDE OF OUR COMMUNITY. SO WE NEED TO THINK ABOUT WAYS THAT WE WOULD SIMPLIFY THAT AND TELL THEM HOW THEY GET ENGAGED AND REALLY FOCUS ON SOMETHING THEY CARE ABOUT. WE TALKED ABOUT GOVERNMENT MODELS TO SUPPORT OR ENCOWERRAGE COMMUNITY ENGAGEMENT SO WE FIRST TALKED ABOUT HOW WE NEED TO HAVE ACCESS TO THE STANDARDS THAT ARE ALREADY THERE, SO IT'S LIKE BIOSHARINGOR KIND OF CRITICAL AND BIOMARTAL AND THESE REGISTRIES OF STANDARDS AND GUIDELINES AND VOCABULARIES, WE NEED THOSE THINGS AND WE NEED THEM WELL DEVELOPED ASK WE NEED THEM MAINTAINED SO THEY HAVE THE POLL FROM THE LATEST FINDINGS SO PEOPLE LIKE KUS FIND THEM. SO WE EMERGE WIDE THIS IDEA THAT WE WE NEED IS A COORDINATION CENTER, SOMETHING THAT WOULD HELP ACT AS A LEVEL BETWEEN THE PEOPLE WHO ARE DEDICATING TO DEVELOPING THE STANDARDS OF PARTICULAR DATA OR WHATEVER AND AND THE SCIENTISTS THAT ARE MAKING PROPOSALS AND THE USE OF THE DATA THAT WILL BE GENERATED TO THAT SO WE KIND OF NEED A GROUP THAT'S SOMEWHAT RESPONSIBLE FOR KEEPING TAPS ON EVERYTHING THAT'S HAPPENING AND WE DON'T FEEL WE HAVE THAT RIGHT NOW. SO THIS COULD BE A POTENTIAL OPPORTUNITY FOR US TO COORDINATE AMONG THAT SO WE TALKED ABOUT ISSUES WITH EARLY ENGAGEMENT AND THEY'RE ON THEIR WAY TO ESTABLISHING THE STANDARD WITHOUT HAVING, WITH THE WIDER COMMUNITY SO NEED TO ESTABLISH SOME KIND OF PROTOCOL TO GET US FROM WHO WILL BE ADOPTED BY THE LARGE COMMUNITIES AND COGNITIVE THAT WILL HAPPEN IN THE FIRST PLACE AND WE DON'T WANT TO CATCH PEOPLE OFFGARD AND GO HOW DID I NOT KNOW ABOUT THIS. WE TALKED ABOUT ROLES IN THE STANDARD ORGANIZATIONS AND DEVELOPMENT AND POTENTIAL AND THE DATA AND TERMINOLOGIES AND DATA STANDARDS SO WE TALK ABOUT INSTANCE A W-THREE C OR ISO OR THESE OTHER ORGANIZATIONS AND DEBATED THE PROS AND CONS OF THEIR APPROACH APPROACHES AND SO WE THINK ABOUT HOW WE WANT TO COMPETE AND IF WE'RE A COMPETING CENTER WE DON'T WANT TO THWART THIS SO WE WANT TO HAVE CONTINUED DEVELOPMENT. SO THERE'S MORE THAN THE HIGH EVERY LEVEL WE HAVE DOING THIS. I--SO WE TALK ABOUT POTENTIAL ROLE OF NIH IS TO--YOU KNOW LOOK AND DEVELOP A TASK FORCE AND LOOK AT WAYS OF MEDIATING THIS, CONVENING GROUPS, ALIGNING EFFORTS AND SO ON BUT THIS MATURED INTO MAYBE A NEED FOR COORDINATING CENTER. BUT I THINK WE NEED CALLS FOR EVALUATING THE CENTERS, BUT THAT MAY BE A WAY TO GET BROADER SETS OF USE CASES THAT WAS ENVISIONED AND DEMONSTRATE VALUE FOR THE INVESTMENT OF DEVELOPING THE STANDARD AND IT OPEN ITS UP AND IT ENGAGES THE COMMUNITY BEYOND THE INITIAL USE CASES THAT WERE PROPOSED AND I JUST HEARD THAT AND THAT WAS A COMMENT THAT WAS MADE IN THE LAST ONE CAN AND IT WAS LIKE OH YEAH, WE HAVE TO HAVE SOMETHING LIKE THAT. SO THE LAST LITTLE BIT IS TALKING ABOUT HOW IT TRANSLATES INTO MAINTENANCE AND SUSTAINABILITY AND WE THINK THERE SHOULD BE SUPPORT FOR THE ADOPTION OF A STANDARD AND THAT COULD BE IN THE DEVELOPMENT OF TOOLS AND TECHNOLOGIES AND COMMUNITY ENGAGEMENT STRATEGIES AND SO ON. ENCOURAGE THE INNOVATION TO SCALE, STANDARDS DEVELOPMENT AND I THINK THIS IS FROM MY OWN EXPERIENCE, IT'S NOT CLEAR HOW YOU SCALE THIS EFFORT AND IF YOU LOOK AT ALL THE DIFFERENT DATA TYPES, ALL THE EFFORT IS MONSTROUSLY LARGE AND COORDINATING THAT IS REALLY UNCLEAR TO ME, ANYWAY. SO I THINK WE NEED TO THINK ABOUT AND INNOVATE ON HOW TO SCALE THIS EFFORT SO IT CAN BE RESPONSIVE AND IT CAN DEVELOP A TIME AND IT CAN DO SO WITHOUT BREAKING THE WHOLE SYSTEM. I DON'T THINK WE HAVE A REALLY GOOD SYSTEM OF HOW TO DO THAT AND WE SHOULD ALSO ENABLE, RIGHT. SO THE LAST POINT IS ENABLE A CONTINUOUS DEVELOPMENT STAND SO AGAIN WE'VE MADE THESE INVESTMENTS AND CORE GROUPS THAT ARE CONFIDENT AND NEW IDEAS COME NOTHING AND WE NEED TO FIGURE OUT HOW TO GET NEW IDEAS TO EXTEND WHAT HAVE ALREADY DONE IN WAYS THAT MAKE SENSE. WE DON'T WANT TO SHOE HORN THING FIST THEY DON'T FIT PROPERLY AND WE MIGHT MAKE DECISIONGOING IN NEW DIRECTIONS BUT THAT'S A CONVERSATION THAT NEEDS TO HAPPEN AND DECISIONS NEED TO BE MADE THAT THIS IS THE MOST VIABLE PATH FOR IT. SO YEAH, IT WAS A GREAT CONVERSATION. ANY COMMENTS, QUESTIONS. SO I WANT TO COME BACK TO ONE OF THE POINTS YOU MADE UP A BOVE ABOUT THE FUNDING. >> HOW IS IT? >> SORRY, CAN'T SEE. >> SO BASICALLY THE IDEA ABOUT MAKING THE FUNDING CALLS MORE SORT OF FRIENDLY TO PEOPLE OUTSIDE OF OUR COMMUNITY AND I ALSO WONDER AND IT WAS NICE HAVING PETER FOX HERE TO TALK ABOUT WHAT THEY'RE DO NOTHING THEIR COMMUNITY. I FEEL LIKE I'VE SEEN A LOT OF SIMILAR WORKSHOPS COMING OUT OF NSF FOR WHAT WE'VE BEEN HAVING IN OUR SERIES OF BD2K WORKSHOPS. YOU KNOW HOW CAN WE PROMOTE SOME OF THAT SORT OF CROSS AGENCY EFFORTS AND INCLUSION OF MORE DIFFERENT PERSPECTIVES IN THOSE EFFORTS BECAUSE I REALLY FEEL LIKE WHEN WE GET PEOPLE TOGETHER FROM THESE DIFFERENT COMMUNEDS, DIFFERENT COMMUNITIES INVOLVE CERTAIN PROBLEMS THAT OTHER COMMUNITIES HAVEN'T AND WE THOUGHT ABOUT THAT AND THAT'S REFLECTED IN THOSE RFAs AS WELL. THE SAME BULLET POINTS, I RESONATE WITH THEM STRONGLY AND I UNDERSCORE PATIENT ADVOCACY GROUPS, DON'T UNDERESTIMATE HOW EFFECTIVE THEY ARE. PART OF OUR MISSION IS TO GET A LOT OF DATA THAT IS GOING TO BE, YOU KNOW SOMEWHAT THE PUBLIC SECTOR TO HAVE THE PEOPLE, THE PATIENTS, THE FAMILIES AND LOOK, I WANT YOU TO GO AHEAD AND USE THIS INFORMATION AND A LOT OF THINGS THAT WOULD BE POSSIBLE BUT YOU COULD LOOK INTO THAT. THAT IS A STRONG MOTIVATOR AND YOU WANT TO TALK ABOUT STANDARDS THAT ALLOW THE INTEROPERABILITY AND AS WELL AS THE INFORMATION AND WHEN IT IS THAT HUMAN CONSENT IS AVAILABLE, YOU WILL WANT THEM. SO, HAVING THEM INVOLVED SO THAT THEY CAN SEE WHAT'S GOING ON AND THEY CAN SEE THIS IS VERY TANGIBLE GOALS THAT THEY CAN SEE BENEFITS FROM, THAT WILL GO A LONG WAY TO MAKING THINGS SUSTAINABLE WITH NIH SAYING, OH WE'RE GETTING PATS ON THE BACK, THAT'S PRETTY NICE, SO THAT FROM THE BEGINNING, THEY SHOULD BE BROUGHT INTO SOME OF THOSE DISCUSSIONS. I KNOW PERSONALLY THEY WANT TO BE PART OF THIS, THEY'RE NOT ALL TECHYS BUT PART OF THAL FINER--MAKE SURE DENATIONAL LIBRARY OF MEDICINIC INFORMATION AND HEALTHCARE FIT TOGETHER, THEY'RE IMPORTANT BUNCH TO HAVE. >> GREAT COMMENT. >> YEAH, WE HAD ALSO MENTIONED, I DON'T THINK I SAID IT SPECIFICALLY BUT WE TALKED ABOUT PUBLIC PRIVATE PARTNERSHIPS AND THE ROLE OF INDUSTRY AND DEVELOPMENT OF STANDARDS AND THAT WAS A CONTENTIOUS ISSUES AND THAT WAS A DISAGREEMENT ABOUT WHAT THE ROLE OF INDUSTRY IS BUT THERE'S LITTLE DOUBT THAT THERE NEEDS TO BE ENGAGEMENT OF THIS THE INDUSTRY AND HOW AND WHAT MECHANISM IS UNCLEAR. >> IT SHOULD BE TOTALLY NONCONTENTIOUS BUT A COMMENT, I WAS REMIND WIDE WE TALK ABOUT THE RFAs ARE GOBBLE-DEE-GOOK IF THE INFORMATION, WE WE WERE TALKING ABOUT THE REPORTING THE INFORMATION TO DOCTORS FROM THE PATIENTS AND WHATEVER LESSONS WE CAN LEARN IN ONE, WE CAN APPLY TO THE OTHER AREA. >> YOU MAY HAVE MENTIONED IT BUT I DIDN'T HEAR IT. >> ONE TOPIC THAT CAME UP QUIETLY WAS THE NOTION OF THE [INDISCERNIBLE] FOR OFFERING A MANUSCRIPT FOR WHAT MAKES A GOOD STANDARD PRACTICAL AND, YOU USEFUL AND I--YOU KNOW IT'S REALLY--COULD BE A DELIVERABLE OUT OF THIS MEETING. >> THANK YOU ANY OTHER COMMENTS? THEN I'M DONE, YES. >> SO I GUESS I START BY JUST THANKING EVERYONE WHO'S STILL HERE FOR COMING AND JUST GIVE EVERYBODY A ROUND OF APPLAUSE. I THINK WE JUST HAVE A FEW MINUTES NOW THAT WE JUST WANT TO KIND OF WRAP UP THINGS AND MAYBE ASK FOR A FEW KEY TAKE HOME MESSAGE FIST ANYBODY HAS ANY SORT OF LIKE, YOU KNOW THE BURNING THING THAT YOU GOT OUT OF THIS SESSION THAT YOU WANT TO MAKE SURE WE LISTEN REALLY LOUDLY TO, LET'S MAKE SURE THAT GETS WRITTEN DOWN NOW AND ALSO LET YOU KNOW A LITTLE BIT ABOUT WHERE WE GO FROM HERE. SO IF THERE'S ANYBODY HAVE ANY SORT OF LIKE, YOU KNOW BRIGHT RED TAKE HOME MESSAGES WE WANT TO LIST UP THERE? WARREN YOUR TIME SUGGEST IMPECCABLE. >> WARREN SHOULD PAY FOR EVERYTHING. >> [LAUGHTER] >> THE HATE TO SAY IT, BUT PEOPLE, MONEY TALKS SO, THE FUNDING, I LOVE THE IDEA OF THE FUNDING FOR EVALUATION OF DATA STANDARDS, PUTTING OUT RFAs FOR THE INFRASTRUCTURE, NOT THE DEVELOPMENT BUT THE INFRASTRUCTURE DEVELOPMENT STUFF, HAVING STUDY SECTIONS THAT REVIEW THAT ARE FAMILIAR WITH THOSE ISSUES AND REALLY LOOK AT THOSE OPPOSE TO THE NORMAL STUDY SECTIONS THAT JUDGE ON HYPOTHESIS DRIVEN-- >> AWESOME, I LOVE ALL THOSE, THOSE ARE GREAT TAKE HOME MESSAGES. DID WE GET THOSE? MAKE SURE THEY'RE GETTING WRITTEN DOWN. ANYBODY ELSE HAVE ANY OTHER HIGH LEVEL--YEAH, GREAT, EXCELLENT. >>> NTHANK YOU FOR INVITING ME INTO YOUR WORLD AND I'M STRUCK BY THE SIMILARITIES AND CHALLENGES WE FACE, IT SEEMS WE SPENT A LOT OF TIME TALKING THE LAST TWO DAYS ABOUT NONTECHNICAL ISSUES ABOUT HOW TO ENGAGE STAKEHOLDERS AND ABOUT HOW TO DO GOOD PROJECT MANAGEMENT AND HOW TO SEE, INVESTIGATE WHY SOME STANDARDS ARE ADOPTED AND WHY SOME ARE NOT ADOPTED AND HOW THESE THINGS--HOW TO FORM PROVIDER--PRIVATE PUBLIC PARTNERSHIPS, ALL THESE THAT ARE CRITICAL TO DEVELOPING GOOD STANDARDS, YET YOU KNOW WE JUST KIND OF DO THEM BY EXPERIENCE OR BY YOU KNOW THAT WORKED THIS OTHER TIME AND SO IF WE COULD FORMALIZE THAT KNOWLEDGE SOMEHOW AND ADVANCE THAT KNOWLEDGE AND RECOGNIZE THAT THE NONTECHNICAL PARTS OF THIS PROCESS ARE JUST AS IMPORTANT, IF NOT MORE IMPORTANT THAN THE TECHNICAL PARTS. >> YES, ONE POINT. AS A EUROPEAN PARTICIPANT, I WANTED TO MAYBE ADD SOMETHING ABOUT CLARIFYING HOW TO INTERACT WITH TRANSLATIONAL EFFORT. THINGS THAT ARE TAKING PLACE OVER IN EUROPE WITH ELIXR, FOR INSTANCE FOR BUILDING INFRASTRUCTURE FOR BIOINFORMATICS HOW DOES IT IMPACT THE FUNDING OPPORTUNITIES OR THE PLAN WE NEED TO LAY OUT TO MAKE SURE THAT THE EFFORTS ARE MEANT TO BE WORLD WIDE AND GLOBAL, SO THAT'S THE ONLY THING I WANTED TO ADD ON THAT PARTICULAR TOPIC. >> I THINK ONE OF THE THINGS I LEARNED IS TO REALLY REITERATE WHAT TOM IS SAYING, THE SIMILARITY OF DEMANDS AND CHALLENGES THAT CONFRONT THE STANDARDS COMMUNITY WHETHER YOU'REOT CLINICL SIDE OR BASIC SCIENCE SIDE OR APPLIED SCIENCE SIDE, I WANT TO REENFORCE A CALL FOR SOME KIND OF FORM IN WHICH COMMUNITIES AND ORGANIZATIONS THAT ARE ENGAGED IN THE PROCESS OF STANDARDS DEVELOPMENT HAVE A PLACE WHERE THEY CAN COORDINATE THEIR ACTIVITIES IN THE CONTEXT OF TRANSLATIONAL RESEARCH AND DISCOVERY RESEARCH BECAUSE IT'S ABUNDANTLY CLEAR THAT MANY OF THE CLINICAL DATA STANDARDS COMMUNITIES DON'T EVEN GRASP THE IMPORTANCE OF HAVING A SLIGHTLY BROADER SCOPE TO ENABLE WHAT THEY DO TO ENABLE DOWN STREAM SECONDARY USE FOR RESEARCH AND I THINK THEY WOULD BE WILLING TO CONSIDER THOSE KINDS OF EXTENSIONS BUT THAT COULD BE REENFORCED AND STRENGTHENED AND PERSISTENT IF THERE IS A FORM IN WHICH THEY CAN CONVENE AND MEET AND SHARE THESE UNDERSTANDINGS AND APPRECIATIONS. I DON'T THINK PEOPLE ARE OPPOSED TO DOING IT IT'S JUST THAT THEY DON'T KNOW AND I THINK I COULD MAKE A COUNTER POINT FOR MANY OF THE OTHER ORGANIZATIONS. >> SO I THINK A LOT OF WHAT I'VE HEARD OVER THE LAST TWO DAYS COULD GO BACK TO THINKING ABOUT MORE OF A LIFE CYCLE APPROACH AND THAT WE HAVE FOCUSED A LOT OF ATTENTION AND YOU KNOW EFFORT ON THE INNOVATIVE ASPECT OF THAT EARLY PART OF--FOR DEVELOPING SOMETHING NEW AND SOMETIMES WE HAVE, YOU KNOW NOT ALWAYS IDENTIFIED AND BROUGHT ALONG STAKEHOLDERS EARLY ENOUGH IN PROTHE PROCESS AND SOMETIMES THAT LEADS TO STUMBLES DOWN THE ROAD, SOME MORE ATTENTION TO THAT, THE EARLY TAJ OF REALLY NURTURING AND FIGHTING THE RIGHT ENVIRONMENT EARLY ON. AND THEN THE OTHER END OF THE LIFE CYCLE TOO, I HEARD AGAIN AND AGAIN, WE PROBABLY DON'T NEED MANY MORE NEW STANDARDS AND THAT SHOULD BE YOUR LAST RESORT OF WHAT HAS BEEN DEFINED AND SUPPORTING AND MAINTAINING AND INTEGRATING COULD FIT OR A WAY TO THINK ABOUT THEM IS THAT WE FOCUS TOO MUCH ON ONE PARTICULAR LIFE STAGE OF CYCLE AND NIH NEDS TO CREATE THE IDEA THAT THERE ARE DIFFERENT POINTS ALONG THAT PROCESS, THAT NEED MORE ATTENTION. >> I THINK ALONG WITH THAT AND PARKING BACK TO THE EARLIER COMMENT, TOO BECOMES BOTH FOR EACH STEP, EACH PART OF THAT LIFE CYCLE THERE'S ALSO A NEED FOR BOTH SOCIAL AND TECHNICAL SUPPORT RIGHT SO SO THAT WE'RE EVALUATING THINGS AND WE'RE COORDINATING STAKEHOLDERS AT THOSE TIMES OF THINGS,. >> WELL, LET US AGAIN THANK YOU ALL. THESE KINDS OF MEETINGS CAN'T WORK WITHOUT PARTICIPATION AND I THINK WE HAVE NO CAUSE FOR COMPLAINT IN TERMS OF PARTICIPATION. THIS HAS BEEN WONDERFUL AND IN TERMS OF ONGOING PROCESS, DO YOU WANT TO ADDRESS THAT? >> NO, I JUST WANT TO--FROM THE NIH, THANK ALL OF YOU FOR YOUR PARTICIPATION. MELISSA AND CHRIS AND ESPECIALLY YOU AND I WILL ADD ASTRID, BECAUSE WITHOUT HER I DON'T THINK THIS MEETING WOULD HAVE COME TOGETHER. SHE HAS REALLY DONE A WONDERFUL JOB. SO WHAT'S GOING TO HAPPEN NEXT? WE WILL DEFINITELY BE INGESTING AND SYNTHESIZING WHAT WE'VE HEARD TODAY OVER THE NEXT TWO MONTHS AND WE WILL HAVE A REPORT THAT WILL BE MADE AVAILABLEOT MAIN NIH BD2K WEBPAGE THAT REFLECTS THE DISCUSSIONS THAT WE'VE HAD HERE AND N AN--SOME SORT OF ORGANIZED FORM. NOW THE FUN PART FOR THE NIH AFTER TODAY IS HOW WE ACT ON AND PRIORITIZE AND THINK MORE DEEPLY ABOUT WHAT WE'VE HEARD ABOUT AND WE WILL BE DOING THAT, THINKING ABOUT INITIAL STEPS WE COULD TAKE SO WE'RE LOOKING FOR SOME OF THE EASIER IMMEDIATE THINGS THAT CAN BE PUT IN PLACE BUT ALSO A LOT OF WHAT WE HEARD IS REALLY MOVING TOWARD A VISION AND WHAT DO WE NEED TO PUT IN PLACE INFRASTRUCTURE WISE THAT WILL ADDRESS SOME OF THE MORE DIFFICULT ISSUES IN A LONGER TERM WAY, SO ALL OF THOSE THINGS ARE ON THE TABLE AND WE WILL DEFINITELY ACT IN SOME FASHION AND PROBABLY NOT JUST IN ONE WAY, BUT THERE WILL A--BITS I NUMBER OF DIFFERENT ACTIONS THAT WILL FLOW FROM THE GOOD WORK THAT ALL OF YOU HAVE DONE HERE TODAY AND PREVIOUS WORKSHOP ON THIS AS WELL AS THE--HAVE A GREAT RESPONSE TO THE REQUEST FOR INFORMATION. SO WE WILL CONTINUE TO ENGAGE WITH THIS COMMUNITY AND AGAIN BY THANKING ALL OF YOU BECAUSE YOU'VE BEEN GREAT IN ADVISING US. [ APPLAUSE ] >> AND SPECIAL THANKS TO CINDY FOR BRINGING US ALL HERE AND REALLY DRIVING US FORWARD FOR NIH AND FOR ALL OF US IN OUR COMMUNITY. [ APPLAUSE ]