>> OKAY, WELCOME EVERYBODY. WE'RE GOING TO BEGIN WITH A PRESENTATION BY IVOR D'SOUZA, TALKING ABOUT THE NLM AND THE CHANGING LANDSCAPE OF I.T. SECURITY. >> GOOD MORNING EVERYONE. >> GOOD MORNING. >> ALL RIGHT, SO TODAY WE'RE GOING TO TALK ABOUT HOW TECHNOLOGY HAS EVOLVED AT THE LIBRARY AND WITH THAT EVOLUTION OF TECHNOLOGY HOW WE ATTACK NEW THREATS SENT TO US AND WHAT WE HAVE DIRECTOR OF NATIONAL INSTITUTE TO CREATE A BALANCED PORTFOLIO OF SECURITY USING PEOPLE, PROCESS AND TECHNOLOGY TO HANDLE THESE TECHNOLOGY CHANGES AND THEIR ASSOCIATED THREAT INTERESTS. BUT FIRST I WANT TO BRIEFLY TALK ABOUT TWO SLIDES TO TALK ABOUT WHY IS SECURITY IMPORTANT TO THE LIBRARY, SO IF WE'RE TALKING ABOUT SECURITY, THIS POPULAR CIA TRIAD MODEL THAT MOST SECURITY PROFESSIONALS KNOW OF, IT HAS CONFIDENTIALITY, INTEGRITY AND PORTABILITY, I'VE MADE THE CONFIDENTIALITY HEXAGONS SMALLER BECAUSE MOST OF OUR NLM'S DATA IS PUBLICLY AVAILABLE AND OPEN. BUT DATA INTEGRITY SEVERY CRITICAL BECAUSE WE COOPERATE WANT TO PEOPLE TO TAMPER OR MODIFY. AND AVAILABILITY IS IMPORTANT BECAUSE WE WANT TO MAXIMIZE OUR SYSTEMS WE DON'T WANT HACKERS TO BLINK OUR SYSTEM SYSTEM DOWN AND NIH TALKS ABOUT NLM BEING A TRUSTED SYSTEM AS PROVIDER AND WHEN YOU TALK ABOUT TRUST, YOU WANT TO MAKE SURE THAT IT'S HIGHLY AVAILABLE TO BE USED. SO MOVING ON TO THE BRIEF EVOLUTION, WE WILL TALK ABOUT HOW TECHNOLOGY HAS EVOLVED AT THE LIBRARY. [LAUGHTER] AND WE'LL TALK ABOUT WHAT SECURITY THREATS, ET CETERA HAVE COME AS A RESULT OF THIS. SO IN THE EARLY 60S, WE GOT INTO COMPUTERS AND THEN OF COURSE IN THE 70S WE GOT THE MAIN FRAME BUT IN THE EARLY YEARS MOST OF OUR FOCUS WAS AROUND OPERATING SYSTEM CYBERSECURITY SO YOU HAVE TO MESS AROUND WITH THE OPERATING SYSTEM AND ALSO KNOWN ASSOCIATED NUMBER AND YOU NEED PHYSICAL ACCESS TO DO SOMETHING TO IT AND EVEN THEN, NLM WAS THE FIRST ORGANIZATION TO JOIN THE PACKET SWITCH, PUBLIC PACKET SWITCH AND EVEN IN THOSE DAYS, IT WASN'T A NETWORK BUT A NETWORK LIKE THE INTERNET IS TODAY AND STILL MOST OF THE THREATS AND THE ATTACKS OF THIS WAS OPERATING SYSTEM BASED EVEN AT THAT TIME. BUT IT WASN'T UP UNTIL THE INTERNET CAME UPON THAT WE STARTED SHIFTING FOCUS FROM PURELY BEING AN OPERATING SYSTEM-FOCUSED FROM A KESURITY TO ALSO WORRYING ABOUT NETWORKS. BECAUSE WHEN WE JOIN THE INTERNET AND THE NLM WAS ONE OF THE FIRST TO JOIN THE INTERNET IN 1988 AND SO WHEN WE JOINED THE INTERNET, BECAUSE HAVE YOU FRIENDLY NEIGHBORS AND HAVE YOU UNFRIENDLY NEIGHBORS AND YOU HAVE TO WORRY ABOUT THE UNFRIENDLY NEIGHBORS OF COURSE, SO THEREIN WE CHANGED THE WAY WE THOUGHT ABOUT SECURITY: OUR FIRST CHALLENGES IN INTERNET SECURITY WERE E-MAIL LISTS, THIS PARTICULAR ONE, I LOVE YOU VIRUS IS POPULARLY KNOWN AS MICROSOFT 8.5 BILLION DOLLARS MISTAKE BECAUSE WHAT THEY HAD DONE WAS E-MAIL READERS LIKE MICROSOFT OUTLOOK AND WOULD TRY TO OPEN EVERY ATTACHMENT THAT WAS SENT TO IT IN THE EARLY SETTINGS. SO MALITIOUS CONTENT WAS SENT IN THESE ATTACHMENTS AND PEOPLE WERE SENT E-MAILS IN ORDER TO FIND OUT WHO ON THE INTERNET ALLOWS THESE, THESE ARE THE KINDS OF THINGS THAT WOULD HAPPEN AND THEN THESE VIRUSES WOULD SPREAD AND WORMS AND THERE WAS A MALITIOUS WORM AS WELL SO THAT EARLY TIME FRAME FROM OF INTERNET WAS MOSTLY E-MAIL-PLAGUED VIRUSES. THE BACK SIDE OF THE FIRST GENERATION OF THE INTERNET WAS MOSTLY STATIC PAGES. THEY HAVE NO APPLICATIONS BEHIND THEM, THEY WERE HARD CODED LINKS. THEY WERE EASIER TO PROTECT, HACKERS THOUGHT THEY WERE EASY BUT IN RETROSPECT, WHEN WE LOOK AT WHERE WE ARE TODAY, THOSE WERE QUITE SIMPLE. SO IT'S EASIER TO PUT A FIRE WALL OR SECURITY LINES AND SAY THESE ARE THE SELECT THINGS THAT WILL LET THROUGH AND EVERYTHING ELSE WE WILL BLOCK. SO IT'S EASIER IN THOSE DAYS. BUT IF I LOOK BACK AND SAY, WE NEVER REALLY HAD A WHOLE BUNCH OF--WE NEVER HAD ANY FULL-TIME SECURITY STAFF UNTIL THE LATE 1990S AND EARLY 2000, WE DID HAVE PEOPLE DABBLING IN SECURITY AS PART-TIME WORK BUT THE TIME YOU TOOK FOCUS AND GOT FULL-TIME SECURITY STAFF MEMBERS WAS IN THE LATE 90S SO WE ENDED UP GETTING NETWORK SECURITY AND NETWORK SECURITY IS IMPORTANT WITH THE ADVENT OF THE INTERNET, TASK TOP SECURITY WAS IMPORTANT BECAUSE VIRUSES WERE ALL OVER THESE AND DESKTOPS WERE VERY DIFFICULT TO CONTROL FROM A CONFIGURURATION AND DESKTOP AND DIFFERENT PIECES OF SOFTWARE AND SO MANY VULNERABILITIES AND THE TOOLS IN THE EARLY DAYS WOULD NOT WORK TO MANAGE THESE DESKTOPS CENTRALLY. IT'S JUST VERY DIFFERENT. YOU HAD TO HAVE PEOPLE RUNNING AROUND AND TRY TO PASS THESE EMPLOYEES AND WE ALSO TOOK OUR OPERATIONS STAFF, WHO WERE MOSTLY DATA FRAME--MAIN FRAME TYPES OF INDIVIDUALS COMING OUT OF THE MAIN FRAME INTO THE DISTRIBUTED COMPUTING AND WE CONVERTED AND TRANSFORMED THEM FROM BEING JUST OPERATORS TO NOW ALSO WATCHING NETWORK EVENTS AND SECURITY EVENTS AND NOW WE HAVE 24-/SEVEN COVERAGE DURING THOSE YEARS. BUT THE NET 2.0 TRANSFORMED EVERYTHING. WE WENT AWAY FROM STATIC PAGES TO VERY RICH AND INTERACTIVE CONTENT. WE HAD THESE THREE TIERS IN WE HAD AND WE HAD TO NOT ONLY WORRY ABOUT THE WEB SERVERS THEMSELVES BUT ALSO APPLICATION SERVERS AND DATABASE SERVER AND IT IS RICHNESS OF THIS MULTITIER HIERARCHY OF COURSE INCREASED THE PROBABILITY THAT WE WOULD HAVE MORE VULNERABILITIES IN OUR SYSTEMS, YEAH? AND SO, WITH THAT, YOU COULD CONCEIVABLY THANK THE PROBABILITY OF PEOPLE OF GETTING ACCESS TO CORE EMISSIONS AND DATA INCREASED. AND OVER THE YEAR FIST YOU LOOK AT IT, EVEN THOUGH THE WEB 2.0 START INDEED THE EARLY 2000S, WE STARTED NOTICING THE SECURITY CHALLENGES OF WEB .2.0 IN THE MID2000S. AND IT'S BEEN INCREASING OVER TIME BUT IT'S NOT IN 2013 FOR THE VERY FIRST TIME THAT WEB ATTACKS HAVE COME TO THE VERY TOP. I MEAN THEY'RE THE MAIN REASON THOUGH FOR DATA BREECHES, MAIN REASON FOR DATA BREECHES AND THEN WHY IS THIS CASE? YOU LOOK AT ALL THE SOFTWARE DEVELOPMENT TOOLS OUT THERE, THIS IS A HUGE COMPLEXITY IN TERMS OF THE FRAMEWORK THAT SOFTWARE DEVELOPERS USE AND THE LIBRARIES, THE INDIVIDUAL DEVELOPMENT AND ALL OF THIS IS GETTING SO COMPLEX AND IT DOES DOES A GOOD JOB BUT IT DOESN'T GIVE ME ENOUGH REAL ESTATE TO LIST EVERYTHING, BUT IT'S SO COMPLEX THAT THERE'S EVOLUTION OF TECHNOLOGY HAPPENING ALL THE TIME, KEEPING THESE THINGS PASSED SO THEY'RE NOT VULNERABLE IS A BIG CHALLENGE. TO LOOK AT CUMULATIVE AND JUST NEW YEAR AFTER YEAR TO KEEP INCREASE YOU CAN SEE THE NUMBER OF VULNERABILITIES THAT ARE THERE FOR PEOPLE TO EXPLOIT IT, WANT TO EXPLOIT. AND THE TOOLS THAT ARE AVAILABLE TO HACKERS ARE SO EASY TO USE AND SO ABUNDANTLY AVAILABLE, THERE ARE YouTUBE VIDEOS THAT COULD MAKE A NOVICE HACKER INTO AN EXPERT IN WEEKS. USED TO BE YOU HAD 20 HAD TO SPEND YEARS TO BE A CODER AND BE A HACKER BUT THAT'S NOT THE CASE ANYMORE. WHEN WEB CONNECTED LOOK AT HACKING AS AN ACTIVITY, AND YOU CAN SEE IT, YOU CAN SEE IT'S REACHED AN ALARMING RATE. HACKING IS NOT JUST MADE THE LIFE OF SECURITY PROFESSIONALS MISERABLE BUT IT'S ALSO CHANGED THE WAY WE LOOK AT COMMON WELL-KNOWN UNIVERSAL TRUTHS WHICH IS THAT ONE. SO I REMEMBER WHEN I WAS LITTLE, YOU WOULD--JUST STOP BY, GIVE A MAN A FISH AND YOU FEED HIM FOR A DAY, TEACH HIM OUT TO FISH AND YOU FEED HIM FOR A LIFETIME. BUT THESE DAYS YOU DON'T TEACH HIM ANYTHING, GIVE HIM ACCESS TO INTERNET, DOWNLOAD THE YouTUBE VIDEOS, DOWNLOAD THE TOOL AND HE WILL TEACH HIMSELF HOW TO HACK AND HE WILL MAKE YOUR LIFE MISERABLE. SO THAT'S WHERE WE ARE. BUT FORTUNATELY, I MEAN WE AT NLM HAVE DEALT WITH THE APPLICATION SECURITY SINCE AROUND 2005 AND WE HAVE MATURURED OUR PROCESSES AS WE'VE GONE ALONG. I WILL SHARE A WITH YOU HOW WE'VE USED A BALANCE COMBINATION OF TECHNOLOGY, PROCESS AND PEOPLE AND COMBINATION IS IMPORTANT AND YOU CANNOT DO IT IN JUST ONE OF THOSE AREAS. TO ILLUSTRATE HOW WE DO IT, I WANTED TO SHARE WITH YOU THIS PICTURE. SO WHAT YOU'RE SEEING DOWN HERE, WHAT YOU SEE, THERE IS A SECURITY GATE AND I WANT TO ASK THE AUDIENCE IF YOU DON'T MIND, PLEASE TELL ME WHAT DO YOU SINGLE WRONG WITH THIS PICTURE. [LAUGHTER] THE FIRST PERSON TO ANSWER GETS A FREE CUP OF COFFEE AT THE NEXT BREAK. >> NO FENCE. >> NO FENCE, CAN YOU SEE TIRE MARKS PEOPLE ARE DRIVING PASSED IT SO THIS SYSTEM WAS BUILT BY PEOPLE, THEY THOUGHT EVERYBODY WAS VERY HONORABLE, AND THEY WOULD FOLLOW THE RULES, OKAY? SO-- >> [INDISCERNIBLE] >> THAT'S PROBABLY TRUE. THEY LOOK TO ME LIKE--[SPEAKING AT ONCE ] YOU COULD BE RIGHT, YOU COULD BE RIGHT. I DON'T KNOW. THAT'S AN INTERESTING PERSPECTIVE. NOW IF YOU LOOK AT WHAT ALLEN TOLD US THE FIRST RECIPIENT OF THE AWARD HAD TO SAY, EVERY PROGRAM HAD AT LEAST TWO PURPOSES, ONE FOR WHICH IT WAS WRITTEN AND ONE FOR WHICH IT WASN'T AND SOMETHING THAT WE AS I.T. SECURITY PROFESSIONALS REMEMBER IS THAT WE HAVE TO WATCH FOR HACKERS WHO ARE GOING TO USE THE SECOND REASON, RIGHT? AS YOU SAW IN THAT PARTICULAR THING. SO LET'S LOOK AT THIS AND LET'S LOOK AT HOW WE USE TECHNOLOGY TO AUTOMATE SOME OF THESE ISSUES THAT WE SEE IN THIS PARTICULAR EXAMPLE. SO THE FIRST THING OF COURSE WE DO IS WE ELIMINATE POINTS OF FAILURE BECAUSE IF THAT ONE GATE STOPS WORKING, NOBODY'S COMING IN AND THAT'S NOT REALLY GOOD, SO YOU HAVE TO HAVE MULTIPLE GATES OUT HERE THAT YOU SEEOT NIH CAMPUS, YOU WANT TO HAVE MULTIPLE GATES. AT THE LIBRARY, WE HAVE MULTIPLE WEB SERVERS, DATABASES, MULTIPLE EVERYTHING, WE HAVE MULTIPLE SITES SO WE HAVE MULTIPLE DATA CENTERS. SO YOU WANT TO HAVE HIGHER AVAILABILITY FOR THE SENSORS. THE OTHER THING WE WANT TO DO, I KNOW ONE OF YOU, MENTIONED WE NEED FENCES AND YOU NEED WALLS AND YOU WANT TO REDUCE THE ATTACKING OF THE PEOPLE, THEY DON'T JUST DRIVE ON THE GRASS AND COME IN. YOU WANT TO BRING THEM ALL TO CHANNELS ONLY SO THAT THEY'RE NOT TAKING ADVANTAGE OF THE FACT THAT THERE'S A FLAT AND AREA GRAPH THAT SAYS COME ON IN. IN OUR WORLD WE HAVE FIRE WALLS AND OTHER TECHNOLOGY ITEMS THAT WE USE TO MAKE SURE THAT PEOPLE HAVE VERY NARROWED FOCUS ACCESS TO OUR SYSTEM. YOU JUST DON'T ALLOW THEM ACCESS TO THINK THEY SHOULDN'T HAVE ACCESS TO. AND THEN YOU BUILT THE FENCE AND THAT GATE IS SO WEAK THAT A TRUCK CAN DRIVE RIGHT THROUGH IT AND KNOCK IT DOWN, THERE'S GOT TO BE SOME OTHER MECHANISMS TO STOP A PERSON FROM COMING IN. SO YOU NEED TO HAVE MULTIPLE LEVELS OF FENCES IN OUR SYSTEM. VALIDATE INPUTS ARE VERY SAFE, WE WANT TO MAKE SURE THAT WHAT PEOPLE ARE BRINGING INTO OUR ENVIRONMENT ARE SAFE. SO IN THIS WORLD, YOU WANT THE CAR TO BE SCREENED TO MAKE SURE IT'S GOT NO EXPLOSIVES, TO MAKE SURE THAT PEOPLE ARE NOT BRINGING AMMUNITION, THINGS OF THAT NATURE. IN OUR WORLD IF PEOPLE WERE TYPING A PASSWORD IN A PARTICULAR FORM, YOU WANT TO MAKE SURE THEY WERE TYPING THE USER NAME, THEY'RE NOT PUTTING IN A CODE THAT CAN BE EXECUTED THAT WOULD SOMEHOW USE VULNERABILITY AND EXECUTE IN OUR SYSTEM AND CREATE HAVOC INTERNALLY, SO IF YOU WANT TO VALIDATE THESE AND THIS IS ONE OF THE BIG ISSUES IN APPLICATION SECURITY AND AS MUCH AS IT SEEMS VERY STRAIGHT FORWARD TO DO WE HAVE SO MANY ALL IT TAKE SYSTEM ONE MORE MISTAKE AND THIS PROBLEM CAN CREATE ISSUES. THE SAME THING IS TRUE ABOUT SANITIZING OUTPUTS, SO WHEN YOU SEND THINGS BACK IN THE SYSTEM, DON'T GIVE AWAY YOUR SECRETS AND YOU CLEAR OUT THE SECRET AND POST A VERY HARMLESS MESSAGE TO THE OUTSIDE AND IF YOU WANT TO ATTACK THE NLM AND STEAL ITS PRICELESS BOOKS FROM THE COLLECTION HERE ARE THE THINGS YOU SHOULD DO AND WE DON'T DO THAT. IF SOMEBODY WERE TO TYPE IN A USER NAME, PASSWORD AND A MESSAGE OUT THAT SAID, WE COULDN'T FIND A PASSWORD IN THIS DATABASE AND THIS TABLE AND THIS LINE AND WAHAVE YOU, THESE TABLES EXIST AND THESE DATABASES EXIST AND THAT'S ONE STEP CLOSER TO TRYING TO FORMULATE A GOOD ATTACK. AND IF BY CHANCE WE WANT TO GET IN, WE WANT TO COMPARTMENTALIZE, WE WANT TO MAKE SURE THAT VISITORS GO INTO THE VISITOR PARKING LOT AND SO ON AND SO FORTH AND IF SOMETHING BAD AND I WON'T--WE HAVE VERY LITTLE TRUST BETWEEN THE VARIOUS GROUPS OF PEOPLE, THE VARIOUS GROUPS OF SYSTEMS, THE VARIOUS GROUPS OF NETWORK, WE KIND OF COMPARTMENTALIZE THEM TO MAKE SURE THAT INCIDENT IF THEY DO HAPPEN WILL BE CONTAINED. AND FINALLY I WANT TO TALK ABOUT THE PRINCIPLE OF LEAST PRIVILEGE SO, EVERYBODY ON THE INSIDE ONLY HAS ACCESS TO THEY NEED TO HAVE ACCESS TO FOR THEIR BUSINESS. THEY NEVER GET RID OF ANY ACCEPTED PRIVILEGE SO EVEN THOUGH I PLAY THE ROLE OF C. I. A. AT THE LIBRARY, IF I NEED SOFTWARE TO REINSTALL ON MY COMPUTER, I HAVE TO ASK SOMEBODY TO DO IT. I DON'T HAVE PRIVILEGES, I DON'T NEED THE PRIVILEGES, IT'S NOT MY ROLE TO HAVE THOSE PRIVILEGES. SO THE BOTTOM LINE IS WE ARE ALWAYS RELUCTANT TO TRUST, AND THE RELOCKET ANNUALSY TO TRUST IS A VERY IMPORTANT ASPECT OF PREVENTING SECURITY ATTACKS, SO WE DON'T BUILD THOSE GATES LIKE YOU SAW. THE SECURITY GATES, THE WAY IT WAS DISPLAYED IN THAT PICTURE. SO HERE'S AN EXAMPLE OF A MAJOR INCIDENT THAT HAPPENED RECENTLY, I THINK THIS INCIDENT BECAUSE IT HAPPENED RECENTLY JUST BEFORE BLACK FRIDAY, RIGHT? TARGET HAD THE LARGEST EXPOSURE OF PAYMENT CAR SYSTEMS IN HISTORY, 40 MILLION--40 MILLION PAYMENT CARDS WERE EXPOSED, 40 MILLION, THAT'S THE LARGEST, EVER. THE TARGET AND THE HVAC WERE LINKED SOMEHOW AND THEY THE HACKER WAS ABLE TO LAUNCH THE ATTACK FROM THE HVAC NETWORK AND CREATE THIS ISSUE. AND WE TRY REALLY HARD, KEPT SAYING TRIED REALLY HARD SO THAT WE DON'T HAVE THIS TRUST IN THESE UNEQUALED SYSTEMS, CREATE SEPARATE ZONES OF TRUST AND PUT IN THE SAME ZONE, THINGS THAT ARE WEAKER OR DIFFERENT NEEDS AND DIFFERENT ZONES SO WE CREATED THESE POCKETS BUT THAT WAS A BIG ISSUE THAT TOOK ADVANTAGE OF THE EXPLOITATION OF TRUST. HERE ARE THE THINGS WE COLLECT DATA AND AT THE LIBRARY WE HAVE TECHNOLOGY THAT IMPLEMENTED BASED ON SECURITY POLICIES AND PEOPLE VIOLATE THE POLICIES, THERE ARE AUTOMATIC BLOCKS THAT HAPPEN, SO WE AUTOMATICALLY BLOCK CERTAIN VIOLATIONALATIONS OF SECURITY POLARIZED POLICY. NOW HERE YOU CAN SEE THE PROFILE OF COUNTRIES OR USER COUNTRIES WHERE ATTACKS ARE SOURCED. SO THIS IS NEVER OVER THE SAME SLIDE, SOMETIMES THERE MAY BE A DIFFERENT IT GENERALLY FLIP FLOPS BETWEEN CHINA AND THE U.S., BUT IT GIVES YOU A SENSE OF WHAT WE'RE DEALING WITH. IT HELPS US FOCUS ON WHICH AREAS OF SOFTWARE WE WANT TO PAY MORE ATTENTION TO. PEOPLE DOING SECURE CODING, WE WANT THEM TO KNOW THAT THESE ARE THE THINGS THAT PEOPLE ARE COMING TO US ON THIS PARTICULAR APPLICATION MORE, ON A DIFFERENT APPLICATION MIGHT BE DIFFERENT AND SO THIS IS LIKE A PROCESS KIND OF THING THAT WE IN OUR SOFTWARE TO MAKE SURE WE KNOW WHERE TO PUT OUR RESOURCES GIVEN THAT RESOURCES ARE ALWAYS GOING TO BE SCARCE AND PRIORITIZE THEM BEST. THE PROCESSES BY THIMS WILL NOT BE GOOD ENOUGH, THIS IS THE REPRESENTATION OF A LOT OF PEOPLE, BUT THERE ARE A TON OF PEOPLE HERE AT THE LIBRARY WHO HAVE BEEN VERY INSTRUMENTAL IN US HAVING A STRONG THERE BECAUSE IT'S THE PEOPLE WHO ACTUALLY PICK THE PROCESSES AND THE TECHNOLOGY REGARDLESS OF THAT IN THE 1990 SYSTEM WHEN WE GOT OUR FIRST OPERATING SYSTEM STAFF THERE, AND THEN LATER ON, WE GOT A NETWORK STAFFERS, YOU KNOW AND THE FULL-TIME STAFF BUT NOT UNTIL 2014, LAST YEAR, THAT WE DECIDED APPLICATION SECURITY WAS JUST GETTING OUT OF WHACK, IT WAS GETTING TOO MUCH AND WE FINALLY OURSELVES AN APPLICATION OF THE SECURITY STAFF. IF I HAVE TO PICK ONE REASON WHY THIS STANDS OUT. WORK IN I.T.,ULE OF THEM HAVE A SIGNIFICANT ROLE IN I.T. SECURITY. IT'S NO LONGER THE CASE THAT SECURITY IS THE WORK OF PEOPLE WHO ARE LABELED SECURITY PROFESSIONALS. IF I'M BUILDING A SECURITY APPLICATION THERE'S A SENSE OF SELF-PRIDE AND--IT REMINDS ME OF A HOMEOWNER, AS A HOMEOWNER I WANT MY HOUSE TO BE SECURE I MAY HAVE ACAN YOU MIN OR SECURITY OR ADT OR WHATEVER MONITOR MY SECURITY BUT I STILL FEEL PERSONALLY RESPONSIBLE FOR LOCKING THE DOORS AND DO ALL THE RIGHT THINGS AND NOT REALLY BANKING ON THOSE PEOPLE TO SECURE MY HOME. IN OUR HOMES, AND THAT'S THE SAME ATTITUDE WE'RE NOW TRANSLATING TO OUR WORKPLACE. WE'RE NOT BANKING ON THESE OTHER PEOPLE TO TAKE CARE OF OUR SYSTEMS. OUR SYSTEMS ARE IMPORTANT TO US AND WE'RE GOING TO DO THE BEST THEY ARE THEMSELVES. SO I'M REALLY PROUD. IT'S BECAUSE OF THESE STAFF, THAT WE HAD A STRONG SECURITY PROGRAM AND ALSO BECAUSE OF THEM THAT I SLEEP VERY WELL AT NIGHT. [LAUGHTER] EVEN MY TEDDY BEAR'S FEELING SECURE. YOU MIGHT REMEMBER THIS LAST YEAR, PRESIDENT OBAMA SIGNED A SECURITY BILL, ONE THAT WAS IMPORTANT TO US WAS THE FSMA ACT. FEDERAL SECURITY MODERNIZATION ACT. AND THE IMPORTANT THING ABOUT THIS ACT WHICH IS A REALLY GOOD ACT WAS IT TOOK THE GOVERNMENT FROM SOLELY THINK TAG SECURITY WAS A ONCE A YEAR ACTIVITY WHERE YOU WOULD DO AN AUDIT ONCE A YEAR AND NOW WE ARE DOING IT ON A CONTINUE BASIS. SO IT'S MOVED US FROM THINKING, IT MOVED--THE FEDERAL GOVERNMENT FROM THINKING THAT SECURITY IS A ONCE A YEAR ACTIVITY TO A CONTINUOUS ACTIVITY. AND IT'S DONE THAT THROUGH THIS PROGRAM CALLED CONTINUOUS DIAGNOSTICS AND MITIGATION WHICH WE CALL CDM. SO NOW IT'S LAW. AND SO IT'S NORMAL COMPLACENCY, I'M COMPLIANT, I HAVE GOOD SECURITY BUT IT'S TRUE. SO WHAT DOES NLM'S TRACK RECORD ON CDM, SO WE STARTED, THE THING CALLED CDM AND THE DATA CALLED AUTHORITY TESTING AND CALLED CONTINUOUS MONITORING SOME OF THE TERMS, PEOPLE, AND NOBODY LIKED THE FACT THAT IT SAID MONITORING BECAUSE IT CREATED PROBLEMS IN THE WORKPLACE THAT PEOPLE WERE BEING MONITORED AND SO IT ENDED UP BEING CONTINUOUS DIAGNOSTICS AND MITIGATION, BUT IN 2005, WE STARTED BY THE TIME IT WAS 2009 WE HAD MATURED ORIGINAL PROGRAM. AND THEN IN 2011 WHEN NIH DECIDED THEY WANT TO DO AN NIH WIDE, THEY ADOPTED OUR PROGRAM. SO THEY SAID, YOU KNOW, NLM, HAVE YOU THE RIGHT THEN, WE WANT TO USE WHAT YOU HAVE. IN 2012 WHEN HHS WANTED TO DO IT ACROSS THE BOARD, THEY INHERITED OURS SO THEY PICKED THE SAME TECHNOLOGY AND PLOT FORM WIC PICKED AND NOW OF COURSE YOU HAVE 2014 WITH THE MODERNIZATION ACT AND IT IS PREPARED AND IT WAS BECAUSE OF SOME OF THE GOOD WORK OUR TEAM HAD DONE EARLIER STARTING IN 2005. I WANT TO TOUCH ON THREE POINTS BECAUSE OF NEXT STEPS, THE FIRST ONE ISAGEILE SOFTWARE DEVELOPMENT FRAMEWORK, MANY OF OUR TEAMS ARE MOVING TO WHAT WE CALL AGILE SOFTWARE DEVELOPMENT SO WE CAN SEE THIS HAPPENING OVER MULTIPLE MONTHS. THERE IS A MOVE TO MOVE SOME OF THOSE SOFTWARE DEVELOPMENT PROGRAMS INTO DELIVERING SOFTWARE WITHIN WEEKS, PERHAPS EVERY TWO WEEKS WE GET SOMETHING NOW. SO WE'RE IN A MORAGEILE PRINT BASED FASHION. THE ADVANTAGE OF THAT IS YOU'RE TESTING EVERY TWO WEEKS, YOU'RE FINDING USER ISSUES EVERY TWO WEEKS, YOU'RE ALSO MAKING SURE THAT YOU'RE NOT BUILDING FEATURES THAT PEOPLE DON'T--PEOPLE DON'T WANT AND YET YOU'RE DETECTING THOSE ISSUES EVERY TWO WEEKS AND MAYBE THREE WEEKS. SO IT'S A VERY INTERESTING PARADIGM. POTENTIALLY COULD MAKE US MORE SECURE BUT IT DOES CHANGE THE WAY WE DO WORK SO WE HAVE TO THEN REARE--ADMINISTRATIVE KHIHIGH TECT PROCESSES SO MAKE SURE WE HAVE THE TOOLS TO HAND THELLIAL CELLS METHODOLOGY SO THAT IS SOMETHING IMPORTANT TO POINT OUT AND PREVIOUSLY I MENTIONED WE HAD THE SOFTWARE DEVELOPMENT TOOLS, WE HAD LOTS OF THEM THAT ARE DEVELOPERS USE TO BUILD SOFTWARE, YET CONTINUALLY EVOLVING, WE'RE IN A PHASE WHERE WE'RE SWAPPING OUT OLDER ARCHITECTURES WITH NEWER OLDER PLATFORMS TO NEWER PLATFORMS SO WITH THAT WE HAVE TO ONCE AGAIN TRAIN OUR STAFF AND MAKE SURE WE HAVE NEW SOFTWARE TOOLS THAT WILL DETECT THESE IN THESE NEW PLATFORMS. NLM FOR MANY YEARS HAS BEEN THE LEADER IN THE FEDERAL GOVERNMENTPdÖ AND THE ADOPTION OF THE NEXT GENERATION INTERNET PROTOCOL WHICH IS IPV SIX, JUST LIKE ANY CONFUSION OF TECHNOLOGY, WE WILL FIND THAT TECHNOLOGY LAGS, OR SECURITY LAGS TECHNOLOGY BY A BIG WAY. SOMETIMES YOU WILL FIND TECHNOLOGY 10 YEARS BEFORE YOU THE SECURE AND MATURE SECURITY COUNTER MEASURE TO APPLY. SO, WE'RE ALWAYS LOOKING TO SEE HOW WE CAN GET AWAY FROM MANUEL PROCESSING OF SECURITY WITH BETTER AUTOMATIC WAYS OF DOING SECURITY IN THIS SPACE. >> [INDISCERNIBLE]. >> SO IPV SIX-- >> YEAH, SO IPV SIX IS THE NEXT GENERATION INTERNET PROTOCOL AND THAT IS, THE EARLY GENERATION OF THE INTERNET, THEY'RE ONLY 4.3 BILLION ADDRESSES. SO YOU COULD HAVE BILLIONS OF ADDRESSES BUT IT WAS CONCEIVED THAT THE INTERNET WAS GOING TO GROW TO THIS EXTENT. AND LOOK AT THE GROWTH OF MOBILE DEVICES AND ALL THE CONSUMER HOME BASED DEVICES AND THEY CALL IT THE INTERNET 4.2 BILLING WOULD BE ADEQUATE, EVERYBODY KNOWS THAT, SO WE ARE NOT IN THIS SPACE WHERE WE HAVE THE, WE HAVE THE NEXT GENERATION INTERNET PROTOCOL WHICH IS CALLED IPV SIX. WHICH HAS TWO WAYS TO 128 ADDRESSES, SO IT'S A PHENOMENAL ADDRESS. AND SO WE DON'T EXPECT TO RUN OUT OF THAT ANYTIME IN THE FUTURE, BUT TO TRANSITION ON THE BACK END FROM VERSION FOUR WHICH IS THE 4.3 BILLION SPAGS TO THIS NOW ONE IS A PROCESS. AND THE NLM, I'M VERY HAPPY TO SAY, HAS BEEN LEADING THE WAY. WE HAVE MADE THIS SHIFT TO WHERE WE CAN SUPPORT BOTH. OKAY? AND SO WE'RE ALSO HELPING OTHER FEDERAL AGENCIES, WE'RE HELPING NIH ACHIEVE THESE GOALS, BUT, FROM A SECURITY PERSPECTIVE, THIS PARTICULAR THING IS VERY CHALLENGING. CHALLENGING IN THE SENSE THAT YOU KNOW YOU--WHEN YOU HAVE MORE ADDRESSES AND YOU HAVE A BIGGER VOLUME OF ADDRESSES THAT YOU HAVE TO PROTECT AGAINST, YOU HAVE TO OUTSCAN EACH OF THESE THINGS SO HAVE YOU MORE ISSUES TO LOOK FOR ANOTHER SET OF ISSUES TO WORRY. >> I DO WANT TO THANK, I WANT TO THANK ALL OUR OPERATION STAFF BECAUSE WITHOUT THEM WE WOULD NOT HAVE BEEN ABLE TO DO THIS. ALL THE PROGRAM MANAGERS, LIKE I SAID, THANKS TO THEM FOR SUPPORTING THE SECURITY PROGRAMS HERE AND ALL THE I.T. AND ALL THE NLM EXECUTIVES HELPING FUND THE PROGRAMS THAT WE HAVE. BUT IT'S VERY SPECIAL THANKS TO WILL FULL-TIME SECURITY STAFF BECAUSE THEY HELP US DEVELOP THESE PRINCIPLES AND SECURITY PRINCIPLES UPON WHICH OUR SECURITY PROGRAM IS BASED AND I ALSO WANT TO THANK OUR USER STAFF BECAUSE THEY KNOW THERE ARE MANY TIMES THAT THEY ARE TEMMED BY BY PHISHING ATTACKS, ET CETERA, BUT THEY SOMEHOW SUPPORT OUR PROGRAMS AND POLICIES. SO THANK YOU VERY MUCH. [ APPLAUSE ] >> SO ONE OF THE THINGS THAT YOU TOUCH ON WHICH YOU FEEL VERY OFTEN IS FROM YOUR POINT OF VIEW THE SECURITY GUYS GET IN THE WAY, THEY STOP ME FROM DOING SOMETHING EFFICIENTLY BECAUSE THEY BUILT ALL THESE FIRE WALLS AND WHAT HAVE YOU. SO, THERE MUST BE--YOU DIDN'T TALK ABOUT IT BUT THERE MUST BE SOME TENSION BETWEEN THE DEMAND FROM EVEN INSIDE SAYING WE WANT MORE PRIVILEGES FOR OURSELVES, I MEAN YOU TALKED ABOUT RESTRICTING PRIVILEGES AND ALSO HOW DO YOU BALANCE THAT. >> SO I WOULD SAY THAT NLM ORGANIZATIONALLY HAS A STRONG CULTURE OF APPRECIATING SECURITY SO THAT'S A VERY IMPORTANT THING. ALLLET WAY TO THE VERY TOP. AND EVERY PROGRAM THAT WE HAVE HERE, REALLY VALUES SECURITY IN THE SENSE THAT THEY REALIZE IT HELPS THEIR BUSINESS, NOBODY WANTS THEIR SYSTEMS TO BE HACKED AND I SEE THAT IN THE DAILY OPERATIONS, HAVING SAID THAT WE'RE ALSO NOT VERY BUREAUCRATIC IN TERMS OF WHAT WE EXPECT OUT OF OUR ORGANIZATION, OUR ORGANIZATION, I WESH I COULD SAY, IS LIKE ANY OTHER ORGANIZATION I'VE BEEN IN BUT WE'RE ALWAYS ON THE LEADING EDGE OF A LOT OF THINGS, I REMEMBER IN 2001 WHEN WE WERE TRYING TO FIND A FIRE WALL PRODUCT TO LIVE UP TO THE--TO PROVIDE SECURITY FOR THE AMOUNT OF BAND WIDTH THROWING OUT THERE, NLM BEING A LARGE PROVIDER OF ONLINE CONTENT IN THE FEDERAL GOVERNMENT, IT WAS HARD TO FIND THAT. WE CHECKED WITH NASA, WE CHECK WIDE ENERGY, I CHECKED WITH EVERYBODY, WE CHECKED WITH THEM. THEY WERE NOT GOING TO BUILD A PRODUCT THAT THEY COULD ONLY SENTENCE TO NLM AND ONE OR TWO OTHER VENDORS AND OTHER CUSTOMERS, AND SO THE OTHERS HAVE THE CHALLENGES AND SO WE HAVE--WE HAVE ALWAYS HAD TO REINVENT WHAT SECURITY MEANS IN OUR SETTING. SO WHEN WE BOTH HAVE THE SIGNS, DMZ FOR EXAMPLE, TO ALLOW US TO HANDLE HIGHER BAND WIDTH OF DATA WE TOOK A DIFFERENT APPROACH. AND I THINK OUR BUSINESS PARTNERS APPRECIATE THAT. WE'RE NOT TRYING TO BE ROAD BLOCKS BUT WE'RE TRYING TO COOPERATIVELY SOLVE PROBLEMS WITH THEM SO WE HAVE DEMONSTRATED OVER TIME THAT WE'RE GOOD PARTNERS. >> I CAN GIVE YOU AN EXAMPLE ABOUT HOW THINGS ARE APPROACHED AT NLM, I WAS TALKING ABOUT IT THE OTHER DAY. ESSENTIALLY PEOPLE LOG OFF THEIR MACHINES A LOST US AND LEAVE THEM ON AT NIGHT AND THEY DO THE SECURITY PATCHING OVERNIGHT AS OPPOSE TO OTHER PLACES WHERE THEY ACTUALLY SAY ONCE A MONTH WE'RE GOING TO BRING ALL THE SYSTEMS DOWN AND WE'RE GOING IT DO SECURITY AND ON ONE HAND PEOPLE SAY, WELL IF YOU LEAVE YOUR MACHINES ON OVERNIGHT LOCKED UP THING BUT NLM THE DECISION IS THAT WE WOULD LOSE SO MUCH PRODUCTIVITY ACROSS THE WHOLE LIBRARY BECAUSE NOBODY CAN DO ANYTHING WITHOUT THEIR MACHINE, THAT WE'RE GOING TO DO THIS OTHER THING AND YOU KNOW, BASICALLY TURN THE PRINTERS OFF BUT WE'RE GOING TO SORT OF ORGANIZE IT SO WE CAN BE UP TO DATE AND AS I'VE HAD POINTED OUT TO ME IN THE OTHER PLACES IF YOU TURN EVERYBODY'S MACHINE OFF, YOU DON'T DO IT AS FREQUENTLY. BUT IF YOU GET SERIOUS PATCHES OR WHATEVER, I MEAN THEY COULD BE UPDATING OUR MACHINES, YOU KNOW THREE OR FOUR TIMES A WEEK AND WE WOULDN'T KNOW IT OR ONCE A WEEK, WHATEVER IT WOULDN'T MATTER, SO, IT'S--I THINK IN GENERAL A LOT OF THOUGHT PUT IN AND THERE'S A FAIR AMOUNT OF INTERACTION WITH STAFF ABOUT HOW CAN WE DO THIS SO THAT WE GET IT DONE AND DON'T-- >> YEAH, NOT ONLY ARE WE RESPONSIBLE FOR SECURITY BUT WE'RE RESPONSIBLE FOR SYSTEMS TOO, SO I'M POSING WHATEVER RULES ON MY OWN SELF, SO I HAVE TO BE REALLY CONSCIENCE OF WHAT IT IS BECAUSE RESPONSIBLE FOR MAKING SURE WORK HAPPENS AND WE REMAIN AGILE. SO SECURITY IS NOT THE ONLY THING I WORRY ABOUT BUT IT IS IMPORTANT AND I WORRY ABOUT IT. NTHIS IS A FABULOUS PROCESS WHAT YOU ALL ARE DOING, HOW DO YOU WORK WITH YOUR THIRD PARTY PARTNERS WHO WANT TO BE ABLE TO HAVE ACCESS TO YOUR SYSTEM BECAUSE OF WORK THAT'S BEING DONE WITHIN THE LIKEERARY? SO YOU HAVE VENDORS THAT WE--THAT YOU WOULD IMPORT RECORDS FROM OR YOU WOULD EXPORT RECORDS TOO BECAUSE YOU MENTION THERE'S LIKE A TWO-WAY [INDISCERNIBLE] SO WHAT TYPES OF BUSINESS PRACTICES OR OTHER SECURITY CRITERIA WOULD YOU-- >> SO FOR US, LIKE I SAID INPUT VALIDATION IS A VERY IMPORTANT THING, ANYTHING WE BRING IN WE HAVE TO BE CONCERNED ABOUT. SO YOU KNOW WE WANT TO WORK WITH THE PARTNERS TO MAKE SURE WHAT WE BRING IN IS SOMETHING THAT WE FEEL IS GOOD. IT'S NOT MALITIOUS. DOESN'T HAVE MALITIOUS CONTENT. AND SO TO THAT EXTENT WE CAN HAVE STRUCTURE, THINGS WE BRING IN, IF THERE ARE FILES THAT COME IN, THEY CAN BE SCANNED FOR MALITIOUS CONTENT, IF THERE ARE THINGS THAT WE BRING IN THAT WE CAN PASS ON THE BACK END, WE PASS THEM TO MAKE SURE THERE'S NO EXECUTABLE CODE IN THEM. SO THERE'S A LOT OF DIFFERENT THINGS THAT HAPPEN AND WE DEAL WITH THEM ON A CASE BY CASE BASIS BECAUSE WE HAVE DIFFERENT DEALINGS WITH DIFFERENT ORGANIZATIONS AND SOMETHING'S COMING THROUGH FTP AND SOMETHING'S COMING THROUGH OTHER MECHANISMS, BUT VALIDATION WILL ALWAYS BE VERY IMPORTANT. >> SO NOW FOR SURFACE EFFECTS YOU DO YOU FIND THOSE--[INDISCERNIBLE]. >> YES, SO, I HAD SET UP A DENIAL OF SERVICE PROTECTION SERVICE TO ACCOMPANY NAMES PROBABLY IN--I'M TRYING--MAYBE IT WAS IN 2007, WE DISCONTINUED THAT SERVICE SOME TWO YEARS AGO BECAUSE WE FOUND IT WASN'T GIVING US A WHOLE LOT OF VALUE, MOSTLY, THE ATTACKS IF WE'RE SEEING ANY ATTACKS THAT WARRANTED THAT SERVICE. IT DOESN'T MEAN THAT IN THE FUTURE IF YOU FIND SOMETHING MORE ROBUST THAT WE WOULDN'T GO AFTER IT, SO IN THE ROLE OF DENIAL OF SERVICE, WHAT WE CAN SEE ARE ZOMBIE ATTACKS THAT COME IN MASS, COME AT THE LIBRARY AND THEN THE TYPE OF SERVICE WE HAD PICKED, WE HAD THE ABILITY TO ROUTE THIS VOLUME NOWS ATTACK THROUGH THIS THIRD PARTY, SO WE THOUGHT THERE WAS A BIG BARRAGE OF ATTACKS, THAT WAY, WE WOULD ROB THAT TRAFFIC TO THE PARTY, THIRD PARTY AND THAT THIRD PARTY WOULD CLEAN UP THE TRAFFIC AND SEND US ONLY THE GOOD TRAFFIC. SO WHAT WE HAD DONE IS WE GOT THE SERVICE AND WE STARTED TESTING THE VENDOR OURSELVES SO WE WOULD FAKE ATTACKS OURSELVES, THAT'S PART OF THE THING BECAUSE IF YOU DON'T FAKE THE ATTACKS YOU WILL NEVER KNOW IF IT WORKS WELL OR NOT SO WE WOULD--WE GOT A SMALL SUBSCRIPTION THROUGH A COMPANY I THINK CALLED RED WOLF IN CANADA OR SOMETHING AND WE STARTED CREATING OUR OWN ATTACKS AND MANY TIMES WE WOULD ACTUALLY TELL THEM THAT WE'RE GOING TO DO AN ATTACK WITHIN THIS PERIOD. AND WE NEVER FOUND THAT THEY RISPONDY LIGHTIS EVEN THOUGH THEY'RE ONE OF THE LEADING PROVIDERS AND DISTRIBUTORS OF THE SERVICE, IT WASN'T VERY GOOD. SO HERE WE'RE TALKING ABOUT LONE ATTACKS. THE FACT THAT WE KNOW HOW TO CRACK, BECAUSE THERE'S KNOWN ATTACKS THAT THIS PROVIDER GIVES US, THEY GIVE US A TOOL SO WE CAN CRACK, TALK ABOUT THE PERFORMANCE ABOUT KNOWN ATTACKS, HOW ARE THEY GOING TO DO AGAINST NEW ATTACKS. SO THAT WAS A CHALLENGE WE FACED. I THINK DENIAL OF SERVICE IS A HUGE PROBLEM FOR ANYBODY. I MEAN IF PEOPLE WANT TO BRING THE WHITE HOUSE DOWN, THEY CAN DO AND IT AND THEY DO IT ALL THE TIME. IT IS AN AREA WE'RE CONCERNED ABOUT. WE DIDN'T FEEL IT WAS RIGHT TO--WE TRIED IT AND WE DIDN'T FEEL IT WAS WORTH THE MONEY SPENT TO KEEP IT, IT WASN'T REALLY SERVING THE PURPOSE. >> [INDISCERNIBLE] >> WE SHUT PEOPLE OFF. >> WELL, YEAH ONE OF THE THINGS WE DO,--THE IDEA IS THAT ONE OF THE THINGS TO DO IS TO [INDISCERNIBLE] YOURSELF, SO WHAT THEY WOULD DO IS TRY TO BRING ONE SIDE DOWN AND THE OTHER SIDE STILL STAYS UP SO WE CAN TRY TO MASK SOME OF THAT, AND THAT'S THE GENERALLY OUR STRATEGY, NOW IF WE WERE MUCH LARGER OF AN ORGANIZATION AND WE HAD A BIGGER FOOTPRINT IN DIFFERENT PARTS OF COUNTRY OR THE WORLD WE COULD CONCEIVABLY HAVE MORE OF THESE SITES. IMU WE HAVE TO--IT'S ALL A RISK BASICKED PROPOSITION, WHAT ARE WE TRYING TO PROTECT AT WHAT COST. >> WE ACTUALLY STARTED BACK IN 92, OUR GUYS FROM D. O. E. AND HE WAS THOUGHT TO BE SLIGHTLY CRAZY BUT IT WORKED VERY WELL. WE MADE THEM PROMISE THAT THEY COULD DO ANYTHING THEY WANTED, ELECTRIC FIRST OF COURSE, BUT THEY COULDN'T COME ON SITE AND PASS OUT HUNDRED DOLLARS BILLS. [LAUGHTER] , THEY HAD TO DO IT REMOTELY, USE INTERNET TO GET IN AND THEY GAVE A LECTURE FIRST. AND THE GOOD EXPERIMENT PEOPLE SAID HA, HA, HA, HA, AND I DIDN'T HAVE TO LEARN ANYTHING FROM MY GUY, THE NEXT MORNING HIS SYSTEM WAS FLATTENED. SO I MEAN YOU--YOU HAVE TO MAKE THE PEOPLE AWARE THAT IT ACTUALLY CAN BE DONE AND IT CAN BE DONE TO ANYBODY. SO WE DID IT TWICE AND I THINK THAT'S RUBBED EVERYBODY'S NOSE IN IT AND GOT USED TO THE GENERAL IDEA THAT IT'S RELATIVELY STRAIGHT FORWARD. >> DENNIS MAY WANT TO COMMENT. I MEAN WE DO HAVE THIS APPROACH OF WHEN--SOMETIMES IT'S JUST THE GRADUATE STUDENTS THAT DIDN'T UNDERSTAND WHAT THEY WERE DOING AND SUDDEN DEATH EVENLY THE TRAFFIC IS COMING IN AT A RATE AND WE JUST SHUT THESE PEOPLE OFF AND SOMETIMES CALL UP AND SAY WHAT'S GOING ON OVER THERE. I DON'T KNOW WHETHER WE HAVE MORE OF THAT THAN WE HAVE OF PEOPLE WHO ARE REALLY TRYING TO DO SOMETHING BAD ON PURPOSE, BUT, GOING INTO THE SYSTEM WE JUST SHUT THEM OFF AND THEN THEY CALL UP AND SAY, HEY, WHAT'S GOING ON? NOBODY IN MY UNIVERSITY CAN GET INTO--AND THEN WE SAY, HEY, YOU BETTER FIND WHOEVER IT IS THAT WAS DOING THAT STUFF NOW. >> MOSTLY SCRIPTS THAT ARE GOING NUTS, INADVERTENTLY. REMEMBER ONE DAY WE SHUT DOWN CAROLYN? THEY HAD ONE OF THESE GOING ON. IT'S GROW NOTHING IMPORTANCE AND GROWING AND ABSORBING MORE AND MORE AND THEY'RE CONCERNED ABOUT SECURITY. >> MORE OF THESE TYPES OF SCRIPT THINGS BECAUSE OF THE TEXT MINING TYPES OF STUDIES THAT ARE BEING DONE AGAIN LARGE CORPORATES OF LIKE CITATIONS OR SOME OF THE IMAGE DATABASES, ET CETERA? >> I WOULD HAVE TO DEFER TO--BECAUSE MOST OF THESE BLOCKS THAT WE DO ARE GENERALLY DONE AT THE HOST LEVEL, WE DON'T DO THEM CENTRALLY BECAUSE WE DON'T WANT TO START UNTRACKING JOURNAL BUT I KNOW NCBI DOES A LOT OF HOST BASE AND IT COULD BE FOR MULTIPLE REASONS AND WE GENERALLY DON'T KNOW THE MODE OF PEOPLE AND THEY BLOCK THAT, I DONE FINISH WE CAN EVER, YOU KNOW FIGURE OUT MOTIVES OF INDIVIDUALS. BUT I WOULD SAY, MANY TIME FIST I GO BACK TO ONE OF THOSE SLIDES, I WANT TO JUST SHOW YOU SO YOU GET A SENSE OF ONE OF THE THINGS WE SEE AS--I DON'T KNOW IF I HAVE THAT EXACT EXAMPLE, LET ME SEE IT. IT'S IT IS NEXT ONE. SO WHEN YOU LOOK AT SOME OF THESE KINDS OF THINGS, SO, LOOK AT THE ITEM THAT SAYS PREDICTABLE RESOURCE LOCATION. THAT'S ONE KIND OF ATTACK THAT IS VERY INTERESTING AND THEN THERE'S THIS THING CALLED [INDISCERNIBLE]. NOW PREDICTABLE RESOURCE LOCATION SAYS, I THINK THESE THINGS EXIST ON YOUR SIDE SO I'M GOING TO TRY THESE DIFFERENT THINGS THAT BASED ON THE PATTERN OF WHAT WE THINK YOU VMAYBE WE WILL GO AFTER THESE AND THESE ARE GENERALLY REMOTELY SCRIPTED. [INDISCERNIBLE] DRIVING IS MOSTLY SOMEBODY DOING IT ONLINE REALTIME USING BROWSERS SO LET'S SAY I CLICK ON NLM AND IT TAKE MEAS TO A URL SAYS:/IVR, AND IT HAS A DATE STAMP LIKE 21502, 11, AND THAT'S TODAY AND THEY SAY THERE'S SOMETHING FOR IVR, BUT IT'S NOT IMPORTANT BUT I WANT TO SEE WHAT DON'S GOT, I WILL REPLACE;nWITH DON, MAYBE DIFFERENT PERSON MORE INFORMATION SO PEOPLE TRY THESE TYPES OF PATTERNS, I DON'T KNOW HOW MUCH OF THAT CAN BE DONE REMOTELY, I KNEW THROUGH SCRIPTING AND BECAUSE IT REQUIRES SOME LEVEL OF KNOWLEDGE TO DO A SCRIPTED ATTACK BUT THAT'S WHERE SOME OF THAT REDICKIBLE RESOURCE LOCATION AND COME INTO PLAY AND THERE'S NOT MUCH WE CAN DO. SO WE TRY NOT TO WORRY ABOUT MOTIVES BECAUSE IF YOU START GETTING INTO THIS, AND GET THE ANALYSIS AND MOTIVES WE BLOCK WHAT WE SEE AS PATTERNS. >> THANK YOU AND IT'S FASCINATING AS ALWAYS. [ APPLAUSE ] >> OUR NEXT PRESENTATION IS HILDA BASTIAN AND SHE WILL UPDATE US ON PUD MED COMMONS. >> I WILL SWITCH GEARS A BIT NOW, WE'VE BEEN TALKING ABOUT PEOPLE AND THEIR THOUGHTS AND WHAT HAPPENS AFTER SOMEBODY HAS PUBLISHED AN ARTICLE THAT'S LIKE A LOT OF IDEAS AND DATA AND IMPLICATIONS FOR OTHER PEOPLE'S RESEARCH AND FOR THEIR PRACTICE IN DIFFERENT AREAS IN THE KINDS OF PUBLICATIONS THAT ARE COMING OUT IN THE BIOMEDICAL LITERATURE. AND PEOPLE START TALKING ABOUT THEM IN THE LIFE OF THAT RESEARCH PROJECT ENDS UP OUT IN THE PUBLIC FOR PEOPLE TO ENGAGE WITH, AFTER PUBLICATIONS, MUCH MORE SO THAN EVEN BEFORE, AND THAT PUBLIC THAT, DISCUSSION IS VERY VIBRANT AND DYNAMIC AND IMPORTANT TO PEOPLE AND IT'S REALLY JUST SCATTERED, ALL OVER THE PLACE, AND THIS--THIS TIME, THOUGH, WE'RE GOING TO BE TALKING ABOUT QUITE SPECIFIC PLACE WHICH IS PUBMED AND THE IDEA OF SAYING, WELL, LET'S TRY AND SEE IF WE CAN OPEN UP THAT PARTICULAR SPACE FOR PEOPLE TO START TO DISCUSS PUBLICATIONS, AND WHAT MIGHT THAT BE, MIGHT THAT LOOK LIKE AND HOW MIGHT THAT WORK. IT WAS DEVELOPED IN RESPONSE TO SCIENTIFIC COMMUNITY INTEREST WITH THE DESIRE OF PEOPLE TO BE ABLE TO BRING DISCUSSIONS TOGETHER HERE FOR DEVELOPING THE COMMONS AND HOW THAT MIGHT WORK. THERE WERE SEVERAL KINDS OF CORE ISSUES THAT PEOPLE REALLY WANTED TO ADDRESS AND CAME UP WITH DIFFERENT APPROACHES TO, AND THESE WERE SOME OF THE DECISIONS THAT THE PEOPLE WHO COULD COMMENT ON PUBMED COMMONS WOULD NEED TO BE OFFICE OF PUBLICATIONS IN PUBMED ONLY AND THAT THEY COULD NOT DO IT ANONYMOUSLY, THAT THEY COULD NOT BE ANY PSEUDONYMS IN THE SYSTEM AND WOWHO WAS DOING THIS AND LOOK FOR CONFLICTS OF INTEREST AND PEOPLE MAY BE MORE THOUGHTFUL AND PUT MORE EFFORT INTO WHAT THEY MIGHT SAY IF THEY WERE--IF PEOPLE WERE GOING TO KNOW WHO IT WAS THAT WAS SAYING THIS. THAT'S POTENTIALLY THE CONTROVERSIAL PATH OF THE WHOLE ISSUE ABOUT DIFFERENT COMMENTING SYSTEMS AND THIS IS DECISION FOR PUBMED COMMONS. IT WAS DECIDED THAT PEOPLE WOULD BE ABLE TO RATE COMMENTS SO THAT MEMBERS COULD NOT JUST MAKE COMMENTS, THEY CAN RATE THE HELPFULNESS OR NOT OF COMMENTS OF OTHER PEOPLE HAVE MADE BUT NOT START DOING A THUMBS UP, THUMBS DOWN TO PUBLICATIONS THEMSELVES. AND NONE OF THE OTHER CORE ISSUES. IT WAS ALSO DECIDED THAT THEY WOULD ONLY BE POST COMMENT MODERATION. SO IF YOU MAKE A COMMENT IN YOUR PUBLIC COMMONS ACCOUNT, IT GOES STRAIGHT ONLINE, IT'S A KIND OF SECURITY CHECKS THAT HAPPEN FIRST, LOOKING FOR SECURITY SO BASICALLY YOU'RE POSTING DIRECTLY ONLINE BUT THEN WE MODERATE IT AFTERWARDS SO WE HAVE THAT ABILITY TO RESPOND TO PEOPLE'S CONCERNS OR COMPLAINTS OR REPORTS BUT ALSO LOOK AT THOSE TO SEE HOW THIS IS GOING. I WILL COME BACK TO THAT IN A MOMENT BECAUSE THAT SECTION IS SORT OF THE SUSTAINABILITY OF A SYSTEM LIKE THIS. HOW CAN YOU ACTUALLY DO THAT IN A WAY THAT IS THE VOLUME GREW THAT THIS WOULD BE MANAGEABLE. AND ONE OF THE OTHER THINGS, PATTERNS IS WHEN PEOPLE MAKE A COMMENT, THEY'RE GIVING US CREATIVE COMMONS LICENSE TO THAT COMMENT AND THEY CAN'T TAKE IT BACK. ONE OF THE OTHER PROCESSES TO THIS WORK THAT WE NEEDED TO SEE, BOTH WAS IT GOING TO BE WORTH WHILE, WAS IT GOING TO BE PROBLEMATIC, WAS IT GOING TO BE SUSTAINABLE, WERE WE GOING TO BE ABLE TO FIND SYSTEMS THAT DIDN'T MAKE THE EXPERIENCE OF USING PUBMED FOR THE MILLIONS OF PEOPLE WHO WERE USING IT EVERY DAY, MISERABLE. THERE'S ACTUALLY ADD SOME KIND OF VALUE TO IT, AND WAS IT POSSIBLE. SO NOW WHEN THE LAST STAGE OF THIS AND WRAPPING UP THE FINAL EVALUATION. WE HAVE A WORKING GROUP THAT HELPS US WITH THIS PROJECT. CHAIRED BY CHRISTOPHER LEE FROM THE NCBI BOARD OF SCIENTIFIC COUNSELORS AND A RANGE OF PEOPLE AND A RANGE OF PEOPLE WITH DIFFERENT EXPERTISE AND PERSPECTIVES HELPING US MOVE THIS PROJECT FORWARD AND THERE HAVE BEEN A MUCH LARGER GROUP IN THE EARLY STAGES LED BY ROB FROM STANFORD WHO HELPED GID--HELPED GID THOSE KINDS OF EARLY DECISIONS AND PROCESSES AND SUFFERS THROUGH THE WHOLE BETA TESTING OF TRYING DIFFERENT OPTIONS. SO THE LARGER PILOT, WHERE WE ACTUALLY STARTED TO KIND OF OPEN THE DOORS TO MORE PEOPLE, BEGAN LATE IN OCTOBER, 2013 BUT IT WAS REALLY NOT VISIBLE TO THE PUBLIC UNTIL CHRISTMAS OF THAT YEAR AND YOU CAN SEE WHAT'S--WHAT'S HAPPENED ACROSS THAT PERIOD OF TIME IN THE CHART AT THE BOTTOM. THAT THE USE WENT UP, WE FOUND THAT WE HAVE ACTUALLY BEEN ABLE TO COME UP WITH TOUCH WOOD, SUSTAINABLE INFRASTRUCTURE THAT MINIMIZES THE MODEL CITIZEN TENTIALLY HARMFUL USE OF THE FORUM. WE FOUND THAT THE QUALITY OF THE COMMENTS IS ACTUALLY BEEN REALLY QUITE HIGH. BUT USE OF THE COMMON SYSTEM QUITE LOW. AND ONE OF THE SURPRISING THINGS ABOUT THAT TO ME WAS THAT THAT WAS ALSO TRUE FOR RATINGS. NOT JUST FOR COMMENTS. WE CURRENTLY GO UP--IT WOULD BE EIGHT AND HALF THOUSAND MEMBERS BY NOW AND 2700 COMMENTS LIVE ON PUBMED. THIS IS WHAT IT LOOKS LIKE IF YOU GO TO PUBMED COMMONS, CAN YOU COME THERE FROM THE PUBMED HOME PAGE AND IT INCLUDES SELECTED COMMENTS THAT ARE GETTING PROFILED UP HIGHER AND THAT'S A--THAT'S A SYSTEM THAT I'LL EXPLAIN ABOUT IN A MINUTE A LITTLE AND YOU CAN ALSO SEE WHAT'S ACTUALLY TRENDING IN TERMS OF ARTICLES ON PUBMED ON THE SIDE. OH AND WHAT WE'VE BEEN FIND SUGGEST THAT THE VAST MAJORITY OF COMMENTING IS ABOUT SOMETHING VERY RECENT ALTHOUGH PEOPLE ARE GOING BACK INTO THE LITERATURE, BUT IT REALLY DOES TEND TO BE ABOUT WHATEVER'S CURRENT. THE SCIENTIFIC CONTRIBUTIONS OF THE COMMENTS HAVE BEEN REALLY ACROSS A BROAD RANGE AND IT'S ACTUALLY BEEN--SOME OF IT IS WHAT WE EXPECTED TO SEE, BUT PEOPLE KEEP COMING UP WITH NEW THINGS. WE THOUGHT THAT WAS A GREAT IDEA, ONCE PEOPLE HAVE GOT THAT KIND OF LIKE A NEW PLACE TO INTERACT AND TO DO THINGS WITH, THEY COME UP UP WITH ALL KINDS OF IDEAS OF WHAT CAN BE DONE, SO FOR EXAMPLE, WE HAVE--WE HAVE SOMEBODY WHOSE INVOLVED, [INDISCERNIBLE] IS INVOLVE WIDE MISIDENTIFIED CELL LINES AND COLLECTING A DATABASE OF THAT, COME NOTHING AND SPENDING TIME, LOOKING FOR ARTICLES WHERE THEY HAVE AN ACTUAL MISIDENTIFIED CELL LINE IN THE RESEARCH AND TAG TAG LITERATURE. THOSE KINDS OF PROCESSES THAT YOU WOULDN'T HAVE NECESSARILY PREDICTED WERE GOING TO HAPPEN. THAT ARE. WE'VE GOT PEOPLE DISCUSSING FUTURE RESEARCH, SO WHAT IS ACTUALLY THE NEXT STEP THAT'S IMPLIED BY WHAT DIFFERENT PEOPLE HAVE DONE. WE HAVE PEOPLE DISCUSSING THE INTERPRETATIONS OF RESULTS INCLUDING IF SOMEBODY HAS REPLICATED OR ATTEMPTED TO REPLICATE A PARTICULAR STUDY AND SOME OF US GO BACK AND GO HERE'S A LINK TO OUR RESULTS FROM TRYING TO REPRODUCE YOUR RESEARCH, WHETHER THEY'RE ACTUALLY ABLE TO CONFIRM THE RESULTS OR ACTUALLY SAYING, WELL, ACTUALLY, AT THE QUESTIONS, SOME OF THAT GOING ON, THERE'S A FAIR AMOUNT OF DISCUSSION ABOUT THE GENERALIZABILITY AND APPLICATION OF RESEARCH RESULTS SO IT'S A REALLY CRITICAL AREA, NOT JUST FOR PEOPLE WHO ARE INVOLVED IN DOING RESEARCH BUT FOR PEOPLE WHO WERE--WHO NEED TO MAKE DECISIONS IN PRACTICE OR POLICY, TO ACTUALLY START TO ENGAGE WITH GETTING MORE INFORMATION FROM AUTHORS AND SO ON THAT THEY NEED. AND THEN THERE'S THE WHOLE AREA OF AUTHORS AND COMMUNITY CURATING AROUND RESEARCH, THAT CAN BE FROM ALL SORTS OF THINGS AND SAYING HERE'S PARTICULAR--HERE'S WHY WHERE I HAD DATA OR ALL SORTS OF ISSUES LIKE THAT, THROUGH TO PEOPLING ALSO GOING BACK AND SAYING YES THAT WAS WORK THAT WE DID IN THE PAST THAT'S STILL BEING CITED A LOT. THERE'S WORK WE'VE DONE LATER THAT WOULD CHANGE, YOU KNOW OUR OPINION OF WHAT WE WOULD SAID AND REFER TO YOU A LATER ARTICLE WE'VE WRITTEN FOR YOU TO LOOK AT. SO HERE'S A COUPLE EXAMPLES UP HERE TO MAKE THAT A BIT MORE CONCRETE ABOUT WHAT THAT ACTUALLY MEANS. THIS WAS A DISCUSSION BETWEEN PEOPLE IN TWO LABS, WHERE THEY WERE TALKING ABOUT THE NEXT STAGE, YOU KNOW WHAT THEY MIGHT TRY NEXT. AND SOMEONE COMING BACK, SAYING WELL, WE'VE ALREADY HURD THIS IS NOT ACTUALLY A FRUITFUL WAY TO GO AND THAT'S ACTUALLY A REALLY QUITE IMPORTANT IDEA WHEN YOU THINK ABOUT IT, THAT THAT WHOLE PROCESS OF SHARING INFORMATION AND PEOPLE KNOWING WHAT MAY HAVE BEEN WORTH WHILE AND SO ON CAN ACTUALLY HAPPEN IF YOU'RE NOT IN A VERY PARTICULAR CLOSE KNIT GROUP OR PARTICIPATING AT PARTICULAR CONFERENCES OR WHATEVER CAN TAKE YOU QUITE A LONG TIME TO FIND OUT IN THE NORMAL COURSE OF EVENTS WHEN SOMEBODY SORT OF PUBLISHES ANOTHER ARTICLE TO YOU LATER, THIS IDEA THAT PERHAPS PEOPLE COULD START SHARE THANKSGIVING KIND OF KNOWLEDGE MORE IN REALTIME. HAS REALLY QUITE SIGNIFICANT IMPLICATIONS FOR IMPROVING THE KIND OF--THE SPEED AT WHICH PEOPLE CAN KIND OF BECOME MORE KNOWLEDGEABLE AND REDUCE A LOT OF RESEARCH WASTE. >> YOU SAY LINKING TO RESULTS OR SOME MORE BULK WAY TO DO THAT? >> WELL, I MEAN THIS IS VERY RUDE OF RAISING PERSONAL COMMUNICATIONS BUT SOMETIMES PEOPLE WILL ALSO LINK TO HERE'S A DATA SET FROM SOMEWHERE ELSE THAT WOULD SHOW AND UNDERPIN WHAT THEY'RE SAYING AND THIS PARTICULAR EXAMPLE IS A CONVERSATION. BUT-- >> [INDISCERNIBLE] SOMEWHERE ELSE, FOR AN EXAMPLE LIKE THAT. >> WELL, PEOPLE WILL SOMETIMES LINK TO DRIAD OR FIXED SHARE FOR PUBLICATIONS, OTHER PUBLICATIONS IN PUBMED, SOMETIME WHEN IS YOU SEE IN THE BIOLOGY AREA AND SEVERAL OTHER AREAS THAT LINK TO CRITICAL PUBLICATIONS THAT ARE IN A JOURNAL THAT'S--COMPLETELY OTHER DISCIPLINE THAT'S NOT WITHIN THE BIOMEDICAL LITERATURE. SO FOR EXAMPLE TMAY BE ABOUT PLANTS OR SOMETHING THAT'S IN A COMPLETELY DIFFERENT KIND OF AREA. OR THEY'RE LINKED TO-- >> IT CAN BE PUBLISHED, IT CAN BE PUBLISHED AND UNPUBLISHED. IT COULD BE WITHIN PUBMED, IT COULD BE OUTSIDE PUBMED. AND THIS NEXT ONE IS AN EXAMPLE OF PEOPLE LINKING TO SOMETHING ON FIGSHARE. THIS IS SOMETHING WE'VE SEEN A LOT OF THIS IS A GOOD EXAMPLE THIS, IS AROUND COMPUTATIONAL WORK, ONE OF THE ADDITIONAL THINGS YOU WILL SEE IS FOR EXAMPLE IS THE SOFTWARE OR DATABASE, DISCUSSED INSIDE A PARTICULAR PIECE OF RESEARCH AND IT'S REALLY QUITE PIVOTAL AND FOUR YEARS DOWN THE TRACK, THAT SOFTWARE HAS BEEN MOVE FRIDAY A PARTICULAR PLACE OR DATA HAS BEEN MOVE PRD A PARTICULAR PLACE AND MOVED SOMEWHERE ELSE, WHAT YOU HAVE INSIDE THE ARTICLE IS EFFECTIVELY A BROKEN LINK. WHICH OF COURSE IS REALLY NOT GOING TO BE ALL THAT HELPFUL SO PEOPLE WILL COME BACK ON A DATABASE HAS BEEN DEPRIVATIONRICATED OR WHATEVER AND SAY OKAY, THAT DATABASE IS NO LONGER THERE, THIS IS WHERE THAT DATABASE IS TO BE FOUND. IN TERMS OF THE INFRASTRUCTURE, THOUGH, AND FINDING A WAY TO MAKE THIS WORK, WE'VE GOT THE TWO SIDES OF IT. ONE IS HOW DO PEOPLE ACTUALLY START USING IT IN A POSITIVE WAY, AND THE OTHER IS HOW DO WE ACTUALLY MINIMIZE THE KIND OF HARM AND DAMAGE THAT CAN HAPPEN SO YOU WANT ON THE ONE HAND PEOPLE TO BE ABLE TO DEFEND THEMSELVES WHEN CERTAIN THING VS BEEN MISINTERPRETED ABOUT THEIR WORK OR WHATEVER, BUT YOU ALSO ACTUALLY DON'T REALLY WANT TO HAVE--SET UP A SYSTEM WHEREBY PEOPLE CONSTANTLY HAVING TO DEFEND THEMSELVES AGAINST ALL SORTS OF CLAIMS AND SO ON. SO, HOW YOU WOULD MANAGE POOR COMMENTS IS PIVOTAL TO A SYSTEM LIKE THIS BEING WORTH WHILE AND SUSTAINABLE, SO WHAT WE FOUND S&P THAT MOST OF THE POOR AND PROBLEMATIC COMMENTS ARE ACTUALLY MADE BY A VERY SMALL NUMBER OF PEOPLE. SMALL NUMBER OF PEOPLE THOUGH TO BE RESPONSIBLE FOR A LOT OF COMMENTS AND THAT'S--THAT'S WHAT WE'VE BEEN SEEING. WE HAVE A SYSTEM WHEREBY WE HAVE CLEAR GUIDELINES THAT HAVE BEEN REFINED ACROSS THE PILOT AND IF THERE ARE EXTENSIVE BREECHES OF THOSE, WE WILL SUSPEND MEMBERSHIP AND THAT'S BEING CRITICAL TO HOW THE SYSTEM WORKS, HOW DO WE DO IT? WE IDENTIFY IT BY COMBINATION OF HUMANS LOOKING AT WHAT'S HAPPENING AND RESPONDING TO PEOPLE'S REPORTS OF CONCERNS BUT ALSO COMPUTATIONALLY SO THAT PROCESS OF HOW THINGS ARE RATED AND PEOPLE'S TRACK RECORDS AND SO ON, VARIETY OF THINGS GOING INTO HELPING US TO ACTUALLY BE ABLE TO DETECT WHAT ARE THE KINDS OF THINGS THAT ARE LIKELY TOW RAISE CONCERNS AND THIS IS GOING TO BE AN ONGOING PROCESS AT THE TIME. ESPECIALLY IF THE NUMBER OF COMMENTS GOES UP OF HOW CAN WE ACTUALLY KIND OF PREDICT AND REACT, PERHAPS HOLD CERTAIN THINGS BACK AND LOOK AT THEM FIRST BEFORE THEY GO ONLINE. WHILE STILL ALLOWING MOST OF THEM TO GO THROUGH. HOW WE DO THIS INVOLVES LOOKING AT A PERSON'S TRACK RECORD AS WELL AS HOW PEOPLE ARE RESPONDING TO AN INDIVIDUAL COMMENT SO HAVE YOU TWO KINDS OF SOURCES INFORMATION THERE, A PARTICULAR INDIVIDUAL COMMENT IS A PATTERN THAT CAN EMERGE. THIS IS A SYSTEM THAT WILL REALLY RELY ON MORE PEOPLE STARTING TO WRITE COMMENTS. SO WRITING FROM OUR POINT OF VIEW IS CRITICAL OF THE COMMENTING IS OF THE SYSTEM IS TO SCALE UP. SO WHAT ARE OUR KEYc8p STRATEGIES? ONE HAS BEEN WORKING TO IMPROVE THE PROFILE OF GOOD COMMENTS. AND I SHOWED YOU THE SNAPSHOT OF THE COMMENTS HOME PAGE LOOKS LIKE AND SO WE'RE TRYING TO BASEOT DIFFERENT KIND OF SCORES THAT ARE FEEDING IN THE INFORMATION AND THE INFORMATION THAT'S FEEDING IN AND GET THE--THE BETTER LINE, THE ONES THAT RATE BET TORE RISE TO THE TOP AND THEN WE TRY TO REDUCE THE VISIBILITY OF THE ONES THAT ARE PROBLEMATIC. WE KNOW THINK IT'S QUITE IMPORTANT, IT'S ONE OF THE ISSUES WE'VE GOT HERE IS IN TERMS OF TRYING TO GET PEOPLE TO COMMENT MORE IN A POSITIVE WAY IS THAT WE HAVE A LOT OF CULTURE CHANGES TO DEAL WITH. PEOPLE ARE VERY BUSY. THEY'RE USED TO DOING PEER REVIEWS EVEN LIKE PRE-PUBLICATION PEER REVIEWS PERHAP WHEN IS THEY DON'T NECESSARILY GET PARTICULAR CREDIT STORED OR WHATEVER AND THAT'S EMBEDDED IN THE CULTURE TO DO THAT. TO ACTUALLY NOT JUST ENGAGE DIRECTLY WITH AN AUTHOR BY E-MAIL TO GET YOUR QUESTIONS ANSWERED WHEN IN FACT LOTS OF OTHER PEOPLE ARE READING WOULD ACTUALLY ALSO HAVE THOSE EXACT SAME QUESTIONS, THAT'S ACTUALLY A CULTURE CHANGE AND TO PUT THE EFFORT INTO POST PUBLICATION REVIEW AND CAPTURING IT AND CAPTURING PEOPLE'S THOUGHTS. THAT'S SOMETHING THAT WE REALLY NEED TO ENCOURAGE. ONE OF THE THINGS WE FEEL IS QUITE IMPORTANT FOR THIS IS, IS THE PROCESS OF ENCOURAGING OFFICE TO RESPOND BECAUSE YES IT'S REALLY VALUABLE IF PEOPLE ADD INFORMATION BUT A LOT OF THE TIME WHAT PEOPLE HAVE IS QUESTIONS OF THE AUTHORS THAT PERHAPS WHAT THEY DID IN THE PARTICULAR EXPERIMENT OR EXACTLY WHAT THE INTERVENTION WAS IF THIS IS MEDICAL CARE, IF SOMEBODY WANTS TO DO IT IF THEY'RE CONVINCED THEY SHOULD DO IT, INFORMATION ABOUT EXACTLY WHAT IS IT PRECISELY THAT THEY SHOULD DO IS FREQUENTLY NOT THERE. BUT THE VALUE THEN OF THIS PROCESS WILL DEPEND ON ALL OF THIS RESPONDING WHICH IS A REAL CHALLENGE. NOT ALL COMMENTS REQUIRE RESPONSE FROM THE AUTHORS BUT FOR THIS TO BE VALUABLE, WE NEED IT SO WE NOW ENCOURAGE AUTHORS TO RESPOND TO COMMENTS ON THEIR WORK, IF THEY GET--IF WE CAN FIND THEM THEY GET AN INDIVIDUAL E-MAIL AND AT THE MOMENT JUST UNDER 10% ARE APPLYING. BUT ONE OF OUR REALLY BIG STRATEGIES AT THE MOMENT, IS ONE THAT I WANTED TO SORT OF SPEND SOME TIME ON AND THAT IS THE IDEA OF JOURNAL CLUBS. WHEN I DID THE SLIDE A COUPLE OF DAYS AGO, THERE WERE STILL SIX, IT WAS VERY, VERY NEW AND THIS WAS ANNOUNCED AROUND CHRISTMAS WHICH IS TOWARDS THE END OF THE FIRST YEAR, FULL OPERATIONS OF THE COMMONS. THEY GOT EIGHT. AND ABOUT THAT MANY IN THE PIPELINE AS WELL. AND THIS IS--THIS IS KIND OF A--WHEN YOU THINK ABOUT WHAT HAPPENS AFTER SOMEBODY HAS PUBLISHED RESEARCH, OTHER THAN THE ISSUE OF PEOPLE DISCUSSING SOMETHING AT A CONFERENCE, ONE OF THE MAJOR THINGS THAT'S HAPPENING ACROSS SCIENCES, AND WE HAVEN'T SEEN IN A FEW OTHER AREAS LIKE PHILOSOPHY AND SO OIT CERTAINLY HAPPENS IN SCIENCE AND MEDICINE IS THAT YOU HAVE A CULTURE OF PEOPLE GETTING TOGETHER IN JOURNAL CLUBS TO DISCUSS THAT PARTICULAR WORK. SOME OF THEM ARE MEETING LIKE EVERY WEEK, SOME OF THEM ARE MEETING EVERY MONTH. AND WHILE A LOT OF THEM ARE ACTUALLY PUBLISHING THE DISCUSSIONS OF THE--POSTING THE RESULTS OF THE DISCUSSIONS ONLINE AT THEIR PARTICULAR UNIVERSITY OR WHATEVER THIS, IS ACTUALLY QUITE A MASSIVE VERBAL LEVEL OF INTELLECTUAL INPUT THAT'S HAPPENING ACROSS THE PLANET THAT ACTUALLY NOT REALLY BEING CAPTURED VERY WELL FOR PEOPLE WHO AREN'T ACTUALLY IN THOSE JOURNAL CLUBS. SO FROM THE POINT OF VIEW OF THINKING, WHERE IS THERE SOME WAY WE COULD ACTUALLY CAPTURE WORK THAT'S HAPPENING, THE--THE ENGAGEMENT WITH RESEARCH IS ALREADY HAPPENING, THAT DOESN'T REQUIRE THIS MASSIVE OF A CHANGE OR WHAT PEOPLE ARE DOING, IT'S THE JOURNAL CLUB TO SEE FOR EXAMPLE IN MEDICINE, JOURNAL CLUBS HAVE BECOME KEY STRATEGY IN CRITICAL SCALES SKILLS AND IT'S MAJOR AND A KIND OF MEDICAL STRATEGY SO HAVE YOU A HUGE AMOUNT OF EFFORT THAT WILL BE ENGAGING WITH MEDICAL LITERATURE AND SAME THING IS HAPPENING IN LABS AS WELL, IN MEDICINE INCREASING ALSO BECOMING A PART OF CONTINUING MEDICAL EDUCATION AND INCREASINGLY PEOPLE ARE ACTUALLY AND PUTTING THESE THINGS INTO THE PUBLIC DOMAIN BUT NOT ANYWHERE THAT'S EASY TO GET TO. SO WE STARTED THIS PROCESS WHERE WITH A RANGE OF ISSUES THAT HAVE TO BE A COUPLE GUARANTORS, COUPLE OF GUARANTOR WHO IS ARE ELIGIBLE INDIVIDUAL PUBMED COMMONS MEMBERS SO WE HAVE LIKE SEVERAL KIND OF HURDLE,S THAT A GROUP HAS TO GO TO AT THE MOMENT IN THIS PILOT PHASE OF TESTING THIS OUT. AND THEY GET THEIR OWN PAGE. THE COMMENTS THAT THEY MAKE WILL BE UNDER THE NAME OF THAT JOURNAL CLUB, SO IT'S THE PUBMED COMMONS ACCOUNT THAT'S GOT A NAME, SOMETHING OR OTHER, SOMETHING OR OTHER JOURNAL CLUB AND WHEN YOU CLICK ON WHO MADE THAT COMMENT, YOU WILL GO TO THEIR PAGE WHERE YOU CAN SEE WHO ARE THESE PEOPLE AND WHETHER THEY COME FROM? THEY THERAPIST IS ONE FOR EVIDENCE BASED MEDICINE GROUP. >> [INDISCERNIBLE]--YOU SAID YOU DIDN'T ALLOW ANONYMOUS, CAN YOU BE ANONYMOUS WITHIN THE JOURNAL CLUB AND SAY THE JOURNAL CLUB SAID X? >> YES BECAUSE WHAT WE'RE SEEING WITH SOME OF THEM WOULD SAY WHO WAS PARTICIPATING WITHIN THE COMMENT, THEY WILL SAY IT. BUT SOME OF THEM--SOME OF THEM DON'T AND IN EFFECT WHAT WE'VE DONE HERE IS WE'VE OUTSOURCED THE MODERATION TO THESE PEOPLE THAT'S REALLY--THAT'S REALLY WHAT WE'VE DONE AND IT'S CLEAR THEY'VE SIGNED ON IF THEY--IF THAT BECOMES PROBLEMATIC THEY WILL LOSE THEIR PERSONAL ACCOUNTS, SO, IT'S JUST OUTSOURCING. AND I THOUGHT I WOULD END WITH THIS SO WE CAN GO INTO DISCUSSION, THIS IS FROM PROFESSOR GARY WARD WHOEE ONE OF THE MEMBERS--WHO'S ONE OF THE MEMBERS OF THE WORD, A WORKING GROUP MEMBER, HE SAID HOW OFTEN DOES A PAPER GET READ CRITICALLY FROM BEGINNING TO END BY 10-12 INFORMED READERS WHO THEN DISCUSS IT AT LENGTH AS A GROUP. THIS KIND OF COLLECTIVE DISCUSSION IS A GREAT WAY TO SURFACE THE STRENGTHS AND WEAKNESSES OF A STUDY AND IDENTIFY CONNECTS OF THIS GROUP. SO THAT'S A WAY TO STRENGTHEN THE PUBMED COMMENTS. SO THANK YOU. [ APPLAUSE ] >> COMMENTS OR QUESTIONS? >> [INDISCERNIBLE]--COMMONS AND WHERE EVERYBODY COMES TOGETHER AS A COMMUNITY SO TO ANSWER YOUR QUESTION, HOW DO YOU SCREEN FOR CONFLICT OF INTEREST IN YOUR MEMBERS? DO YOU TYPICALLY ASK THEM OR JUST BY BEING NOT ANONYMOUS TO SURE SURE THERE ISN'T SOME [INDISCERNIBLE]. >> YEAH, THAT WHOLE ISSUE OF CONFLICT OF INTEREST IS A NASTY ONE, AND CRITICALLY IMPORTANT, ONE OF THE GUIDELINES, SAY THAT YOU HAVE TO DECLARE YOUR CONFLICT OF INTEREST. BUT IT'S DIFFICULT, EVEN FOR JOURNALS WILL EVEN SEE SOMETIME WHEN IS SOMETHING'S BEEN THROUGH, THERE'S CERTIFY YOU WILL GO TO TRIAL AND INVESTIGATE, CERTAINLY BUT NOT PUTTING THAT KIND OF EFFORT IN, AND SO, YOU--YOUR COMMENT ABOUT THEM BEING IDENTIFIED IS ACTUALLY IN THE END, THE MAIN--THE MAIN STRATEGIC PLAN EDGEY THAT WE'VE GOT AT THE MOMENT WHICH IS TO ENABLE OTHER PEOPLE TO KIND OF INVESTIGATE AND IDENTIFY WHO WERE GOING TO PUT THE TIME IN AND NOTIFY US. SO THAT'S HAPPENED. SOMETIMES, WHERE WE NEED TO GO BACK AND CONFLICT OF INTEREST BECOMES AN ISSUE AND YOU COULD EDIT YOUR COMMENT ON PUBMED COMMONS SO ONE OF THE THINGS WE HAVE DONE, TOO IS TO GO BACK TO SOMEBODY AND SAY, THIS IS HAS COME UP, COULD YOU PLEASE DECLARE THIS AND THEN THEY GO IN AND THEY DECLARE THEIR CONFLICT OF INTEREST. IT WAS PROBLEMATIC WE WOULD HAVE A WHOLE OTHER RANGE AND SOME OF THE TIME IT BECOMES CLEAR FROM A PATENT, YOU KNOW THAT THE FIRST COMMENT THAT'S NOT SO CLEAR, THE SECOND COMMENT IT'S STILL NOT SO CLEAR. IF THERE'S KIND OF 15 THINGS AND THEY'RE ALL THE SAME AND AN INDIVIDUAL'S BEING TARGETED OR PARTICULAR TOPIC IS BEING TARGETED, YOU DETECT THAT PATTERN OVER TIME BUT IT IS A REALLY--THE REASON THERE'S A GREAT SOLUTION TO THE MOMENT BUT I THINK IT WILL CONTINUALLY BE AN ISSUE. >> CARRY ON. >> AFTER THAT ORGANIZING THE [INDISCERNIBLE] AND THEN AND [INDISCERNIBLE] DAVID AND SO VERY LIVELY DISCUSSION. >> SO,--SO YOU MENTIONED THAT THERE'S A RATHER LOW USAGE OF THEM, SO WHAT DO YOU CONSIDER USAGE AND DO YOU TRACK ANY SUCH COLLABORATIONS THAT COME ABOUT, SO THE DECISION IS A GREAT SOURCE OF FINDING WHAT [INDISCERNIBLE] IS? >> SO FOR USAGE, PEOPLE JOIN SUGGEST NOT THAT MUCH OF A PROBLEM, WE HAVE REASONABLE GROWTH IN MEMBERSHIP BUT NOT THE--IT DOESN'T REALLY CONVERT TO A LOT OF ACTIVITY. >> USAGE AND BOTH OF THOSE THINGS, AND EACH OF THEM, IS IMPORTANT, AND SO THE SECOND PART OF YOUR QUESTION? >> KD--SALLY >> --[INDISCERNIBLE]--COLLABORATE , OKAY, LET'S COLLABORATE THAT WOULD BE A GREAT EFFORT? , IT WOULD BE A GREAT OUTCOME. THE THING WE MAY BE ABLE TO START TO PICK UP, ONE OF THE MAIN WAYS YOU SEE THIS HAPPENING IS ACTUALLY OUTSIDE BECAUSE WHAT'S HAPPENING TO WUB MED COMMON SYSTEM SIMILAR TO A LOT OF THESE KINDS OF THINGS, PEOPLE MAKE A COMMENT AND THERE'S A LOT OF DISCUSSION ABOUT THE COMMENTS, THE DISCUSSION IS NOT ON PUBMED COMMENTS. THE DISCUSSION IS HAPPENING BY E-MAIL. AND THOSE KINDS OF THINGS. SO THERE'S ONLY--WHICH IS ACTUALLY WHEN YOU THINK ABOUT THIS IS NOT A BAD THING BECAUSE IF THERE'S A COMMENT STRING WITH 50 COMMENTS ON TPEOPLE MAY NOT NECESSARILY READ IT BUT IT DOES MAKE THAT ISSUE WITH HOW DO YOU TRACK THESE THINGS AND ACTUALLY SAY THIS IS A RESULT OF THIS BECAUSE THEY ALSO MAY ADMIT AT A CONFERENCE OR WHATEVER SO THIS IS SOMETHING THAT TO SOME EXTENT WILL BE ABLE TO PICK OUT, WE'VE SEEN COMPANIES GO BADLY, AND OCCASIONS WHEN WE'VE SEEN THINGS GO REALLY WELL BUT THIS IS SOMETHING THAT ULTIMATELY WILL BE A QUESTION FOR PEOPLE TO RESEARCH PROPERLY. >> AND AS A FOLLOW UP TO THAT, THERE ARE SERVICES THAT FOR EXAMPLE, IN ALTERNATE METRICS FOR EXAMPLE WHICH IS PUT OUT BY--FORMAT ANALYTIC SYSTEM ONE KIND OF SERVICE ASK THEY AGGREGATE WHERE ALL OF THESE COMMENTS ARE COMING FROM BECAUSE THERE'S D. O. I. WITH THAT ARTICLE, CAN YOU PUT THAT--HELPS TO TRACK WHAT IS SORT OF BEING SAID ON THAT SO THESE KINDS OF COMMERCIAL SERVICES ARE MADE AVAILABLE FROM PUBLISHERS BUT IT'S AN INTERESTING NEW AREA TO SEE HOW THOSE AGGREGATIONS WILL HAPPEN AND BE ABLE TO TRACK THAT BECAUSE IN ORDER TO GET HIGHER USE YOU NEED THESE KINDS OF SERVICES BECAUSE OTHERWISE EVERYONE'S GOING TO HAVE TO KROLL A TWITTER ACCOUNT AND ALL THAT, AND THESE SERVICES YOU PUT IN YOUR NAME AND IT'S LIKE OH, THESE ICONS YOU CAN CLICK OUT TO THESE THINGS. >> THE COMMENTS ON THE COMMENTS ELEMENT ARE KIND OF AN INDIVIDUAL LINK AND WE'VE GOT KIND OF AN NLM CITATION FORMAT FOR IT SO POTENTIALLY OVER TIME THERE, 'S BEEN A COUPLE CITATIONS OF PUBMED COMMONS BUT POTENTIALLY OVERTIME SOME OF THESE MIGHT BE VISIBLE IF PEOPLE ARE CITING COMMENTS OR MENTIONING THEM. THIS IS PART OF LATER WORK. >> SO WHO DOES THE MODERATION? IN GENERAL? WHO DOES THE MODERATION? >> WELL, AS I SAID THE COMPUTER'S DOING HEAVY LIFTING WHICH WE REALLY APPRECIATE BUT THEY HAVE A COLLEAGUE WHO REALLY DOES SPEND A CONSIDERABLE AMOUNT OF TIME, SHE WORKS ON PUBMED COMMONS FULL-TIME, SPENDS A CONSIDERABLE AMOUNT OF TIME ON THAT AND THEN WHEN WE KIND OF--BUT THEN THERE ARE ISSUES, WE SEE THERE ARE ISSUES, WE PASS THEM UP TO NCBI TO MAKE THE CALL. >> SO I THINK THIS SEXER SENTIALLY A GOOD IDEA BUT I'M WONDERING WHAT THE COMPETITION IS? BECAUSE I KNOW A LOT OF DISCUSSION I HAVE IN MY OWN DISCUSSION GROUPS AND PROBABLY IN OTHERS BUT YOU KNOW ARTICLES AND SO ON, AND THE SORT OF USERS THAT CARE MOST ABOUT PARTICULAR ARTICLE MIGHT BE EITHER [INDISCERNIBLE] OR DISCIPLINE WHERE PUBMED COMMENTS IS THE UNIVERSE SO I WONDER IF YOU MIGHT LOOK AT IT RATHER--YOU KNOW I GUESS I'M READING THE QUESTION OF HOW LONG ONE WOULD KNOW WHAT THE POTENTIAL FOR INCREASING USAGE IS VERSES THE SPECIAL [INDISCERNIBLE]. >> I THINK THAT ISSUE ABOUT DISCUSSIONS OF CARING ARE ON THE VENUES, WOULD BT WE SEE THE COMPETITION IN THAT SENSE. DISCUSSION IS GOING TO HAPPEN ALL OVER THE PLACE AND THAT'S GREAT AND IT SHOULD, MY PERSONAL HOPE MORE WOULD BE THAT AT LEAST SOME OF THE MOST VALUABLE STEPS WOULD COME HOME TO ROOST HERE AND TO SOME EXTENT THAT WILL RELYOT PEOPLE WHO KIND OF LIKE START TO ACTUALLY THINK ABOUT IT, EVEN THINK OF IT, THAT IT'S IN THAT CONSCIENCENESS OR IT'S SOMETHING THAT WOULD BE GOOD TO APPEND TO THE RECORD OF THIS PARTICULAR PIECE OF RESEARCH. I THINK THAT WILL BE A PROCESS, IT WILL TAKE TIME FOR PEOPLE TO START TO VALUE WITH IT AND HOPEFULLY IT WILL BE LIKE A LOT OF OTHER THINGS THERE WILL BE SOME EARLY ADOPTERS AND THE THING WILL GROW. IF IT DOESN'T, THAT'S NOT NECESSARILY THE END OF THE WORLD, IS IT, I MEAN AS LONG AS THERE IS--WHAT'S HAPPEN SUGGEST VALUABLE AND THE OPPORTUNITY IS THERE. >> SO WHAT I WAS WAS WONDERING IS IF THERE'S A WAY TO BE HOSTING THE DISCUSSIONS FOR SOME OF THE SPECIAL GROUPS, ONE OF THE SERVICES THAT PUBMED BRINGS THAT I SEE WE WILL TREAT ARTICLES IS RELATED ARTICLES. AND SO JUST THE ABILITY TO MAYBE HAVE A DISCUSSION THAT ACTUALLY PULLS IN RELATED THINGS ON THE TOPIC. >> IT LOOKS LIKE HE HAS AN ANSWER? >> YEAH [INDISCERNIBLE]--I MEAN, THE HISTORY OF THIS PROJECT IS THAT WITH THE LAST ALMOST AT THIS POINT 10 YEARS WE'VE HAD VARIOUS ADVISORY GROUPS, NUMBERS OF THEM, MANY WELL KNOWN SCIENTISTS, NOBEL PRIZE WINNERS, YOU KNOW REALLY, YOU SHOULD REALLY COUNT [INDISCERNIBLE] AND I ALWAYS [INDISCERNIBLE], YOU KNOW IT JUST SEEMS TO ME THIS WOULD BE TERRIFIC. AND IT CONTINUES WITH THE 32 CRESCENDO BEFORE WE STARTED THIS AND I REFERRED SOME FOLKS TO ACTUALLY SPEAK TO HAROLD VARMIS ACROSS THAT AND LET HIM--YOU KNOW [INDISCERNIBLE] AND IT'S REALLY NOT SOMETHING THAT [INDISCERNIBLE]. AT THE LIBRARY WE'RE LOOKING FOR AT ALL. HAROLD GOT ENTHUSIASTIC AND AT THIS POINT WE HAVE FOUR PRESENTATIONS WITH TWO DIRECTORS WITH THE STEERING COMMITTEE OF [INDISCERNIBLE] AND THE SENSE USE THE INTERNET CITATIONS ANDO DIFFERENT WAYS THEY'RE WRITTEN THAT THE NOTION OF NIH PROVIDED A CHANNEL FOR THIS IS FOR TRANSPARENCY AND THAT WE SHOULD GIVE IT A SHOT. AND AT THE TIME I THOUGHT THERE WAS LIKE ON THE ONE HAND THERE'S DISASTER THAT HAVE ALL THE CRAZY THINGS AND SO MUCH WORK BECAUSE OF ALL THE NEGATIVE STUFFOT OTHER HAND IF THAT DOESN'T HAPPEN, WELL THEN [INDISCERNIBLE] AND ALL THIS CONTENT AND TELL BE GREAT. INSTEAD WE HAD IN RETROSPECT A SORT OF OBVIOUS RESULT, WHICH IS REDON'T HAVE A LOT OF TRAFFIC, BUT MOST OF THE STUFF WE HAVE IS ACTUALLY, YOU KNOW VERY USEFUL AND SOME OF IT WAS REALLY EXCELLENT AND I THINK THAT'S BECAUSE CURRENTLY, THE CURRENCY IN BIOMEDICAL COMMUNICATION IS PUBLICATION TO GET CREDITS FOR GRANT TENURE PROMOTION WHATEVER AND THAT'S THE ONLY THING, THAT'S THE ONLY INCENTIVE RIGHT NOW THAT YOU KNOW IF I'M GOING TO PUT AN EFFORT, IF I PUT MY NAME OUT THERE, I NEED TO GIVE SOMETHING BACK SUBSTANTIAL, NOT ONLY IS IT FOR US TO NOT BE ADVANTAGEOUS TO PUBLICLY BE OUT THERE COMMENTING AND WHATEVER, BUT FOR MANY PEOPLE YOU SEE THE EFFECTS OF IT, AND THE FACT THAT [INDISCERNIBLE]. AND SO, THERE'S ONLY A FRACTION OF THE COMMUNITY THAT'S GOING TO WALK TO ENGAGE ESPECIALLY AS THEY POINTED OUT BECAUSE YOU CAN'T BE ANONYMOUS. SO THE FACT THAT IT'S NOT [INDISCERNIBLE] USE IN RETROSPECT IS ACTUALLY CONSISTENT. I MEAN NIH DOESN'T GIVE YOU CREDIT FOR [INDISCERNIBLE]. WE SORT OF ASKED THEM TO CONSIDER WAYS TO DO THAT. [INDISCERNIBLE] BETWEEN US, THE FACT THAT YOU CAN'T BE ANONYMOUS IN THE SYSTEM ALSO EXPLAINED WHY VERY FEW PEOPLE ARE WILLING TO GO OUT THERE AND DO STUFF WHICH IS WHAT YOU WANT WHAT WE HAVE A CHAN THEY'LL'S OPEN TO THE COMMUNITY AND AT THIS POINT THE COST TO NLM OR MINIMAL, THERE HAVE BEEN USEFUL THINGS THAT HAVE BEEN OUT OUT, BUT I THINK MAYBE THIS WILL GROW, AT THE CURRENT TIME IT'S SORT OF LIKE THE BENEFIT FAR OUTWAYS THE COST AND I REALLY BELIEVE AND IT'S INTERESTING TO CONTINUE DISCUSSIONINSTITUTE DIRECTORS WHERE THEY WERE SAYING, YOU KNOW TRANSPARENT THIS, IS THE FORM OF TRANSPARENCY, AND IT'S REALLY IMPORTANT. THAT NIH WOULD HAVE IT ON THE SAME PLACE THAT ALL THESE OTHER PAPERS AND SO FORTH COME IN ON. SO, I THINK IT'S AN EXPERIMENT WHICH I THINK AT THIS POINT HAS BEEN WORTH WHILE BUT IT'S CLEARLY AT THIS POINT NOT REVOLUTIONIZING BIOMEDICAL COMMUNICATION AND I THINK THE SAME REASON WE HAVE A LOT OF OTHER ISSUES WITH IT, INCENTIVES ARE PARTICULARLY CREATE INCENTIVES FOR OTHER THINGS, THEY'RE NOT GOING TO HAPPEN. >> [INDISCERNIBLE]. >> THANK YOU. VERY GOOD PRESENTATION. INTRIGUING, I HAVE TWO PARTS TO MY QUESTION, THE FIRST RELATES TO HOW THIS IS BEING DEPLOYED. I REALLY SEE TERRIFIC OPPORTUNITIES HERE FOR TRAINING PROGRAMS SO WE'RE CLOSE WITHIN TRAINING PROGRAMS COULD BE SHARED WITH OTHER TRAINING PROGRAMS AND [INDISCERNIBLE] DIALOGUE, ACTUALLY, THAT WILL BE CROSS CULTURAL BOUNDS AS WELL, SO I DON'T FINISH ANY THOUGHT HAS BEEN MADE OF THAT, AND HOW THEY MIGHT BE PROMOTED. THE OTHER IS MORE TECHNICAL THROUGH THE EMPHASIZING GOOD AND DEEMPHASIZING POOR COMMENTS AND I'M JUST CURIOUS, ANY SUBSTANTIVE, VERSES NONSUBSTANTIVE COMMENTS OR POSITIVE VERSES NEGATIVE COMMENTS AND HOW THEY'RE--AND HOW WOULD THAT DETERMINE, WHO MAKES THE DETERMINATION WHETHER IT'S THE COMMENT? >> SO I'LL ANSWER THAT ONE FIRST AND GO BACK TO THE OTHER. I MEAN WHAT WE'RE RELYING ON IS FOR EXAMPLE, I MEAN--SAY FOR EXAMPLE SOMEBODY HAS HAD--I'M NOT SAY THANKSGIVING HAPPENED TO ANYBODY YOU IT'S AN EXTREME EXAMPLE, SAY YOU HAVE A PERSON WHO HAS MADE FIVE COMMENTS. EVERY SINGLE ONE OF THEM HAS RESULTED IN COMPLAINT FROM THE AUTHORS AND DOWN RATING USING THE RATINGS SYSTEM OR WHATEVER, AT A CERTAIN POINT IT'S CLEAR THAT PERHAPS THE SIXTH TIME THEY MAKE A COMMENT THAT MIGHT BE A PROBLEMATIC COMMENT AS WELL, SO IT'S NOT A PROCESS OF TRYING TO LOOK AT WHETHER THE COMMENT IS SUBSTANTIVE OR NOT, IT'S MORE RELATED TO THOSE THINGS AS SOMEBODY ACCUSING PEOPLE EVER SCIENTIFIC MISCONDUCT AND THOSE KINDS OF-- >> IT'S A JUDGMENT CALL. >> IT'S A JUDGMENT CALL AND FOR IT TO BECOME ANY BETTER, THAN THAT, WOULD REQUIRE A LOT OF PEOPLE TO BE ENGAGED IN RATING AND THEN IT MIGHT START TO BECOME MORE ABOUT THOSE--ABOUT THOSE OTHER THINGS. >> IT'S NOT BASED ON WHETHER THEY WERE CRITICAL OF THE METHODOLOGY OR WHATEVER, IT'S MORE THE WAY THEY MAKE THE COMMENTS IN A WAY, YOU KNOW OVER TIME BECAUSE YOU KNOW I THINK MOST PEOPLE IF THEY FEEL THEY'RE--AND I SEE MANY OF THESE, YOU KNOW SOMEBODY SAYING, WELL IT'S AN INTERESTING PAPER BUT WE WONDER ABOUT METHODS, X, Y, ZOR DOES THAT STATISTICAL MEASURE APPLY TO THIS SINCHE. >> IT'S TECH NO LOGICAL OVER RIGOROUS. >> YES AND NOT TO FAST THIS, IS MY FAVORITE COMMENT THAT MY UNCLE HARRY EVER WROTE KIND OF THING WILL THAT WILL GET A FEW THUMBS DOWN AND THE KINDS OF THINGS WHERE SOMEBODY WRITES AND THEY SAID I HAVEN'T READ THIS ARTICLE BUT I RECORD BLAH, BLAH, BLAH. YOU KNOW THOSE KINDS OF THINGS, THE SYSTEM OF RELYING ON RATERS AND SO ON THAT IS CAPTURING A LITTLE BIT AND USING MORE WITHIN A LOT MORE WRITING GOING ON APPROXIMATE IN TERMS OF JOURNAL CLUBS, I THINK THAT'S REALLY IMPORTANT POINT ABOUT THAT, UNTAPPED POTENTIAL FOR THEM TO START TALKING TO EACH OTHER. THAT IS SOMETHING THAT WE HOPE AS WE GET MORE OF THEM THAT WOULD BE LOOKING AT, LOOKING AT SORT OF FACEBOOK AND A FEW OTHER PLACES TO ACTUALLY START TO GET THEM TO ENGAGE WITH EACH OTHER, AND ALL THE ONE ON TWITTER SOME OF THEM AND YOU CAN SEE THAT POINT THAT GETS TO AND SOME CONTROVERSIAL PAPERS COME OUT OF WHATEVER, STARTING TO SAY, HEY, YOU GUYS LOOKING AT THIS ONE AND WHATEVER, SO, IF THIS COMMUNITY DEVELOPS, THAT WOULD BE A REALLY QUITE IMPORTANT ONE. AND IN TERMS OF THAT PROCESS THAT ALSO--THAT GOES BACK TO THE PREVIOUS QUESTION ABOUT HOSTING AND SO ON. YOU HAVE PEOPLE WHO'VE BEEN WANTING TO CAPTURE, THE CRITICAL APPRAISAL, PEOPLE HAVE DONE A STUDY BEFORE THE JOURNAL CLUB ACTUALLY HAPPENED, FOR EXAMPLE, IN MEDICINE, IN ORDER TO HAVE A JOURNAL CLUB BE ACCREDITED THEY NEED TO ACTUALLY BE RELEASING THEIR FUNDING SOMEWHERE AND FOR SOME PEAL --PEOPLE THEY ARE LOOKING AT WELL, THIS IS EASIER THAN TRYING TO GET A UNIVERSITY TO DEVELOP A THENOT WEB SITE OR WHATEVER, WOULD BE RELATIVELY EASY IN THE END TO CAPTURE IT AND PUT IT THERE, BUT YOU'RE RIGHT. THAT JOURNAL CLUB TO JOURNAL CLUB COMMUNICATION IS ACTUALLY A CONSPICUOUS ABNORMALITIES ABSENCE IN THE SCIENCE SYSTEM. IT SEEMS TO BE A SOCIETY FOR HYMN EVERYTHING ELSE. >> [INDISCERNIBLE]--POTENTIAL SLOWLY TO BECOME IMPORTANT FOR A SCIENTIFIC COMMUNICATION. IT REMINDS ME OF SOMETHING LIZ THOMAS SAID BACK IN THE DAYS BEFORE THE INTERNET AND THE SLOW COMMUNICATION AMONG THE PEOPLES OF THE EARTH AND HE SAID THAT ULTIMATELY WE WILL BECOME ONE GIGANTIC BRAIN AND THIS IS A STEP SCIENTIFICALLY. BUT ANOTHER SENSE THOUGH, THIS IS AN EVOLUTION AND LETTERS TO THE EDITOR AND THEY MAY THINK OF THAT, WHAT WAS THE HISTORY OF THE LETTERS TO THE EDITOR, NOT TOO FOUNDED, I THOUGHT MAYBE GEORGE MIGHT KNOW SOMETHING ABOUT LETTERS TO THE EDITOR STORY BACK IN ENGLAND AND THE EARLY JOURNALS, I DON'T KNOW. >> WELL WE HAVE BMJ ALL THE WAY BACK TO 18-SOMETHING-OR-OTHER. SO WE COULD TRY TO USE PMC BACK FILES TO SEE IF AT LEAST WE COULD SAY, WELL WE KNOW THERE WAS ONE IN 1812 OR WHATEVER IT WAS THAT IT STARTED. >> THE TRANSACTION TO THE REAL SOCIETY. >> I WAS GOING TO SAY, SO CHRISTOPHER AND THEN DON. >> YEAH. >> YOU KNOW CIRCLE BACK TO SOMETHING GAIL RAISED EARLIER WHEN SHE WAS REFERENCING PUB, WE WERE IMPLICATING USDA, AND CAPTURING ALL THE RESEARCHERS ARE DOG WITHIN USDA AND THEIR AREAS OF EXPERTISE AND PUBLICATIONS AND AS PART OF THE LACK OF A BETTER TERM, DATA WRANGLING ALLOWED THIS. WE'RE WORKING WITH AN OUTFIT OUT OF ENGLAND CALLED SIM PLEATICS, AND ONE OF THEM IS CALLED METRICS AND AND WE STARTED LOOKING AT AS THEY WORK ON THESE ARTICLES THEY AUTOMATICALLY TELL YOU WHERE THE BLOGS ARE, IT'S BEEN POSTED WHERE COMMENTS ARE COMING ON IT, WE STARTED SAYING, WHAT WAS--WHAT WOULD BE THE EFFECT IF WE STARTED LOOKING AT GIVE THEM UP, SET A D. O. I.s FOR USDA PUBLICATIONS. IT'S AX STOWNDING TO LOOK AT THAT 60 DAY--ASTOUNDING TO LOOK AT THAT GROUP OF PUBLICATIONS TO GET A SENSE OF THE CHANGING NATURE OF COMMUNICATION, WHERE THINGS ARE COMMENTED ON, AND YES, THEY!q TRADITIONAL FORMS ARE STILL PREDOMINANT BUT IT'S--IT'S SURPRISING WHERE THINGS, WHAT GETS PICKED UP AND WHERE IT GETS PICKED UP AND WHY. AND I THINK IT'S IMPORTANT THAT WE MONITOR FOR NO OTHER REASON THAN WHAT'S GOT A LOT OF TRAFFIC IN THESE NEW AREAS NOW ARE FOR LACK OF A BETTER TERM DODGEY SCIENCE. IT'S FREE, SOMETIMES IT'S VERY OPINION DRIVEN AND FACT BASED AND SO I THINK WE NEED TO LOOK AT HOW--HOW DO WE--WHAT ARE THE OTHER FORMS THAT GET OUR RESEARCH OUT AND WHAT'S--WHAT IS IT'S EFFECT. I LIKE WHAT YOU'RE DOING AND PARTICULARLY THE FACT THAT NO ONE CAN BE ANONYMOUS. THAT'S ABSOLUTELY CRITICAL FOR HIGH QUALITY. >> YEAH. >> YOU'RE STILL GOING TO GET CONTROLS BUT THEY'RE GOING TO HAVE TO THINK TWICE BEFORE THEY. >> QUESTION AND DAVID HAD COMMENTS CAN AND-- >> I HAD NO COMMENT. [LAUGHTER] >> OKAY. >> I'M JUST RECALLING THAT YOU'RE APPROPRIATELY SHOWING AN ATTITUDE ABOUT THIS WHOLE THING TO BEGIN WITH AND [INDISCERNIBLE] BUT I WONDER DO YOU SEE ANY EXCHANGES HERE ABOUT REPRODUCIBILITY? OR IF YOU WANTED TO COULD YOU SEED IT AND GET SEVEN A THING FOR THE DISCUSSION GOING? >> SO THERE WAS ONE I THINK TO [INDISCERNIBLE] EXACTLY ON THAT, WHERE SCIENTISTS POST AND APPRECIATE GETTING THE DATA FROM THE TALK OF THE [INDISCERNIBLE] REPEATED SOMETHING ELSE [INDISCERNIBLE] SO WE SEE PRODUCIBILITY, COMMENTS, WE HAD ONE--WE HAD ANECDOTES AND THERE WAS ONE THAT WAS INTERESTING AND SOMEBODY PUBLISHED THE PARCH AND SCIENTISTS COMMENT ON THE PHYSICAL PROBLEM, THE ANALYSIS, AND THE OFFER GOT BACK AND ACKNOWLEDGED THAT THEY WERE WRONG AND WE'VE SEEN OTHERS LIKE THAT. WHICH ARE VERY INTERESTING AND IT'S--IT'S A NICE THING TO DO WHEN WE SEE THAT WRITING IN THE CHAIN. >> I GUESS ONE LAST POINT WHICH SOMEBODY MENTIONED, ONE OF THE REASONS WHY WE'RE GETTING VERY STRONG INTEREST TO MAKE THIS AVAILABLE AS A CHANNEL WAS THAT PEOPLE POINTED OUT, HOW FREQUENTLY AND I FRANKLY WASN'T THAT AWARE OF, PEOPLE WRITE A LETTER TO THE EDITOR AND IT'S REFUSED. THEY WON'T--AND THERE'S REALLY NOTHING WRONG WITH IT. YOU KNOW IT'S LIKE YOU'RE NOT TALKING ABOUT [INDISCERNIBLE] UNPROFESSIONALLY OR WHATEVER, IT'S PURE SUBSTANTIVE COMMENT YET FOR SOME REASON, THEY THE EDITOR DOESN'T PUBLISH THAT SO THAT WAS SOMETHING WHERE THE JOURNAL PUBLISH IT AND THEY SEE IT AND AGAIN, I THINK THAT WE'VE SEEN SOME EXTREMELY LONG, IN MY MIND PREVIOUSLY PREPARED, YOU KNOW FAIRLY QUICKLY WHERE PROBABLY WOULD HAPPEN WAS, IT WENT [INDISCERNIBLE] FOR A LETTER TO THE EDITOR AND PUBLISH [INDISCERNIBLE]. >> WONDERFUL PROJECT. THANK YOU VERY MUCH. [ APPLAUSE ] >> WE'LL TAKE A 15 MINUTE BREAK AND RECONVENE AT 10:50. >>> OKAY, SO THE NEXT ITEM WILL BE QUITE BRIEF. I WILL BE RETIRING AS CHAIR AT THE NEXT MEETING IN MAY, AND SO WE HAVE TO ANNOUNCE THE APPOINTMENT OF THE NOMINATING COMMITTEE FOR THE NEXT BOARD CHAIR, AND IT CONSISTS OF KATHRYN MENDEL HAUL, CHARLES RICE, AND KATHY. OKAY. [LAUGHTER] THAT DIDN'T TAKE TEN MINUTES. [LAUGHTER] >> Dr. Trudy MacKay: SO NEXT WE'LL HAVE -- [LAUGHTER] >> Dr. Trudy MacKay: NEXT WE HAVE TWO PRESENTATIONS. BY ALAN CAPELMAN, AND ON THE ENVIRONMENTAL HEALTH ANIMATION. SO I THINK THIS IS GOING TO BE A LOT OF FUN, BUT LET ME TELL YOU A LITTLE BIT ABOUT OUR GUEST SPEAKER. ALEX CASTLEMAN HAS A DOCTORATE IN HUMAN CONDITIONS AND LEARNING, MASTER'S, HE'S A SENIOR ANALYST IN THE POSITION OF SPECIALIZED INFORMATION SERVICES, SHE LEADS A K-12 TEAM THAT DEVELOPED SCIENCE AND HEALTH EDUCATION INFLUENCES, INCLUDING ANIMATION AND GAMES. SOUNDS LI SHE ALSO DIRECTS PROGRAMo+s&Ö EVALUATION EFFORTS FOR THE DIVISION. IN ADDITION, SHE CONDUCTS RESEARCH INTO CONSUMER HEALTH INFORMATION AND PERCEPTUAL UNDERSTANDING OF HEALTH AS WELL AS DEALING WITH HEALTH INNOVATION. HER PUBLICATIONS HAVE INCLUDED MANY INFORMATICS JOURNALS. NEXT WE HAVE A GRADUATE FROM THE UNIVERSITY OF MARYLAND WITH A PASSION FOR NUTRITIONAL EATING BEHAVIORS, SHE CURRENTLY INTERNS AT THE NATIONAL LIBRARY OF MEDICINE WHERE SHE DEVELOPED HIGH QUALITY ENVIRONMENTAL HEALTH INTERVENTIONS. SHE CONDUCTS NUTRITION RELATED RESEARCH AND MASTER'S IN PUBLIC HEALTH. WE LOOK VERY MUCH FORWARD TO YOUR PRESENTATION.W„Ny >> GOOD MORNING. IT IS A PLEASURE TO BE HERE, AND THIS INNOVATION -- THIS PRESENTATION IS ABOUT ENVIRONMENTAL HEALTH ANIMATION FOR THE SCIENCE CLASSROOMS. THEY ARE PRIMARILY TARGETED AT MIDDLE SCHOOL STUDENTS, AND I WOULD LIKE TO START THIS PRESENTATION BY INTRODUCING SOMEBODY NAMED DAVID AND HIS PECULIAR LITTLE PROBLEM. DAVID IS NOT AN ADOLESCENT. [MUSIC] * * [BACKGROUNDI [SNORING] >> OH, NO! THE HOUSE, IT'S INFESTED! QUICK! * >> WHAT DO WE DO? RIGHT AWAY! [LAUGHING] >> HEY, HOLD UP THERE, MR. PHOBIA. >> ME? >> DO YOU EVEN KNOW WHAT THE STUFF IN THAT CAN IS? >> YEAH, IT SAYS RIGHT HERE. >> WELL, DO YOU KNOW HOW TO USE IT DO YOU KNOW WHAT IT CAN DO TO YOU? >> MAKE ME LOOK LIKE AN ACTION HERO WHILE I CLEAN UP MY HOUSE. >> YEAH, WHY DON'T YOU GIVE THAT TO ME FOR A SECOND. >> HEY! MY AWESOME SPRAY! MY BATHROOM! AM I HAVING AN EXISTENTIAL CRISIS HERE? >> CALM DOWN, WE NEED TO HAVE A CHAT ABOUT PESTICIDES. >> PESTERBRIDES? >> NO, PESTICIDES. THEY'RE ANY CHEMICAL TO PREVENT, REPEL OR DESTROY PESTS. >> RIGHT. PESTS. MEANING TINY PENGUINS. >> WELL, IN THIS CASE, SURE, BUT PESTS AREN'T JUST PENGUINS, PESTS ARE ANY UNWANTED ORGANISM. PESTS CAN SPREAD DISEASE, COMPETE FOR RESOURCES AND OTHER ORGANISMS, CAUSE PROPERTY AND CROP DAMAGE OR JUST GROSS USI OUT. >> LIKE MICROPENGUINS. >> YOU KNOW IT, TIGER. BASICALLY A PEST IS AN ORGANISM WE DON'T WANT AROUND FOR ONE REASON OR ANOTHER. >> OKAY. SO DAVID IS ABOUT TO LEARN A FEW THINGS AND RECONSIDER HIS APPROACH. WE'RE GOING TO NOW TALK ABOUT OUR ANIMATION PROGRAM AND WHY WE'RE DOING THIS, BUT BEFORE I GET INTO THIS, I WOULD LIKE TO GIVE A LITTLE BIT OF AN OVERVIEW OF OUR DIVISION AND ITS MISSION AND HOW THIS ANIMATION PROGRAM RELATES TO ITS MISSION. SO A BIT OF HISTORY. SO I REPRESENT THE DIVISION OF SPECIALIZED INFORMATION SERVICES, AND OUR MISSION HAS A NUMBER OF THINGS, WE DEVELOP RESOURCES IN ENVIRONMENTAL HEALTH AND TOXICOLOGY, CHEMICAL AND DRUG INFORMATION, INFORMATION ABOUT HIV/AIDS, DISASTER INFORMATION. OUR ANIMATIONS KIND OF BRING TOGETHER TWO ASPECTS OF OUR MISSION. ONE IS DEVELOPING ENVIRONMENTAL HEALTH AND TOXICOLOGY RESOURCES. SO WE'VE BEEN DOING THIS FOR A WHILE. WE ARE THE DIVISION THAT PUTS TOGETHER TOX MAT DATABASES, ONE OF THEM FOR EXAMPLE IS A HAZARDOUS SUBSTANCE DATA BANK. YOU MAY BE FAMILIAR WITH TOXMAIL WHICH IS A GEOGRAPHIC INFORMATION SYSTEM THAT REPRESENTS TOXIC RELEASES, THIS IS HOUSEHOLD TOXIC DATABASE, PEOPLE CAN LOOK UP INFORMATION FOR COMMON CONSUMER BRANDS. THIS IS TEXTILE RESOURCE ABOUT CHEMICALS IN EVERYDAY PLACES. SO WE DO A LOT OF THINGS ON THE ONE HAND. ON THE OTHER HAND, ANOTHER COMPONENT OF OUR MISSION THAT'S RELATED TO THE ANIMATIONS OR THAT MADE US GO IN THE ANIMATIONS IS OUTREACH TO SPECIAL POPULATIONS, SO WE WORK A LOT WITH DIFFERENT GROUPS. THIS IS OUR HEALTH WEBSITE, THIS IS RELT REACH, WHICH COLLECTS -- THIS IS HEALTH REACH, WHICH COLLECTS AND ORGANIZES INFORMATION WITH MULTILINGUAL, MULTICULTURAL HEALTH RESOURCES FOR HEALTH PROVIDERS. WE DO A LOT OF WORK WITH CAREGIVERS, AND SO ON. SO WHEN THESE TWO INTERSECT, WE GET OUR KIDS AND HEALTH PROGRAM, BECAUSE TEACHERS AND STUDENTS IS AN IMPORTANT POPULATION WE WANT TO TARGET. AS FAR AS DELIVERING ENVIRONMENTAL HEALTH INFORMATION. THE NATURAL PLACE TO DO THIS IS THE CLASSROOM, AND THIS IS BECAUSE MORE AND MORE ENVIRONMENTAL SCIENCE IS TAUGHT AT THE MIDDLE SCHOOL LEVEL AND AT THE HIGH SCHOOL LEVEL ON THE ONE HAND. ON THE OTHER HAND, WE'RE FINDING THAT THIS SURPRISING HEALTH CONNECTION IS ALSO NOT MADE, SO STUDENTS ARE LEARNING ABOUT VARIOUS CHEMICALS, ABOUT POLLUTION, BUT THERE IS NO WHAT'S IN IT FOR ME CONNECTION TO HEALTH EFFECTS, WHICH IS VERY IMPORTANT, AND THIS IS WHERE WE WOULD LIKE TO STEP IN. SO THIS IS THE FIRST WEBSITE THAT WE DEVELOPED SPECIFICALLY FOR THE CLASSROOM FOR STUDENTS AND TEACHERS. THIS IS THE MIDDLE SCHOOL ENVIRONMENTAL HEALTH STUDENT PORTAL, AND IT'S FOR MIDDLE SCHOOL BECAUSE IT NOT ONLY REPRESENTS TOPICS THAT ARE RELATED TO MIDDLE SCHOOL TOPICS, BUT IT ACTUALLY REFLECTS THE MIDDLE SCHOOL CURRICULUM, AND WE KNOW THIS BECAUSE WE DEVELOPED THIS BASED ON A NUMBER OF FOCUS GROUPS WITH TEACHERS AND WITH TEACHER COLLABORATORS. SO IT HAS FOUR MAJOR AREAS: AIR POLLUTION, CHEMICALS, CLIMATE CHANGE, AND WATER POLLUTION. AND THIS HAS A NUMBER OF SUBTOPICS, AND ALL OF THEM RELATE TO HUMAN HEALTH. THE RESOURCES HERE ARE NOT JUST STRAIGHT INFORMATION WEBSITE, BASED ON TEACHER REQUESTS WE HAVE A LOT OF EXPERIMENTS, GAMES, VIDEOS. AND THESE ARE JUST SOME OF THOSE PAGES TO GIVE YOU A SENSE WHAT THEY LOOK LIKE ONCE YOU CLICK. SO THIS IS OUR EXPERIMENT AND PROJECTS PAGE, GAMES AND ACTIVITIES, VIDEOS, AND HERE I WOULD LIKE TO TALK A LITTLE BIT MORE ABOUT VIDEOS BECAUSE IN FOCUS GROUPS AND MORE INFORMAL ED CONFERENCES, WE HEAR IT OVER AND OVER AGAIN, THE TEACHERS LOVE SHOWING VIDEOS IN THE CLASSROOM, THEY SAID THIS IS GREAT THAT YOU HAVE THIS, BUT WE WOULD LIKE TO HAVE MORE VIDEOS ON THIS WEBSITE. AND IT MADE A LOT OF SENSE TO US BECAUSE WE'RE THINKING OF COURSE, VIDEOS, THEY CAN RELATE TO ADOLESCENT CULTURE AND USE HUMOR AND MAKE THINGS INTERESTING. THEY ALSO MORE IMPORTANTLY CAN MAKE VERY DIFFICULT INVISIBLE PROCESSES VISIBLE AND MAKE BIG IDEAS SIMPLE AND CONCRETE. SO WE'RE GOING TO LINK THIS TO OUR WEBSITE AND IT WILL BE GREAT. HOWEVER, ONCE WE SET OUT TO MOVE THINGS THAT WE'RE FINDING LOOKED MORE LIKE THIS. A LOT OF PROFESSIONALLY DRESSED PEOPLE GIVING FORMAL PRESENTATIONS, AND NOT TO PUT DOWN THIS GENERAL HE WERE AT ALL, THIS IS NOT SOMETHING THAT KIDS 12 YEARS OLD AT THE EDGE OF THEIR SEAT. [LAUGHTER] >> SO WE DECIDED, OKAY, WHAT ARE WE GOING TO DO? WE DECIDED WE'RE GOING TO TRY AND CREATE OUR OWNYU% AND WE'VE NEVER DONE THIS BEFORE, BUT THERE WERE TWO THINGS THAT KIND OF CAME TOGETHER TO MAKE IT POSSIBLE. ONE WAS THE CURRENT DEVELOPMENT OF ANIMATING TECHNOLOGY. TEN YEARS AGO, THIS WOULD NOT BE POSSIBLE, BUT RIGHT NOW THERE ARE SO MANY FREE OR LOW COST RELATIVELY EASY TO USE ONLINE ANIMATION TOOLS THAT DO NOT REQUIRE PROFESSIONAL EXPERTISE. THAT WAS ONE THING. THE OTHER THING WAS WE'RE VERY FORTUNATE TO HAVE SEVERAL INTERNSHIP PROGRAMS RELATED TO OUR DIVISION. ONE IS IN THE ADOPT A SCHOOL PROGRAM WHERE HIGH SCHOOL SCIENCE TEACHERS AND STUDENTS COME TO WORK WITH US. THE OTHER IS THE PROGRAM THEJ THE UNIVERSITY COLLEGE COMMUNITY HEALTH PROGRAM, SO THE STUDENTS WE HAVE IN THESE PROGRAMS OR THE TEACHER INTERNS USUALLY ARE VERY IT-SAVVY AND VERY INTERESTED IN WORKING WITH TECHNOLOGY, SO WE THOUGHT, WELL, WE'LL TRY IT. AND WE WILL TALK MORE ABOUT THE PROCESS, BUT I JUST WANT TO SAY THAT FROM OUR PERSPECTIVE, IT WAS VERY SUCCESSFUL, AND I WOULD LIKE TO GIVE A LITTLE BIT OF A PREVIEW OF THE ANIMATION THAT WE HAVE TO DATE. SO SILENT BUT DEADLY, TOXICITY OF LEAD. THIS WAS SPEARHEADED BY DR. WENDY SPARKS, WHO IS A CHEMISTRY TEACHER AT MONTGOMERY HIGH SCHOOL, IT'S ONE OF THE LOCAL HIGH SCHOOLS, AND IT STARTED AS PAN ADVENTURE. PROFESSOR TAKES AN INTERGALACTIC SUPERHERO LEAD MAN, WHO IS A LITTLE BIT OBNOXIOUS, HE COMMITS GOOD DEEDS ON EARTH BUT HE'S VERY PROUD OF HIS LEAD SUIT, BECAUSE IT HAS A LOW MELTING POINT, THERE'S A LITTLE PROBLEM WITH HIS LEAD SUIT, IT'S TOXIC TO HUMANS. AT FIRST HE'S DISMISSIVE ABOUT IT, BUT THE PROFESSOR USES HER CHARM TO CONVINCE HIM TO REDEFINE HIMSELF INTO SOMETHING LEADLY. LESS> AND THIS ANIMATION, THIS IS THE FIRST ANIMATION THAT WAS CREATED. IT TALKS ABOUT THE PROS OF INDUSTRIAL DESALINIZATION, LINKING IT TO WHY IT'S IMPORTANT TO CONSERVATIVE WATER. SO I MENTIONED AT THE BEGINNING WHY WE WOULD LIKE TO CREATE ANIMATIONS, RELATING TO THE ADOLESCENT CULTURE. THIS IS A SPOOF ON THE POPULAR MOVIE THE AVENGERS, AND YOU'RE PROBABLY RECOGNIZING SOME OF THE CHARACTERS. FOR EXAMPLE, ADMIRAL AMERICA, OF COURSE, IS CAPTAIN AMERICA'S REFERENCE, AND BULK REFERS TO HULK, AND WE SHOW IT TO KIDS AND TEACHERS AND THEY LIKE IT. WE ALSO TRY ON USE HUMOR, SO HERE DAVID RUSHES TO A STORE TO LOAD UP ON PESTICIDES, AND THE STORE IS TRYING TO CONVINCE HIM TO GET A BONUS CARD BY PROVIDING HIS E-MAIL, SOCIAL SECURITY CODE AND GENETIC FINGERPRINT. [LAUGHTER] >> DAVID IS NOT IMPRESSED. BUT MORE IMPORTANTLY, WHAT WE TRY TO DO IS SHOW INVISIBLE PROCESSES AND MAKE THEM CLEAR FOR STUDENTS. SO HERE PROFESSOR FLOMBOM IS EXPLAINING TO LEADMAN HOW LEAD GETS INTO THE HUMAN BODY AND SHE'S EXPLAINING HOW CHEMICAL MOLECULES CAN GET THROUGH THE SKIN OR THE LUNGS OR THE INTESTINES. SIMILARLY HERE, THE SCIENTIST IS EXPLAINING TO ZERO HOW OZONE IS FORMED FROM OXYGEN/y MOLECULES AND INDIVIDUAL ATOMS TO MAKE O3 AND THIS IS ALL ANIMATED, SO THE MOLECULES ARE MOVING, THEY'RE COMING TOGETHER, IT'S ALL VERY VISUAL. SO I WOULD LIKE TO CONCLUDE MY PART OF THE PRESENTATION BY SHOWING YOU A DIGEST THAT HAS A LITTLE BIT OF HIGHLIGHTS FROM ALL OF THE VIDEOS I'VE MENTIONED SO FAR. *UI55– >> AND WHERE ARE WE NOW? >> IT WAS MUCH LOWER DURING THIS TIME PERIOD UP UNTIL THE 19THz CENTURY, A MECHANICAL PRINTING PRESS, COMBINED SEVERAL INTERESTING TECHNOLOGIES TO CREATE THIS PRESS THAT USED LEAD BASE AND POUREDltG FORM THE PRINTING BOX. >> THOSE DON'T LOOK SO GOOD. I THOUGHT LEAD WASN'T ABSORBED WELL THROUGH THE SKIN. >> THAT'S TRUE. >> THE DANGER COMES OFF THE ITEMS IF THEY ARE HANDLED AND NOT WASHING YOUR HANDS AND TOUCHING YOUR LIPS AFTER WORKING WITH THE METAL. * >> HEY, CAN'T YOU GIVE ME A HEADS-UP? >> OOPS, YOU ARE WELCOME. >> FINE, WHERE WAS I? ZOOR UPÖ< DESCRIBING HOW ATMOSPHERIC OZONE IS FORMED. >> THAT'S RIGHT. FIRST, HERE IS THE STRATOSPHERE, IT'S NATURALLY FORMED WHEN ULTRAVIOLET RAYS INTERACT WITH OXYGEN GAS, O2, IN THE STRATOSPHERE. THE PHOTONS FROM THE UV RAYS BREAK IT INTO TWO SEPARATE SINGLE OXYGEN ATOMS, AN INDIVIDUAL OXYGEN ATOM BONDS WITH AN O2 TO MAKE O3 OROZONE. * >> HOW DO ALL THESE PARTICLES GET IN THE AIR ANYWAY? >> CONSTRUCTION, MINING, COMBUSTION CAN RELEASE PARTICLES. >> COMBUSTION, LIKE BONFIRES, OR MAKING YOUR CAR RUN. >> YES, BURNING FOSSIL FUELS WILL RELEASE PARTICLES IN THE AIR. THESE CARS RUN ON KALE. ALSO BONFIRES AND VOLCANIC ACTIVITY CAN CREATE PARTICLES IN THE AIR NATURALLY. >> VOLCANIC ACTIVITY, A GIANT WHICH I CHICKEN, IN THE THEME SONG! * >> THEN THE MERCURY GETS INTO FISH BY GOING UP THROUGH THEW FOOD CHAIN THROUGH A PROCESS CALLED DIOMAG NECESSITY FICTION, FIRST -- MAGNIFICATION. TINY AQUATIC ANIMALS EAT IT, THEN THE FISH FEED ON THEM, THEN IT GETS PASSED ON TO THEM. BIGGER FISH EAT THE SMALLER FISH AND ABSORB THE METHYLMERCURY, SO ON AND SO FORTH. THE HIGHER UP THE FOOD CHAIN WE G THE MORE MERCURY OF FISH ACCUMULATES. * >> EASTIEST WAY TO -- THE EASIEST WAY TO AVOID PESTICIDES IN YOUR FOOD IS THOROUGHLY RINSE YOUR PRODUCE BEFORE EATING ANYTHING. PESTICIDES CAN BE VERY EFFECTIVE AGAINST CERTAIN PESTS, BUT YOU WILL NEVER KILL ALL OF THEM. >> WELL, SURE, BUT THAT'S JUST TWO OF THEM. >> OKAY. BUT WHY DO YOU THINK THEY SURVIVE? >> I DON'T KNOW, THEY GO TO YOGA EVERY DAY? >> THEY SURVIVE BECAUSE THERE'S A VARIATION IN THIS POPULATION OF CATERPILLARS, THEY POSSESS SOME TRAIT THAT ALLOW THEM TO SURVIVE BETTER THAN OTHERS, THIS TRAIT THAT MAKES THEM IMMUNE TO THE PESTICIDE CAN LIKELY BE PASSED ONTO THEIR OFFSPRING, SO THAT MEANS THEIR BABIES WILL BE -- >> SUPER AWESOME MEGA CATERPILLARS! >> WELL, MORE LIKELY TO SURVIVE THE SPRAY, YES. SO THAT MEANS -- >> WE NEED MORE SPRAY, MORE TOXIC SPRAY! * >> OKAY. SO IN CONCLUSION, I WOULD LIKE TO THANK A LOT OF PEOPLE WHO PARTICIPATED IN THIS PROJECT BECAUSE IT IS A BIG PROJECT, AND THAT WAS SHOWING NAMES OF OUR WONDERFUL INTERNS WHO WERE MAKING THE ANIMATIONS, BUT THE TRUTH IS THERE WERE A LOT OF OTHER PEOPLE BEHIND THE SCENES, OUR SIS K-12 TEAM AND THEN PEOPLE HELPING WITH TECHNICAL ASPECTS,m.Öj PEOPLE DONATING THEIR VOICES TO ANIMATE THE ANIMATIONS. SO IT WAS A FUN EFFORT AND A LARGE GROUP EFFORT. AND NOW I WOULD LIKE TO PASS THE MICROPHONE TO QUEEN ALIKI WHO IS OUR CURRENT INTERN, SHE'LL TALK ABOUT THE ANIMATION THAT SHE IS DEVELOPING RIGHT NOW. >>> GOOD MORNING. AS DR. KESSMAN MENTIONED, MY NAME IS QUEEN ALIKI, I'M A CURRENT INTERN AT THE UNIVERSITY OF, BACHELOR OF SCIENCE IN COMMUNITY HEALTH, I'M VERY INTERESTED IN NUTRITIONAL EATING BEHAVIORS IN CHILDREN AND ADOLESCENTS AND I'M CURRENTLY DOING AN ARSENIC ANIMATION. I STARTED WITHOUT ANY ANIMATION EXPERIENCE BUT RECEIVED GENEROUS SUPPORT FROM MANY WITHIN THE K-12 GROUP. SOME OF MY FAVORITE ASPECTS ABOUT CREATING ANIMATIONS IS THE ABILITY TO USE MY CREATIVE LICENSE, AS WELL AS JUST PROVIDING EASY TO UNDERSTAND CULTURALLY RELEVANT APPLICABLE HEALTH EDUCATION.  ON SO I CHOSE TO DO THE TOPIC X ARSENIC BECAUSE ARSENIC IN FOOD HAS BEEN IN THE NEWS RECENTLY, PARTICULARLY ARSENIC IN RICE AND APPLE JUICE. I'M VERY PASSIONATE ABOUT NUTRITION, I WANT TO DELVE MORE INTO THE SUBJECT. ALSO AT THE NATIONAL LIBRARY OF MEDICINE IT'S IMPORTANT TO PROVIDE CONSUMERS INFORMATION THAT BLENDS SCIENCE WITH ITS PRACTICAL APPLICATIONS. SO THE FIRST STEP OF THE ANIMATION PROCESS IS DOING BACKGROUND RESEARCH. I'M NOT AN ENVIRONMENTAL HEALTH EXPERT BUT DID KNOW THERE WAS A CONNECTION WITH ARSENIC AND FOOD. I USED MANY OF THE NATIONAL LIBRARY OF MEDICINE'S RESOURCES, SUCH AS ENVIRONMENTAL HEALTH STUDENT PORTAL AND TOX TOWN AND ALSO USED OTHER FEDERAL AGENCIES SUCH AS THE EPA AND FDA. AFTER THE BACKGROUND RESEARCH I USE IT HAD TO CREATE THE STORYBOARD. THE STORYBOARD REALLY BLENDS THE BACKGROUND RESEARCH WITH THE ENGAGING, IT'S A BEAUTIFUL MARRIAGE IF YOU WOULD LIKE TO MAKE THAT ANALOGY. WITH THE STORYBOARD AT THE SAME TIME YOU DO WANT TO BALANCE THAT SCIENTIFIC CONTENT WITH THE ENTERTAINING STORY. AGAIN, IT'S GREAT BECAUSE YOU CAN USE A CREATIVE LICENSE WITH IT, PARTICULARLY WITH CHARACTER DEVELOPMENT. SO THESE ARE THE CHARACTERS OF THE ARSENIC ANIMATION. AND I KNEW WHEN I WAS CREATING THEM THAT I WANTED THEM TO BE INDEPENDENT INQUISITIVE, HUMOROUS ADOLESCENTS, I WANT THERE TO BE A NATURAL DIALOGUE WHERE THEY'RE GOING OFF THEIR BACKGROUND KNOWLEDGE. SO FOR EXAMPLE HERE, THEY'RE TALKING ABOUT ORGANIC AND INORGANIC ARSENIC COMPOUNDS, WHICH THEY MAY HAVE LEARNED IN CHEMISTRY CLASS. I WANTED THE ADOLESCENTS TO BE REALISTIC, SO I GAVE THEM EACH DISTINCT PERSONALITIES. SO AFTER THE STORYBOARD COMES THE ACTUAL ANIMATING PROCESS. SO THE ANIMATION VISUALLY DISPLAYS THE STORYBOARD. WE USE THE TOOL "GO ANIMATE" AND WE PREVIOUSLY USED A FREE TOOL, BUT THE FREE TOOL LACKED THE CAIP ABILITIES THIS WE DESIRED -- THE CAPE THABILITIES WE DESIRED, SO WE SWITCHED TO "GO ANIMATE" WHICH IS LOW COST, MANY FEATURES. MANY FEATURES INCLUDE DIFFERENT CHARACTER AND BACKGROUND SCENES. NEXT COMES AUDIO RECORDING. SOME OF YOU MIGHT BE FAMILIAR WITH THE PERSON HERE, VICTOR SID, SENIOR COMPUTER SCIENTIST AT SIS. WITH THE AUDIO RECORDING IT BRINGSTH ANIMATION TO LIFE WITH REALISTIC VOICES, WITH EMOTIONS AND TONES. SO WITH AUDIO RECORDING YOU RECORD, ED IT, AND THEN SYNC THE AUDIO TO THE ACTUAL ANIMATION. SO I'M DOING ARSENIC ANIMATION, AND IT'S STILL IN PRODUCTION, BUT I DID WANT TO GIVE YOU ALL A PREVIEW, SO IT WILL BE SHOWN IN THE FORM OF A COMIC. SO THE ANIMATION STARTS WITH THE THREE FRIENDS, GLORIA, GRACE AND CALEB. GRACE TELLS HER FRIENDS THAT SHE'S READ ONLINE THAT ARSENIC IS IN APPLE JUICE AND RICE. THE THREE FRIENDS ARE UNSURE SO DECIDE TOIN VEST. GRACE FINDS THE ENVIRONMENTAL HEALTH PORTAL, ARSENIC IS IN THE EARTH, WATER AND AIR. CALEB ALSO POINTS OUT ON THE SITE THAT HE KNOWS ARSENIC IS IN THE ENVIRONMENT THROUGH THINGS LIKE FARMING AND MINING. FROM THERE, THE FRIENDS DISCUSS INORGANIC AND ORGANIC ARSENIC COMPOUNDS. VISUALLY DEMONSTRATING HOW INORGANIC ARSENIC COMBINES AND HOW INORGANIC ARSENIC IS HARMFUL TO HUMAN HEALTH. GRACE PIGGYBACKS OFF OF THE IDEA BY TELLING A REAL ARSENIC1SE. POISONING. SO CHARLES FRANCIS HALL WAS PERFECTLY HEALTHY UNTIL HE DRANK A CUP OF COFFEE ONE DAY AND DIED. NO ONE SUSPECTED FOUL PLAY UNTIL ALMOST 100 YEARS LATER WHEN A BUY OG RAFER -- BIOGRAPHER LOOKED INTO IT. HE DISCOVERED THAT HALL MIGHT HAVE BEEN POISONED BY HIS CREW. THE THREE FRIENDS DISCUSS ARSENIC'S HEALTH EFFECTS. CALEB ASKS HOW IN THE WORLD IS ARSENIC IN FOOD? GRACE CONNECTS THAT BECAUSE ARSENIC IS IN EARTH AND WATER THAT PLANTS MIGHT PICK IT UP THROUGH THE ROOTS AND THAT SOME PLANTS SUCH AS APPLES OR RICE MAY PICK UP ARSENIC MORE READILY THAN OTHERS. FROM THERE, CALEB FINDS THAT THE EPA'S RECOMMENDED AMOUNT OF ARABS NICK IN THE> THE RECOMMENDED AMOUNT OF ARSENIC IN THE WATER IS THIS AMOUNT. [LAUGHTER] IT'S A LONG NUMBER. [LAUGHTER] HE ALSO DISCUSSES THE FDA'S TESTING ON ARSENIC IN APPLE JUICE AND FINDS THAT THE FDA SAYS THAT APPLE JUICE IS SAFE FOR CONSUMPTION. NEXT THEY FIND WHAT FDA SAYS ABOUT ARSENIC IN RICE. FDA'S LEVELS OF ARSENIC IN RICE/+O(v MAKES CALEB HAPPY, HE GOES ON AND ON ABOUT HOW HE'S GOING TO EAT HIS RICE. GLORIA ADVISES CALEB TO PERHAPS EAT DIFFERENT GRAINS TO WHICH HE RECOUNTS A STORY WHERE HE ASKED HIS MOM WHAT WAS FOR BREAKFAST, LUNCH AND DINNER, ALL WITH THE SAME RESPONSE: RICE. WELL, DINNER WAS DIFFERENT. IT WAS RICE AND SO THE MYSTERY HAS BEEN SOLVED AND PUT AT EASE, GLORIA TELLS GRACE THANKS FOR LOOKING OUT FOR US, GRACE, BUT RELAX, AND YOU CAN'T TRUST EVERYTHING YOU REED ON THE INTERNET. YOU CAN'T TRUST EVERYTHING YOU READ ON THE INTERNET. CALEB ASKS IF THE ENVIRONMENTAL HEALTH PORTAL CAN HELP WITH HIS HOMEWORK, SINCE HE STILL HAVMENT DONE IT. GRACE ADVISES THAT HE FINISH IT BEFORE LUNCH ENDS, BUT CALEB INSISTS THAT FIRST LET HIM FINISH HIS RICE. SO DR. TESSMAN AND I ARE HONORED TO PRESENT FOR YOU ALL. WE'RE SO THANKFUL FOR THE OPPORTUNITY TO BE HERE, AND WE WOULD BE HAPPY TO ANSWER ANY QUESTIONS YOU MAY HAVE. [APPLAUSE] >> QUESTIONS? HENRY? >> YES, THANK YOU SO MUCH MORE DOING IT. I THINK IT'S FOR THE LEARNING STYLE OF THE KIDS OF TODAY, AND WE PEOPLE HAVE TO REALIZE THAT YOUNG PEOPLE ARE NOT THE PEOPLE LIKE HOW WE TRAINED, AND WE HAVEb TO TRAIN THEM DIFFERENTLY. AND USING THESE ANIMATIONS BRINGS IN SCIENCE, BRINGS IN ENGLISH, AND MATH, THERE'S MATH RIGHT THERE. YOU'RE TEACHING IT. SO I THINK YOU'VE REALLY BROUGHT ENVIRONMENTAL SCIENCE AND LEARNING TOGETHER VERY ELEGANTLY AND I APPLAUD YOU FOR THAT. QUESTION. >> WE DO DEVELOP IT FOR PARTICULAR SCHOOLS, WE DEVELOP THE RESOURCES AND THEN CONDUCT OUR OUTREACH TRYING TO PROMOTE IT NATIONALLY.A2 >> HAVE ANY TESTS BEEN RUN THAT CONFIRM MAYBE THE ANIMATING IF IT'S MADE A DIFFERENCE IN THE LEARNING? >> IF THERE'S A DIFFERENCE IN LEARNING BETWEEN -- >> THE CONTROL GROUP THAT DID NOT USE ANIMATION AND THE GROUP THAT USED ANIMATION FOR THE SAME TOPIC. >> YOU KNOW, I'M VERY GLAD YOU'RE ASKING BECAUSE WE WERE ALL ACTUALLY JUST BRAINSTORMING THESE THINGS. THIS PROGRAM IS ABOUT A YEAR AND A HALF OLD, AND SO WE WERE DEVELOPING ANIMATIONS GRADUALLY. WE JUST FINISHED APPLAUDING -- JUST FINISHED MAKING ALL OF THEM 503 COMPLIANT, NOW IS THE TIME TO PROMOTE THEM AND TO TEST THEM. SOME OF THEM HAVE BEEN OUT FOR A WHILE, BUT THIS IS THE TIME WHEN ALL OF THEM ARE IN THE PUBLIC DOMAIN. >> EXCELLENT. >> VERYDrœW CREATIVE. PENGUINS, MICROPENGUINS. I THINK I SAW A POLAR BEAR IN THERE. VERY, VERY CREATIVE. I HAVE A QUESTION, THIS IS KIND OF VISIONING FOR THE FUTURE. SO HOW WOULD YOU ENVISION THE ROLE DIFFERENT THAN OTHER AGENCIES TO DEVELOP RESOURCES AND MATERIALS, AND I'M REALLY THINKING ABOUT CONTINUING THE DISCUSSION WE HAD LAST NIGHT AND THE THINKING THAT, YOU KNOW, WHAT WOULD YOU DO IF YOU HAD -- >> MY VISION BASICALLY, AND THIS IS GLOBAL, THIS VISUALIZATION, USING ALL OF THE MASSIVE AMOUNTS OF DATA,¢ VISUALIZE EVERYTHING. THIS IS PART OF THE DISCUSSION THAT WE'VE HAD. AND THAT WOULD BE A COMPONENT OF HOW ARE WE POSITIONING THINGS FOR THE FUTURE. I MEAN, OUR BASIC OBJECTIVE IS PROVIDING INFORMATION AND ALL THESE OTHER THINGS ARE TOOLS, THEY'RE DIFFERENT TOOLS, AND WE HAVE TO BE WISE ENOUGH TO IDENTIFY THE TOOLS THAT ARE GOING TO WORK TO HELP THIS, AND A LOT OF THAT FALLS BY THE WAYSIDE. >> ANYONE ELSE? >> THIS IS REALLY FANTASTIC. WHEN I THINK ABOUT 20 YEARS WHEN WE DID THE FIRST EXHIBITION AND WHAT A BATTLE IT WAS TO GET THE ATTITUDES TO UNDERSTAND THAT IT'S INTERESTING AND ENGAGING TO PEOPLE AND NOT BORING, SO THEY FOLLOW THROUGH, THAT WOULD BE A GOOD THING, AND, YOU KNOW, SOME OF THE MEMBERS WE LITERALLY BATTLED. I JUST THINK YOU GUYS WE'VE GOT TO CONTINUE THIS. IT'S A REAL IMPORTANT MISSION. >> TO PIGGYBACK ON THAT, AMERICA IS FALLING BEHIND IN SCIENCE AND MATH, AND WE HAVE TO FIND WAYS FOR US TO CLOSE THAT GAP BETWEEN US AND OTHER COUNTRIES. IF WE DON'T, WE'RE NOT GOING TO BE THE LEADER THAT WE WERE IN THE PAST. I THINK THESE KINDS OF THINGS WILL HELP MOTIVATE OUR KIDS TO STAY INTERESTED IN SCIENCE AND MATH. OUTREACH, I THINK THAT'S A CRITICAL ELEMENT. I TOLD A STORY IN OTHER GROUPS AROUND THIS TABLE THAT I GOT INTERESTED IN THE LIBRARY OUTREACH, IT WAS THE BOOK MOBILE. BEEN DOING IT SINCE THE BEGINNING OF THE TIME. AND WE HAVE TO REMEMBER THAT, BECAUSE LOOK AT THE OUTREACH OF TODAY, WE HAVE TO CONTINUE TO DO THAT. >> YEAH. THERE WAS A CONVERSATION YESTERDAY ABOUT THE WORD "OUTREACH" WHICH MAYBE IS A LITTLE BIT SOFT AND PEOPLE THINK IT'S PR, BUT I THINK ENGAGEMENT, PUBLIC AND YOUTH ENGAGEMENT IS REALLY WHAT YOU'RE DOING. >> IN THIS CASE, YES. >> WELL, THAT'S WHAT WE WANT. WE WANT TO ENGAGE PEOPLE ACTIVELY, NOT JUST REACH OUT AND PUT IT THERE. >> I WOULD LIKE TO TAKE THE OPPORTUNITY TO SPECIFICALLY THANK STEE -- THANK STEPHEN GAIL, THEY WERE EXTREMELY SUPPORTIVE OF TRYING IT OUT AND MORE THAN TRYING IT OUT. >> ONE OF THE THINGS THAT I SEE IN THIS IS SOMETHING THAT THERE'S NOT A LONG HISTORY IN DOING. WE ARE BOTH A WHOLESALER AND A RETAILER OF INFORMATION, AND ONE OF THE THINGS THAT WE DO WHEN WE PROVIDE ATw WE PROVIDE IT TO A PARTICULAR GROUP OR WE, YOU ) ONLINE ACCESS TO OUR OWN DATABASES, WHICH ARE ALSO AVAILABLE TO ANYBODY ELSE, MOST OF IT, IF THEY WANT TO DO SOMETHING WITH DEMONSTRATE HOW TO TAKE ADVANTAGE OF THE DATA6.sT„ THAT WE HAVE. AND I CAN TELL YOU, I MEAN, FROM THE DAYS OF INTERNET GRATEFUL MED, I THINK ANYBODY ELSE WHO REASONABLY FIGURED OUT HOW TO USE A REASONABLY USEFUL PC INTERFACE TO MEDLINE OR IN DOCTOR'S OFFICES OR IN CLINICS OR THE PUBLIC, THEN WE WOULD NOT HAVE HAD TO DO THAT. AND THERE'S ALSO, WE HAVE A HISTORY OF PEOPLE WHO PROVIDE ACCESS BUT THEY'RE NOT ACTUALLY TAKING ADVANTAGE OF THE INTELLIGENCE THAT'S BEEN INTO THE SYSTEM. WE HAVE THESE DATABASES, WE KNOW ALL THIS STUFF, WE HAVE ALL THESE OTHER THINGS, AND I THINK THAT IN SOME SENSE, PART OF OUR ROLE IS TO GIVE EVERYBODY ELSE A GOOD IDEA OF HOW TO TAKE EFFECTIVE ADVANTAGE OF ALL THE INFORMATION WE HAVE. I THERE DEVELOPING ANIMATIONS LIKE THIS, WOULDN'T WE JUST APPLAUD THEM AND SAY OH GREAT, THEY'RE POINTING TO OUR STUFF, IT'S WONDERFUL. BUT I THINK THAT WE HAVE TO CONTINUE TO HAVE OUR OWN IDEAS BECAUSE WE ACTUALLY UNDERSTAND THE INFORMATION WE HAVE. I MEAN, WE KNOW THE DEPTH AND THE BREADTH OF IT, AND IF WE DON'T DO THIS IN SOME WAYS TO SHOW PEOPLE, THEN WE'VE JUST GOT A LOT OF EXPERIENCE THAT IT WON'T NECESSARILY SPONTANEOUSLY HAPPEN THAT PEOPLE DEVELOP THE TYPES OF INTERFACES AND SPECIALIZED TOOLS THAT WILL REALLY TAKE ADVANTAGE OF WHAT THE TAXPAYER IS INVESTED IN, WHICH IS ESSENTIALLY THE GATHERING OF ALL OF THIS GOOD INFORMATION. >> DO YOU HAVE A COMMENT? >> YES. I JUST HAVE A QUESTION. I DON'T KNOW IF YOU WERE THERE YESTERDAY, WE SAW A VERY MOVING MOVIE THAT WAS MADE HERE ABOUT MICHAEL DEBAKE, AND I'M WONDERING WHETHER THERE'S ANY INTERACTION BETWEEN THE PEOPLE WHO MAKE THAT MOVIE AND THE KINDS OF ANIMATIONS THAT YOU'RE MAKING. I MEAN, IS THERE ELSEWHERE -- IS THERE EXPERTISE ELSEWHERE IN NLM THAT YOU MAYBE HAVE NEVER TALKED TO? >> THERE IS DEFINITELY A LOT OF EXPERTISE THAT WE HAVEN'T TAPPED. THAT'S AN EXCELLENT COMMENT. YES. >> OKAY. WHAT IS THE PRIMARY WAY THAT TEACHERS BECOME AVAILABLE, BECOME AWARE OF THE AVAILABILITY OF THESE WONDERFUL ANIMATIONS? >> SO WE DO A LOT OF OUTREACH TO TEACHERS IN ALL WAYS POSSIBLE. WE PRESENT AT A NUMBER OF NATIONAL CONFERENCES. WE PROMOTE IT THROUGH SOCIAL MEDIA. WE HAVE OUR TEACHER CONTACTS, WE ALSO GO TO LOCAL SCHOOLS, BUT OF COURSE THAT'S A RETAIL. BUT WE ALSO TRY TO -- NOT TRY, WE ALSO PUBLISH IN EDUCATIONAL MAGAZINES AND JOURNALS. >> MY QUESTION WAS, I THINK YOU KIND OF ANSWERED IT, BUT I WAS WONDERING, IS THIS AVAILABLE THROUGH LIKE YOUTUBE, AND I THINK ABOUT, YOU KNOW, THE FUTURE, LOOKING AT IT ON TV AND SINGING THE SONG IN MY MIND, CONJUNCTION JUNCTION. AND IT WAS THE VAIBLG OF -- AND IT WAS THE AVAILABILITY OF THAT. AND I CAN SING IT, BUT I WON'T. [LAUGHTER] >> SO THAT WAY AS WELL. >> IN TWO PLACES. ONE IS THE ENVIRONMENTAL HEALTH STUDENT PORTAL AND ALSO THE NLM YOUTUBE CHANNEL. >> VERY GOOD. >> SO I JUST WANT TO MAKE A COMMENT. THIS HAS BEEN A LONG TRA DIPTION IN THE HISTORY OF -- A LONG TRADITION IN THE HISTORY OF MEDICINE DIVISION, I'M SURE YOU KNOW ABOUT THE TOOTHBRUSH ANIMATION, IT'S LIKE ONE OF DISANY'S FIRST ANIMATIONS -- DISNEY'S FIRST ANIMATIONS, YOU HAVE TO LOOK IT UP IN THE ARCHIVES, IT'S A LITTLE ANIMATED TOOTHBRUSH, IT HAD A VERY CUTE PERSONALITY. SO THIS GOES BACK A LONG WAY, AND WE'RE TAKING IT INTO THE 21ST CENTURY. CONGRATULATIONS. >> THANK YOU. >> THE QUESTION OF GETTING IT VIEWED IS VERY IMPORTANT. THE TOX STUFF IS AT LEAST TEN YEARS OLD AND IT WAS DONE, AS YOU KNOW, INSIDE SIS. AND IT WAS USED IN THIS COUNTY. BUT WE HAVE DATA THAT WAS USED IN THIS PLACE. BUT ALSO IT'S AIMED AT A YOUNGER CHILD, IN JUNIOR HIGH SCHOOL. SO YOU'RE DOING TWO THINGS. ONE IS YOU'RE CHOOSING TO TARGET AN OLDER CHILD, AND SECONDLY IT'S STARTING OUT NATIONALLY. I THINK YOU HAVE TO JUSTIFY BOTH OF THOSE THINGS. I MEAN, STARTING OUT NATIONALLY, SCIENCE, THAT'S WHAT I KNOW, SO I THINK YOU HAVE TO FOCUS ON THE USE OF THESE THINGS, NOT JUST THE QUALITY MUCH THE QUALITY LOOKS TO ME PRETTY NICE, BUT WE HAVE TO GET IT LIKED BY THE SCHOOL, BY THE TEACHERS, BY THE SUPERINTENDENTS OR SOMEBODY IN THE SCHOOL SYSTEM OR IT WILL NEVER GETFcn THE FACT THAT IT'S UP HERE IS NOTHING, THAT DOESN'T GET IT USED IN THE CLASS. THE TEACHERS AND THE SUPERINTENDENTS, THE PRINCIPALS HAVE TO KNOW ABOUT THESE SOMEPLACE, NOT NATIONALLY. SOMEPLACE. DOESN'T HAVE TO BE MONTEREY GOM RI COUNTY -- DOESN'T HAVE TO BE MONTGOMERY COUNTY, BUT I DON'T KNOW WHY YOU WOULDN'T START THERE, THAT'S WHERE WE ARE. >> ABSOLUTELY. WE'RE AT THE BEGINNING OF THIS ROAD. WE ARE HOLDING A FOCUS GROUP WITH MONTGOMERY COUNTY TEACHERS NEXT WEEK, I BELIEVE.%idp„ ALSO WHAT WE'RE TRYING TO DO -- SO THERE WERE TWO TEACH ERPZ IN THE LOCAL SCHOOLS, ONE FROM MONTGOMERY AND ONE FROM WALTER JOHNSON THAT PARTICIPATED IN DEVELOPING THIS ANIMATION, SO THIS IS THE EASIEST SCHOOLS TO GET INTO LOCALLY. AND WE ARE STARTING TO DO THIS. >> I DON'T ACTUALLY KNOW, I MEAN , I DON'T KNOW IF IT'S SUPERINTENDENTS OR PRINCIPALS OR BOARDS OF GOVERNORS OR WHERE THE DECISIONS LIKE THAT ARE MADE. THE SCHOOL THAT CHOOSES THE TEXTBOOK. MAYBE NOT THAT BAD SHAPE AS MARYLAND. >> APPROVED TEXTBOOKS, THERE'S A SYSTEM FOR DOING THAT. AND THERE ARE ONLY FOUR OR FIVE NATIONAL TEXTBOOKS OF MIDDLE SCHOOL ENVIRONMENTAL SCIENCE. THEY ALL HAVE A TEACHER'S GUIDE THAT GOES WITH THEM. IT COSTS NOTHING TO CONVINCE THE EDITOR AND PUBLISHER TO ALLUDE TO THESE IN THE NEXT TEACHER'S GUIDE, IN A HALF DOZEN ENVIRONMENTAL TEXTBOOKS THAT ARE YOUR MARKET. AND YOU CAN DO THE SAME THING WITH THE GRADESCHOOL, TOX TOWN. THAT PUTS IT IN THE HAND OF EVERY TEACHER IN THE COUNTRY TEACHING THE SUBJECT. AND SINCE YOU'RE MAKING THEM AVAILABLE PRO BONO, THERE'S NO REASON FOR THEM NOT TO MAKE ALLUSION TO IT. [LAUGHTER] >> DO THEY COME IN DIFFERENT LANGUAGES? SPANISH? >> NO, NOT YET. >> OKAY. EARLY ON. >> I THINK WE SO HAVE WE SOLVED THAT PROBLEM AND NOW WE'RE MOVING ON? OR WHAT'S THE LOGIC? ZOONG I THE LOGIC IS. >> I THINK THE LOGIC IS, I MEAN, AS I UNDE2wn FOCUS IN THE SCHOOL SYSTEM ON ENVIRONMENTAL SCIENCE AND HEALTH, AND THAT OBVIOUSLY IS ONE OF OUR MAIN AREAS. SO YOU'RE LINING UP OUR INTEREST IN ENVIRONMENTAL HEALTH RESOURCES WITH THE PIECE OF THE K-12 CURRICULUM THAT FOCUSES ON THAT ISSUE. >> LIKE ALL THE BAD STUFF UNDER THE KITCHEN SINK, THAT'S KIDS EATING AND DRINKING THAT KIND OF STUFF AND SHOWING UP IN THE PEDIATRIC WARD, SO THAT IS A JOB THAT'S NOT FINISHED. >> YOU MIGHT THINK OF MAKING PART OF THE WEBSITE OF NLM RESOURCES. >> AND IN FACT, WE ACTUALLY HAVE r A -- WE HAVE BROUGHT TOGETHER RESOURCES FOR TEACHERS AND SO FORTH BECAUSE WE HAVE THEM IN MULTIPLE PLACES. A LOT OF THEM ARE COMING OUT OF THE ENVIRONMENTAL HEALTH PORTAL AND SO FORTH AND WE ALSO AS MANY PEOPLE KNOW HAVE SOME EXCELLENT ONES THAT ARE CONNECTED TO SPECIFIC EXHIBITIONS THAT HAVE BEEN DEVELOPED WITH INTERACTION BETWEEN THE HISTORY OF MEDICINE DIVISION AND TEACHERS AND STUDENTS AS WELL. SO WE DO ACTUALLY HAVE A PLACE. MAYBE WE HAVE A NEED TO TAKE ANOTHER LOOK AT IT AND MAKE SURE THAT IT IS AS ACCESSIBLE AND WONDERFUL AS IT COULD BE. THERE ARE THE NATIONAL ASSOCIATIONS OF STEM TEACHERS AND WE DO A FAIR AMOUNT OF INTERACTION WITH THEM. I THINK DALE'S SUGGESTION ABOUT THE TEXTBOOKS IS FABULOUS. WE REALLY SHOULD PURSUE THAT. >> I WAS JUST GOING TO ADD, THIS IS NOT JUST A ONE-TIME PROGRAM. IT'S AN EVOLVING EPIC, AND IT'S VALUE ADDED BECAUSE WE HAVE STUDENTS AND INTERNS ROTATING THROUGH, JUMPING IN TO ASSIST THIS PROGRAM WITH TEACH ERB'S. THERE'S ALSO A SECONDARY INTEREST BEYOND PEOPLE AND THE SCIENCES -- SECONDARY INTEREST TO YOUNG PEOPLE AND THE SCIENCES, AND I HEAR A LOT OF COMMENTS FROM THE STUDENTS SAYING, YOU KNOW, I THINK I WANT TO GO INTO ENVIRONMENTAL HEALTH, I THINK I WANT TO CHANGE MY MAJOR. >> I THINK THAT'S A GREAT SEGWAY TO WHAT I WAS GOING TO COMMENT ON. ONE IS MORE OF A QUESTION, HOW DO YOU GET FEEDBACK FROM INDIVIDUALS AND TEACHERS WHO USE THIS THAT AREN'T LOCAL? A MEAN, THOSEU I THE OTHER ONE IS LOOKING AT OUTCOMES, WHAT KIND OF OUTCOMES ARE OCCURRING SEXUAL HARASSMENT WHAT KIND OF OUTCOMES ARE OCCURRING AS A RESULT OF THIS? SOME OF THEM ARE GOING TO BE LONG-TERM. I THINK IT WILL BE CHALLENGING, BUT I THINK THAT WILL CHALLENGE THE ARGUMENT, LOOKING AT THE OUTCOMES, AND WHAT'S THE FEEDBACK? >> I THINK YOU'RE ASKING EXCELLENT QUESTIONS, AND AGAIN, BECAUSE WE'RE AT THE VERY BEGINNING OF THIS PROGRAM, WE'RE JUST STARTING TO THINK ABOUT AND TO LOOK AT THAT. ONE METHOD OF COURSE IS TO LOOK AT THE VIEWS, SO WE CAN LOOK AT THE NUMBER OF VIEWS ON THE YOUTUBE CHANNEL AND ON OUR PORTAL, AND THAT'S VERY IMPRESSIVE, ACTUALLY. BUT BEYOND THAT, WE DEFINITELY WANT TO LOOK AT THE LEARNING OUTCOMES AND WORK WITH TEACHERS MORE, SO WE'RE STARTING OUR FOCUS GROUPS. WE WOULD LIKE TO DEVELOP SOMETHING WHERE IT'S USED IN THE CLASSROOM AND WE COLLABORATE WITH TEACHERS ON TRACKING SOME LEARNING MEASURES. >> AND I WOULD SAY NOT JUST LOOKING AT ONE-TIME USE, IT'S THE REPETITIVE USE, THEY LIKE IT, AND THEY'RE USE IT GO AGAIN, THEY FOUND IT VALUABLE -- AND THEY'RE USING IT AGAIN, THEY FOUND IT VALUABLE. >> ENVIRONMENTAL HEAL+ >> YES. >> THANK YOU.œ [APPLAUSE] >> THANK YOU. [APPLAUSE] TRUD TRUD. >> Dr. Trudy MacKay: SO ANYONE WHO HAS TIME TO STICK AROUND AND TALK ABOUT THEIR RESPONSE TO THIS INFORMATION. ANYONE THAT HAS TO GO CAN GO, BUT I THINK IT'S NICE THAT WE'RE ALL HERE, SINCE WE'RE ALL HERE, IF WE HAVE TIME TO SAY A LITTLE. >> RIGHT HERE. >> RIGHT HERE, YEAH. [LUNCH RECESS]