>> GOOD MORNING. I'M CAROL CHRISTIAN AND I'M GOING TO INTRODUCE OUR FIRST SPEAKER TODAY, GEORGE SANTANGELO WAS CALLED TO A MEETING WITH THE NIH DIRECTOR AND HE WILL BE A FEW MINUTES LATE. IT'S MY PLEASURE TO INTRODUCE OUR FIRST SPEAKER TODAY, DR. KATY BORNER OF INDIANA UNIVERSITY, THE PROFESSOR OF INFORMATION SCIENCE P AT THE SCHOOL OF LIBRARY AND INFORMATION SCIENCE, AT JUNGT PROFESSOR AT THE SCHOOL FOR INFORMATICS AND COMPUTING. ADJUNCT PROFESSOR DEPARTMENT OF STATISTICS IN THE COLLEGE OF ARTS AND SCIENCES. CORE FACULTY OF COGNITIVE SCIENCE, MEMBER OF ADVANCED VISUALIZATION LABORATORY AND FOUNDING DIRECTOR OF THE CYBER INFRASTRUCTURE FOR NETWORK SCIENCE CENTER AT THE INDIANA UNIVERSITY. SHE'S VISITING PROFESSOR AS ROYAL NETHERLANDS OF ARTS AN SCIENCE, CURATOR OF INTERNATIONAL PLACES AND SPACES, MAPPING SCIENCE EXHIBIT. SHE HAS AN MS IN ELECTRICAL ENGINEERING FROM THE UNIVERSITY OF TECHNOLOGY AND LIVE -- AND A Ph.D. IN COMPUTER SCIENCE FROM UNIVERSITY OF KEISER. SHE IS A FELLOW IN THE AMERICAN ASSOCIATION FOR ADVANCEMENT OF SCIENCE. SO WITH NO FURTHER ADIEU, PLEASE COME FORWARD. THANK YOU VERY MUCH. [APPLAUSE] >> GOOD MORNING, LADIES AND GENTLEMEN. I HOPE YOU ENJOY THE MAPS OUTSIDE IN THE NATCHER AUDITORIUM, THERE ARE 70 MAPS OF SCIENCE ON DISPLAY THERE. ARE ACTUALLY INCLUDED AN IMAGE, PHOTO FOR THOSE WHICH JOIN US REMOTELY. THESE MAPS HAVE BEEN GENERATED OVER THE LAST SEVEN YEARS, A TEN YEAR ITERATION EFFORT TO BRING KNOWLEDGE OF MAP SCIENCE TO A GENERAL AUDIENCE, NOT ONLY FROM ONE TO THE THE NEXT BUT ALSO TO PUBLIC LIBRARY, TO SCIENCE ACADEMIES, TO PUBLIC SPACES. THE TALK TODAY I HAVE PREPARED WILL TRY TO COMMUNICATE THAT THESE MAPS ARE GENERATED USING VERY DIFFERENT DATA SETS, VERY DIFFERENT WORK FLOWS OF PRE-PROCESSING THE DATA, CLEANING THE DATA, CLEANING THE DATA, CLEANING THE DATA, DOING MORE CLEANING OF THE DATA AND THEN GOING ON TO MINING THE DATA AND LAYING THEM OUT, VISUALIZING THEM, THEY ARE ALSO DESIGNED FOR QUITE DIFFERENT USER GROUPS, A MAP OF SCIENCE DESIGNED FOR CHILDREN, THIS IS SIX YEARS ITERATION OF THE EXHIBIT, SCIENCE MAPS THAT ARE TRULY INTERESTING TO CHILDREN. THOSE MAPS WILL BE VERY DIFFERENT FROM MAPS THAT ARE DESIGNED FOR ECONOMIC DECISION MAKERS OR FROM MAPS THAT ARE DESIGNED FOR RESEARCHERS OF SCIENCE POLICY MAKERS. THE EXHIBIT ALSO HAS INTERACTIVE ELEMENTS, YOU HAVE PUZ MAPS TO CHILDREN, YOU HAVE ILLUMINATED GLOBES, THESE ARE CALLED ILLUMINATED DIAGRAMS SET UP WHICH I WILL SHOW YOU MORE LATER ON. IT GOES TO MUSEUMS AND UNIVERSITIES. HERE YOU SEE A DISPLAY AT THE STANFORD UNIVERSITY. IN THE EXHIBIT OF DIVERSE MAPS ONE IS SHOWN HERE, IT'S GENERATED BY DICK KLAVANS AND KEVIN BOYACK IN THIS ROOM THAT IS UNIQUE FOR THESE PRESENTATIONS, A NUMBER OF MAP MAKERS ARE ALL HERE, I FEATURE THOSE MAPS BECAUSE YOU CAN LATER GO TO THEM AND ASK THEM ABOUT ALL THE DETAILS. I WON'T BE ABLE TO DO THIS ALL IN MY TALK HERE. IT'S A VISUALIZATION OF 7.2 MILLION SCHOLARLY DOCUMENTS FROM APPEARING IN OVER 16,000 JOURNALS WITHIN A FIVE YEAR TIME SPAN. THESE JOURNALS ARE ASSIGNED TO 554 SUB DISCIPLINES OF SCIENCE WHICH ARE AGGRAVATED INTO 13 LARGER DISCIPLINES WHICH ARE HERE, COLOR CODED AND LABELED. THE OTHER TWO JOURNALS ARE SIMILAR TO EACH OTHER, LIKELY TO BE IN THE SAME SUB DISCIPLINE AND DISCIPLINES AND SUB DISCIPLINES WHICH ARE SIMILAR TO EACH OTHER AGAIN, THEY ARE PUSHED AND PULLED TOWARD EACH OTHER AND THOSE NOT SIMILAR, THEY ARE PUSHED APART. AND THERE ARE MANY DIFFERENT MEASURES THAT CAN BE USED TO GENERATE A MAP OF SCIENCE AND ONE VERY SPECIFIC ELABORATE PROCESS WAS USED HERE. IF YOU HAVE A FIVE YEAR DATA SET YOU CAN ALSO LOOK AT THE PUSH AND PULL OF THOSE CONTINENTS OF SCIENCE IF YOU WISH, HOW THEY ARE ACTUALLY ATTRACTING AND REPELLING EACH OTHER, IF YOU WISH. THIS IS DEMONSTRATED DOWN HERE. IN THESE DIFFERENT -- IN THESE DIFFERENT OVERLAYS. JUST LIKE A MAP OF -- IT HELPS YOU TO OVERLAY POLITICAL BOUNDARIES, LETS YOU SHOW YOUR TRAVELS ACROSS THE GLOBE, LETS YOU IDENTIFY WHAT MINUTE L RALES ARE TO BE FOUND. YOU CAN USE MAP OF SCIENCE TO OVERLAY OTHER DATA AND THIS IS WHAT MY TALK IS ALSO ABOUT. BEFORE THIS I WOULD LIKE TO LET YOU KNOW THAT THIS MAP YOU JUST SAW RECENTLY UPDATED TO CAPTURE THE LAST TEN YEARS OF MAP OF SCIENCE AND SCOPE OF DATA. NOW YOU HAVE 25,000 JOURNALS CAPTURED AND THESE SUB DISCIPLINES PLACED ON THE SURFACE OF THE SPHERE, THIS SPHERE IS THEN MAPPED TO TWO DIMENSIONS USING A PROJECTION. NODE IS#G EXTENSIVE LIST OF JOURNAL NAMES AN KEY PHRASES OF METADATA USED TO SCIENCE LOCATE JOURNALS, IF YOU HAVE JOURNAL DATA. THANKS TO KEY WORDS YOU CAN LOCATE NON-JOURNAL DATA THAT HAPPENS IN GRANTS. AGAIN, IF YOU HAVE FOR INSTANCE ALL YOUR PUBLICATIONS IN A NOTE FILE OR IN A TABLE, YOU CAN READ IN THIS TABLE AND OVERLAY YOUR CAREER O TRAJECTORY OVER THIS LANDSCAPE OF SCIENCE. YOU CAN DATE IT AND YOU KNOW HOW OFTEN YOU HAVE TO GO BACK TO AN OLE AREA OF SCIENCE BECAUSE THEY STILL WANT YOU BACK FOR KEYNOTES. THOUGH YOU HAVE GONE ON TO DO NEW THINGS. THE UCSD MAP OF SCIENCE IS USED IN DIFFERENT SERVICES. SOME WHICH I WILL SHOW YOU. HERE YOU HAVE A MAP INTERFACE WHICH HELPS SUSTAINABILITY RESEARCHERS TO GET AN UNDERSTANDING WHAT PAPERS PATENTS GRANTS EXIST AND WHO IS WORKING ON THEM, THE VIVO FOR FACEBOOK ALSO SEARCHES IT AND HE WILL SEVERE'S COMMERCIAL PRODUCT IS USING A CIRCULAR MAP BASE MAP TO HELP PEOPLE UNDERSTAND STRENGTH AND OPPORTUNITIES. ANOTHER TYPE OF MAP AS SHOWN YESTERDAY BY BOLIN, THIS IS BASED ON CITATION DATA BUT DOWNLOAD ACTIVITY DATA. HERE YOU WILL SEE THAT FOR INSTANCE CLINICAL RESEARCH OCCUPIES A MUCH LARGER AREA OF SCIENCE BECAUSE YOU HAVE LOTS OF PEOPLE WHICH BENEFIT DEEPLY FROM PUBLICATIONS MADE AVAILABLE VIA MED LINE TO USE IN DOCTOR'S OFFICES TO SAVE LIVES BUT U THEY MIGHT NOT PUBLISH A PAPER AND THEREFORE WOULDN'T FIGHT OTHER PAPERS. YOU HAVE HERE THE USAGE OF SCIENCE THAT'S WHAT REALLY GETS AT THE QUESTION OF OUTPUT AND OUTCOME. MAPS CAN ALSO SHOW CYCLES OF SCIENCE, FEEDBACK LOOPS AN DELAYS IN THE SCIENCE SYSTEM. WE ARE IN DESPERATE NEED TO UNDERSTAND BETTER WHAT THOSE FEEDBACK CYCLES ARE AND WE WANT POSITIVE FEEDBACK CYCLES, NOT THE NEGATIVE ONES. AND WE WANT TO POSSIBLY REDUCE BUT I THINK THEY'RE ALSO THERE'SSYMES IT -- TIMES IT TAKES TO DEVELOP RESEARCH IDEA OR MAKE A GOOD PRODUCT. WHAT YOU SEE HERE IS A MAP THAT ACTUALLY MADE IT AS THE YELLOW ARROW GRAPH ON THE RIGHT HAND SIDE, AND WAS THEN REVISED FOR THE EXHIBIT TO BE LEGIBLE TO MUCH LARGER AUDIENCE BY THE COUNCIL FOR CHEMICAL RESEARCH. THERE ARE TWO REPORTS BEHIND THIS MAP. THE MAP IS ONE PICTURE BUT THERE'S A LOT OF WORK IN THE BACKGROUND. IN ORDER TO GET THE RIGHT DATA, IN ORDER TO GET THE INSIGHTS. WHAT YOU SEE HERE IS THAT THERE'S ONE BILLION FEDERAL FUNDING WHICH GOES INTO CHEMISTRY BASIC RESEARCH, FOUNDATIONAL RESEARCH. THAT 1 BILLION IS MATCHED BY THE CHEMICAL INDUSTRY, 5 BILLION OF INDUSTRY FUNDING. WHICH THEN RESULTS IN DOLLAR 10 BILLION CHEMISTRY OPERATING INCOME AND THEN VIA GROWTH AND GNP AND JOBS CREATED RESULTING IN $8 BILLION OF TAXES. AND OUT OF THOSE 1 BILLION AGAIN IS SPENT BY FEDERAL FUNDING. FOR CHEMISTRY RESEARCH. BELOW THESE NUMBERS YOU ALSO HAVE THE TIME SPANS WHICH TYPICALLY TAKE TO DO FOUNDATIONAL RESEARCH. HERE IN THIS CASE THIS PARTICULAR AREA OF RESEARCH IS FOUR TO FIVE YEARS. INVENTION DEVELOPMENT IS NINE TO ELEVEN YEARS AND TECHNOLOGY COMMERCIALIZATION IS FIVE YEARS. SO 20 YEARS HAVE TO PASS BEFORE YOU GET FROM THE INITIAL IDEA TO A PRODUCT THAT SELLS AN PROVIDES PRODUCTS AND INCOME. IF AS A FUNDING OFFICER YOU WOULD LIKE TO HAVE SOMETHING WITHIN TWO YEARS THAT GOES FROM IDEA TO A PRODUCT, THAT MIGHT ACTUALLY NOT BE FEASIBLE. YOU EITHER START PICKING PROJECTS WHICH ARE DOWN HERE OR YOU GET TO A CERTAIN DEGREE DOWN THIS PIPELINE. THERE ARE DELAYS IN THE SYSTEM AND I THINK IMPORTANT FOR US TO UNDERSTAND THAT. IN ANOTHER WORK BY (INAUDIBLE) PRESENTED YESTERDAY WE CAN GET A DEEPER UNDERSTANDING OF THE EMERGENCE OF A FIELD HERE IN NANOTECHNOLOGY. DIFFERENT JOURNALS ARE REPRESENTED HERE IN THOSE TIME SLIDES WHICH GO FROM 1998 TO 2003. I SHOULD POINT OUT ALL THESE MAPS ARE 30 BY 24-INCH, 300 BPI, SOMETHING WHICH THOSE PROJECTORS CANNOT POSSIBLY REPRODUCE HERE. BUT YOU SEE THE DIFFERENT TIME SLICES PASSING BY. SCIENCE IS THE PINKISH PURPLISH DOT, NATURE IS ORANGE, THE NANOTECHNOLOGY JOURNAL IS GREEN AND THE NANOLETTERS MUCH MORE PRESTIGIOUS IN BLUE. SO AS YOU -- AS THE FIELD EVOLVES IN THE BEGINNING THERE ARE NO JOURNALS DIRECTLY ON NANOTECHNOLOGY AND THOUGH MUCH RESEARCH IS PUBLISHED IN SCIENCE AND NATURE AND LATER ON IN ALSO NANOTECHNOLOGY. BUT WHEN THE JOURNAL GETS CREATED A LOT OF HIGH QUALITY RESEARCH IS PUBLISHED THERE SO YOU SEE THAT BETWEEN THE CENTRALITY AND THE DEGREE OF THOSE FOUR JOURNALS CHANGES OVER TIME CONSIDERABLY AND ALSO THE KIND OF OFF THE FEEL IN TERMS OF JOURNALS AND HOW THEY ARE USED TO COMMUNICATE RESULTS. YOU CAN ALSO USE FUNDING DATA TO CREATE MAPS OF SCIENCE. HERE NET TALLY, CHOCK LABS AND DAVE KNEWMAN AT UC IRVINE CREATED A TOPIC MAP OF NIH GRANTS FROM 2007. HERE YOU DON'T HAVE DOWNLOAD ACTIVITY, YOU DON'T HAVE CITATION ACTIVITY, YOU HAVE ABSTRACTS THAT YOU CAN USE TO DO TOPICAL ANALYSIS. YOU HAVE DOLLAR AMOUNTS, YOU KNOW OUT OF WHICH IC THEY WERE FUNDED, ET CETERA. THE TOPIC MODELING TO USE TO CLUSTER THOSE PROJECTS TOGETHER THAT HAVE SIMILAR TOPICS ASSOCIATED WITH THEM. AND YOU CAN THEN DO OVERLAYS OVER THIS BASE MAP YOU SEE HERE IN THE MIDDLE IN TERMS OF DIFFERENT ICs THIS IS NIGMS AND OTHERS. ZOOM INTO CERTAIN AREAS, AND YOU CAN GET AN UNDERSTANDING OF WHAT RESEARCH IS REPRESENTED BY AN AREA. THIS IS ALSO TO LEARN THE BASE AS YOU MIGHT REMEMBER WHEN YOU FIRST LEARNED THE MAP OF OUR WORK, IT TOOK A WHILE UNTIL YOU REALIZED AS LITTLE THE SCANDINAVIAN, ET CETERA. JUST AS YOU LEARN THE IMPORTANT MAP OF THE WORLD IT'S IMPORTANT TO LEARN THE BASE MAPS OF SCIENCE OR AT SOME POINT JUST ONE SO INTERPRET DATA OVERLAYS. THE MAP IS ALSO AVAILABLE ONLINE FOR THOSE OF YOU WHICH ARE AVAILABLE -- WHICH -- WIDOW HAVE INTERNET ACCESS. YOU MIGHT GO TO NIHMAPS.ORG AND START BROWSING AND SEEING YOUR PORTFOLIO IN NEW WAYS. YOU SEE AN INTERACTIVE BROWSER THAT ALLOWS YOU TO ENTER KEY WORDS TOPIC WORDS THAT SHOWS YOU WHERE THAT RESEARCH LAYS. YOU CAN ALSO SEE RELEVANT TOPICS, RELEVANT GRANTS, YOU CAN GO ON TO ADDITIONAL PAGES WHICH SHOW YOU THE INVESTIGATORS AND INSTITUTES WHICH FUND CERTAIN TYPES OF RESEARCH. SO IT'S IF I GO OVER HERE, I CAN ACTUALLY NOW AFTER IT COMES UP. PROCESS MAPS, WE HAVE A BASE MAP OF SCIENCE, YOU CAN DO OVERLAYS EXPERTISE AREAS IF YOU WISH AND THEke FUNDING OF DIFFER ENT INSTITUTES AT NIH. YOU CAN ZOOM INTO THIS MAP AND GET MORE DETAILS. HOW MANY OF YOU HAVE BEEN USING THIS MAP BEFORE? SHOW OF HANDS? INTERESTING. I WOULD BE VERY INTERESTED FOR YOUR FEEDBACK ON IT. DOES THIS HELP YOU MAKE DECISIONS? IF NOT, WHY NOT? WHAT DO YOU NEED IN IN YOU ARE YOUR DATA DECISION MAKING TO FACILITATE CERTAIN TYPES OF ANALYSIS TO GIVE YOU THE INSIGHTS YOU NEED IN YOUR WORK. THE EXHIBIT YOU SEE OUTSIDE IN A SMALL POSTER FORMAT ALSO TRAVELS WELL IN TRAINS SO IF YOU HAVE A TRAIN STATION YOU CAN COLLABORATE WITH THE MAX PLANK SOCIETY THEY'RE VERY INTERESTED TO GET THIS TO COME TO THE U.S.. IT'S A GREAT WAY TO COMMUNICATE WHAT NIH OR IN THIS CASE HERE MAX PLANK SOCIETY HAS BEEN DOING FOR SOCIETY. YOU HAVE TO MAKE SURE NO EXHIBIT IN THIS TRAIN IS MUCH MORE INTERESTING THAN ALL OTHERS. BECAUSE THEN IT'S A BOTTLENECK. WHAT YOU WANT IS A FLOW OF PEOPLE CONTINUOUSLY THROUGH ALL THE DISTANCE COACHES. HERE YOU SEE THE ILLUMINATED DIAGRAM WHICH IS A SETUP WHERE YOU HAVE A MAP OF SCIENCE AND MAP OF THE WORLD AND TOUCH PANEL DISPLAY. YOU CAN SELECT ANY AREA ON THE MAP OF THE WORLD AND SEE WHAT KIND OF RESEARCH IS CONDUCTED ON THE MAP OF SCIENCE USING THESE OVERLAYS YOU JUST SAW ON THE NIH MAP. YOU CAN ALSO SELECT ANY AREA IN THE MAP OF SCIENCE SEE WHO IS CONDUCTING THIS RESEARCH OVERLAID ON THE MAP OF THE WORLD. THE RECENTLY UPDATED DISPLAY TO HAVE THE UCSD MAP OF SINCE AT THE BASE MAP. YOU ALSO NOW HAVE A KEYBOARD WHERE YOU CAN ENTER YOUR OWN NAME TO SEE YOURSELF IN BOTH MAPS. YOU CAN USE PEOPLE BUTTONS AN BUTTONS FOR MORE INTERDISCIPLINARY AREAS TO SEE THEM BOTH OVERLAID OVER BOTH MAPS. AND THESE PATTERNS. THE TALK ENTITLES MULTI-LEVEL SCIENCE MAPS. YOU HAVE BEEN SEEING MAPS THAT ARE ONE LEVEL EXCEPT FOR THE LITTLE (INAUDIBLE) INTO THE NIH MAP. IN MANY CASES PEOPLE WOULD LIKE TO GET A GLOBAL YEFER VIEW FIRST THEN MOVE INTO DETAILS AN FILTER OUT CERTAIN YEARS, CERTAIN TOPIC AREAS, CERTAIN PIs, BUT ALSO CLICK ON ONE OF THOSE MANY DOTS AND GET MORE DETAILS ON DEMAND. HERE I HAVE THREE EXAMPLE FOR YOU WHICH YOU'RE WELCOME THE PLAY WITH, ALL THESE SYSTEMS ARE AVAILABLE ONLINE, WE'RE INTERESTED IN FEEDBACK, IF YOU WANT TO KNOW THOSE COSTS THEY'RE READY FOR DEPLOYMENT AND YOU CAN HOPEFULLY ENVISION HOW THEY WOULD LOOK LIKE IF YOU FEED IN YOUR OWN DATA. ONE OF THE SERVICES I'M GOING TO BRING TO YOUR ATTENTION IS THE FACEBOOK FOR RESEARCHER. IT'S ACTUALLY A $12 MILLION PROJECT FUNDED OUT OF NIH AND THE IDEA IS TO CREATE AN INFRASTRUCTURE FOR THE NATION AND ULTIMATELY HOPEFULLY INTERNATIONALLY, WHERE INSTITUTIONS SUCH AS INDIANA UNIVERSITY OR NIH CAN DOWNLOAD A FREE PIECE OF SOFTWARE, INSTALL ON THEIR CAMPUS, CONNECT TO HUMAN RESOURCE DATABASES, CONNECT TO SPONSORED RESEARCH DATABASES, COURSE CREDIT DATABASES NOT RELEVANT FOR NIH BUT VERY RELEVANT FOR UNIVERSITIES. CONNECTED TO SCHOLARLY WORK DATABASES FOR EVERY PERSON ON YOUR CAMPUS, YOU HAVE INFORMATION ON THE ACTIVITIES. SOME UNIVERSITIES EVEN SO FAR TO CONNECT TO FACULTY SUMMARY REPORTS. THIS GIVES YOU HIGH QUALITY DATA WHICH CAN BE RENDER AS WEB PAGES BUT ALSO EXPOSED TO THE SEMANTIC WEB SO THAT ANYBODY CAN DOWNLOAD IT BUT ALSO MACHINE READABLE FORMAT SO YOU CAN ACTUALLY BUILD SERVICES ON IT. WHAT YOU SEE HERE IS THE COVERAGE, THE CURRENT COVERAGE OF DIFFERENT SYSTEM TYPES ACROSS THE U.S. AND A SIMILAR SYSTEM CALLED (INAUDIBLE) EXPERT. THAT IS THE HARVARD CATALYST PROFILE SYSTEM WHICH HAS VERY NICE FEATURES ALSO. IT'S NOW DEPLOYED IN DIFFERENT UNIVERSITIES, YOU HAVE THE STANFORD SYSTEM, WHICH IS STANFORD AND YOU HAVE THE VIVO NATIONAL RESEARCHER NETWORK DEPLOYED AT DIFFERENT INSTITUTIONS. THESE SYSTEMS HAVE DIFFERENT HOLES IN TERMS OF PEOPLE, PUBLICATIONS, PATTERN, FUNDING AND COURSES. SOME DECIDED TO LOAD FUNDING DOLLAR AMOUNTS. SOME HAVE ACCESS TO CITATION DATA AND LOADED IT. OTHER VERSUS NOT DONE THIS. IF YOU GO TO NIHEDU YOU CAN ZOOM INTO CERTAIN AREAS, YOU CAN CLICK ON THOSE SYMBOLS ON WEB PAGE AND YOU GETTER MORE INFORMATION ON WHAT THEY ACTUALLY LOADED TO THE DETAIL THAT YOU GET TO PEOPLEnp WEB PAGE AND YOU SEE THEIR COURSES, PAPERS, FUNDING LISTED. THIS FOR THE FIRST TIME GIVES US THE MICROLEVEL DATA WHICH IS CONNECTED TO PEOPLE TO THE ACTIVE ELEMENTS IN THE SCIENCE SYSTEM. NOT PASSIVE HE WILLN'TS SUCH AS PAPERS AND THEIR CITATIONS TYPICALLY STUDIED IN (INAUDIBLE) METRICS. WHAT I'M INTERESTED IN IS TO UNDERSTAND CONNECTIONS OF PEOPLE AND THEIR CO-PI SHIP, CO-INVENTORSHIPS AND THEY ARE ALSO ACTIVITIES IN TERMS OF FOR INSTANCE TEACHING. IF YOU GO TO ONE OF THE (INAUDIBLE) INSTANCES YOU GET DIFFERENT TYPES OF VISUALIZATION, ALL THESE VISUALIZATIONS HAVE TO BE UNDERSTANDABLE WITHOUT YOU READING A MANUAL. HERE OFF MAP OF SCIENCE WITH OVERLAYS OF PUBLICATION ACTIVITY BY THE UNIVERSITY OF FLORIDA. WHAT YOU CAN DO IS GO DOWN THE ORGANIZATIONAL HIERARCHY FROM THE UNIVERSITY TO THE DIFFERENT COLLEGES TO THE DEPARTMENT TO THE CENTERS IN THE DISCIPLINARY ONES FOR INSTANCE, SINGLE FACULTY MEMBERS AND YOU WILL GET TO THE SEE THE NUMBER OF PAPERS PER YEAR, NUMBER OF P GRANTS PER YEAR IF CITATIONS OR DOLLAR AMOUNTS ARE LOADED YOU CAN SEE THOSE AS WELL. THEIR COLLABORATION NETWORKS, CO-PI AND CO-MENTORSHIP NETWORK AND ALSO THE EXPERTISE PROFILES OVERLAID OVER THE MAP OF SCIENCE. IN ANOTHER WORK WHICH I ALREADY BRIEFLY MENTIONED WE TRY TO GIVE SUSTAINABILITY RESEARCH, AN OVERVIEW WHAT TYPES OF FUNDING EXIST. WHAT PUBLICATIONS WHAT PUBLICATIONS CAME OUT AND PATENTS EXIST. WE TALK BIOFUEL AN BIOMASS RESEARCH. WE LOADED DATA FROM 7 SOURCES OVERLAID ON THE MAP OF U.S. IN THIS CASE AND OVERLAID ON MAP O SCIENCE. YOU'RE WELCOME TO EXPLORE THOSE SYSTEMS AND POTENTIALLY ENVISION YOUR OWN DATA IN THERE. YOU CAN SWITCH BETWEEN A LOT AMOUNT OF FUNDING OR SIMPLE COUNT FOR PROJECTS. YOU HAVE CITATION COUNTS OF COUNT OF NUMBER OF PAPERS AND CITATIONS ACCOUNT FOR PATTERNS ALSO. YOU CAN RESTRICT THE YEAR RANGE AND SEARCH BY KEY WORDS. YOU ALSO SEE SYMBOLS WHICH ARE DIAMONDS FOR PATTERNS WHICH ARE PILES OF MONEY, TRIANGLES AND SQUARE PAPERS. YOU CAN ZOOM INTO CERTAIN AREAS OF INTEREST FOR YOU. HERE IS THE STATE LEVEL F. YOU ZOOM FARTHER DOWN YOU GET CITY LEVEL. YOU CAN SEARCH FOR INSTANCE FOR AND THE RESULT OF THIS RESEARCH IS IN THE MIDDLE OF THE COUNTRY. IF YOU SEARCH FOR OIL IT'S MOSTLY ON COASTLINE AND YOU CAN SEARCH MORE INTERESTING TERMS AS WELL. IF YOU ARE INTERESTED TO THEN UNDERSTAND HOW MANY PUBLICATIONS, FOR INSTANCE COME OUT OF CERTAIN STATE, YOU CAN HOVER OVER THE SYMBOL AND YOU WILL GET TO SEE THE DETAILED RECORD INFORMATION ON THE RIGHT HAND SIDE AND YOU CAN CLICK ON THEM AND YOU END UP IN FRONT OF THE INFORMATION BRIDGE WHICH CAN THEN LET YOU READ ABOUT THAT PARTICULAR PAPER. YOU CAN GO OVER TO THE MAP OF SCIENCE. AND HOVERING OVER THE SYMBOL YOU GET MORE INFORMATION ON TYPES OF FUNDING THAT EXIST HERE IN BIOLOGY. YOU CAN CLICK ON IT TO GET LISTING OF ALL OF THOSE ON THE RIGHT HAND SIDE, YOU CAN ZOOM INTO THE 554 SUB DISCIPLINES AN CLICK ON THE ICONS FOR MORE DETAIL. YOU CAN SELECT FOR INSTANCE ONE OF THE PATENTS AND THIS BRINGS YOU TO THE USPDO WEBSITE OFF THAT PARTICULAR PATENT SO YOU CAN GO FROM TOP DOWN TO THE CREATE DATA SETS RECORDS WHICH ARE BEHIND THIS MAP. WE RECENTLY P DEVELOPED A SIMILAR INTERFACE, SIMILAR IN FUNCTIONALITY FOR GENE THERAPY. HERE WE ALSO ADDED CLINICAL TRIALS DATA SO YOU NOW HAVE FOUR ICONS SO THERE'S A LIMIT OF HOW MANY DIFFERENT TYPES OF DATA YOU CAN ULTIMATELY HAVE. IN THIS PARTICULAR CASE YOU HAVE FOUR DIFFERENT TYPES FUNDING PUBLICATION PATENT AND CLINICAL TRIALS DATA. AGAIN FEEL FREE TO EXPLORE IT. AND AGAIN F YOU NOW CLICK ON A CLINICAL TRIALS RECORD, YOU WILL GO TO THE CLINICALTRIALS.GOV DATABASE. I WANT TO ALSO TELL YOU THAT YOU CAN DO SOME OF THESE ANALYSIS VISUAL SAIGS USING YOUR OWN DATA, RUNNING YOUR OWN ANALYSIS, AND IDENTIFY OVERLAP GAPS AN AND E MRNLGING AREAS IN YOUR OFFICE. THEN YOU HAVE TO INTERPRET RESULTS. THE SCIENCE OF SCIENCE TO -- THERE WILL BE A TUTORIAL THIS AFTERNOON, FUNDED BY JULIA LANES SCIENCE OF SCIENCE INNOVATION PROGRAM AT NSF. IT ALLOWS YOU TO DO NETWORK VISUALIZATIONS TO DO THE SCIENCE THAT OVERLAYS YOU HAVE JUST SEEN TO DO HORIZONTAL BAR GRAPHS WHICH YOU CAN FOR INSTANCE USE TO MAP THE ENTIRE PORTFOLIO OF NIH. EVERYTHING. YOU HAVE THESE, ONE FOR EACH PROJECT. ON THE LEFT-HAND SIDE YOU HAVE TEXT LABEL, YOU HAVE THE AREA SIZE OF THE BAR ENCODED BY A NUMBER, THE TOTAL AMOUNT. AND YOU HAVE A START AN ENDING POINT. YOU CAN COLOR CODE THOSE BARS BY ADDITIONAL ATTRIBUTES, FOR INSTANCE MALE OR FEMALE LEAD PIs OR ICs OR TYPES OF PROJECTS FUNDED, ET CETERA. IT SCALES VERY WELL, MOST OF THOSE VISUALIZATIONS ARE WRITTEN INTO POSTSCRIPT FILE AN THEREFORE YOU MIGHT GET RATHER LARGE MAPS IF YOU'RE INTERESTED. YOU CAN DO GEO MAPS, HIERARCHICAL ANALYSIS OF COLLABORATION NETWORKS WHICH ARE REPRESENTED FOR INSTANCE AT THE CIRCLE OF HIERARCHIES IN THE LOWER RIGHT. IT COMES WITH EXTENSIVE DOCUMENTATION, TWO YEARS AGO I GAVE 12 TUTORIALS, 12, TWO HOUR TUTORIALS OF 24 HOURS TUTORIALS RECORDED STILL AVAILABLE HERE. THOSE WHICH ARE PART OF NIH HAVE AN EASIER WAY TO GET TO THEM. IF YOU GO TO THE SIDE 2, OUR WEBSITE YOU WILL FINE EXTENSIVE TUTORIALS HOW TO RUN THESE ANALYSIS. YOU BASICALLY TAKE YOUR HAND AND TELL EXACTLY WHAT STEPS YOU HAVE TO GO THROUGH TO GET TO YOUR INTEREST -- TO THE ANALYSIS AND VISUALIZATION YOU'RE INTERESTED IN. THE TOOL IS USED BY DIFFERENT FUNDING AGENCIES HERE IN THE US. ALSO GAVE TUTORIAL IN OACD AND ANOTHER ONE IN BERLIN AND GERMANY. AND MULTIPLE FUNDING AGENCIES ARE USING THIS TOOL TO DO SOMETHING WHICH THEY COULDN'T DO BEFORE. THEY CAN COMPARE RESULTS. TYPICALLY FUNDING AGENCIES CANNOT SHARE INTERNAL DATA HOLDINGS. NOW THEY HAVE A TOOL AND DOCUMENTED WORK FLOWS THAT THEY CAN APPLY THEY CAN GET TO RESULTS THAT CAN BE COMPARED ACROSS DIFFERENT FUNDING AGENCIES. THE TOOL REGISTRATIONS COME FROM 73 COUNTRIES AND MANY DIFFERENT PROFESSIONS, AND IS FASCINATING TO READ WHAT THESE PEOPLE TRY TO DO AND WANT TO DO WITH TOOL. THERE'S ENORMOUS NEED TO GET MORE ADVANCED DATA ANALYSIS AND VISUALIZATION TOOLS INTO THE HANDS OF PROFESSIONALS BUT ALSO INTO THE HANDS OF DECISION MAKERS. AND PROFESSIONAL DECISION MAKERS. THE TOOL USES A DISTINCTION OF TYPES OF LEVELS OF ANALYSIS AND LEVEL -- TYPES OF ANALYSIS AN LEVELS OF ANALYSIS WHICH HOPEFULLY ALSO LETS YOU DIVIDE UP THE SPACE OF POSSIBLE WORK FLOWS. IT SPORTS ANSWERING WHEN WHERE WHAT AND WITH WITH WHOM QUESTIONS, SUCH AS THE FIRST COLUMN THERE. IN ADDITION TO STATISTICAL ANALYSIS, THERE IS ALSO A BRIDGE TO ALL AVAILABLE AS A TOOL. YOU GET TO RUN TEMPORAL ANALYSIS, GEOSPATIAL ANALYSIS, BENEFITING MUCH FROM WORK IN GEOGRAPHY AND CARTOGRAPHY. LINGUISTIC ANALYSIS AND NETWORK ANALYSIS USING NEW ALGORITHMS AND EXISTING AL GOR RITES FROM SOCIAL SCIENCES AND INFO METRICS. YOU CAN RUN THESE ANALYSIS AT MULTIPLE LEVELS AT THE MICROLEVEL WORKING WITH ONE TO 100 RECORDS, A SMALL PORTFOLIO, YOU CAN RUN AT LARGER LEVELS, 100 TO 10,000 RECORDS. YOU CAN RUN AT THE GLOBAL LEVEL ALL OF SCIENCE ALL OVER THE WORLD AND THAT'S RELEVANT IF YOU WANT TO UNDERSTAND GLOBAL BRAIN DRAINS OR OVERLAYS OF ACTIVITY PATTERNS, OVERLAID OVER A MAP OF SCIENCE. OVER THE YEARS WE HAVE DONE A NUMBER OF ANALYSIS AN I WANT TO SHOW YOU THEM HERE. THESE TYPES OF ANALYSIS CAN BE SORTED INTO THE GRID WHICH YOU HAVE HERE AND NOW. OBVIOUSLY YOU CAN DO EITHER CENTRIC ANALYSIS. SO YOU TAKE ONE PI AN DOWNLOAD ALL OF HIS OR HER GRANTS. HERE YOU SEE ME IN THE MIDDLE WITH ALL THE FUNDING I HAD BETWEEN 2001 AND 2006. AND ALL THE COLLABORATORS WHICH ARE PARTICIPATING IN THOSE PROJECTS. THE FUNDING HERE IS COLOR CODED BY THE YEAR. AND THE YOU CAN ALSO DO THIS FOR PUBLICATIONS AND YOU CAN ALSO DO IT FOR PEOPLE AND WHAT FUNDING THEY ARE ACTUALLY BENEFITING FROM. YOU CAN DO A LOT OF BIMODAL NETWORKS IF YOU'RE INTERESTED. YOU CAN TAKE A CO-OCCURRENCE NETWORKS. HERE APPEARING ON THE SAME PAPER, AND YOU CAN ANIMATE THEM OVER TIME. THIS IS ONE CONFERENCE OUT OF THREE IN THE INFORMATION VISUALIZATION TOPIC AREA. EACH NOTE REPRESENTS ONE ULCER. ULCER INCREASES WHEN THEY COLLABORATE. THEY INCREASE IN SIZE IF THEY PUBLISH MORE PAPERS OVER TIME. THE COLOR CODING OF NODES IS BASED ON NUMBER OF CITATIONS THEY HAVE MANAGED TO ACCUMULATE. AND AS YOU SEE THERE ARE SOME NODES WHICH ARE HIGH DENSITY INTERCONNECTED SO THIS IS A MANY ALPHA PAPER WHICH CAME INTO INTIS TENSE AN RESULTS IN A FULLY -- EXISTENCE AND RESULTS IN A CONNECTED PEAK. YOU CAN (INAUDIBLE) WORKING CLOSE. THIS IS INCOMING YUT GOING STREAM OF STUDENTS AND YOU HAVE NATIONAL LABS LIKE -- THAT ACTUALLY MANAGE TO HAVE MULTIPLE RESEARCHERS IN THE SAME AREA WHICH YOU WOULDN'T FIND AT THE UNIVERSITY. SO TYPICALLY EACH UNIVERSITY HAS ONE PROFESSOR WITH INFORMATION VISUALIZATION BUT YOU HAVE MUCH DIFFERENT COLLABORATION NETWORKS. AND WHAT YOU WANT IN HERE IS A LARGE NODE WHICH IS RATHER DARK. SO VINCE SCHNEIDERMAN TOOK THE MAP, FRAMED IT, DRAW A SCAT OVER HERE, ONE OF MY HEROES IN INFORMATION VISUALIZATION. HE PUBLISHED A LOT OF PAPERS RECENTLY SO NOBODY HAD A CHANCE TO ACTUALLY CITE HIM. HE ALSO SEES THAT MANY OF THOSE NETWORKS ARE CONNECTED. OFTENTIMES ONLY ONE PERSON WHO WORKED AT BOTH INSTITUTIONS WITH THE CONNECTING POINT OR WORKING WITH SOME OF IN THIS COLLEAGUES. SO THESE NETWORKS ARE STILL VERY LOCAL, VERY MUCH CONFINEDDED TO CERTAIN GEOLOCATIONS. I EXPECT TO SEE THIS IN OTHER AREAS OF SCIENCE ALSO. WE ALSO COLLABORATED WITH NCI TO IDENTIFY DIFFERENCES IN COMMONALITIES OFF DIFFERENCE TYPES OF FUNDING SO HERE YOU HAVE ONE INVESTIGATOR INITIATED RESEARCH PROJECT EACH LABELED BY P ID AND COLOR CODED IN A DIFFERENT WAY. AND YOU HAVE P TURK FUNDING, BIG CENTERS RECEIVING FUNDING FOR RESEARCH BUT ALSO FOR TRAINING POST DOCS FOR RUNNING WORKSHOPS, FOR CREATING INFRASTRUCTURE, ET CETERA. SO THESE NUMBERS UP HERE MIGHT BE HARD TO COMPARE BETWEEN RO-1s AND CENTERS BECAUSE I THINK CENTERS HAVE DIFFERENT FUNCTIONALITY AND SERVICES TO FULFILL. BUT IF YOU LOOK AT THE CO-AUTHORSHIP NETWORKS WHICH YOU CAN DO BETWEEN THE FUNDING AND THE PUBLICATIONS YOU CAN START DRAWING COLLABORATION NETWORKS BASED ON PUBLICATIONS THAT ACKNOWLEDGE CERTAIN FUNDING. AND WHAT YOU SEE IS NCI THESE NETWORKS ARE MUCH MORE -- MUCH LESS CONNECTED, WHEREAS THIS ONE IS NICELY INTERCONNECTED. IF YOU ARE INTERESTED IN EFFECTIVE COMMUNICATION OF NEW IDEAS THEY TRAVEL FASTER IN HERE. HOWEVER I WOULD ALSO ASSUME THAT ONE VISITOR FROM TIME TO TIME VISIT ONE OF THESE CENTERS AND THEREFORE -- BUT IT'S VERY INTERESTING TO LOOK AT WHAT NETWORKS EVOLVE GIVEN DIFFERENT TYPES OF FUNDING. THE OFFICE STARTED TO COLLABORATE AND WRITE UP WORK WITH ROBIN VONGER AND HER TEAM AT THE REPORTING BRANCH AND WE TOOK PUBLICATIONS AT THE NIH AND WE STARTED TO OVERLAY OVER A MAP OF SCIENCE AND LOOK AT THE EVOLUTION OVER TIME. THIS IS WORK IN PROGRESS BUT YOU MIGHT LIKE TO WATCH OUT FOR THE RESULTS. YOU CAN ALSO LOOK AT THIS ACTIVITY. SO SOME AREAS YOU MIGHT UNDERSTAND CERTAIN KEY TERMS OUT OF NOWHERE ARE WIDELY USED SO THEY EXPERIENCE SUDDEN INCREASE IN FREQUENCY. HERE WE USE THE DETECTION ALGORITHM TO IDENTIFY THOSE 50 WORDS WHICH EXPERIENCE INCREASE BUT ALSO HAS BEEN IN WIDESPREAD USAGE. WE USE THE 20 YEAR DATA SET OF PNAS PUBLICATIONS, AND WHAT YOU SEE HERE ARE THE 50 TERMS EACH LABELED WITH THE TERMS ITSELF. THE CIRCLE SIZE IS THE FIRST WEIGHT SO HOW MUCH INCREASE WOULD SUDDENLY EXPERIENCE. THE CIRCLE COLOR IS THE FIRST ONSET AND THE RING COLOR IS THE YEAR OF THE MAXIMUM WORD COUNT. THE COLOR CODE IS OVER HERE. YOU SEE TYPICALLY THE INTERIOR OF THE CIRCLE IS LIGHTER THAN THE CIRCLE RING MEANING IT REALLY DID FIRST AN EXPERIENCE WIDESPREAD USAGE. WOULD BE INTERESTING TO FOLD INTO 2002 INTO TODAY DATA AND SEE GROWTH IN PROTEIN AN MODELS AN GENE EXPRESSION REGULATION IS REALLY WIDELY USED IN THAT TIME FRAME. THERE IS A CERTAIN PREDICTABILITY IN HERE AND IT WOULD BE INTERESTING TO RUN THIS TYPE OF ANALYSIS OVER OTHER DATA SETS. THESE DATA HAVE TO HAVE A TIME STAMP, YOU CAN IMAGINE RUNNING THIS RUNNING TO A BOX OR RUNNING IT DOWNSTREAM. THESE TUTORIAL IS ALSO USED AT NATIONAL SCIENCE FOUNDATION. THEY WERE INTERESTED TO ANALYZE THE 2,085 COGNITIVE NEUROSCIENCE PROJECTS FUNDED BETWEEN 2007 AND 2011. THEY USE THE NIH MAP TO IDENTIFY THESE AN PROPOSES BECAUSE THEY ARE NOW USING THIS TOOL INTERNAL TO NSF. THEY HAVE ACCESS TO PROPOSALS WHICH PROBABLY I WOULD -- I HAVE ACCESS TO. BUT GIVEN THESE TOOLS ARE FREELY AVAILABLE THESE TYPES OF ANALYSIS BECOME POSSIBLE. AS YOU READ ON EACH IS UP TO FOUR TOPICS AND THEN THE LINES REPRESENT CO-OCCURRENCE OF CONNECTED TOPICS WITHIN AN AWARD. THEN YOU CAN SCALE THE NODES AND THEY'RE SCALED BY THE NUMBER OF AWARDS AND LINES ARE SCALED BY NUMBER OF CO-OCCURRENCE. IF YOU LOOK AT THE MAP YOU GET AN OVERVIEW THESE AREAS OF RESEARCH INTERRELATE TO THE EACH OTHER. IF YOU WOULD LIKE TO KNOW MORE ABOUT THIS ANALYSIS, INTERESTED IN FEEDBACK BUT ALSO AVAILABLE TO ANSWER QUESTIONS. ULTIMATELY THEY IDENTIFY NEW WAY OF CHARACTERIZING AN UNDERSTANDING THE NSF PORTFOLIO. THEY WERE ABLE TO ANALYZE THE CONTENT OF AWARDS AN PROPOSALS INDEPENDENTLY AND YOU CAN COMPARING WHAT WAS PROPOSED, WHAT WAS FUNDING AND WHAT WAS THE DIFFERENCE IN TOPIC COVERAGE BETWEEN THE TWO. BUT THEY WERE ALSO INTERESTED TO IDENTIFY AREAS OF PARALLEL OR POTENTIALLY COLLABORATIVE RESEARCH FUNDED BY DIFFERENT INSTITUTIONAL STRUCTURES. SIMILAR QUESTIONS MIGHT ALSO APPEAR HERE AT NIH. NOOA ALSO STARTED TO USE THE TOOL AND VERY INTERESTED TO LOOK AT PUBLICATION SUPPORTED BY OFFICE OF OCEAN EXPLORATION AN RESEARCH. HERE YOU HAVE A TOPIC NETWORK. YOU SEE HOW THESE DIFFERENT NODES ARE COLOR CODED BY DIFFERENT TOPICS. AND TO HIGHLIGHT THE CLUSTERING BASED ON NUMBER OF COLLABORATIONS BETWEEN ALPHAS. THE ALPHA NAMES ARE OMITTED HERE. BUT IF YOU WANT TO READ MORE ABOUT IT OR TALK WITH CHRIS, HE'S ALSO AVAILABLE. I THINK IT'S IMPORTANT THAT IN D.C. BUT LATER INTERNATIONAL, THERE'S A COHORT OF PEOPLE EVOLVING THAT USE THESE TOOLS AN WHICH CAN HELP EACH OTHER AND WHICH CAN CAN BENEFIT FROM EACH OTHER'S EXPERTISE. THE JAMES MCDONALD FOUNDATION ALSO IS USING THESE TOOLS TO LOOK AT AND TRACE PROSPECTIVELY THE DEVELOPMENT OF A FIELD WHICH THEY HAD INVESTED QUITE HEAVILY IN AND THEY WANTED TO SEE WHAT IMPACT THEY'RE -- THEIR FUNDING HAD ON THE FIELD. AND (INAUDIBLE) DID THIS WORK AND THERE'S PUBLICATION THAT YOU CAN ACCESS AND YOU CAN CONTACT HIM FOR DETAILS. VERY NEW WORK LETS YOU INTERACTIVELY EXPLORE THE EXPORTER REPORTER DATA IN NEW WAYS WHICH ALSO IS VISUALIZATIONS YOU HAVE JUST SEEN. IN A UNIQUE COLLABORATION OF OUR TEAM AS THE CYBER INFRASTRUCTURE FOR NETWORK SCIENCE CENTER AT IU AND NETTA HERE IN D.C. WE STARTED TO DEVELOP NEW INTERFACES FOR THE REPORTER DATA. AND NETTA DID A GOOD JOB TO MAKE IT OAZIER TO ANSWER THESE WHEN, WHERE, WHAT AND WITH WHOM QUESTIONS. THEY ALSO BACK WITH AN INTERFACE THAT LETS YOU GO ONE, TWO, THREE, SO YOU FIRST CHOOSE THE DATA SET, CHOOSE ANALYSIS THEN YOU HIT THE VISUALIZATION BUTTON. THERE'S NUMBER OF WORK FLOWS EASY TO USE WHICH WE BELIEVE ARE ALREADY SHOWING THE UTILITY AND THE POTENTIAL BEHIND THESE TYPES OF VISUAL INTERFACES FOR DIGITAL LIBRARIES, TO A LARGER AUDIENCE AND IF YOU'RE INTERESTED TO EXPLORE IT, I WOULD BE HAPPY TO CONNECT YOU TO THE TEAM. WE ARE VERY INTERESTED IN YOUR FEEDBACK. IT'S ONLY VIA THE LOOP OF FEEDBACK TO US THAT WE CAN ACTUALLY IMPROVE THOSE TOOLS. HERE YOU CAN CHOOSE THE DATA SET FROM REPORTER OR EXPORTER. YOU CAN THEN CHOOSE AN ANALYSIS AND ALSO ENTER FRAMES AND YOU CAN VISUALIZE IT FOR INSTANCE USING THESE HORIZONTAL BAR GRAPHS OR USING A MAP OF THE U.S. OR USING THE MAP OF SCIENCE. OR USING THE BIMODAL VISUALIZATIONS OF NETWORKS WHERE YOU HAVE FOR INSTANCE PIs AND PROJECTS AND THEY'RE ALL INTERRELATIONSHIPS. HERE IS A CIRCLE AREA, IS THE TOTAL AMOUNT FOR THE PIs BUT ALSO FOR THE PROJECT. THOSE OF YOU WHICH CAME HERE TO LEARN MORE ABOUT IDENTIFICATION OF EMERGING RESEARCH AREAS WHICH I THINK THE NEXT SPEAKER ALSO GO INTO DETAILS IN, WE DID RESEARCH IN LAST YEAR, MAYBE PROPOSED THREE INDICATORS IN COMBINATION TO TRY TO GET AT EMERGING RESEARCH AREAS. ONE SUDDEN INCREASE IN FREQUENCY OF SPECIFIC WORK, JUST LIKE THE BIRTH ANALYSIS I SHOWED YOU. OTHER IS A NUMBER AND SPEED WHICH NEW AUTHORS ARE ATRACKED TO AN EMERGING AREA. IF YOU HAVE PUBLICATION DATA YOU CAN EASILY IDENTIFY HOW MANY UNIQUE ALPHA NAMES ARE APPEARING IN YOUR DATA SET EACH YEAR AND THEIR ISSUES TRANSMITTED, ET CETERA BUT YOU CAN STILL SEE IT'S A NUMBER OF NEW AUTHORS COMING DOWN. BUT ALSO CHANGES IN THE DISCIPLINARITY OF CITED REFERENCES. IN THE BEGINNING OF THE FIELD, THERE ARE TYPICALLY PAPERS WHICH HAVE RESEARCH OR PAPERS HAVING IN YOUR FIRST TIME SLIDE MIGHT COLLECTIVELY CITE DI RER RESEARCH. A TZ FIELD MATURES IT CAN CITE ITSELF BUT IT'S NOT POSSIBLE IN THE FIRST TIME SLIDE. SO OVER TIME YOU WILL SEE MORE AN MORE DISCIPLINARY CITATIONS TO WORK. AND BASED ON THIS YOU CAN TRY TO IDENTIFY WHAT STAGE OF LIFE IF YOU WISH THE RESEARCH FIELD IS IN. WE USED IN A DATA SET OF PNAS PUBLICATIONS BUT ALSO METRICS PUBLICATIONS. AND DID TRY TO TRACK FOR DIFFERENT FIELDS USING KEY WORDS. IT WAS INTERESTING TO SEE THE APPEARANCE OF NEW AUTHORS WAS REALLY A VERY STRONG INDICATOR RESEARCH AREA IS EMERGING. WE HAVE SUDDEN INCREASE IN DIVERSITY OF CITED REFERENCES FOR THREE OF THE FOUR AREAS. WITH APPEARANCE OF NEW AUTHORS, HAPPENING SIMULTANEOUSLY. IT WAS INTERESTING TO SEE THAT MAJOR KEY WORDS DID BURST LATER. IT TAKE AS WHILE BEFORE THE COMMUNITY DEVELOPS THE SAME LANGUAGE BECAUSE THEY'RE ALL COMING FROM DIFFERENCE AREAS OF VERGE AND THEY MIGHT USE DIFFERENCE WORK TO REFER TO THE VERY SAME CONCEPT. THAT MIGHT BE A REASON YOU DONE HAVE THESE BURSTS IN THE VERY BEGINNING OF THE TIME DEVICES. I WANTED TO LEAVE YOU WITH A GENERAL COMMENT ON MAPS OF SCIENCE. SO I HAVE BEEN SHOWING YOU THE UCSD MAP OF SCIENCE, ALSO AVAILABLE A CIRCLE MAP AND LATEST ALSO MADE AVAILABLE AND ALAN PORTER MIGHT TEACH ABOUT IT, THIS SCIENCE MAP OF I CATEGORIES WHICH CAN BE USED TO DO OVERLAYS OF DATA. THERE IS (INDISCERNIBLE) SOME OF YOU MIGHT USE, IT GIVES YOU KIND OF A HEAT MAP OF DIFFERENT TOPIC AREAS. THERE'S SCIENCE METRICS, A MAJOR EFFORT TO ACTUALLY HAVE MULTIPLE LANGUAGE SCIENCE ONTOLOGIES UP IN MONTREAL. AND THE NIH MAP I SHOWED. THERE ARE MANY AGENCIES I BELIEVE OF NSF WORKING ON THIS TO GET THEIR OWN TOPIC MAP FAIR GRANT PORTFOLIO. THERE ARE AGENCIES IN EUROPE WHICH LEARNING FROM NIH ALSO ARE IN THE PROCESS OF CREATING THEIR OWN TOPIC MAPS. THE MORE SCIENCE MAPS YOU HAVE THE MORE YOU NEED TO LEARN. IF YOU WOULD LIKE TO COMPARE TOPIC MAP OVERLAYS DONE USING AN NIH MAP THIS TOPIC MAP OVERLAYS USING NSF MAP WHICH YOU DONE SEE HERE, IT WILL BE HARD BUZZ THEY BOTH USE A VERY DIFFERENT BASE MAP. IF YOU TAKE NIH FUNDING AND OVERLAY OVER A GENERAL MAP OF SCIENCE, IT WILL BE EASY TO COMPARE BECAUSE YOU CAN GO BACK AND FORTH BETWEEN THE DIFFERENCE OVERLAYS JUST LIKE FOR THE DIFFERENT INSTITUTES ON THE INTERACTIVE NIH MAP. SO IT WILL DEPEND ON THE -- WHICH YOU NEED THESE MAPS, IF YOU DECIDE TO CREATE A MAP OF YOUR OWN PORTFOLIO, OR IF YOU TAKE A GENERAL MAP OF SCIENCE, ONE OF THOSE, TO OVER LAY YOUR PORTFOLIO SO THAT YOU CAN COMPARE IT WITH OTHER AGENCIES SO THAT YOU CAN IDENTIFY QOAFER LAPS, GAPS, POTENTIAL COLLABORATION OPPORTUNITIES. ULTIMATELY WHAT WE SHALL DO RESEARCH PROJECT THAT I DON'T HAVE FUNNING FOR. VERY INTERESTING IN GETTING THEM TO ALIGN THESE DIFFERENCE MAPS OR SUBSET OF THEM SO YOU CAN DO CROSSWALKS IF YOU WISH. THERE ARE OTHER SCIENCE AN TECHNOLOGY ENGINEERING ONTOLOGIES SUCH AS DELIVERY SUCH AS MESH, SUCH AS THE ACN CLASSIFICATION HIERARCHY AND OTHERS WHICH ARE RELEVANT HERE. YOU WANT TO TAKE DATA IN THOSE DATABASE, ORGANIZED IN THESE ONTOLOGIES TAXONOMIES CLASSIFICATION HIERARCHIES AND OVERLAY THEM ON THOSE MAPS. SO YOU CAN SEE ON A MAP NOT ONLY WHERE FUNDING GOES BUT WHERE PUBLICATIONS AND PATENTS COME OUT, WHERE SCIENCE NEWS ARE, WHERE CERTAIN GRAD STUDENTS ARE PRODUCED, WHERE DROPS ARE OPENING THIS YEAR AND LAST YEAR, ET CETERA. IF YOU HAVE SO MANY DIVERSE MAPS IT'S HARD. SO HAVING THESE CROSS WALKS IS VERY IMPORTANT. IF YOU ARE INTERESTED TO SEE THE FORMULAS WHICH ARE BEHIND THIS, I HAVE REVIEW ARTICLES WITH ME BUT ALSO ONLINE SO PLEASE FEEL FREE TO DIG INTO THE MASS AND TO NOT ONLY ENYOU THE TOOLS BUT ALSO HELP US IMPROVE THE APPROACHES WE'RE USING BY GIVING US FEEDBACK IN TERMS OF HOW USEFUL THESE SERVICES, THESE TOOLS ARE REALLY FOR YOUR DECISION MAKING. THERE'S ALSO A NEW BOOK ON MODELS OF SCIENCE DYNAMICS FORESHADOWING RESEARCH ON NOT ONLY ANSWERING WHEN, WHERE, WHAT AND WITH WHOM BUT ALSO TRYING TO ANSWER WHY QUESTIONS, WHY DO WE HAVE THESE DELAYS IN THE SCIENCE SYSTEM, WHY DO WE HAVE CERTAIN STRUCTURES IN THE SCIENCE SYSTEM. IF YOU WOULD LIKE TO READ A REVIEW OF MAJOR MODELS, AGENT BASED MODEL, STOCHASTIC MODEL, GAME THEORETIC MODELS, THAT'S A GOOD WAY TO START THIS. THERE IS RESEARCH AT NIH FUNDED TO STUDY CAREER TRAJECTORIES AN MODEL THEM THAT'S ALL I HAD FOR YOU TODAY IN THE MORNING. IN THE AFTERNOON THAT'S A A TUTORIAL. THANK YOU FOR YOUR ATTENTION AND I I'M HAPPY TO ANSWER QUESTIONS. [APPLAUSE] >> I WAS INTERESTED IN LOOKING AT THE BIG BUCKET THAT YOU PUT DIFFERENT SCIENTIFIC DISCIPLINES IN. HOW DO YOU DECIDE WHAT THOSE ARE? IS THERE CONSENSUS? SEEMS THE DIFFERENT MAPPING TOOLS GROUP THINGS DIFFERENTLY WHICH CAN CAUSE PROBLEMS. >> SOME OF THE MAPS YOU SAW TODAY ARE DRIVEN BY THE DATA, THEY'RE PURELY DATA DRIVEN INCLUDING THE U UCSD MAP OF SCIENCE. OTHER MAPS ARE DRIVEN USING EXISTING TAXONOMIES, CLASSIFICATION HIERARCHIES SO IF YOU HAVE A CLASSIFICATION HIERARCHY YOU CAN TAKE THE ROOT NODE AS YOUR MAP OF SCIENCE. THE LEVEL CATEGORIES DIFFERENT CONTINENT, SECOND LEVEL COUNTRY, ET CETERA. DEPENDING ON WHAT DATA YOU USE, DEPENDING ON WHAT WORK FLOW YOU USE, YOU MIGHT END UP HAVING VERY DIFFERENT MAPS OF SCIENCE. ONE MAJOR RESEARCH GOAL WHICH I SEE FOR OUR SCIENTIST MATRITIONS AND ALSO DATA MINING VISUALIZATION EXPERTS IS TO DESIGN THE BEST MAP OF SCIENCE WHICH HAS TO BE LOCALLY AN GLOBALLY ACCURATE AND KEVIN BOYACK AND DICK KLAVANS HAVE RUN A NUMBER OF NAILSIS TO GET TO THE BEST MAP WHICH IS ARE HIGHLY ACCURATE LOCALLY THROUGH ALSO GLOBALLY AND THEY STARTED TO USE HUMAN SUBJECTS, EXPERTS WHICH ACTUALLY UNDERSTAND DIFFERENCE AREAS OF SCIENCE, MAYBE NOT ALL OF SCIENCE BECAUSE THERE'S NOBODY OUT THERE WHO WOULD KNOW ABOUT ALL OF SCIENCE. TO VALIDATE THOSE MAPS. WE HAVE ALSO RUN DIFFERENCE TYPES OF ANALYSIS WHERE WE LOOK AT DIFFERENT SIMILARITY MEASURES AN COMPARED THEM AND IDENTIFIED THE ONE THAT LEADS TO THE BEST, DIFFERENCE CRITERIA THERE. WE HAVE RUN STUDIES WHICH COMPARE DIFFERENT LINKAGE BASED AN TEXT BASED ALGORITHMS IF YOU STILL NEED CITATION LINKAGES. IF YOU DONE THEN YOU DON'T HAVE TO BUY ALPHA (INAUDIBLE) SCIENCE DATA. IF YOU CAN JUST USE OPEN FREE MED LINE DATA AND ANYBODY CAN REPLICATE THOSE STUDIES WHICH IS WHAT YOU WOULD LIKE TO SEE. THAT DOESN'T REALLY ANSWER YOUR QUESTION HOW YOU GOT TO THOSE 13 BECAUSE I WOULD HAVE TO EXPLAIN IN DETAIL OR KEVIN AND DICK WOULD HAVE TO, WHAT DATA WENT INTO THE MAP AND HOW THIS PROCESS WORKS IN DETAIL WHICH LED TO THOSE 13 LARGE SCALE DISCIPLINES. >> THANKS, I REALLY ENJOYED YOUR PRESENTATION. ONE INHERENT FEATURE OF MAPS IS THAT THEY EITHER ARE VERY RETROSPECTIVE OR AT BEST LOOKING AT THINGS THAT ARE GOING ON RIGHT NOW. IS THERE A -- IN YOUR EXAMINATION OF TRENDS THAT YOU FELT YOU COULD MAKE PREDICTIONS, CAN YOU LOOK AT THE GAPS IN SCIENCE AND SAY WE FILLED THAT IN? HAVE YOU SEEN ANYTHING THAT WOULD GIVE ANY ADVICE ABOUT HOW WE WOULD APPROACH FUTURE INVESTMENTS IN SCIENCE? SINCE OUR GAMBLE IS ON WHAT WILL PRODUCE GREAT RESULTS GOING FORWARD. IS THERE ANY HINT FROM MAPS ABOUT THAT? >> YOU'RE CORRECT. THE MAPS GENERATED BASED ON PUBLICATION AND CITATION DATA SHOW PAST ACTIVITY, TWO, THREE YEARS BACK IN TIME BECAUSE IT TAKE AS WHILE BEFORE THE CITATIONS ACCUMULATE. YOU ALSO CORRECT THESE MAPS SHOW WHERE FUNNING GOES. IF YOU PUT FUNDING IN AN AREA, THERE'S PEOPLE HIRED, THEY PUBLISH OR PERISH. THESE PAPERS HAVE TO GIVE AWAY CITATION COUNTS, THE CURRENCY OF SCIENCE. AND MAPS LIKE THE ONING ON FLOOR USE DOWNLOAD ACTIVITY ACCOUNTS ARE REALLY INTERESTING BECAUSE THEY SHOW HOW SCIENCE IS USED. IN GENERAL I WOULD ARGUE 80% OF SCIENCE IS NORMAL SCIENCE AND YOU CAN PREDICT THIS NORMAL SCIENCE. THIS IS ENOUGH TO HIRE THE MANY POST-DOCS OR APPLY IN SO MUCH RESEARCH. TYPICALLY BASED ON WHAT WAS PREVIOUSLY DONE IN THAT AREA, YOU CAN KIND OF PREDICT HOW MANY PAPERS ARE GOING TO COME OUT BASED ON WHERE THOSE PAPERS ARE TYPICALLY PUBLISHED, BY THE DISTRIBUTION, THEY WILL GIVE A WAY SO MANY CITATION COUNTS, THESE HAVE TO GO SOMEWHERE, THESE ARE PROBLEMISTIC MODELS. WHAT YOU'RE INTERESTED IN IS THE 20 OTHER PERCENT, THE NON-NORMAL SCIENCE. WHICH ARE IMPACTED BY STIMULUS FUNDING WHICH ARE IMPACTED BY SCIENCE POLICY DECISIONS, IMPACTED BY BREAK THROUGH RESULTS WHICH ENABLE OTHER RESEARCH TO COME INTO EXISTENCE AND CREATE SWEET SPOTS WHERE IDEAS MAKE TO PRODUCTS. THOSE ARE MUCH HARDER TO PREDICT. HOWEVER, GETTING AT THE 80% WOULD BE VERY INTERESTING AND SOME OF THE APPROACHES WE HAVE HEARD YESTERDAY, THESE IN VIVO EXPERIMENTS WHERE YOU HAVE ONE SET OF PROJECTS GETTING ONE TREATMENT OR PIs GETTING TREATMENT OR OTHER SET OF PIs GETTING OTHER TREATMENT ARE VERY INTERESTING. THERE'S ALSO A LOT OF OPPORTUNITY IN COMPARING DIFFERENT FUNDING MECHANISMS ACROSS DIFFERENT COUNTRIES BECAUSE SOME COUNTRIES HAVE A WATER CAN MODEL THAT EVERYBODY GETS A LITTLE BIT OF MONEY. OTHER IN THE U.S. YOU YOU HAVE A LOT OF CENTER FUNDING. IT WOULD BE INTERESTING TO COMPARE FOR THE SAME RESEARCH AREA WHAT THESE REALLY RESULT IN. AS FOR THE IDENTIFYING AREAS OF OPPORTUNITY, I THINK THE NEXT SPEAKER HAS MORE TO SAY THERE. >> AS I UNDERSTAND AMONG TRANSLATORS THERE'S LATIN SAYING, (SPEAKING LATIN) WHICH MEANS TRANSLATOR TRADER. IT MEANS IF YOU TAKE ONE PIECE OF INFORMATION EXPRESSED IN A CERTAIN LANGUAGE AND TRANSLATE TO ANOTHER LANGUAGE YOU WILL INEVITABLY BETRAY THE MEANING AND SEMANTICS OF THE ORIGINAL BECAUSE MOST LANGUAGE IT'S REALLY DIFFICULT TO TRANSLATE ONE LANGUAGE TO THE NEXT WITHOUT LOSING MEANING. IN YOUR CASE WHAT YOU HAVE IS HIGHLY MULTI-DIMENSIONAL DATA THAT U YOU'RE TRANSLATING TO THE INDIVIDUAL, EXPRESSIONS WHICH BY THE WAY ARE FANTASTICICALLY BEAUTIFUL. AND VERY INFORMATIVE. THE QUESTION I ASK IS HOW DO YOU DEAL WITH THAT ISSUE? EVENTUALLY THESE MAPS WILL BE USED AND ARE ALREADY BEEN USED BY HUMAN OPERATORS. AND WHEN PEOPLE DESIGN INTERFACE, THE SAME IS TRUE FOR A COCKPIT OR CAR. PEOPLE SPEND TIME DEALING WITH HUMAN FACTORS, MAKING SURE THE INFORMATION IS TRANSLATED TO SOMETHING THE HUMAN OPERATOR PERCEIVES AND CAN CAN RELY ON TO MAKE ACCURATE DECISIONS. SO THE QUESTION IS IT'S A CONVOLUTED QUESTION BUT NOT WHETHER YOU CAN PRODUCE SOMETHING VISUALLY ATTRACTIVE BUT ALSO HELP HUMAN OPERATORS MAKE ACCURATE AND TRUE DECISIONS AN TO REDUCE HUMAN ERROR, IF YOU WILL. I HAVE WONDERING HOW YOU SEE THAT PART OF THIS RESEARCH EVOLVING OVER TIME. >> VERY GOOD POINT. GOOD QUESTION. SO IN THE EXAMPLE OF THE UCSD MAP OF SCIENCE, YOU SAW THERE ARE MILLIONS OF PAPERS, ALL OF THEM, MANY, MANY CITATION REFERENCES USED IN THE CASE OF THE NIH MAP YOU HAVE AN ENTIRE PORTFOLIO FOR ONE NOW MULTIPLE YEARS AND ALL OF THEIR ROOTS ASSOCIATED. THIS IS AN ENORMOUS SIZE MATRIX OF WORDS BY PROJECTS. HEIDI MENTIONAL SPACE. UNFORTUNATELY WE AS HUMAN BEINGS ARE BAD AT DEALING WITH HEIDI MENTIONAL SPACES. YOU BREAK THEM DOWN TO TWO DIMENSIONS. VERY HEIDI MENTIONAL SPACE, TWO DIMENSIONS, OF COURSE YOU'RE GOING TO LOSE INFORMATION THERE. THE HOPE IS TO PRESERVE THE IMPORTANT STRUCTURES IN THE DATA AND THERE HAVE BEEN MANY EXTENSIVE RESEARCH PROJECTS WHICH REALLY TRY TO GET DATA MINING TECHNIQUES WHICH GIVE THIS TO YOU. I I MENTION VALIDATION STUDIES DONE WHICH ALSO TRY TO ENSURE THAT YOU PRESERVE THIS MAIN STRUCTURE. WHAT HAPPENS IS OFTENTIMES ONE JOURNAL MIGHT BE ASSOCIATED WITH MULTIPLE SUB DISCIPLINES. FOR INSTANCE IN THE USCD MAP OF SCIENCE, SCIENCE AND NATUREND OTHER MULTI-DISCIPLINARY JOURNALS ARE ASSOCIATED WITH MANY MANY OF O THOSE NODES THERE. IF YOU THEN START TO GO FROM ONE MAP TO THE NEXT TO THE NEXT, YOU HAVE END TO END MAPPING SO IT'S WORSE, YOU NOW ARE TAKING THAT HIGHLY COMPRESSED SPACE, TWO DIMENSION AND TRYING TO CONNECT IT TO ANOTHER EQUALLY -- IT'S VERY TRICKY AN REQUIRES ADVANCED DATA ANALYSIS TECHNIQUES WHICH EXIST OR MAYBE HAVE TO STILL BE DEVELOPED. I WOULD ARGUE THAT GIVEN WE CAN'T DEAL WITH A HEIDI MNAL SPACE, HAVING THE SPACE TO TALK ABOUT THESE THINGS IS ALREADY USEFUL AND WHAT WE DO HERE IS LIKE ON GOOGLE YOU SEARCH FOR TERMS, SO YOU SEARCH TOPICS, YOU SEARCH WORDS APPEARING IN TITLE ABSTRACT OR FULL TEXT AND YOU SEE THINGS HIGHLIGHTED ON THE MAP OF SCIENCE. AND YOU SEE ALL OF THEM WHICH HAVE THE KEY WORD WE WON'T MISANY OF THOSE. BUT YOU SEE WHERE THEY CLUSTER. IF THE MAP IS DONE CORRECTLY, IDEALLY THEN THEY'RE REALLY GROUP THOSING TO WHICH SHOULD BE GROUPED TOGETHER OR SUBJECT MATTER EXPERTS WILL GO IN AND LET YOU VERY QUICKLY KNOW THAT THIS IS -- THIS CAN'T BE RIGHT AND THEN YOU HAVE TO BACK TO YOUR DRAWING BOARD TO FIGURE OUT HOW TO DO A BETTER MAP. NOT SURE I FULLY ANSWERED THIS QUESTION BECAUSE I THINK THIS IS ACTIVE RESEARCH BUT ULTIMATELY WHAT I WOULD LIKE TO SEE IS REALLY MAPS OF SCIENCE THAT ARE AS RIGOROUSLY DESIGNED AS THE MAPS OF THE WORLD TODAY. AND THERE IS RESEARCH INTO DOING THIS RIGHT AN ULTIMATELY AN INFRASTRUCTURE WHICH COLLECTS NOT ONLY COLLECTS PAPERS PATTERNS GRANTS AND INFORMATION DATA STREAM INFORMATION ON GRADUATING AN CAREER TRAJECTORIES OF MANY PEOPLE IN THE SCIENCE SYSTEM. THESE MAPS SHOULD NOT ONLY BE USEFUL FOR INTERNAL DECISION MAKING BUT THEY SHOULD BE AVAILABLE FOR ANYBODY TO SEE. I THINK WOULD CREATE A NEW AWARENESS OF NOT ONLY THE BEAUTY OF SCIENCE BUT PRNS OF SCIENCE AN POTENTIALLY ALSO MAKE DECISIONS IN TERMS OF RESEARCH AREAS TO GO INTO, WHAT RESEARCH TO SUPPORT, HOW TO CONNECT WITH EXPERTS THAT WORK ON CERTAIN TOPICS RELEVANT FOR WORK AND GET A MORE HOLISTIC UNDERSTANDING AND APPRECIATION OF SCIENCE. >> THANK YOU. >> THANK YOU. [APPLAUSE] >> OUR NEXT SPEAKER IS DR. KEVIN BOYACK, PRESIDENT OF SCITEC STRATEGIES. HE SPENT 17 YEARS AT SANDI RKEN NATIONAL LABORATORIES WHERE E HE WORKED IN COMBUSTION, TRANSPORT PROCESSES SOCIOECONOMIC WAR GAMING AN SCIENCE MAPPING. SINCE YOUR HONORING SCITECH HIS WORK CENTERED ON DEVELOPING MORE ACCURATE GLOBAL MAPS OF SCIENCE. HE'S PUBLISHED NEARLY 30 ARTICLES DEALING WITH VARIOUS ASPECTS OF SCIENCE MAPPING AND RELATED METRICS. CURRENTLY HIS INTERESTS INCLUDE APPLICATION TO FULL TEXT THE SCIENCE MAPPING AND BIBLIOMETRICS AND DETAILED STRUCTURE AND DYNAMIC OF SCIENCE AND IDENTIFICATION OF EMERGING AREAS. >> IT'S A PLEASURE TO BE HERE THIS MORNING AND TO HAVE A CHANCE TO TALK A LITTLE BIT ABOUT SOME OF THE WORK THAT I HAVE DONE ALONG WITH DICK KLAVANS WHO IS MY PARTNER IN CRIME SO TO SPEAK. AND HE WILL BE ONE OF THE BREAK OUT SPEAKERS THIS AFTERNOON. SO SOME OF THE DETAIL THAT I'M NOT GOING TO GO THROUGH IN THIS TALK WILL BE AVAILABLE IN THE BREAK-OUT SESSION THIS AFTERNOON. I HAVE BEEN ASKED TO SPEAK ABOUT GAPS AN OVERLAPS AND HOW TO IDENTIFY THOSE IN RESEARCH PORTFOLIO USING SCIENCE MAPPING TECHNIQUES. SO I'M GOING TO TALK A BIT TO THAT BUT ALSO TALK ABOUT SOME OTHER THINGS AS WELL. SO TO SOME EXTENT WHICH YOU'LL HEAR FROM ME OVERLAPS WHICH YOU HEARD FROM KATIE. SHE GAVE A WONDERFUL OVERVIEW WITH DETAIL HOW SCIENCE MAPS CAN BE USED IN PRACTICE AND ESPECIALLY TO LOOK AT PORTFOLIOS. SO BRIEF AGENDA WHAT I'M GOING TO TALK ABOUT TODAY. I'LL START WITH A GEOGRAPHIC ANALOGY, KATY DID THAT AS WELL. THEN I'M GOING TO TALK ABOUT A GLOBAL MAP OF SCIENCE, MAP OF SCIENCE THAT WE HAVE BEEN USING HERE FOR THE LAST SHORT PERIOD OF TIME, DICK AND I GENERATED MANY MAPS OF SCIENCE OVER THE YEARS. SOME OF THOSE YOU'LL SEE IN THE POSTERS OUTSIDE ON THE WALLS. MANY OF THEM WERE SHOWN IN KATY'S PRESENTATION. BUT I'LL SHOW YOU A NEW IRMAP THAT WE JUST STARTED USING. THEN WE'LL TALK OVERLAPS AN GAPS. AND ALSO ABOUT IDENTIFYING POTENTIALLY TRANSFORMATIVE RESEARCH. I'M GOING TO EQUATE THE WORD GAPS WITH POTENTIALLY TRANSFORMATIVE RESEARCH AND WITH OTHER TERMS THAT YOU'LL SEE LATER ON IN THE TALK. THEN WE'LL TALK ABOUT ADDITIONAL MAP USES AND SUMMARIZE. SO GEOGRAPHIC MAPS, WE'RE ALL FAMILIAR WITH THEM, WE USE THEM EVERY DAY, MOST OF YOU NOW HAVE THESE PHONES AND YOU CAN TAKE A LOOK AT A MAP AND THE FACT THAT WHEN WE'RE WALKING BACK TO THE HOTEL LAST NIGHT DEWEY HUGHES WILL SPEAK TO YOU HERE IN ANOTHER HOUR OR SO AT HIS GPS OUT AND WE WERE FOLLOWING A PATH TO THE HOTEL. SO WE KNOW HOW TO USE MAPS. AND MAPS SHOW EXISTENCE OF OBJECTS. THEY SHOW THE LOCATIONS OF THOSE OBJECTS AND THEY SHOW THE DISTANCES BETWEEN THOSE OBJECTS. AND THE MAPS CAN PROVIDE -- THEY'RE A TEMPLATE WHICH YOU CAN SHOW OTHER THINGS SO THINGS LIKE RESOURCES AN PRODUCTION, AND LOOK AT OVERP LAPS AN GAPS BETWEEN FEATURES. AN EXAMPLE OF THAT, THIS IS A THE OUTLINE HERE AN LINES, SOMETHING WE'RE FAMILIAR WITH, MAP OF THE UNITED STATES. BUT THIS SHOWS PRODUCTION OF WHEAT. THOSE OF YOU IN FRONT PROBABLY CAN'T SEE THIS BUT THERE ARE 6 TYPES OF WHEAT SHOWN ON THIS MAP FROM SEVERAL TYPES, RED WHEAT, WHITE WHEAT AND DURHAM. THE COLORS YOU CAN SEE WHERE THE DIFFERENT TYPES OF WHEAT IN THE COUNTRY ARE PRODUCED. THIS MAP WB USED TO SHOW OVERLAPS. IT MAYBE HARD TO SEE BUT IN THE AREAS HIGHLIGHTED YOU HAVE MULTIPLE COLORS. SO YOU CAN SEE IF YOU LOOK AT THIS ON A PIECE OF PAPER YOU HAVE OVERLAPS IN THE TYPES OF WHEAT THAT ARE GROWN IN PARTICULAR SECTIONS OF THE COUNTRY. LIKEWISE YOU CAN SEE AREAS WHERE THERE'S NOTHING. THESE ARE GAPS. THERE'S LITTLE OR NOT ENOUGH WHEAT GROWN IN IOWA TO SHOW UP ON THIS TYPE OF MAP. THE FOUR CORNERS REGION, SAME TYPE OF THING. ONE QUESTION ONE MIGHT ASK, WHY ARE THE GAPS THERE? THERE ARE REASONS FOR THAT. ONE WOULD BE THE AREA IS NOT SUITABLE FOR GROWING WHEAT. I LIVE NEAR THE FOUR CORNERS REGION. YOU WON'T GROW MUCH WHEAT OUT THERE BECAUSE OF THE NATURE, THIS IS RED ROCK COUNTRY. JUST TRY TO DIG A HOLE IN THAT STUFF, NOT VERY EASY. IOWA IS PROBABLY SUITABLE FOR WHEAT PRODUCTION. BUT THERE IS A LOT OF CORN PRODUCTION. SOME CASES IT'S A CHOICE MADE. BUT IN SOME CASES IT'S BECAUSE IT HASN'T BEEN TRIED. WE'LL COME BACK TO HA LATER. SCIENCE MAPS CAN BE USED THE SAME WAY. I MENTIONED THESE THINGS FOR GEOGRAPHIC MAP BUT A SCIENCE MAP INSTEAD OF SHOWING THE PHYSICAL SPACE THIS TWO DIMENSIONAL SPACE WE ARE FAMILIAR WITH AND DISTANCES MEAN SOMETHING PHYSICAL, SCIENCE MAP SHOWS AN ABSTRACT SPACE BUT ALSO SHOWS EXISTENCE OF PARTICULAR OBJECTS, SHOWS RELATIVE LOCATIONS, RELATIVE DISTANCES BETWEEN THEM. BECAUSE SCIENCE AS WE HEARD FROM KATY IS A VERY HIGHLY DIMENSIONAL SYSTEM. WHEN YOU COMPRESS DOWN TO TWO DIMENSIONS ON A MAP, LOCATIONS AND DISTANCES ARE NO LONGER ABSOLUTE. BUT THEY DO MEAN SOMETHING. THEY'RE RELATIVE TO EACH OTHER. SO THE WAY TO READ THE SCIENCE MAP IS TO THINK OF IT IN TERMS OF PROXIMITIES OVERLAPS AN GAPS. RATHER THAN PRECISE DISTANCE, LOCATIONS. THESE MAPS PROVIDE A BASIS OR TEMPLATE UPON WHICH YOU CAN SHOW OTHER INFORMATION. THINGS LIKE RESOURCES WHICH IN OUR CASE MIGHT BE FUNDING. THINGS LIKE PRODUCTION WHICH IN OUR CASE MIGHT BE PATENTS OR ARTICLES THAT CAME FROM THESE GRANTS. AND WE CAN SHOW OVERLAPS AN GAPS IN THESE FUTURES.y;– SO THESE THINGS HERE, THIS IS PART OF PORTFOLIO ANALYSIS. LOOKING AT GAPS AN OVERLAPS, THAT -- THOSE ARE PORTFOLIO QUESTIONS. LOOK AT RESOURCES AN PRODUCTION, THOSE ARE PORTFOLIO QUESTIONS. SO SCIENCE MAPS ARE A GREAT WAY ARE A GREAT WAY TO ANSWER PORTFOLIO QUESTIONS. KATY ELUDED TO THIS IN ONE OF HER SLIDES BUT THERE ARE CHOICES THE MAKE WHEN GENERATING A MAP OF SCIENCE OR DOING A PORTFOLIO ANALYSIS. THERE'S INHERENT RELATIONSHIP BETWEEN SCALE AND WHAT YOU MIGHT CALL EVOLUTION OR STABILITY. SO ON THE LEFT WE HAVE THE SCALE F. YOU TAKE SCIENCE AND DIVIDE INTO 10, 15 CATEGORIES, SCALE THERE YOU'RE TALKING ABOUT IS HUNDREDS OF THOUSANDS OF ARTICLES PER YEAR. OR HUP THOUSANDS ARTICLES PER YEAR PER CATEGORY. IF YOU HAVE THAT SIZE OF CATEGORY, IT WON'T CHANGE MUCH IN TIME. A FIELD THE SIZE OF MEDICINE OR FIELD THE SIZE OF PHYSICS WILL NOT BE BORN OR DIE IN OUR LIFETIME. SO THERE'S NO MORPHOLOGICAL CHANGE AT THE LEVEL OF WHAT THE SCIENCE AN ENGINEERING UNDERSTOOD KAYTORS REPORTS REPORT ON IT. IF YOU GO TO THE SUB DISCIPLINE LEVEL OR SUB DISCIPLINE LEVEL, WHAT WE CALL THE DISCIPLINE LEVEL, THAT BRINGS SOMETHING ELSE UP, EVEN AMONG SCIENCE MAPPING COMMUNITY, WE USE DIFFERENCE WORDS, THAT'S SOMETHING WE NEED TO CONVERGENCE ON, IF YOU TAKE AND DIVIDE SUB DISCIPLINES OR SUBJECT CATEGORIES, SEVERAL HUNDRED NOW YOU LOOK AT STRUCTURES ON ORDER OF TENS OF THOUSANDS OF ARTICLES PER YEAR. AND YOU MAY HAVE A DISCIPLINE BORN EACH YEAR OR EACH FEW YEARS BUT IT DOESN'T HAPPEN -- IT'S NOT HAPPENING SYSTEMATICALLY TO LARGE PORTIONS OF THE DATA. IF YOU BREAK DOWN FURTHER TO SPECIALTIES OR COLLEGES OR SCHOOLS OF THOUGHT, NOW WE DEAL WITH STRUCTURES ON THE ORDER OF A COUPLE OF HUNDRED ARTICLES PER YEAR. YOU WILL SEE MORE CHANGE IN THOSE. YOU WILL SEE BIRTHS AND DEATH REGULARLY. IF YOU BREAK DOWN FURTHER AT THE PROBLEM LEVEL, ON THE TOPIC LEVEL WHERE YOU HAVE 15, 20 ARTICLES PER TOPIC PER YEAR, THERE IS ENORMOUS CHANGE. ALL WHO DO RESEARCH KNOW THIS. AS RESEARCHERS WE TRY SOMETHING, IT WILL WORK OR DOESN'T WORK BUT WE HOP AROUND. A LOT OF RESEARCHERS ARE CONTINUALLY MOVING THEIR RESEARCH. SO TOPICS AT THAT LEVEL HAVE A GREAT DEAL OF INSTABILITY. CHOOSING THE LEVEL YOU'RE GOING TO OPERATE AT IS CHOOSING THE STABILITY LEVEL OF DATA YOU'RE LOOKING AT. AND MOST SCIENCE STUDIES DONE TO DATE HAVE BEEN DONE UP IN THIS RANGE. SO WE SEE SCIENCE AN ENGINEERING REPORTS COMING OUT EVERY COUPLE OF YEARS AND WE SEE THE GROWTH CURVES BUT NOT BIRTH AND DEATH. WE, DICK AND I ARE INTERESTED IN MODELING THIS LEVEL. INSTABILITY IS NOT SOMETHING MOST ARE COMFORTABLE WITH. P IF WE DO THE RESEARCH WE REALIZE THAT. BUT WHEN TRYING TO TRACK IT, INSTABILITY IS HARD TO DEAL WITH. SO I ENCOURAGE YOU, MANAGING PROGRAMS YOU SEE STABL IN THE RESEARCH GOING ON UNDER YOUR PURVIEW. I WOULD URGE YOU THOI ABOUT ANALYSIS AT THIS LEVEL. THAT'S WHAT WE'RE INTERESTED IN. LET'S JUMP INTO THIS, THIS IS THE MOST RECENT MAP OF SCIENCE, A BIG BLOB. BUT THERE'S 116,000 DOTS ON THIS PLOT. EACH DOT REPRESENTS ONE TOPIC IN SCIENCE. SOMETHING ABOUT 15 PAPERS. THIS MAP WAS GENERATED FROM DATA FROM 2010, ABOUT 1.7 MILLION ARTICLES. I WON'T GET INTO TECHNIQUE THAT COULD TAKE A LONG TIME. BUT SUFFICE IT TO SAY THERE'S A LOT OF MATH BEHIND THIS AND THIS LAY OUT REPRESENTS THE PROXIMITY RELATIONSHIPS BETWEEN 116,000 CLUSTERS OF ARTICLES. AND WE HAVE COLORED THINGS OVER HERE SO YOU CAN SEE DOMINANT PATTERNS IN DATA. IF WE START HERE, THE DARK RED IS OUR INFECTIOUS DISEASE RELATED TYPES OF TOPICS. TO THE LIGHTER RED, THESE ARE MEDICAL SPECIALTIES. MOVING AROUND THE CIRCLE WE GO THROUGH THE NEUROSCIENCES AN SOCIAL SIGNS, COMPUTER SCIENCE, PHYSICS ANi,u MATT ARE -- MATH ARE PURPLE, ENGINEERING, CHEMISTRY AND BUY LOGICAL AND EARTH SCIENCES. THIS IS KATY IN HER NEXT TO LAST SLIDE SHOWED SEVERAL DIFFERENCE MAPS OF SCIENCE. ONE THING SHE DIDN'T POINT OUT IS THAT ALMOST ALL OF THOSE MAPS HAD A COMMON ORDERING OF MAJOR FIELDS. THIS MAP PRESERVES THAT COMMON ORDERING OF MAJOR FIELDSCH WHEN WE LOOK AT MAPS OF SCIENCE WHETHER DONE BASED ON JOURNALS OR PAPERS, SCIENCE IS ROBUST AT THAT SCALE. SO THE ORDERING OF FIELDS REMAINS THE SAME REGARDLESS OF THE TECHNIQUE USED THE MAP, WHICH IS COMFORTING FOR US BECAUSE THERE'S STABL AT THAT HIGHEST LEVEL. AMONG SCIENCE MAKERS, THERE'S -- THERE ARE A VARIETY OF METH LOGICAL CHOICES. THOSE SPEAKING ON THIS PROGRAM, YESTERDAY AND TODAY, WE ALL DO THINGS DIFFERENTLY. BUT THERE ARE THINGS COMMON AMONG US AN THOSE ARE VERY STRONG. SO FIRST OF ALL THERE'S A COMMON PHILOSOPHICAL APPROACH AMONG ALL OF US. KATY AND I CO-AUTHORED FOR YEARS AND CHAOMEI AND I WORKED SEVERAL YEARS AGO. YOU'LL HEAR FROM ALAN THIS AFTERNOON. THIS IS A FAIRLY TIGHT GROUP, THOUGH WE HAVE DIFFERENCES IN PROCESS WE HAVE SIMILAR PHILOSOPHY, THAT PHILOSOPHY COMES THROW IN THE SENSE WE'RE ALL TRYING TO USE THE SCIENTIFIC METHODS TO DO SCIENCE MAPPING. WHAT THAT MEANS TO ME IS WE'RE STARTING WITH A HYPOTHESIS, STARTING WITH A RESEARCH QUESTION. THEN WE FINE THE DATA AND EVOLVE PROCESSES TO TRY TO DO THE MOST ACCURATE JOB OF SCIENCE MAPPING THAT WE CAN. AT THE LEVELS THAT WE HAVE CHOSEN TO ANSWER THE QUESTIONS WE'RE ASKING. NOW, BY CONTRAST, I GET A BUNCH OF PAPERS TO REVIEW, WHERE PEOPLE HAVE A DATA SET AN THEY'RE LOOKING FOR A PROBLEM TO ANSWER. THAT'S NOT THE WAY TO GO ABOUT IT. THOSE OF US HERE ARE TRULY TRYING TO FOLLOW THE SCIENTIFIC METHOD, AS WE GENERATE THESE MAPS. WE ARE INTERESTED IN COMING TO A MORE COMMON MAPPING BASIS AND IN USING MORE COMMON TERMINOLOGY. THE THING THAT I THINK -- IT'S NOT THAT IT DISTINGUISHES US, DICK AND I OVER THE PAST TEN YEARS HAVE BEEN STANLEY POUNDING ON -- CONSTANTLY POUNDING ACCURACY. SO WE HAVE NOT BEEN DEVELOPING TOOLS THE TO HAND OVER THE FENCE TO PEOPLE. RATHER DOING RESEARCH TO TRY TO UNDERSTAND WHY THE DIFFERENT METHODOLOGICAL CHOICES, WHAT AFFECT THAT HAS ON THE ACCURACY. SO THAT HAS DRIVEN THE WAY WE DO THINGS. SO WE TEND TO MAP SCIENCE GLOBALLY. WE DO THE WHOLE THING. INSTEAD OF TAKING SMALLER SETS AN MAPPING THEM. WE WORK AT THE TOPIC LEVEL, WE ARE ACTIVELY WORKING ON PREDICTIVE MODELING AND TRYING TO MODEL THIS COMBINATION OF STABILITY AND INSTABILITY. OKAY. SO HAVING SAID THAT, LET'S TALK ABOUT OVERLAPS AN GAPS. SO OVERLAPS, KATY SHOWED YOU SEVERAL OVERLAP TYPES OF MAPS AND DIDN'T CALL IT THAT SO I'M GOING TO GIVE YOU A COUPLE OF EXAMPLES YOU YOU'LL RECOGNIZE THESE. OVERLAPS CAN BE SHOWN ON A SINGLE MAP OR SIDE BY SIDE MAPS. THEY ARE EASY TO CALCULATE AND CAN BE DONE AT A VARIETY OF LEVELS. NOW, THIS IS AN OLDER MAP THAT WE GENERATED ABOUT 2004 OR 5 WE GENERATED THIS MAP T. BASE MAP, THE GRAY DOTS ARE JOURNAL CATEGORIES. BUT WHAT WE'RE SHOWING HERE IS AN OVERLAY OF THE NIH PORTFOLIO FROM 2005 VERSUS THE OVERLAY OF AN NSF PORTFOLIO. YOU MIGHT ASK HOW ARE YOU GOING TO FIND OVERLAPS? AS YOU LOOK SIDE BY SIDE, VISUALLY YOU'RE GOING TO FIND AREAS ON THE MAP WERE THERE CIRCLES IN BOTH MAPS OR HIGHLIGHTED INFORMATION IN BOTH MAPS. SO YOU CAN SEE FOUR OF THOSE HERE THAT I HAVE POINTED OUT. YOU ALSO SEE AREAS WHERE NIH HAS THE MAJORITY FUNDING VERSUS NSF WITH MAJORITY OF FUNDING. SO THIS IS WAY TO LOOK AT OVERLAPS. HERE IS ANOTHER WAY OF LOOKING AT OVERLAPS. THIS IS ENTIRELY DIFFERENT MAP FROM EITHER THE BRIEFIUS TWO I HAVE SHOWN YOU, THIS IS A TOPIC LEVEL MAP, SHOWN UNDERNEATH THE COLORED AREAS. WHAT WE DID HERE, THIS IS A PROJICTD FOR THE NSF CHEMISTRY DIVISION A COUPLE OF YEARS AGO. THEY WERE RIGHT IN THE MIDDLE OF CHANGING THEIR PROGRAM STRUCTURE. SO WHAT THEY WANTED TO KNOW AS THEY WENT TO RETREAT, WHAT IS THE OVERLAP BETWEEN OUR PROGRAM AREAS. SO WE TALKED WITH THE PROGRAM OFFICERS THERE, AND THEY HAD SIX DIFFERENT PROGRAM AREAS. THESE PROGRAM AREAS ARE DISCIPLINARY. IF YOU LOOK AT THESE, YOU HAVE EXPERIMENTAL AND PHYSICAL CHEMISTRY, ANALYTICAL AN SURFACE IN ANOTHER, THEORETICAL AND -- ORGANIC AND SYNTHETIC CHEMISTRY AN ORGANIC CHEMISTRY. SO THESE LOCK LIKE THE WAY YOU DEFINE DEPARTMENTS AT A UNIVERSITY OR MAYBE SUB DEPARTMENTS. TRATITIONAL TYPES OF PROGRAM AREAS. SO WE TALKED TO PROGRAM OFFICERS WHO SELF-IDENTIFIED ON THIS MAP AREAS THEY CONSIDERED CORE TO THEIR PROGRAM AREAS. IT WAS BASED ON BOTH TOPICS, PEOPLE THEY WERE FUNDING, AND SO ON. YOU CAN SEE A TREMENDOUS AM OF OVERLAP BETWEEN PERCEPTION OF PROGRAM OFFICERS, THEIR CORE AREAS. THEY TOOK THIS INFORMATION, THIS MAP WITH OTHER INFORMATION INTO THEIR RETREAT, AND REORGANIZEDDED THEIR PROGRAM AREAS. IT'S INTERESTING, YOU PROBABLY CAN'T READ THIS BUT THE PROGRAM AREAS AN NAMES NO LONGER LOOK LIKE SUB DISCIPLINES. THEY DON'T LOOK LIKE TRADITIONAL UNIVERSAL TYPES OF BOUNDARIES. WE HAVE CHEMICAL STRUCTURE DYNAMIC AN MECHANISM. MEASUREMENT AND IMAGING. IF I WERE TO TAKE THESE MAPS NOW AND STACK THEM ON TOP OF EACH OTHER, THE OVERLAP BETWEEN THESE PROGRAM AREAS IS NOW FAR LESS THAN IT WAS BEFORE THE EXERCISE THEY WENT THROUGH SO MAPS LIKE THIS CAN BE USED TO SHOW OVERLAPS BETWEEN PROGRAM AREAS AN HELP YOU DISAM BIG WAIT THEM. IF YOU SO CHOOSE. GAPS, GAPS ARE DIFFERENT. WE TALKED ABOUT SOME OF THE OTHERS AND QUESTIONS HAVE COME UP, TALKED ABOUT THAT. IDENTIFYING GAPS ASSUMES THAT YOU KNOW YOUR TARGET TO SOME DEGREE. YOU HAVE SOME SORT OF A REFERENCE STANDARD. THE PROBLEM IS SOMETIMES THE STANDARD IS NOT WELL DEFINED. SOMETHING IN OUR HEADS, OR MAYBE TEXT WRITTEN ON PAPER BUT IN TERMS OF SOMETHING THAT CAN BE MAPPED, IT'S DIFFICULT TO IDENTIFY WHAT THE TARGET IS OR WHAT THE EXPECTED VALUE IS. TO SOME DEGREE GAP IS DIFFERENCE BETWEEN ACTUAL AN EXPECTED OR HOPED FOR OUTCOME. NOW I'M GOING THE TALK DIFFERENT GAPS, SURE THERE ARE OTHERS BUT THE TWO TYPES THE FIRST IS A COUNTER LAP, COUNTER PART TO OVERLAPS. WHAT I'M GOING TO CALL A MISMATCH GAP OR A COVERAGE GAP. THOSE ARE FAIRLY EASY TO IDENTIFY AND CAN BE SHOWN ON A MAP IN THE SAME WAY OVERLAP CAN. THE SECOND KIND OF GAP, I'M GOING TO CALL TRANSFORMATIVE RESEARCH. THAT'S THE QUESTION RICHARD ASKED A FEW MINUTES AGO, HOW DO YOU IDENTIFY FROM THE WHITE SPACES WHAT CAN HAPPEN IN THE FUTURE. IT'S A VERY DIFFICULT QUESTION. I'M GOING TO SHOW YOU THAT I I'M NOT GOING TO ANSWER THAT QUESTION DIRECTLY HERE. WE'LL GET TO THAT IN A MINUTE. LET'S TALK ABOUT THE MISMATCH GAPS. THIS MAP IS BACK THE OUR LARGE SCIENCE MAP, THE MOST RECENT ONE WITH 1 THE 16,000 CLUSTER ON IT. WHAT YOU CAN SEE IN THE LIGHT GRAY, SHOWS WHERE THOSE CLUSTERS ARE. THIS IS AN I DON'T HAVE LAY HERE THAT SHOWS THE SYSTEM WORK WE DID FOR CSR SIX MONTHS AGO. WHERE WE COMPARED EXPERTISE OF STUDY SECTIONS TO THE PROPOSALS COMING INTO THOSE STUDY SECTIONS. SO WHERE YOU HAVE EXPERTISE AN PROPOSALS THAT OVERLAP, THESE FLUORESCENCE TYPE OF PROGRAM. IF YOU TAKE BLUE AND USE IT TO REPRESENT EXPERTISE, TAKE THE PINK TO REPRESENT WHERE THE PROPOSALS ARE, IF YOU MIX COLORIOUS GET PURPLE SO THAT'S SHOWING OVERLAP BETWEEN EXPERTISE AN PROPOSALS IN THE STUDY SECTION. I SHOULD MENTION EXPERTISE IS ONLY ON STANDING STUDY SECTION MEMBERS, EXPERTISE IS UNDERSTOOD REPRESENTED IN THIS OVERLAP BUZZ WE LIMITED IT TO THE STANDING SECTION NUMBERS. SO TREMENDOUS OVERLAP BETWEEN EXPERTISE AN PROPOSALS. THERE'S ALSO MISMATCHES. YOU CAN SEE AREAS OF PINK WHERE THERE WERE PROPOSAL THERE WAS TEXT IN A PROPOSAL, WHERE THERE WAS NOT EXPERT SITTING ON THAT PANEL WHO PUBLISHED IN THAT TOPIC. THERE WERE EXPERTS NEXT DOOR SO YOU CAN SEE THE BLUE AN PINK CAN BE RIGHT NEXT TO EACH OTHER. AT THIS HIGHLY GRANULAR LEVEL YOU CAN SHOW WHERE THE MISMATCHES WERE AT A VERY TOPICAL LEVEL. THAT MAY OR MAY NOT BE A FACTOR IN HOW THINGS GO. IN TERMS OF REVIEWS. ANOTHER EXAMPLE FOR STUDY SECTION WHERE NEEDS OF THE STUDY SECTION ARE FAR MORE BROAD, SPREAD OVER ALL OF SCIENCE OR OVER MUCH OF SCIENCE. YOU CAN SEE THAT THIS PARTICULAR SECTION HAS EXPERTISE AND NEEDS VERY STRONG IN COMPUTER SCIENCES AND ENGINEERING SO IT'S MUCH HARDER SO COVER THAT SPACE, BECAUSE IT'S NOT TOPICALLY LOCAL AS IN THE OTHER STUDY SECTION. SO GAPS AS POTENTIALLY TRANSFORMATIVE RESEARCH. THERE'S BUZZ WORDS OUT THERE, AND MEMBERSES -- DIFFERENT SCIENCE MAPPERS AND BIBLIOMATRITIONS TALK IN DIFFERENT TERMS BUT YOU'LL HEAR FROM CHAOMEI ABOUT STRUCTURED HOLES. IT'S A SIMILAR CONCEPT TO FINDING TRANSFORMATIVE RESEARCH OR FINDING GAPS. YOU'RE GOING TO HEAR FROM ALAN PORTER LATER ABOUT INTERDISCIPLINARITY. VERY SIMILAR TOPIC ONCE AGAIN. THE FUSE PROGRAM, TALKING ABOUT THAT, ALL ABOUT IDENTIFYING EMERGING TOPICS, DICK AND I WILL TALK ABOUT HOT SCIENCE. THESE ARE OVERLAPPING TOPICS SO IT'S A HOT AREA OF RESEARCH. IF WE'RE IDENTIFYING TRANSFORMATIVE TOPICS WHAT ARE WE LOOKING FOR? HOW DO WE DO THIS? SHOULD WE EVALUATE THE WHITE SPACES? I'LL BE HONEST. WE DONE KNOW WHAT WHITE SPACE MEANS. WE JUST DON'T. BECAUSE WE'RE DEALING WITH AN ABSTRACT MAP. WE KNOW WHAT THE PORTIONS OF THE MAP MEANS BUT THE UNOCCUPIED PORTION BECAUSE OF THE DIMENSIONAL NATURE THAT GOES INTO IT COULD MEAN THERE'S SOMETHING THERE IN BETWEEN COUPLE OF OTHER THINGS THAT HAVE BEEN DONE BUT MIGHT NOT. SO LOOKING AT WHITE SPACE, IT'S KIND OF LIKE LOOK AT THE WHEAT MAP. WHY ISN'T THERE WHEAT GROWING THERE? IT MIGHT NOT BE FERTILE GROUND FOR THAT PARTICULAR TYPE OF THING. OR IT MIGHT NOT HAVE BEEN TRIED. AND THAT'S WHERE RESEARCHERS COME INTO THE PICTURE. THE RESEARCHERS ARE THE PEOPLE WHO USE WHAT THEY HAVE BEEN DOING FOR A LIFETIME AND WHAT THEY HAVE BEEN READING TO DETERMINE WHERE THEY WANT TO GO NEXT. I DON'T KNOW ANY BETTER WAY TO GO TO THE NEXT THING OTHER THAN USING OUR HUMAN CAPABILITIES. SO THIS IS NOT WHAT WE DO. HERE IS WHAT WE DO. WE TAKE OUR MAP, FIRST WE WANT TO GET A PICTURE OF BOTH THINGS THAT WE FAILED OR BLIND ALLIES THAT PEOPLE ARE GONE UP AND BACKED OUT OF AND PEOPLE THAT COMPLETED WORK IN THE PAST. I'LL SHOW YOU AN OVERLAY HERE OF THE TOPICS THAT PEOPLE ABAN DOONED -- DONNED -- ABANDONED IN THE TWO YEARS PREEFIOUS TO THE 2010 MAP. THIS IS CONCEPTUAL OF TRIANGULATED THINGS SO THEY OTHER APPEARING IN THE WHITE SPACE BUT THEY'RE REALLY NOT. YOU CAN MAP ACTIVITY OVER TIME. SO WE WANT TO KNOW WHAT HAPPENED IN THE PAST. BUT THEN WHAT WE DO IS WE LOOK FOR TOPICS WITH HIGH NOVELTY. I HAVE TAKEN NOVEL SCIENCE HERE AN COVERED USING THREE DIFFERENT COLORS. SO YOU CAN SEE DARK BROWN, WHAT WE'RE GOING TO CALL NOT NOVEL SCIENCE BUT IT'S VERY PROVEN THESE ARE TEMPORAL HISTORY -- THESE AREN'T THE TOPICS IN OUR MAP THAT HAVE A VERY LONG HISTORY, A VERY LONG UNBROKEN HISTORY. SO THE SCIENCE IS NOT REALLY NOVEL. IT MAYBE PRODUCTIVE. IT IS CERTAINLY PROVEN. THE GROUND YOU CAN SEE PEPPERED IN THERE ARE TOPICS THAT HAVE A THREE, FOUR, FIVE, SIX YEAR HISTORY, SO THEY'RE NOVEL STILL. AND THEY'RE BEING PRODUCTIVE CURRENTLY. SO IT'S A PROVEN TECHNOLOGY BUT THERE'S STILL NOVELTY TO IT BECAUSE IT'S PRETTY YOUNG. NOW, THE PINK, THE LIGHT PINK, ARE THOSE TOPICS THAT JUST STARTED THIS YEAR OR ARE TWO YEARS OLD. WE DON'T KNOW WHAT WILL HAPPEN TO THOSE NEXT YEAR. BUT THEY ARE ARE REALLY NOVEL. THEY'RE UNPROVEN. WE JUST DON'T KNOW WHICH WILL CONTINUE. WE'RE WORKING ON METRICS TO TRY TO PREDICT WHICH OF THOSE ONE AND TWO-YEAR-OLD TOPICS WILL CONTINUE INTO THE FUTURE BUT THE TRUTH IS, HALF WON'T BE IN THE MAP. BECAUSE OF THE WAY RESEARCH IS DONE. SO I'M GOING TO THROW OUT A TEASER HERE, THIS IS WORK THAT WE ARE DOING LAST WEEK. SO WE HAVE TO VALIDATE THIS AND VERIFY SOME OF THESE THINGS BUT THESE ARE AREAS WITH WORK IN THE LAST TWO YEARS WHERE THAT LINE OF RESEARCH JUST STARTED. THAT ARE POTENTIALLY TRANSFORMATIVE OPPORTUNITIES. THIS ONE HERE IS IN REGENERATIVE MEDICINE. AND WE ALL KNOW THAT'S AN UP AND COMING TOPIC. MY GUT SAYS THAT'S LIKELY TO BE AROUND IN THE FUTURE BUT WE'RE WORKING ON METRICS ON THOSE THINGS. SO KEEP YOUR EYES OUT WHAT WE'LL PUBLISH IN THE NEXT WHILE, BECAUSE WE'RE WORKING ON THISSEN HAVING FUN, JUST A REALLY FUN THING TO WORK ON. A FEW ADDITIONAL MAP USES, THESE ARE SLIDES I EDITED LAST NIGHT BECAUSE THEY SPEAK TO QUESTIONS THAT CAME UP YESTERDAY. CAN YOU USE A MAP LIKE THIS TO SHOW WHERE NIH IMPACT IS? LET'S TAKE GRANTS ARTICLE LINKAGES. KATY SHOWED A MAP LIKE THIS ON THE UCSD MAP. SO WHERE DOES NIH FUND RESEARCH THAT'S PUBLISHED? RIGHT HERE. I WILLING TO L BACK BUT NOW WE'LL SHOW WHERE PAPERS ARE THAT EXPLICITLY MENTION THE NIH GRANT. WE'RE GOING TO COLOR CODE THEM BY THE INSTITUTE. AT THIS LEVEL THAT'S HARD TO LEAD. SO ONE CAN ZOOM INND LOOK AT THOSE THINGS MORE CLOSELY, BY CLUSTER, BY CLUSTER, DO ANALYSIS, BUT THESE ARE THE LINKAGES FOR NCI, FOR INSTANCE, YOU CAN SEE HOW BROAD THE AREA OF SCIENCE IS, THAT NCI HAS AN IMPACT OVER. HEART, LUNG AND BLOOD. SAME TYPE OF THING. IF I TOGGLE BACK AND FORTH YOU CAN START NOW TO LOOK AT OVERLAPS BETWEEN THOSE INSTITUTES. THIS IS ACTUAL DATA. WHAT ELSE CAN WE DO? WE TALKED ABOUT WHEN THE IMPACT PEOPLE WERE SPEAKING IMPACT YESTERDAY WE WERE TALKING VARIOUS NORMALIZATIONS, A MAP LIKE THIS CAN BE USED TO PROVIDE THE BASIS FOR NORMALIZATION. SO YOU CAN TAKE A GROUP LIKE THIS AND YOU CAN CALL THIS A CATEGORY, THIS ONE IS INTERESTING. YOU SEE ALL THE DIFFERENCE COLORS IN HERE, THIS IS A HIGHLY INTERDISCIPLINARY AREA OF RESEARCH INHERENTLY, YOU'VE GOT CHEMISTRY, ENGINEERING BIOLOGY, EARTH SCIENCES AN HEALTH SCIENCES. YOU HAVE GOT PAPERS FROM JOURNALS IN ALL OF THOSE AREAS THAT ARE BEING PUBLISHED ON A SIMILAR SET OF TOPICS. I FIND THAT FASCINATING. YOU LOOK AT THIS AREA RIGHT HERE, THIS IS A COMBINATION OF BIOLOGY AND INFECTIOUS DISEASE WORK. IF YOU LOOK AT THE NE NEXT AREA HERE, THIS IS INFECTIOUS DISEASE BUT WITH SOME BIOTECHNOLOGY MIXED IN. SOME AREAS OF SCIENCE ARE INHERENTLY MONODISCIPLINARY OR TIGHTLY DISCIPLINARY. SUCH AS THIS UP HERE IN THE UPPER LEFT. BUT SOME OF OUR INHERENTLY INTERDISCIPLINARY. I FINE THAT REALLY FASCINATING. BUT THIS COULD BE -- YOU CAN SET THIS UP AS A UNIT FOR NORMALIZATION AND DO IMPACT CALCULATIONS BASED ON THAT. THESE AREAS IN THE MAP OF SCIENCE CAN BE USED TO IDENTIFY TOPICS, POTENTIAL COLLABORATIONS, ET CETERA, BECAUSE YOU CAN TAKE ANY ONE OF THESE AREAS AND LOOK AT WHAT THE TOPIC DISTRIBUTION IS WHO THE KEY CITED AUTHORS ARE, KEY CURRENT AUTHORS AND SO ON. SO IF U YOU'RE PUBLISHING IN ONE OF THESE AREAS AND WANT TO LOOK FOR POTENTIAL COLLABORATORS, LOOKING FOR OTHERS WHO PUBLISH IN THAT POTENTIAL TOPIC OR NEIGHBORING TOPICS, IT'S A BUILT IN SYSTEM LOOKING AT THE SCIENCE AT THIS GRANULAR LEVEL. FINALLY I'LL MENTION, I WON'T GO INTO HOW WE GENERATE THIS BUT TOPIC HISTORIES AT THE AGES OF TOPICS, YOU CAN LOOK AT CONCENTRATIONS OF DIFFERENT KEY WORDS OVER THESE TOPICS. THIS HAPPENS TO BE AN EXAMPLE SHOWING GRAPH IN TEMPORAL STRUCTURES RELATED TO THAT LARGE MAP OF SCIENCE. WHAT'S INTERESTING IS GRAPHENE AS A COMPOUND HAS BEEN AROUND A LOT OF YEARS. BUT IT WASN'T UNTIL 2000 YOU CAN SEE A LIGHT PINK DOT HERE BUT IT WASN'T UNTIL 2006 AFTER SOMEBODY DISCOVERED A STABLE STRUCTURE, STABLE STRUCTURE FOR GRAPHING THAT IT TOOK OFF. GRAPHING RESEARCH WENT THROUGH THE ROOF AFTER THAT. AND SPAWNED A VARIETY OF GRAPHENE RELATED RESEARCH BECAUSE OF A KEY DISCOVERY. TEMPORAL HISTORIES CAN BE USED TO TELL THOSE STORIES. SOMETHING ELSE I SHOULD MENTION, AND DICK WILL TALK ABOUT MORE IN HIS BREAK-OUT SESSION IS THAT WE HAVE TAKEN HISTORIES LIKE THIS AND PUT THEM IN FRONT OF KEY RESEARCHERS, IN FRONT OF PROGRAM OFFICERS. PROGRAM OFFICERS TELL US THE STORIES RELATED TO THESE. WE CAN SHOW THEM THE CONTENT OF THESE THINGS, BUZZ THEY HAVE BEEN AROUND THAT FIELD. I DON'T KNOW THE STORIES BUT THEY CAN TELL THE STORIES OF WHY THIS THREAT ENDED HERE. THEY CAN TELL THE STORY WHY THIS NODE HERE SPAWNED RESEARCH OVER HERE AN WHY THIS WAS KEY IN SPAWNING THIS RESEARCH. THEY KNOW THOSE STORIES. SO WE'VE GONE A LONG WAY TOWARDS VALIDATION IS TOO STRONG A WORD BUT ANECDOTAL EVIDENCE THAT PEOPLE -- RESEARCHERS AND PROGRAM OFFICERS UNDERSTAND SCIENCE AT THIS LEVEL AND UNDERSTAND THE STORIES. THE STORIES CORRELATE WITH TEMPORAL ACTIVITY WE SEE. SO TO SUMMARIZE, I'M NOT OVER TIME YET. SCIENCE MAPPING IS REALLY I THINK VERY USEFUL AND VERY EFFECTIVE TOOL AND PORTFOLIO ANALYSIS. THE ACCURACY AND DETAIL TO ME ARE CRITICAL. THE IF THE MAP ISN'T ACCURATE, NOT GOING THE LEAD TO ACCURATE DECISIONS OR DETECTION OF EMERGING TOPICS. ACURA IS IS CRITICAL AND ALL OF US PRESENTING ARE TRYING TO MAKE THE MOST ACCURATE MAPS POSSIBLE. AND GENERATE PROCESS THAT COMES UP WITH ACCURATE MAPS. TRADITIONAL GAPS AN OVERLAPS ARE SHOWN ON THESE MAPS, THAT'S SIMPLE TO DO. AND THE OVERLAYS ARE VERY INFORMATIVE. WE'RE MAKING STRIDES TOWARD BEING ABLE TO SHOW GAPS OR TO LOCATE THE RESEARCH THAT HAS POTENTIAL TO BE TRANSFORMATIVE IN THE FUTURE. WHICH IS WHAT I SAID DOWN HERE. WE'RE LOOKING AT THE IDEA OF NOVELTY, VERY RECENT STRUCTURES. NOT TOTALLY CERTAIN WHERE THAT'S GOING, ITS AN ACTIVE AREA OF RESEARCH, WE'RE HAPPY WITH WAY THAT'S GOING AND EXCITED FOR THE FUTURE. IF SLIDES GET PASSED OUT YOU'LL GET THIS AND AND WANT TO ACKNOWLEDGE WE HAD NSF FUNDING TO LOOK AT EMERGING TOPICS AND WORKING AS PART OF THE FUSE PROGRAM AS WELL. WITH THAT, I THANK YOU FOR YOUR ATTENTION, IT'S BEEN A PLEASURE TO BE ABLE TO TELL YOU ABOUT THE RESEARCH WE'RE DOING, IT'S A HECK OF A LOT OF FUN. BE HAPPY TO ANSWER QUESTIONS. [APPLAUSE] >> MICHAEL LAUER FROM NHLBI. FASCINATING. THANK YOU VERY MUCH. I HAVE TWO QUESTIONS. ONE QUESTION, WHEN YOU SAY ON YOUR MAP POTENTIAL NEW AREAS WHEN YOU SAY UNPROVEN, DOES UNPROVEN SIMPLY MEAN THE FIELD HAS NOT CONTINUED TO EXIST FOR A CERTAIN LENGTH OF TIME OR IS THERE SOMETHING DEEPER TO THIS? >> IT MEANS IT'S VERY NEW, A VERY NEW TOPIC. WE DON'T KNOW IF IT WILL LAST OR GOING TO DIE. IT IS UNPROVEN IN THAT SENSE. >> GREAT. THE OTHER QUESTION IS ON THESE WHITE AREAS. I'M THINKING HOW WE AT NIH MIGHT USE A MAP LIKE YOURS TO UNDERSTAND THESE WHITE AREAS. ONE WAY WE COULD DO THIS, WE SEE WHERE THE WHITE AIR CRAS ARE, WE SEE WHO IS AROUND THE AREAS, THOSE ARE THE AREAS WE ATTACK, THOSE ARE THE SCIENTISTS PROFESSIONALS THAT WE CARRIED THAT BECAUSE THEY MIGHT HAVE THE GREATEST UNDERSTANDING ABOUT WHY THE WHITE AREAS ARE WHITE? >> PROXIMITY DOES MEAN SOMETHING. YOU CAN'T DO THAT FROM AREAS CLEAR ACROSS THE MAP BUT IF YOU HAVE CONCENTRATION HERE AN CONCENTRATION HERE, YES, THE PEOPLE THAT ARE IN THOSE AREAS WOULD PROBABLY BE THE BEST ONES TO KNOW IF THERE'S UNFOUND ISLAND SITTING BETWEEN THEM. SOMETHING ELSE THAT MAY GO TOWARDS THAT OR ANSWERING THIS QUESTION, IS IT WOULD BE FASCINATING TO TAKE THE DOWNLOAD ACTIVITY, THE USAGE ACTIVITY, AND OVERLAY ON THESE MAPS. I THINK OVERLAYING THE USAGE ACTIVITY AND -- SO IF YOU HAVE GOT USAGE ACTIVITY THAT IS LINKING THE EDGES OF TWO OF THOSE GROUPS, THAT IS I THINK A STRONG INDICATOR THAT THERE MIGHT BE SOMETHING UNFOUND SITTING IN THE MIDDLE. SO I THINK USAGE PATTERNS COULD BE USED FOR THAT TYPE OF THING. >> THANK YOU. >> JENNIFER (INAUDIBLE) NCI. MY QUESTION IS HOW TO LOOK AT THE GAPS TEMPORALLY OR HOW YOU GET INFORMATION ABOUT THE STAGES OF RESEARCH FOR THESE DIFFERENCE TOPICS. SO OBVIOUSLY YOU CAN LOOK TO SEE WHEN THE TOPICS COMPARING WITH THINGS. >> OKAY. TO BEEFILY EXPLAIN OUR PROCESS, WE GENERATE MAP EACH YEAR P APPROXIMATE LINK THEM BACK TOGETHER A FEW TIMESSEN WE LINK THEM BACKWARDS BASED ON REFERENCES IN THE OVERLAY STRUCTURES SO THAT'S HOW WE GENERATE HISTORY GOING BACK. SO WE CAN TELL WHEN A TOPIC WAS INTERRUPTED GOING BACK. IT GIVES US A CHANCE TO DETERMINE THE AGE OF THAT PARTICULAR RESEARCH THREAT. >> THEN I GUESS IN TERMS OF WHEN YOU START SLAYING APART, NOW THERE ARE CLINICAL STUDIES OR CLINICAL TRIALS BEING DONE FOR THIS THING WE STARTED STUDYING A LONG TIME AGO SO WE CAN IDENTIFY WHERE IN THE DRUG DEVELOPMENT PIPELINE. >> WE HAVEN'T LINKED THAT DATA IN BUT THAT CAN BE DONE. DICK TELLS ME HE DID THAT 15 YEARS AGO BEFORE I GOT INVOLVED. (OFF MIC) >> (INAUDIBLE) NIMS. I HAVE A QUESTION ON DATA SET. FROM YOUR TALK, BY THE WAY, FASCINATING. I'M VERY IMPRESSED. FROM YOUR TALK, MY UNDERSTANDING IS THAT YOU WORK ON VERY LARGE DATA SETS. RIGHT? >> WE USE ALL SCOPISTS. >> BECAUSE -- SO OFTEN TIME IT IS DATA SETS ARE NOT PERFECT, THEN WE HAVE HUMAN FACTORS, WE HAVE HOW WE DEFINE THE TERMS, WE ARE DOING THINGS SO MY UNDERSTANDING, THIS IS BIGGER THAN THE DATA SETS, THE MORE -- THE LESS THESE FACTORS WILL AFFECT OUTCOME. SO MY QUESTION ACTUALLY IS HOW SMALL CAN YOU GO THE DATA SETS? WE ARE TALKING NIH NISF DIVISION OF NISF MAYBE INSTITUTING A NIH PROGRAM LEVEL WHERE SOME SMALL PROGRAM HAVE 20 GRANTS AND SOME LARGE HAVE 300 GRANTS. SO I GUESS THAT'S WHERE -- IF WE APPLY THIS TO OVERLAP, IT MAY NOT BE -- THE ARROW BAR MAY NOT BE THE SAME AND OUTCOME MAYBE VERY INFLUENCED BY THE DATA SET. >> I THINK YOU'RE RIGHT BUT THAT'S MITIGATED TO SOME EXTENT BY USING A CONTEXTUAL MAP. OUR MAP HERE IS A GLOBAL MAP, CONTEXTUAL BECAUSE IT CONTAINS ALL THE INFORMATION. SO THEN IF A SMALLER DATA SET COMES IN, WHETHER BASED ON 20 GRANTS, 300 GRANTS OR FIVE GRANTS, IF ONE COMES IN AND OVERLAYS THAT, USUALLY THINGS ARE LOCALLY FAIRLY CONCENTRATED. FROM A SMALLER PROGRAM CONCENTRATED IN A SMALLER AREA, FROM A LARGER PROGRAM THEY'RE CONCENTRATED OVER MORE AREA. SO SOME OF THE EFFECTS OF SIZE ARE MITIGATED WHEN YOU USE A FINE GRAIN SYSTEM AS A TEMPLATE FOR OVERLAYING AND LOOKING AT THAT INFORMATION. >> THANKS TO THE SPEAKERS FOR SOME REALLY INTERESTING TOOLS. AS A NOVICE USER COMING IN, BEGINNING TO USE SOME OF THESE TOOLS, I GUESS I HAVE A COMMENT OR REQUEST MAYBE. SO THESE TWO DIMENSIONAL MAPS REDUCE HEIDI MENTIONALTIES OUT OF TWO DIMENSIONS. AS I USE MAPPING TOOLS, I HAVE OFTEN FOUND MYSELF WANTING TO LOOK AT SOME OTHER DIMENSION. YOU GAVE COUPLE OF EXAMPLE PES, MATURITY INDEX OR TEMPORAL BUT TOOLS THAT ALLOW US TO ROTATE PRINCIPLE COMPONENTS OR MAN FOALTDS OF THE DATA TO SEE RELATIVE RELATIONSHIPS ARE USEFUL FOR USERS. >> I AGREE WITH YOUR COMMENT AND ANECDOTE. WE HAVEN'T PURSUED IT FULLY BUT A MAP LIKE THIS ACCOUNTS FOR 30% VARIANTS IN THE SYSTEM THAT LEAVES VARIANTS. IF WE WERE TO TAKE DIMENSIONS OUT AND MANGE ANOTHER MAP IT WOULD LOOK DIFFERENT. AND THINGS THAT ARE CLOSE TOGETHER IN THIS MAP MIGHT NOT BE CLOSE TOGETHER. SO IF YOU TAKE THESE DIMENSIONS OUT THE NEXT WOULD BE INSTRUMENTATION, SOMETHING LIKE THAT. AND YOU FIND THINGS LUMPED INTO COMPUTER SCIENCE YOU DIDN'T EXPECT OR THINGS LUMPED WITH WITH VARIOUS INSTRUMENTS USED ACROSS DISCIPLINES. SO I AGREE WITH YOU. THAT IS BEYOND STATE-OF-THE-ART BECAUSE DIRECTION WE THOUGHT ABOUT. AND IT WOULD BE FASCINATING TO HAVE 3, 4 MAPS CLICKED BETWEEN BASED ON DIFFERENT COMPONENTS. >> IT MIGHT BE HELPFUL IN FINDING NEW GAPS, NEW WHITE SPACE. >> YOU'RE RIGHT. THAT COULD BE A WAY THAT -- WILL BEING FOR THINGS CLOSE TOGETHER IN ONE SET OF DIMENSIONS THAT WEREN'T IN ANOTHER IS A POSSIBLE WAY TO CHARACTERIZE THE WHITE SPACE. >> (INDISCERNIBLE) AS I LOOK AT YOUR PICTURE, LARGE EXTENT ABILITY TO ANALYZE OR USE THEM FOR PURPOSES TO DRAW CONCLUSIONS, NEED TO BE ABLE TO GET RELIABLE DATA. AND STABLE TOPIC MAPPING THAT WORKS ACROSS, SHAKING HEADS, YOU CAN TELL ME WHY YOU DON'T, THAT WOULD BE WONDERFUL. I UNDERSTAND THIS COULD BE LARGE ANSWER SO IF YOU JUST SKETCH, HOW DO YOU GET FINE GRAIN CLASSIFICATION AT THAT LEVEL (INDISCERNIBLE)? >> AS I MENTION WE LINK THROUGH TIME. AND THERE ARE SOME RESEARCH PROBLEMS WHERE THE OVERLAP BETWEEN REFERENCE STRUCTURES IS VERY HIGH. THE INSTRUMENTATION BASE. THERE IS STABILITY AT THAT LEVEL. >> LET ME MAKE SURE I ASK THE RIGHT QUESTION. THE FIRST THING HAS TO BE ABLE THE TO DO IS IDENTIFY WHAT POINT IS, WHAT YOUR TOPIC IS. FROM WHAT YOU SAID I GET THE IMPRESSION YOU'RE LOOKING AT OVERLAPPING THE CITATION REGENERAL. >> WE'RE USING A CO-CITATION ANALYSIS. COST CITATION ANALYSIS TO GENERATE CLUSTERS, WE THEN USE TEXT TO POSITION THE CLUSTERS. , OKAY. YEAH. I GUESS WE'LL TALK AFTERWARDS. >> MY NAME IS MARK SHERRETT FROM NHLBI. HERE WITH ME AS THEY TROOB TRY TO ARTICULATE. IN BASEBALL THEY USE STATISTICS TO FIGURE OUT WHEN BASEBALL PLAYERS REACH A CERTAIN AGE AND PRODUCTIVITY WILL FALL OFF. ANDEEOLOGISTS DO THE SAME FOR MINES. CAN YOU USE THESE MAPS TO FIGURE OUT WHICH AREAS OF SCIENCE HAVE BEEN PLAYED OUT IN THEIR PRODUCTIVITY IS GOING TO FALL OFF? IS THERE A CERTAIN OR NEW TOPICS, IS THERE A DISTRIBUTION OF TIME THAT SHOWS THAT REACH A PEAK OF PRODUCTIVITY AND MAYBE WE SHOULD BE DIRECTING FUNDS TO NEW AREAS RATHER THAN THESE OLDER AREAS WHICH ARE PROBABLY MINED OUT? DOES THAT MAKE SENSE? >> GREAT QUESTION. WE'RE NOT THERE YET. BUT THAT IS WHAT WE'RE WORKING ON. WE'RE LOOKING FOR ANSWER TO THOSE QUESTIONS. AND COMING ONE THOSE METRICS A PRIORI CANNOT BE DONE THAT'S WHERE WE HAVE TO DO INTERVIEWS AND GET THE HUMAN EXPERT KNOWLEDGE ON THESE AREAS. DICK WILL SHOW IN THE TUTORIAL LATER ON WHERE -- WHAT'S INTERESTING IS THESE AREAS OF NON-NOVEL VERSUS NOVEL PROVEN AND NOVEL UNPROVEN HAVE DIFFERENT CHARACTERISTICS IN TERMS OF WHAT PROGRAM OFFICERS AND WHAT EXPERTS CONSIDER HOT AND COLD SCIENCE SO WE MOVE TOWARDS FINING INDICATORS FED BACK FROM THAT TYPE OF ENTERRUE PROCESS BUT IT WILL TAKE MORE DATA TO SOLIDIFY THAT. WE ARE HEADING THAT DIRECTION. >> I HAVE TWO QUESTIONS. FIRST IS THAT MAPS ASSUME EXISTENCE OF COMMON FRAME OF REFERENCE. YOU CAN'T BUILD MAPS IF YOU DONE HAVE EQUIVALENT OF GPS COORDINATES. IN TOPIC MAPS THEY KEEP CHANGING. I THINK YOU NEED TO FIND A WAY TO HANG THEM ON TO UNCHANGING REFERENCE FRAMES SUCH AS DISEASE. HOW MUCH DO DISEASES COME AROUND? THEY HAPPEN BUT ONE IN A COUPLE OF DECADES. >> FAIR. >> ONTOLOGIES ARE A WAY TO DO THAT. THERE'S LOTS OF THEM ACROSS THE STREET NLM MAINTAINS THE BEST ONTOLOGY COLLECTION OUT THERE. SO THAT'S ONE. THE SECOND ONE IS WHY IS THE NEED TO IDENTIFY GAPS, WHY WE IMMEDIATE TO DO IT IN A FULLY AUTOMATED MANNER? YOU MADE A COMMENT IF YOU ASK LEADING SCIENTISTS FACE TO FACE, THEY TELL YOU, THIS IS DEAD, THIS IS HOT. WHY NOT FIND THE TOP 100 POTENTIALLY CHANGING TOPICS AND DO A SURVEY? IT IS A MERIT OF MAN POWER AND PROBABLY ALSO THE FACT THAT WE LIKE TO DO MAPS. I'LL ADDRESS CHANGING NATURE OF THE MAPS. VERY TRUE, SO WHAT WE NEED TO DO AND WHAT WE HAVEN'T GOTTEN TO IS TAKING OUR TEMPORAL THREADS AND HAVING A HISTORY BACK 50 YEARS AN MAPPING THE WHOLE THING. THAT WILL COME CLOSER TO SHOWING, HAVING THE BROADER PICTURE WITHIN TO HANG EVERYTHING. OTHER THING I WOULD MENTION, MY IDEA OR SORE EYE ONTOLOGIES, MESH IS WONDERFUL. BUT I VIEW IT BETTER FOR RETRIEVAL THAN MAPPING. WE HAVE DONE EXPERIMENT WHERE IS WE USE MESH TERMS AS A BASIS FOR GENERATING MAPS. THE MAPS ARE LESS ACCURATE IF YOU BASE IT ONCO OCCURRENCE OF MESH JERMS THAN IF YOU USE THE FULL -- THE TEXT OF THE TITLE AND ABSTRACTS OR CITATION RECORD. WE HAVE DONE THAT, WE HAVE USED VARIOUS METRICS TO MEASURE THAT. ONE IS THE COHERENCE OF THE TEXT AND REFERENCE RECORDS, ONE IS WE TAKE THE GRANT ARTICLE LINKAGE, IF YOU ASSUME THAT FROM AN RO-1, THE ARTICLES THAT REFERENCE THAT SHOULD BE MORE CONCENTRATED RATHER THAN MORE DISPERSED IN YOUR SOLUTION. YOU EXPECT THEM TOPICALLY SIMILAR IF THEY COME FROM ONE GRANT. THE NUMBERS BASING MAPPING ON MESH GO WAY DOWN. SO MESH IS WONDERFUL FOR RETRIEVAL, NOT AS GOOD FOR MAPPING. >> ONE SMALL COMMENT. WHILE IT IS TRUE THAT THINGS LIKE MESH TERMS MAY NOT GIVE YOU AS PRECISE ANY GIVEN SLICE IT WOULD ALLOW YOU FAR MORE STABLE TEMPORALLY. BECAUSE IF YOU GO BASED ON REFERENCE SETS YOUR ABLE TO BE STABLE ACROSS, 2, 3, 4 YEAR BOUNDARY IS GOING TO BE VERY LOW. >> WE'RE OKAY WITH INSTABILITY. >> WE CAN PICK THIS UP AT PANEL QUESTION LATER. WE'RE GOING TO TAKE A SHORT BREAK. SINCE WE'RE RUNNING A LITTLE BEHIND WE'LL RECONVENE AT 10:35. [APPLAUSE] I' D LIKE TO INTRODUCE CHAOMEI CHEN ASSOCIATE PROFESSOR OT THE COLLEGE OF INFORMATION TECH NOL AT DREXE UNIVERSITY. HE'S A SCHOLAR AT (INAUDIBLE) UNIVERSITY OF CHINA. AND HE'S THE FOUNDER AND EDITOR IN CHIEF OF THE JOURNAL INFORMATION VISUALIZATION. HE PUBLISHED OVER 180 PEER REVIEW PUBLICATIONS IN MULTIPLE DISCIPLINES OF COMPUTER SCIENCE AN INFORMATION SCIENCE. AND CREATED THE WIDELY USED SOFTWARE SITE SPACE FOR VISUALIZING AND AND LYING EMERGING TRENDS IN SCIENTIFIC LITERATURE. SITES BASE USED BY USERS IN OVER 3800 CITIES AND 105 COUNTRIES. THE TITLE OF HIS TALK IS DETECTING POTENTIALLY TRANSFORMATIVE RESEARCH, THEORY AND EXAMPLES OF STRUCTURAL VARIATION. >> HI, HELLO, EVERYONE. SO I'M GOING TO TALK ABOUT THREE THINGS TODAY. THE MAIN TOPIC HERE IS TO INTRODUCE VISUAL ANALYTICS APPROACH THAT COULD HELP US TO IDENTIFY TRANSFORMATIVE POTENTIAL. I'LL EXPLAIN WHAT IS THAT. INTUITIVELY YOU CAN SEE THIS IS A -- THERE'S A WHITE SPACE OR BLUE SPACE OR (INAUDIBLE) CENTER THIS IS THE WORK DONE FEW YEARS AGO. I WILL EXPLAIN THE THEORY BEHIND WHAT MEASUREMENT METRICS WE CAN DEVELOP AND SOME STATISTICAL METHOD TO VERIFY TO GIVE CONFIDENCE HOW VARIABLE CAN GO IN THIS DIRECTION. SO IN CASE I RUN OUT OF TIME HERE FOR FURTHER DETAIL YOU CAN EMAIL ME, YOU CAN LOOK AT THE PAPER IN THE BOOK AS WELL. SO HERE IS THE PROBLEM. WE'RE GOING TO ANALYZE THE SCIENTIFIC KNOWLEDGE HOW DOES SCIENCE IMPROVE OR ADVANCE. SECOND ONE IS WHAT CAN WE SAY? WHAT CAN WE LEARN FROM THE PAST AND PRESENT TO BETTER PREPARE FOR THE FUTURE. THIS IS BIG -- THE LOWER LEVEL OPERATIONAL LEVEL AUTOMATED GOAL IS TO PRODUCE TOOLS TO HELP US WORK AT THE LEVEL OF OUR THINKING, WE'RE THINKING ABOUT CONCEPTS, THINKING ABOUT POLICIES, RELATING TO THAT LEVEL. SO MUCH FURTHER DOWN THAN SPEAKERS. WE LOOK AT THE KNOWLEDGE STRUCTURE AT THE PRESENTATION LEVEL PUBLICATION LEVEL AND PATENT LEVEL. SO THIS IS THE KEY MESSAGE THAT AGAIN, I WILL FOLLOW-UP WITH EXAMPLES. HERE IS THE KEY QUESTION WE'RE ASKING. IF WE THINK ABOUT KNOWLEDGE AS A SYSTEM, HOW DO WE MEASURE IMPACT? THE IMPACT HERE, I TRANSLATED THIS TO A QUESTION, IS TO IDENTIFY INFORMATION THAT WILL ALTER OR CHANGE THE STRUCTURE. KNOWLEDGE STRUCTURE VERY ABSTRACT STRUCTURE. THE SECOND PART CLOSELY RELATED IS HOW THIS INFORMATION OR BELIEF IS SPREAD AND INFUSED OVER OUR KNOWLEDGE STRUCTURE. SO THESE ARE THE CORE QUESTIONS. SO I'LL START WITH SOME OBSERVATIONS ABOUT THE DIFFERENCES BETWEEN EXTRINSIC METRICS AND INTRINSIC. SO CITATIONS FOR EXAMPLE, FOR PUBLICATION HOW MANY TIMES CITED, ONE OF THE EXAMPLES OF EXTRINSIC. THE CITATIONS INCREASE HOW FAST FREQUENCY LIKE THE FIRST AND ALSO TIME ONLINE,COULDN'T LOAD IT, COLLECT IT AND SO ON. AND ALSO PRESTIGIOUS AUTHORS, PRESTIGIOUS JOURNALS PUBLISHED. ALL THESE THINGS ARE VALUABLE GIVEN THE USEFUL INFORMATION BUT ON THE OTHER HAND, THEY ARE LOOKING AT SOMETHING OUTSIDE OR SOMETHING EXTERNAL FOR THE CONCEPT IN TERMS OF KNOWLEDGE. SO THEY -- WHAT THEY HAVE IN COMMON APART FROM THOSE STRENGTHS, ONE OF THE THINGS WE'RE LOOKING FOR IF WE WANTED TO ASK THE QUESTION, WHAT HAPPENS TO A PARTICULAR IDEA? THIS TYPE OF MEASURE UNLIKELY TO TELL US ANYTHING MORE SPECIFIC ABOUT THAT. THAT IS ONE SIDE. SO IN CONTRAST WE'RE LOOKING FOR SOME INDICATORS OR MEASURES TELL US MORE ABOUT THE CONCEPT, IDEA, KNOWLEDGE ITSELF. SO SEVERAL THINGS ABOUT THE CONCEPT WE CAN MEASURE OR WE WANT TO MEASURE THE NOVELTY THIS IS A NEW CONCEPT OR RELATIONSHIP IS NEW RELATIONSHIP, NOVEL RELATION. WE ALSO HAVE SOME OTHER MEASURES IN TERMS OF STRUCTURE, HOW CRITICAL ONE PARTICULAR CONCEPT IN THIS PARTICULAR STRUCTURE. HOW IMPORTANT THIS CONCEPT OR LINK, CONNECTION IS TO DO WITH OUR UNDERSTANDING OF THE WHOLE SYSTEM. SO IN TERMS OF ADVANTAGES OR THIS IS MORE RELEVANT TO OUR QUESTION IS WE CAN FIGURE OUT SOMETHING THAT'S MORE DIRECTLY ANSWER OUR QUESTION, WE'RE LOOKING FOR A NEW CONCEPT OR PAPER TO EXPLAIN WHY THIS PAPER IS GETTING PARTICULAR HIGHLY CITED HISTORY. SO THIS IS THE EXERCISE I'M GOING THE USE IN FOLLOWING PART OF THE TALK. SO THIS IS ONE OF MANY WAYS TO LOOK T THE LITERATURE, LOOK AT THE INFORMATION. BUT THIS IS PARTICULARLY RELEVANT FOR HOW WE IDENTIFY TRANSFORMATIVE POTENTIAL. TO BEGIN WORK THIS IS A VERY SIMPLE DIAGRAM. WE ARE LOOKING T T A KNOWLEDGE SYSTEM AS A PERFECT CIRCLE AS IS NOW OR BEFORE. MY HANDS ARE BUSY WITH CONTROLS HERE. THIS IS BEFORE IN A SIMPLE CASE WHAT ONE CAN EXPECT IS AFTER SOMETIME AFTER A FEW MONTHS OR FEW YEARS, WE CAN EXPECT THE SYSTEM IS NOTHING REALLY SURPRISING HAPPEN, IT WILL BE MORE OR LESS -- REMAIN THE SAME SHAPE AND LED WITH BIGGER, MORE PUBLICATIONS, THIS WILL BE LIKE NORMAL SCIENCE. IN A MORE REALISTIC MODEL WE EXPECT THIS. THIS IS WHAT WE HAVE RIGHT NOW OR SO FAR BASED ON OUR UNDERSTANDING, MAPS. UP TO A CERTAIN POINT OR TOMORROW, THIS IS -- THE SYSTEM IS UNPREDICTABLE, IT IS A VERY ODD SHAPE. THE ORIGINAL STRUCTURE HAS CHANGED. SO THIS IS THE HIGH LEVEL CONCEPTUALIZATION I HAVE IN THIS DIRECTION. SO WE'LL WORK OUT COMPUTATIONAL APPROACH AND SEE IF WE CAN DO SOMETHING BASED ON THE ACCOUNT INFORMATION AND IDENTIFY THOSE SIGNALS THAT CAN HELP US TO BE BETTER PREPARED FOR THOSE CHANGES. SO WE'RE LOOKING AT THE SYSTEM POINT OF VIEW. SO THERE ARE LOTS OF INFORMATION AS SOURCE OF INTERVENTION THAT WILL CAUSE THE SYSTEM TO PERFECT CIRCLE TO BECOME DIFFERENT SHAPES SO FOR EXAMPLE, POLICY IS ONE MAJOR FACTOR, IT WILL INFLUENCE WHERE PEOPLE HAVE -- CHOOSE, WHAT KIND OF AREAS. PATTERNS AVAILABLE PATTERNS AN NEW IDEAS PRESENTED IN PATTERNS -- PATENTS AN GRANTS, PEOPLE DOING WHAT. ALSO PAPERS, ALSO GOING TO TALK ABOUT BRIEFLY ABOUT RETRACTION, IMPACT OF RETRACTIONS WHEN PEOPLE PUBLISH THEIR WORK FOR SOME REASONS LATER ON MOST OF THE TIME THE WORK STAY VALID. SOME PAPERS AS I'LL SHOW YOU IN EXAMPLES, THEY HAVE TO RETRACT, THEY HAVE TO PULL OUT FROM THE LITERATURE. SO IT LEAVES SOME GAPS AND HOLE AS WELL. THESE FACTORS COULD BE USED AS A ONE WAY TO INTERPRET THE FRAMEWORK. HOW WE IDENTIFY POTENTIAL TRANSFORMATIVE WORK SO ALL THE FACTORS ARE POTENTIALLY TRIGGERS. SO THIS IS AN EXAMPLE BUT ULTIMATELY I WANT TO ACHIEVE THIS GOAL, LOOK AT THE SCIENTIFIC LITERATURE, LOOK AT CURRENT KNOWLEDGE, LOTS OF NOISE BUT CAN WE BOIL DOWN TO VERY SIMPLE, VERY CLEAR LOGIC RELATIONS IN TERMS OF CONNECTS BETWEEN CONCEPTS SO THAT WE CAN SEE CLEARLY MIX PROPOSAL, NEXT PAPER COMING OUT WHAT WAY, WHERE EXACTLY WE'RE MAKING CHANGES. SO IN THIS EXAMPLE WE START FROM NUMBER ONE AT THE LOWER LEFT CORNER SO THIS IS ABOUT IN THIS AREA BEFORE SEPTEMBER 11 THE JOURNAL BELIEVE TO GET POST TRAUMATIC STRESS DISORDER THE VICTIMS HAVE TO BE ON SITE. THEY HAVE TO WITNESS THE TRAGEDIES DIRECTLY. SO IF WE FORMULATE THIS KNOWLEDGE IN THIS FORMAL STRUCTURE IT IS LIKE A PERSON HZ THE TO WITNESS TRAUMA AND THAT'S A CONDITION THAT LEADS TO POST TRAUMATIC STRESS DISORDER. IN 2002 AFTER SEPTEMBER 11 THERE IS A PAPER, SIGNAL SENDING TO THE SYSTEM FROM HERE AS NOT ONLY THE POSSIBILITY. THERE IS ANOTHER POSSIBILITY THAT YOU CAN ALSO GET POST TRAUMATIC DISORDER WITHOUT BEING ON SITE. SO THIS IS A MAJOR PAPER STUDY YOU COULD WATCH A VIDEO, YOU COULD WATCH TV AND GET THE SAME EFFECT. SO IN THIS WAY THE STRUCTURE BEFORE IS CHANGED BY ADDING ONE MORE POSSIBILITY, ONE MORE LINK HERE. SO OF COURSE, IF YOU THINK ABOUT PUBLICATION, NOISE, GOOD NEWS R NOISE, BAD NOISE, BUT THE ABILITY TO TRACK DOWN TO THIS LEVEL IS REALLY UNCONSUMING FOR LOTS OF PEOPLE, FOR DOMAIN EXPERTS. SO EVENTUALLY I WANT TO LOOK AT THIS INFORMATION FLOWING INTO THE SYSTEM APPROXIMATE MEASURE WHAT IS THE TYPE OF INFORMATION LIKE THIS TYPE THAT WILL CAUSE THIS STRUCTURE CHANGE, IN CHANGE OF KNOWLEDGE STRUCTURE. SO THIS IS GOAL. HERE IS ANOTHER EXAMPLE IN THE PAST SUPPORTING SOME OF OUR HYPOTHESIS. THEY WILL PUT THESE HYPOTHESES TOGETHER TO FORMULATE A THEORY. THE THEORY WILL GUIDE US HOW TO DERIVE METRICS AND HOW TO GET THE COMPUTATIONAL APPROACH WORKING. THIS IS THE NIH NSF TRACES PRODUCT BUT THE POINT I CAN TAKE FROM HERE IS MULTIPLE STRAINS THIS DEVELOPMENT EVENTUALLY SEE THIS STREAM DOWN HERE CAN ONLY HAPPEN BECAUSE YOU HAD THE RIGHT INFORMATION AND THE RIGHT TIME. IN OTHER WORDS YOU HAD THE OPPORTUNITY TO BRIDGE STREAMS, CONVERGENCE AT VERY END. SO THIS IS A VERY IMPRESSIVE CONTRIBUTION FROM THAT ANALYSIS. ANOTHER EXAMPLE L SHOWS, THERE'S SOMETHING IMPORTANT LOOKING FOR CONNECTIONS, THERE'S SOME POSITIONS MORE IMPORTANT THAN OTHERS. IN THIS EXAMPLE WE'RE LOOKING AT PHILOSOPHERS. PHILOSOPHERS STAY REPRESENT THEIR OWN RULES OF FORCE ARE MORE IMPORTANT IN THE HISTORY. THAT'S LIKE SAYING SOMETHING, IS THIS A PEEK MOUNTAIN, IS THIS ANOTHER PEAK MOUNTAIN. IN BETWEEN THERE'S VALLEYS AND IF YOU CAN BRIDGE ACROSS VALLEYS YOU PROBABLY HAVE SOMETHING SPECIAL. THIS IS A SPECIAL CONNECTION HERE. ANOTHER SOURCE IS COMING FROM SOCIAL NETWORK. YOU'RE LOOKING AT THE SOCIAL NETWORK, HERE IS ANOTHER GROUP. THIS IS THE LINK THAT STANDS OUT, IT'S A VERY IMPORTANT KIND OF LINK AND THIS IS WHAT IS CALLED A STRUCTURAL WHOLE THEORY IN SOCIAL NETWORKS. NETWORKS OF PEOPLE. SO WE EXTEND THIS BUN SO INSTEAD OF IN ADDITION TO THE HUMAN CONNECTIONS, WE'RE LOOKING AT WE HAVE THE SIMILAR EFFECT BY LOOKING AT THOUGHTS AS CONCEPTS. YOU HAVE KNOWLEDGE, YOU HAVE CONCEPT, YOU HAVE ISLANDS LIKE THIS. BUT WHAT DOES IT MEAN? IF YOU SEE A CONNECTION THAT BRIDGE DIFFERENT CONTINENTS OF OF IDEAS. SO THESE ARE THE CONTINENTS. THIS IS A SYSTEM. WE ARE LOOKING FOR MEASUREMENT OF THE TYPES OF SIGNALS, TRYING TO MEASURE OR EVALUATE HOW LIKELY COMING IN -- INCOMING SIGNAL WILL CHANGE THE STRUCTURE, MAKE FUNDAMENTAL CHANGES. SO WE'LL WORK OUT VARIABLES. WE REPRESENT KNOWLEDGE STRUCTURE WHOLE SYSTEM AS NETWORK. WE'RE LOOKING AT SEVERAL STRUCTURE COMPONENTS OR CLUSTERS. SO THESE ARE CLUSTERS REPRESENTS A SCHOOL OF THOUGHT OR A STREAM. WE'RE LOOKING AT STRUCTURAL HOLDS IN THIS SENSE. THIS IS THE STRUCTURAL HOLE THAT YOU HAVE THE CONNECTIVITY BETWEEN DIFFERENT ISLANDS OR CONTINENTS. WE'RE LOOKING AT A NEW SIGNAL. IF IT CAN HAVE THIS TYPE OF THING. FIRST AT LONG BRIDGE BETWEEN TWO CONTINENT, A SECOND ONE IN A BRIDGE ALREADY THERE, MAYBE A SECOND BRIDGE. THE SIGNAL COULD REINFORCE WHAT IS A REALLY QUITE SOLID CONNECTION. SO HERE, OR LOCALLY IS NOT FOUND BEFORE BUT IS A MORE PREDICTABLE THAN THE LONGER ONES. THE LONGER ONES REALLY SURPRISE BUT THE SHORTER ONE IS SOMEHOW LOCAL -- IS NEW BUT YOU CAN RIDE IN A WAY BY THE CON TEXT. WE'RE LOOKING AT THIS AND DERIVE SOME METRICS. THOSE METRICS GIVE US THIS MEASURE. THE INFORMATION COMING IN SCORE VERY HIGH WHICH MEANS THIS IS THE STRUCTURE IN THIS WAY, THE WHOLE SYSTEM WILL BE AFFECTED. IF THE MEASUREMENT HERE TELL US SOMETHING REALLY IS USEFUL, REALLY EXPLAIN WHAT'S HAPPENING, WE ALSO EXPECT THAT IF WE LET THE TIME RUN A WHILE, COMPARE WITH WITH LATER METRICS EXTRINSIC METRICS LIKE CITATIONS, WE EXPECT ONE OF THOSE LINKS IS PROBABLY GIVE US SOME SIGNAL, WE'LL TRY THAT, TO FINE OUT IF THIS IS POSSIBLE THEN WE COULD USE (INAUDIBLE) TO ALL THE PUBLICATIONS AS SOON AS THEY GET PUBLISHED OR EVEN BEFORE. AS SOON AS YOU GET A COPY TO THAT YOU CAN FEED INTO THE SYSTEM AND I WILL GENERATE SOME INDICATORS, THIS IS THE CHALLENGE, THIS IS THE POTENTIAL YOU COULD HAVE AND ALSO IT WILL PINPOINT EXACTLY WHICH LINK YOU ARE MAKING NEW INFORMATION IS INTRODUCING TO THE SYSTEM. SO THIS IS AN ALTERNATIVE VIEW OF THIS THEORY. THE THINKING. SO EACH SPECIALTY OR TOPIC COULD DEVELOP LIKE THIS. THE INFORMATION HERE LOOKING FOR TRANSFORMATIVE IS THE BRIDGING DIFFERENT STREAMS OR DIFFERENT SPECIALTIES. SOMETIMES OF COURSE IMPORTANT CAN HAPPEN AT A SINGLE STREAM BUT WE DON'T CALL THAT TRANSFORMATIVE. INCREMENTAL CONTINUE WITHIN EXISTING FRAMEWORK. TO BRIDGE DIFFERENCE SPECIALTY IT TAKES SOMETHING REVOLUTIONARY, YOU NEED TO BUILD A CONCEPTUAL NETWORK. THE LINKAGE AND THINGS WE TE RIEF SEVERAL -- DERIVE SEVERAL MEASURES TO THAT DO THIS EMPERIMENTAL TEST. WE DO THIS AS STREAM OF PUBLICATIONS AND MEASURE TO WHAT EXTENT THEY CHANGE THESE THREE SYSTEM LEVEL MEASURES. WE TRY TO USE THIS TO BUILD A MODEL TO PREDICT THE CITATIONS IN THE FUTURE. REWOIND BECOME A FEW YEARS. SUCH A CHANGE MEASURE IS LIKE THIS AND IN FEW YEARS LATER WHAT IS THE CITATION. SO WE HAVE THE NETWORK DIAGRAM LIKE THIS, IF YOU MAKE A LINK ACROSS DIFFERENT EYE LANDS YOU GET MORE SCORE IN STRUCTURAL VARIATION HERE. EVENTUALLY FOR EACH PUBLICATION IN THIS CASE THEY CAN MAKE ALL SORTS OF CONNECTIONS. ALL TOGETHER EACH HZ A SCORE SUMMARIZED. THIS IS THE FIRST EXAMPLE WE TRIED. WE KNOW SMALL WORLD NETWORKS AND MR. CXFC OR LESS STARTED WITH ONE PAPER. WE LOOK AT THE SNAP SHOP BEFORE PUBLICATION OF THAT PAPER AND LOOKING AT THE LINKS OR THE NEW LINKS ADDED BY PAPER PARTICULARLY TO THE LANDSCAPE CONNECTIONS THAT WERE NOT FOUND IN THE CONTEXT OF THE LITERATURE JUST BEFORE THAT PUBLICATION. IN OTHER WORDS THE SIGNALS SEND BY THIS PAPER IS MAKING CONNECTIONS OF DIFFERENT ISLANDS. HERE IS ONE ISLAND SOMEHOW CONNECTED TO HERE, ANOTHER BIGGER ISLAND HERE. IN THIS EXAMPLE WE KNOW THIS IS TRANSFORMATIVE ALREADY, WE'RE LOOKING BACK TO SEE WHERE THE LINKS ARE ADDED AT THAT TIME. THIS IS THE SAME NETWORK. WE AGGREGATE THIS TOGETHER BASED ON STRENGTHS, THE TWO PAPERS CO-CITED CLOSELY STRONGLY THEY WILL BE ALONG THE AIM ISLAND -- SAME ISLAND. SO WE'RE LOOKING AT ALL THIS SAYING THIS IS BEFORE THE PUBLICATION TO HAVE THE NEW PAPER, SO BEFORE WE RECEIVE THAT SIGNAL, BEFORE THE SYSTEM RECEIVE SIGNAL. THIS IS AFTER. AFTER THE PUBLICATION OF THAT, WE LOOK AT FURTHER PUBLICATION DOWNSTREAM P PUB 15 PAPERS THAT MOST LIKELY -- HIGHLY CITED, THOSE PAPERS REALLY FOLLOW-UP THIS TRANSFORMATIVE DIRECTION. SO INTERESTING THING THIS IS A PA ATTORNEY THE PAPERS FOLLOW THE INITIAL PUBLICATION IS STRENGTHENING THE CONNECTIONS TO THE ISLAND. IN THIS WAY WE CAN CONCEPTUALIZE TRANSFORMATIVE RESEARCH LIKE SOMETHING SOMEONE BUILD A BRIDGE FOR AND LOTS OFD FOLLOWERS AND WE CAN SEE THAT AS ALSO AS KNOWLEDGE DIFFUSION OR HOW KNOWLEDGE IS SPREAD OVER KNOWLEDGE STRUCTURE. PERCEIVE RISK ACROSS THIS ROAD BETWEEN THE TWO ISLANDS. AFTER THAT SOMEBODY DEMONSTRATE THIS IS POSSIBLE, THIS IS CONCEIVABLE AN PERCEIVE MUCH LOWER, AND OTHER PEOPLE CAN FOLLOW THE THRESHOLD. SO PUTTING THIS IN A STATISTICAL MODEL. THE DEPENDENT VARIABLE IS THE CITATION COUNTS FOR PAPERS, WE USE THESE MEASURES IN THE PAST PEOPLE HAVE BEEN USING MEASURES SUCH AS THE NUMBER OF CO-AUTHORS ON A PAPER, TEND TO PREDICT HIGHLY CITED PAPER, YOU HAVE MORE AUTHOR, YOU CAN EXPECT CITATION WILL BE HIGH. TO SOME EXTENT. THE NUMBER OF REFERENCES LOTS OF REFERENCES INDICATOR. LOTS OF STUDIES SUPPORT THAT. THE REASON WASN'T CLEAR. SO THE NUMBER OF PAGE, THERE'S ANOTHER INTERESTING ONE. THE LONGER PAPERS HAVE MORE CITATIONS. IN ORDER TO COME PAIR WITH REFERENCE TO EXTRINSIC MEASURES HOW DOES THE STRUCTURAL VARIATION MEASURES TELL US, THE RATIOS ARE OURS HERE, WE NEED TO FOCUS ON THIS NUMBER. A NUMBER GREATER THAN ONE IS GOOD. WHICH MEANS IF THE VARIABLE INCREASE IN THAT VARIABLE, AND WE CAN EXPECT CITATIONS ALSO INCREASE. SO GREATER THAN ONE THE STRONGER THE PREDICTIVE POWER. SO FOR EXAMPLE, THIS IS CLUSTER LINKAGE, HOW STRONGLY, HOW FREQUENTLY THIS SIGNAL IS BUILDING BRIDGES BETWEEN CLUSTERS. SO THIS ONE PROVIDE LIKE THREE TIMES HIGHER PREDICTION THAN OTHER INDICATORS. VERY SENTIVE SIGNAL, MEASURE. WE CAN EXPECT THE DIFFERENCE WAYS TO USE DIFFERENT UNITS OF ANALYSIS, CITED REFERENCES, CONCEPTS OR KEY WORDS, NOUNS. THIS IS A GENERIC FRAMEWORK WE'RE LOOKING AT, THIS IS THE FIRST EXAMPLE VERY PROMISING, WE CAN IDENTIFY NEW METRICS AND MEASURE INCOMING INFORMATION. THIS ONE WE DONE HAVE TO WAIT MANY YEARS TO GET DOWNLOADS AND SIGHATIONS. AS SOON AS WE GET THAT, IT CONTAINS ALL INFORMATION WE NEED TO WORK OUT HOW MANY NEW LINK, HOW MANY NOVEL LINKS TO BUILD THIS. SO ANOTHER EXAMPLE IS TO VERIFY IT COULD HAPPEN IN REALITY. THIS IS ANOTHER EXAMPLE BEFORE THIS PUBLICATION. IMAGINE PUBLICATIONS COMING IN LOOKING AT WHAT'S HAPPENING BEFORE THIS POINT, YOU LOOK AT THE MAP, WHERE THE ACTIVITIES WHERE MOST PEOPLE -- WHERE MOST PEOPLE ARE DOING THINGS DIFFERENTLY FROM YESTERDAY. SO SO THERE'S LOSS OF CONTINUITY, PEOPLE DOING SAME THINGS OR SIMILAR THINGS BUT ON THE OTHER HAND WE CAN IDENTIFY THE POINT OF THOSE AREAS OF PEOPLE DOING DIFFERENTLY THAN BEFORE. THIS IS SOMETHING NEW, SOMETHING QUITE DIFFERENT AS FAR AS RIGHT CONTEXT HERE THE BASIS THEY'RE TELLING US. SO THIS IS PART OF THE KNOWLEDGE IS VISUAL ANALYTICS CAN GUIDE US WITH DIFFERENT INFORMATION, I WAS HOPING THIS BECOME A TOOL FOR INDIVIDUAL SCIENTISTS THAT CAN USE THIS ONE TO IDENTIFY EXACTLY WHERE THE NEW PUBLICATION DIFFERS IN TERMS OF KNOWLEDGE STRUCTURE. SO THIS IS REGION RA ACTIVE MEDICINE, THE COLOR INDICATES THE CITATION. THE WARMEST COLOR INDICATE IT IS MOST RECENT CITATION WHERE IT GOES. SO HERE IS THE RED ONE, HERE IS A RED ONE AS WELL. THERE'S SOME OLDER PUBLICATIONS, OLDER TOPICS. SO FROM HERE WE CAN ALSO ANALYZE THE SYSTEM LEVEL AND MEASURE THE TEMPERATURE OF THIS SYSTEM, HERE IS THE SYSTEM MEASURE USING ONE OF THE POPULARITY AND NEUTRALITY, CONVERGENCE AN THOSE THINGS SO WE CAN MEASURE SYSTEM -- TEMPERATURE IS MORE OR LESS STABLE FOR THE FIRST FEW YEARS, AND SUDDENLY THERE'S A DROP. SO THE CONCEPT FOR (INAUDIBLE) IS IF YOU ADD NEW BRIDGES YOU REDUCE THE MODULARITY. MODULARITY IS THE NETWORK DIVIDED NETWORK INTO ANALYTIC PIECES BUT IF YOU ADD MORE CONNECTS TO IT, DESTROY THAT MORE AN MORE MODULARITY SO SYSTEM WIDE IT DECKS TWO THINGS. TWO EVENTS. TWO TIME POINTS. THE SYSTEM HAS FUNDAMENTALLY CHANGED THE STRUCTURE. SO NOW WE KNOW THAT BEFORE WE GET TO LOWER LEVEL WE KNOW THAT TWO THINGS HAPPENED. VERY SIGNIFICANTLY REVOLUTIONARY, WE CAN EXPECT BUT EXACTLY WHAT WILL HAPPEN, WE HAVE TO DRILL DOWN. THIS IS A MUCH LOWER LEVEL NETWORK. THIS ONE SHOWS THE HISTORY THE COOL COLORS THE COAL COLORS ARE LONG TIME AGO, THE WARMER COLORS ARE MORE RESEN. THE THINGS IN RED, THEY ARE THE ONES THAT ATTRACT THE CITATION BURST, SUDDEN INCREASE, ACCELERATION, PEOPLE STEP DOWN INTO THIS TOPIC. SO THESE ARE THE TITLES THEY WRITE FROM PAPERS THAT'S SITING ALL -- CITING THESE NUMBERS OR PAPERS. SO IN OTHER WORDS LOOKING AT IMPACT IN THIS WAY BECAUSE SOMETHING IS IMPORTANT THERE, FOR EXAMPLE SOMETHING -- YOU'RE LOCKING AT WHO IS LOOKING AT THAT AREA, AND WHAT ARE THEY WRITING. SO THESE ARE LABELS EXTRACTED FROM THE CITERS. THE PEOPLE CITING THIS AREA. SO THIS WILL TELL US THE TOP YOU KNOW AREAS THEY'RE WORKING ON. YOU CAN ALSO SEE A TREND PEOPLE HOW FAST PEOPLE MOVING, INSTABILITY WITH PEOPLE CHANGE THEIR RESEARCH FOCUS AT THIS LEVEL. SO IN TERMS OF A TOOL LEVEL WE CAN ANALYZE WHAT IS THE NUMBER OF THIS CLUSTER, GROUP OF CO-CITED REFERENCES? THESE ARE MEMBERS AND THESE ARE CITERS, THESE GUYS CITE ALL THESE MEMBERS SO WE HIGHLIGHT WHAT IS IN COMMON HERE, THIS BECOMES A LABEL OF THAT CLUSTER OF THE WHOLE. SO THIS IS THE MOST INTERESTING CLUSTER, THE -- CO-AUTHOR, THE MOST RESEN ONE AS WELL. IN TERMS OF MEASUREMENT, EACH NEAL PAPER WE CAN PUT A PRICE TAG ON IT IN TERMS OF TRANSFORMATIVE POTENTIAL. SO WE CAN SELECT THOSE AND SHOW -- SHOW ME WHERE EXACTLY WHY YOU ARE POTENTIAL, YOU HAD POTENTIAL. BECAUSE THIS PARTICULAR PAPER LINKING, EACH (INAUDIBLE) IS A ONE GROUP CORRESPOND TO TOPIC. EACH ONE THE LINE DRAWING DIFFERENT TOPIC AN EXAMPLE AS WE SEE IN EARLIER EXAMPLES, WE HAVE IN THEORY OR PRINCIPLE DOING SOMETHING UNUSUAL. SO AT THIS POINT WE CAN SAY THEY'RE DOING SOMETHING UNUSUAL BUT WHETHER THIS UNUSUAL IN A GOOD WAY OR BAD WAY WE COULDN'T TELL BECAUSE DATA DOESN'T PROVIDE US ANY INFORMATION LIKE THAT. WE FOUND SOME COUNTER EXAMPLES DOING SOMETHING UNUSUAL BECAUSE OF SOME OTHER REASON, IS NOT TRULY TRANSFORMATIVE. THE POINT IS WE CAN NARROW DOWN THE LIST TO GIVE US A SHORT LIST, THE PEOPLE VERY SUSPICIOUS AND DOING SOMETHING DIFFERENT FROM H WE KNOW WHAT WE EXPECT. THEN WE CAN ASK EXPERTS OR DO FURTHER STUDY TO FIGURE OUT WHAT IS THE REASON. ALONG THIS LINE, MY PHILOSOPHY KEEP AN OPEN MINE EVEN YOU COULDN'T EXPLAIN WHY IT IS GOOD TO DO THIS WAY. WE BETTER FIND OUT WHAT IS THE POSSIBILITY, HOW WE CHANGE TO OUR EXISTING SYSTEM. SO THIS IS AN EXAMPLE, SOMETIMES THE STAR SHOWS THIS IS THE PAPER ITSELF. SO IF YOU LAID A TIME FLOW FOR A WHILE, THIS STARRED SITE ADDING THE NEW LINK HERE, AND ITSELF AS THAT IS AN IMPORTANT PART OF THE LANDSCAPE. SO IN THIS WAY WE CAN SEE THE CONTRIBUTION OF STAR IS TOWARDS THE CLUSTER NUMBER NINE ITSELF FELL INTO A DIFFERENCE CLUSTER. SO IN THIS WAY RESEARCHERS OR SCIENTISTS INDIVIDUALLY YOU CAN LOOK AT ALL THIS FOR NEW PUBLICATIONS, YOU LOOK AT WHAT IS NEW PUBLICATION ADDING ALL THIS DIFFERENT LINKS. IN PARTICULAR THIS IS A SOME SORT OFFER WAY TO LOOK AT LITERATURE. THE FIRST STEP YOU LOOK AT THE CITATIONS OR LINKS ADDED BY THE NEW PAPER, IF THAT SURPRISE YOU, YOU CAN GO AN READ IT. OTHERWISE YOU CAN WAIT FOR SECOND PAPER. SO THESE ARE CITATIONS PROVIDED BY ONE PARTICULAR PAPER CONNECTING THESE TWO ISLANDS, AGAIN, WE'RE LOOKING AT ONE PARTICULAR CASE IS THAT NEW BRIDGES IF F YOU ADD MORE NEW BRIDGES TO THE ISLANDS, BETWEEN THESE ISLANDS, OUR EXPECTATION IS THERE'S SOMETHING IMPORTANT TO GIVE IT SOMETHING NEW. SO THIS IS ANOTHER WAY TO LOOK AT THIS, EACH LINE HERE IS ONE CLUSTER, WE CAN SEE EACH DEVELOPMENT OF A TOPIC. SO AGAIN, THE CROSS -- THE LINES3J THAT CROSS DIFFERENT LINES, THAT CROSS DIFFERENT CLUSTERS OR SPECIALTIES, THEY HAVE HIGHER TRANSFORMATIVE POTENTIAL. AT THE START, A STAR MEANS THIS IS THE PAPER RESPONSIBLE FOR THOSED IDEAS. THESE IDEAS ARE NEW IDEAS, NEVER FOUND IN OUR DATA SET. SO THIS IS ANOTHER EXAMPLE. THIS PAPER ACTUAL THROUGH ADDING THOUGH LINES. SOME SOLID LINES WHICH MEANS IS REPEATING SOME OTHER LINES FOUND IN THE LITERATURE BUT THE OTHER LINES ARE NEW. O IN THIS WAY YOU CAN EXAM PAPERS ONE BY ONE AND YOU CAN LEAVE THEM PATENTS AND OTHER (INAUDIBLE). SO THIS IS GOING DOWN FURTHER LEVEL. OUR INTERESTING -- INTERESTED IN THE RELATIONSHIPS BETWEEN CONCEPTS AND HOW DO WE GENERATE THIS CONCEPT RELATION DIRECTLY FROM THE ABSTRACT OR PAPERS. SO WE'RE BUILDING THIS TREE, THE TREE WILL HIGHLIGHT THE KEY TOPIC OR KEY THINGS TO TALK ABOUT. THIS IS NO TYPE OF MODELING INVOLVED. PURELY BASED ON NATURAL LANGUAGE PROCESSING. TH WON'T TELL US REPROGRAMMING THAT'S A HIGH LEVEL CONCEPT UNBENEATH, THERE ARE LOTS OF OTHER WORKS USED BY AUTHORS AND SCIENTISTS IN THIS CONTEXT. SO WE CAN HAVE THIS SEVERAL LEVEL, THIS COULD BE A TREE WE CAN START FROM THE BEGINNING OF THE TREE AND UNDERSTAND WHAT IS THE TOPIC OF THIS CLUSTER, WHAT IS THE TOPIC OF THIS SPECIALTY. AND THEN WE CAN UNDERSTAND WHAT IS THE LINKAGE BETWEEN DIFFERENT CONCEPTS. SO THAT WAS CITATION, NOW I'M GOING TO TALK ABOUT RETRACTION. RETRACTION IS DEFINED BY THE FORMAL ANNOUNCEMENT CERTAIN PAPER HAS TO BE RETRACTED, THE REASON SOMETIMES IS TECHNICAL ERROR, SOMETIMES THERE'S MORE COMPLEX REASONS BEHIND. SO THE REASON FOR THIS, WE LOOK AT PUBMED TO SEE HOW MANY PAPERS GET INCLUDED IN PUBMED EVERY YEAR IS GROWING. ALSO WE LOOK AT THE BLUE LINES ARE RETRACTION NOTED. RETRACTION NOTICED OR ANNOUNCED EVERY YEAR. SO MORE PROBLEMATIC PAPERS, AND THE ACTUAL PAPER RETRACTED THIS IS THE -- THIS ONE IS NOT GOING DOWN, THE DATA HAS NOT ARRIVED YET. SO WE DID SOME STUDIES. AN AVERAGE IT TAKES TWO YEARS FOR PUBLICATION THE GET RETRACTED IF THERE'S SOMETHING WRONG WITH IT AND ANOTHER TWO YEARS BEFORE EXITATION TO LOSE CITATION BECAUSE OF THAT. SO THE TIME GUIDE IS TWO YEARS PLUS TWO YEARS. SIGNIFICANT PROBLEM POTENTIALLY THAT COULD DAMAGE EVERYTHING WE BUILT ON SUCH LITERATURE. SO THIS IS A DIAGRAM, A MAP. THE RED DOTS ARE (INAUDIBLE) THE RED DOTS ARE RETRACTED ARTICLES. IN OTHER WORDS, RED DOTS ARE THINGS WE SHOULDN'T BELIEVE. AND SUBSEQUENT PUBLICATIONS, IF THEY REFERENCE OR BUILD DOWN RED DOTS, THEY SHOULD BE ALSO RETRACT TRACTED IN A SENSE. HOWEVER THE CURRENT PRACTICE ONLY START AT THE FIRST STEP. RETRACTED PAPERS THEY PULL IT OUT, THE PAPERS AFTERWARDS CAN STILL CITE THOSE PAPERS AND STILL FEEL THEIR RETRACTED THEORY, RETRACTED MODEL, RETRACTED DESIGN. SO LOOK AT THIS CLOSELY, THE SIZE RED DOTS IS THE CITATIONS RECEIVED BY RETRACTED PAPERS. THE (INAUDIBLE) IS THE CO-CITED PAPERS. SO FOR EXAMPLE, HERE THE MOST DANGEROUS THING IS IF YOU HAVE A RETRACTED PAPER SIT IN THE MIDDLE OF AN ISLAND, ALL THE OTHER PART OF ISLAND NOT RETRACTED. SO IN OTHER WORDS, THIS ONE HAS POTENTIAL TO DAMAGE THE WHOLE ISLAND BECAUSE EVERYBODY IS (INAUDIBLE) WITH THAT. AT THIS POINT WE DONE KNOW THE CITATIONS POSITIVE OR NEGATIVE IS THE AUTHOR AWARE OF RETRACTION. SO WE HAVE FURTHER DRIVERS EVEN DOWNWARD IF YOU LOOK AT THE DETAIL. THE SENTENCE, THEY HAVE MENTIONED THIS. SO THIS IS A MAP JUST TO SHOW THE EXTENT WE HAVE THIS TYPE OF PROBLEM. SO THE SINGLE BLACK DOT AT THE BEGINNING -- CENTER HERE IS RETRACTED PAPER AND HOWEVER, THE RED DOTS ARE ONES CITING OR -- CITING THIS BLACK DOT. CITING AS I MENTION COULD BE SEVERAL STAGES. CITING BEFORE THE RETRACTION CITING AFTER THE RETRACTION WITHOUT WORRYING THIS IS RETRACTEDDED OR CITING AFTERWARDS AS A BAD EXAMPLE, SOMEBODY ELSE HAS DONE STUDIES RETRACTED. THEN THE BLUISH DOT THEY CITE RED ONES, THEY ARE THE SECOND DEGREE -- THEIR INFLUENCE INDIRECTLY BY TWO STEPS. BUT WE CAN LOOK AT THIS, IT'S REALLY HARD TO TRACE EVEN THE ONE'S GETTING HOTTER AND HARDER TO TRIS. SO LOTS OF PUBLICATION BEFORE WE CAN HAVE TIME TO ANALYZE THEM, NO MATTER RETRACTED OR NOT, IT BECOMES MORE CHALLENGING TO FIGURE OUT THE FOUNDATION OF OUR WORK IS IT VIOLATED OR IS IT NOT VIOLATED? SO THIS IS JUST SAYING THAT TRANSFORMATIVE, IDENTIFY TRANSFORMATIVE WHITE SPACE AND (INAUDIBLE) DARK SPACE, WE ALSO NEED TO PAY ATTENTION TO THIS TYPE OF THING. SO THEY NEED TO BE -- LOOK AT IN AN INTEGRATIVE FRAMEWORK. SO THIS IS A TIME LINE, AGAIN, BUT WHAT IS THE WORRY, WORRYING FROM IS THAT THE RED CIRCLES ARE CITATION FIRST WHICH MEANS CITATIONS ACCELERATED AND THE BLUE LABELS ARE RETRACTED. THIS BECOMES PROBLEMATIC BECAUSE OF THE MOST ACTIVE RESEARCH AREAS RETRACTED, WE HAVE TO BE VERY CAREFUL ON ONE HAND LOOKING AT HIGHLY CITED PAPERS. ON THE OTHER HAND THE NEIGHBORS NEXT TO US MAYBE COULD BE A TRACK. SO WE'RE LOOKING AT THE SENTENCES, THE CITING OF THE RETRACTED PAPER, TRYING TO FIGURE OUT, SUPPOSE THIS IS THE BEGINNING OF THE PAPER GETTING ALL THE CITATIONS, IN THE MIDDLE SOMEWHERE IS RETRACTED. WHAT IS THE USAGE OF THAT? WHAT IS THE EXACT TERM YOU HAD BEEN USING? SO THIS IS THE INFAMOUS PAPER RETRACTED THE KOREAN PUBLISHED IN SCIENCE. SO PUBLISH HERE, RETRACTED HERE. AIM RUNNING OUT OF TIME? OKAY. WE HAVE -- GO THROUGH THIS VERY QUICKLY. SAME THING APPLIED TO PATENT CITATIONS LOOKING AT THE DIFFERENT PATENTS FROM PICTURES. SO THE SAME FRAMEWORK LOOKING FOR SOMETHING UNUSUAL WE'RE LOOKING FOR. I'LLUe>A HERE. THANK YOU. [APPLAUSE] I HAVE SOME TIME TO TAKE SOME QUESTIONS. >> YOU EAR MAKING AN ASSUMPTION WHEN YOU LOOK AT THE PAPERS AND THE INFERENCE THAT IT IS >> YOU'RE MAKING AN ASSUMPTION THAT ONLY ONE PAPER IS INFLUENCING THE OTHERS BUT THERE ARE OTHER THINGS GOING ON TAME. COULD YOU G INTO THAT AS WELL? BECAUSE IT'S NOT A SINGLE PAPER THAT IS INFLUENCING THE FIELDS. >> YOU'RE RIGHT. WE START FROM SOMEWHERE. THE THINGS REALLY COMPLEX, IT COULD BE A PAPER, IT COULD BE SIMULTANEOUSLY A BUNCH OF PAPERS. YES. WE'RE DEALING WITH ONE PAPER FIRST. WE'LL EXPAND THE APPROACH. SO THEORETICALLY IT'S THAT YOU CAN SEE THE 15 PAPERS FOLLOW, THEY BUILD BRIDGES SORT OF TOGETHER. >> HI, HAVE YOU HAD AN OPPORTUNITY TO THINK ABOUT THE NATURE OF THE INVESTIGATORS THAT ARE MORE LIKELY ACROSS DISCIPLINES? LOOK AT THE PEOPLE, THE CHARACTERISTICS OF THEIR TRAINING OR SOMETHING LIKE THAT THAT MAKE PEOPLE MORE LIKELY TO BRIDGE DIFFERENT PLANS THAN OTHERS IN >> YEAH, IN A WAY THAT THE FACT PEOPLE USE BEFORE THE NUMBER OF AUTHORS ON A PAPER PREDICTING THAT. IS IN A WAY THE NUMBER OF DISCIPLINE, NUMBER OF DISCIPLINES, YOU CAN EXPECT PEOPLE MAY BRING IN DIFFERENCE IDEAS. SO THE BOTTOM LINE, IF YOU HAVE DIFFERENT IDEAS MERGED FROM DIFFERENT PLACES, YOU HAVE A NEW IDEA THAT CAN ACCOMMODATE THOSE EXISTING IDEAS. SO YOU COULD IN A WAY DERIVE FROM NUMBER OF CO-AUTHORS, NUMBER OF INSTITUTIONS, BUT WE'RE LOOKING MORE DIRECTLY AT THE USING THIS CO-CITATION CLUSTERS, THERE IS A NOTION CALLED A CONCEPT ASSEMBLE. THESE ARE THE GROUP OF DOCUMENTS CITED TOGETHER, THEY REPRESENT SOME UNDERLYING CONFIDENCE. THAT'S DIFFERENT CONFIDENCE MERGING. >> OKAY. WE CAN CONTINUE THE DISCUSSION IN CHAOMEI'S BREAK-OUT SESSION. [APPLAUSE] >> THE NEXT SPEAKER IS DEWEY MURDICK. DEWEY IS MANAGER OF THE FUSE PROGRAM WHICH FOCUSES RESEARCH ON THE DEVELOPMENT AND VALIDATION OF INDICATORS OF TECHNICAL EMERGENCE FROM INFORMATION FOUND IN THE FULL TEXT OF SCIENTIFIC TECHNICAL AND PATENT LITERATURE. DEWEY ALSO SERVED AS AWARD-WINNING SCIENTIFIC AND TECHNICAL IN THE IN THE U.S. ARMY WHERE HE ASSESSED TECH CAP KALABLES AN SUPPORT DECISION MAKERS IN DODD. HE'S DEVELOPED EXPERTISE IN SCIENTIFIC COMPUTING, COMPLEX MODELING AND SIMULATION, TECHNICAL ANALYSIS, METHODS OF PRACTICES AND TEXT ANALYTICS. AND THE TITLE IS FINDING PATTERNS OF EMERGENCE IN SCIENCE AND TECHNOLOGY. >> THANK YOU VERY MUCH FOR THE INVITATION TO SPEAK AND I'M STANDING BETWEEN YOU AND LUNCH SO HOPEFULLY WON'T TAKE TOO LONG. AS FORMER ANALYST I FELT VERY KEENLY THE NEED FOR NEW DIMENSIONS TO ENABLE ANALYSIS TO MOVE BEYOND JUST THE KEY WORDS CITATIONS BECAUSE A LOT OF DIMENSIONS ARE MISSED WITHIN THOSE PARTICULAR FEATURES AT LEAST FOR BIBLIOMETRIC ANALYSIS. THERE'S OTHER TYPES TO SO I'LL COMMENT BRIEFLY AND TRY TO CONNECT TO OTHERS. AS COMPUTATIONAL PHYSICIST I A LOT OF MODELING SIMULATION METHODS RELEVANT UNDERSTANDING THE CAPABILITIES OF AUTHORS ARE BURIED VERY MUCH FURTHER IN THE CONTEXT OF THE FULL TEXT LITERATURE AND THE MEANING MIND THE CITATIONS AND SOME OTHER FEATURES THAT ARE NOT READILY ANALYZED IN THE STATE OF THE ART TODAY. AT LEAST AS OF A LITTLE WHILE AGO. THIS KEEN NEED FOR THIS TYPE OF DIMENSION ANALYSIS TO DISCRIMINATE BETWEEN ACTIVITY IN A PARTICULAR AREA, AN TRUE CAPABILITY IN A GIVEN AREA, WAS VERY IMPLICATIONS OF CAPABILITIES EMERGING WERE LEFT TO HUMAN PROCESS. NOT SUGGESTING THAT THINK SHOULDN'T BE A HUMAN PROCESS BUT THEY WERE -- THE LARGE SCALE DATA ANALYTICS COULD BE SUPPORT THAT -- COULDN'T SUPPORT THAT MANY REGARDS. MOVING ANALYST IN A SECOND DIMENSION ANALYST AT LEAST IN MY GROUP OF ANALYSTS NEEDED TO MOVE BEYOND, JUST MOVING BEYOND SEARCH. IN SEARCH YOU'RE PRNTED WITH A KISS OF IF I NEED TO KNOW ABOUT SOMETHING I CAN SEARCH FOR IT AND FIND GREAT INFORMATION. IF I KNOW IT'S CELL PHONE OR ELECTRONIC WALLET OR PARTICULAR TECHNOLOGY I CAN SEARCH FOR THIS AND GET INFORMATION THAT HELPS ME UNDERSTAND THAT AREA BETTER OR UNDERSTAND WHAT'S GOING ON IN THE AREA. HOW DO YOU SEARCH FOR SOMETHING YOU DONE KNOW THE NAME FOR? THE NAME HASN'T BEEN CRAFTED OR NO COALESCE SENSE ON THAT PARTICULAR TERM OR IT'S A CAPABILITY YOU'RE INTERESTED IN BUT NO IDEA WHERE IT'S MANIFEST. WHICH AREAS. HOW DO YOU INTERACT WITH DATA IN THAT WAY? THIS IS ONE THING THAT'S MOTIVATING THIS PARTICULAR PROGRAM. SO IN THE INTRODUCTION I AM PROGRAM MANAGER, EEL TELL YOU BRIEFLY ABOUT IARPA. IARPA IS A PROGRAM WHERE EVERYTHING WE INVEST IN IS HIGH RISK, HIGH PAY OFF. A LITTLE EXAWING EVERY ONCE IN A WHILE BECAUSE EVERYTHING IS REALLY EXCITING AN NEW. BUT IT PROVIDES A VERY NEAT VENUE FOR INTERACTING WITH PEOPLE WITH NEW IDEAS TO TRY TO FORCE THAT AND HOME THAT. WE HAVE GOT CONSTRUCTS TO HELP PEOPLE WORK WITH, THE QUESTIONS ARE USEFUL STRUCTURED TO KNOW WHETHER SOMETHING IS INNOVATIVE OR NOT. AND WHETHER THE RIGHT TIME TO INVEST IN THIS AREA. I HIT THERE'S BACK UP SLIDES, SLIDES ARE DISTRIBUTED. LET ME GIVE YOU SOME SENSE OF THE THREE THRUSTS OF OUR ORGANIZATION. INSIDETIVE ANAL -- INSIDESIVE ANALYSIS, COLLECTION OF DATA. SMART COLLECTION WHICH ARE ALL RELEVANT TO THE AREA THAT WE'RE MOTIVATED FROM. SO AS YOU MIGHT HAVE GUESSED, INTELLIGENCE, ADVANCED RESEARCH PRODUCTS ACTIVITY IS INTELLIGENCE ORGANIZATION FUNDED FOR THE IDEA BEING ABLE TO MOVE INTO DRAMATICALLY NEW AREAS AN CAPABILITIES SO PROGRAMS ARE VERY MUCH OPEN AS IS MINE. PART OF THE OFFICE OF DIRECTOR OF NATIONAL INTELLIGENCE. WHO IS HEARD OF IARPA BEFORE THIS? FAIR ENOUGH SORKS IT'S NOT A COMPLETE ENIGMA. THE POINT YOU WERE MENTIONING IS WE'RE FAIR AN OPEN ORGANIZATION. WHICH IS NOT NECESSARILY THE MOST COMMON ELEMENT AND THIS IS A NEW EXPERIMENT OF ACTUALLY TRYING THE TRULY ENGAGE, WE'RE VERY -- I'M PROUD OF OUR SELECTION PROCESS AND HOW WE DO THINGS IN A FAIR OPEN WAY. ONE THING THAT'S IMPORTANT TO KNOW ABOUT IARPA, THERE WAS A COMMENT THAT JOHAN MENTIONEDDED, BASICALLY BEFORE A PROGRAM STARTS ARE TO ENGINEER FAILURE POINTS, HOW WILL WE KNOW WHETHER THIS PROGRAM WILL BE SUCCESSFUL. HOW WILL WE KNOW WHETHER IT WILL BE A FAILURE? AND WHAT DIMENSIONS OF FAILURE POINTS THAT WE HAVE, SO THAT'S SOMETHING IARPA INVESTS IN BEFORE A PROGRAM STARTS TO UNDERSTAND THOSE PARTICULAR DIMENSIONS SO A REALLY SETTING PROCESS. BUT IF YOU WANT MORE QUESTIONS TO IARPA, MY POINT IS NOT TO TALK ABOUT IT BUT TO GIVE A CONTEXT OF THE REGION WE'RE STARTING ON. JUST A BRIEF OUTLINE, I'M GOING TO GIVE YOU A SENSE OF THE (INAUDIBLE) PROGRAM, TALK ABOUT SOME OF THE POTENTIAL IMPLICATIONS. WHAT WE'RE DOING IS MORE SOLID IMPLICATIONS, BECAUSE WE'RE SEEING IF WE'RE -- IF THIS IS POSSIBLE. ARE WE ABLE TO DO THE THINGS THAT WE'RE SENDING -- SETTING OUT TO TRY TO DO. AT THE FUSE PROGRAM MANAGER MY GOAL IS TO STRUCTURE A RESEARCH PROBLEM TO TRULY CAPTURE THE IMAGINATION OF RESEARCHERS AND SEE IF WE CAN GO FORWARD. SO THE FUSE PROGRAM FORESIGHT UNDERSTANDING FROM SCIENCE EXPEDITION, REALLY TRYING TO GET THE CONCEPT OF CAN WE DEVELOP SYSTEM TO AID NAILSIS, NOT TRYING TO REPLACE HUMAN BUT TRIED TO AID ANALYST TO GIVE THEM RELIABLE EARLY DETECTION OF EMERGING CAPABILITIES IN THE SCIENTIFIC CAPABILITIES REGARD, ACROSS DISCIPLINES, ACROSS LANGUAGES. ONE CORE HYPOTHESES IS FULL TEXT CONTENT OF SCIENTIFIC TEBCAL PA TEN LITERATURE WILL MAKE A DIFFERENCE. WE'RE STARTING WITH SCIENTIFIC PATENT LITERATURE, MOVING TO INFORMAL DISCOURSE. SO THE FORMAL DISCOURSE IS PEER REVIEW LITERATURE, FILE PATENTS. AND THE -- TECHNICAL LITERATURE EMBODIES THIS TRADE JOURNALS OR MORE SOCIAL MEDIAER OTHER PARTICULAR ASPECTS TO GIVE YOU THE CONCEPT OF DISCOURSE HAPPENING BETWEEN SCIENTISTS. AS I MENTIONED, FOREIGN LANGUAGES, WEAR NOT JUST INTERESTED IN WHAT'S HAPPENING IN OUR FAVORITE TOP TIER JOURNALS, THE U.S. PTO PATENT OFFICE. WE'RE VERY MUCH INTERESTED IN WHAT'S HAPPENING ACROSS THE WORLD. THE DISCOURSE, THE INNOVATIONS GOING ON VERY EXCITING INNOVATION I MIGHT ADD AROUND THE WORLD. THE LANGUAGE WE SELECTED WERE REPRESENTATIVE, ENGLISH CHINESE, GERMAN, WE GAVE THE PERFORMERS CHOICES ON WHICH LANGUAGES THEY WANTED TO WORK ON, MOST CHOSE THOSE THREE LANGUAGES, JAPANESE, RUSSIAN AN KOREAN AND SPANISH, THE SIZE OF THE FONT IS THE ENTHUSIASM WE WERE ABLE TO GENERATE WITH THOSE PARTICULAR LANGUAGES. NOVELTY IS HERE, CAN WE DISCOVER PATTERNS OF E EMERGENCE AN CONNECTIONS SPEED KALE AN COMPREHENSIVENESS THAT MOVES BEYOND WHAT A HUMAN CAN DO. CAN WE FIND TECHNICAL AREAS THAT ARE REALLY EMERGENT, COMMUNITIES ARE PRACTICE THAT HAVE CHARACTERISTICS NEW AREAS AS OPPOSED TO AREAS TRIED AND TRUE. THE USE CASE IN GENERAL IS ALERT ANALYSTS TO SUBMERGENT CAPABILITY WITH SUFFICIENT EXPLANATORY EVIDENCE THAT ALLOWS THEM TO GO TO THIR BOSS IF YOU MAY AND JUSTIFY FURTHER ANALYSIS. SO THEY FOR EXAMPLE, I HAVE DISCOVERED THIS PARTICULAR AREA, I THINK I NEED TWO WEEKS OF ANALYSIS BECAUSE THIS IS ONLY GIVING ME A VECTOR, A POINTER, THIS IS REALLY IMPORTANT AREA BECAUSE OF THESE DIMENSIONSCH THIS IS THE NATURE OF COMMUNITY OF PRACTICE. THERE IS AN INTERESTING DEBATE, THIS IS PROPOSED AS ALTERNATIVE TO THIS PARTICULAR APPROACH BEFORE. IT'S SHOWING PEOPLE POSTULATING THIS IS A APPLICABLE TO THIS PARTICULAR NEED, PARTICULAR THINKING FROM NIH PERSPECTIVE, A PARTICULAR TREATMENT FOR A PARTICULAR AILMENT. THESE KINDS OF QUESTIONS WITH EVIDENCE THAT ALLOWS AN ANALYST TO TAKE THAT AND MOVE FORWARD, IF ANALYSTS CAN'T UNDERSTAND WHY THIS TECHNOLOGY OR THIS SECTOR IS MORE IMPORTANT IT'S NOT GOING TO DO ANY GOOD. AS AN ANALYST BEFORE THERE'S ALWAYS SOMEONE YELLING IN MY EAR SAYING THIS IS MORE IMPORTANT F. YOU ONLY FOLLOW WHAT I WAS DOING YOU KNOW YOUR WORLD WOULD BE BETTER. SO WITH ALL THE COMPETING VOICES YELLING AT ME, I DON'T NEED A COMPUTER YELLING AT ME WITH NO JUSTIFICATION WHY I SHOULD -- SO THAT PART IS VERY KEY ELEMENT. DATA SOURCES YOU ARE FAMILIAR WITH, THERE'S LITERATURE GOING ON, THIS IS A QUICK SUMMARY OF LANGUAGE SECTORS AN WORLDWIDE PATENTSCH THIS IS A COMBINATION OF SCOPISTS, WEB OF SCIENCE, DIFFERENCE COLORS REPRESENT THE SAME LITERATURE BASE. LOT OF CONTENT GOING ON IN MULTIPLE LANGUAGES, WHETHER ALL THAT INNOVATION, WHETHER THE ACTIVITY IS TRULY INNOVATION IS A DISCUSSION WE CAN ALWAYS HAVE. THERE IS A LOT OF INTERESTING IDEAS, THERE'S REASON TO LOOK AT JUST -- MOVING BEYOND JUST ENGLISH. MIEST STATE IS -- ESTIMATE IS 800,000 DOCUMENTS PER MONTH ARE ENTERED TO THE FORMAL SCIENTIFIC DISCOURSE. THAT'S A LOT OF CONTENT. I'M NOT THAT GOOD A READER TO BE ABLE TO PROCESS THAT. HOW DO YOU ACTUALLY HANDLE THAT LEVEL -- THIS IS NOT RIDICULOUS, THERE ARE OTHER SECTORS MUCH MORE CON CONTENT COMING THROUGH BUT 800,000 DOCUMENTS IS SIZABLE. SO I'M GOING TO INTRODUCE A CONCEPT, PERHAPS U YOU'RE FAMILIAR WITH IT, I'M FOLLOWING A COUPLE OF SPEAKER WHOSE LAID OUT SOME OF THESE CONCEPTS ALREADY. BUT THIS CONCEPT OF TECHNICAL HORIZON SCANNING, WHEN YOU ACTUALLY WANT TO SCAN THE HORIZON FOR A PARTICULAR -- WHERE DO WE INVEST, WHAT PARTICULAR SECTOR DO E WANT TO BE INVOLVED IN, WHAT PARTICULAR SECTOR HOLDS PROMISE. THIS PROCESS RIGHT NOW IS GENERALLY REQUIRES SUBSTANTIAL EXPERTISE, MUCH OF HUMAN INVOLVEMENT NOT THAT THAT'S WRONG BUT IT'S EXPENSIVE, THEREFORE WE CAN'T DO A LOT OF THEM. THERE IS A MALL NUMBER OF TOPICS TRIMMED, PRUNED HEAVILY SO YOU'RE LOOKING AT PARTICULAR SUB DISCIPLINE. IT'S SUBJECTED TO LIMITED VALIDATION. BASICALLY A BUNCH OF PEOPLE AROUND A TABLE GRUNT HAPPILY WHEN THEY FELT THIS IS ESTABLISHED CRITERIA FOR GOODNESS. SOMETIMES SO THERE'S A WIDE VARIETY OF WAYS OF DOING THAT, SOME GROUPS ARE MORE QUANTITATIVE THAN OTHERS BUT THERE'S CERTAINLY BETWEEN DIFFERENT STUDIES NO UNIFORM ANYMORETY OR WAYS OF COMPARISON THIS HORIZON TO SCAN, HOW DO YOU COMPARE THAT. SO THIS TABLE OF TODAY THE MANUAL PROCESS, SELECTED COVERAGE, UPDATED INFREQUENTLY, ONCE THINGS DOESN'T GET DONE FOR YEARS UNLESS IT'S A HIGH PRIORITY TOPIC AREA. MONTHS TO PRODUCE, SOMETIMES YEARS, DEPENDING ON THE NATURE OF THE PARTICULAR STUDY. IT'S AN AD HOC EVALUATION. WE'RE SAYING WE WANT THIS DO THIS AUTOMATICALLY. WE WANT TO DO A COMPLETE LITERATURE COVERAGE, BEST WE CAN. UPDATED BASICALLY, WHENEVER NEW DATA COMES TO THE SYSTEM IT'S UPDATED. 24 HOURS PERFORMANCE METRIC, MAYBE SURE ALGORITHMS DON'T TAKE INFINITY, NOT TOO PRODUCTIVE OR TEN THOUSAND YEARS IN SOME CASES. SOME ALGORITHMS ARE THAT SCALABLE AND FORMAL MODELS OF EMERGENCE. THIS IS REALLY KEY, TO DEVELOP A SCIENTIFICALLY VALID MODELS OF EMERGENCE POSTULATED. TESTED THROUGH A SERIES OF EXPERIMENTS, SO WE START TO MAKE CONCLUSIONS WHETHER OR NOT WE CAN MEASURE EMERGENCE. FIRST WHAT IS EMERGENCE, DIMENSIONS OF EMERGENCE AND HOW TO TEST THAT. THIS IS REALLY THE PURPOSE OF THIS PROGRAM. E WALLVATION, PARTICULARLY -- EVALUATION, PARTICULARLY A THREE STAGE PROGRAM EVALUATION RIGHT NOW, FOCUSED VERY MUCH ON CASE STUDIES, EXPERT JUDGMENT AND QUANTITATIVE TESTS SO WE HAVE A LOT OF CASE STUDIES, I'LL BRIEFLY TELL YOU WHAT SOME OF THOSE STUDIES ARE. WE HAVE INTERVIEWED, WE HAVE RIGOROUS INTERVIEW PROCESS WHICH WE SELECT PEOPLE FROM GOVERNMENT, ACADEMIA, INDUSTRY AND A PARTICULAR SECTOR WHO ARE INVOLVED, WHO MAKE A COMMENT ON THE TIME LINE TO INDICATE WHAT WAS THE E EMERGENT STATE AT GIVEN TIMES. THEN WE ROLL THOSE UP ACROSS MULTIPLE EXPERTS AND PROVIDE A REFERENCE BASELINE TO EVALUATE SYSTEMS AGAINST. THIS IS ACTUALLY A VERY INSIGHTFUL AND LOTS OF THINGS LEARNED, HOW RIGID ARE THE EXPENSIVE PROCESS, VERY TIME CONSUMING. WE FOUND -- HOW DO I SAY THIS NICELY, EXPERTS HAVE STRENGTHS AND WEAKNESSES. SOME OF THE INSIGHTS GAINED FROM EXPERTS DON'T PROVIDE SMFT TEMPORAL RESOLUTION AND DISCRIMINATION OF FAILURE. PARTICULARLY NEGATIVE EXAMPLES. GETTING SOMEONE INVESTED IN PARTICULAR AREA TO SAY THAT THEIR AREA IS NOT EMERGENT, THERE'S PROBLEMS WITH THAT. THERE ARE PEOPLE WHO WILL SAY THAT WE FEEL LIKE THIS FEEL MOVED ON TO SAY THIS FEEL DIDN'T EMERGE IS A HARD ONE. SO THERE'S PROBLEMS HERE, NOT TRYING TO INDICT THEM. WE'RE LOOKING AT NEW WAYS TO EVALUATE NEW STANDARDS. ANYWAY, WE WANT THIS RELIABLE TRANSPARENT CAPABILITY TO STAND CONTINUALLY, TO SPEED READ, MILLIONS AND MILLIONS OF PAGES OF CONTENT, PROVIDE INDICATORS THAT WOULD BE INDICATIVE OF EMERGENT. SO THERE'S FOUR LAKES. I'LL BRIEFLY TALK TO THE LEGS. THE FIRST ONE IS TECHNICAL EMERGENCE THEORIES, ASKING RESEARCHERS TO FORMULATE HYPOTHESES, TEST THEM, AND PUBLISH THEORIES HOW EMERGENCE INVOLVES. OR PROGRESSES, DEPENDING WHETHER IT'S A PROCESS OR STAGE BASED VIEW. MULTI-LINGUAL AND NOISY, NOISE IS A KEY ELEMENT BECAUSE THIS IS NOT, THEY'RE NOT ALL (INAUDIBLE) FULL TEXT LITERATURE, WE WANT TO EXTRACT USABLE FEATURES FROM THE GIVEN DOCUMENT AND CONNECT ACROSS DOCUMENTS. THESE INTERESTING THINGS LIKE RHETORICAL CITATIONS, FOR EXAMPLE WE TALKED ABOUT CITATIONS WE CAN'T TELL IF IT'S NEGATIVE OR POSITIVE. WHAT IS THE RHETORICAL STANCE OF THE SETTING AUTHOR? ARE THEY SAYING THIS IS SOMETHING THAT CONTRIBUTES TO THEIR BACKGROUND WORK, IS IT SOMETHING THEIR DON TRASSING WITH, IS IT SOMETHING THEY'RE SAYING THERE'S A LOT OF WORK GOING ON OVER HERE? AND I CAN DECIDE IT OR THESE DISCRIMINATORY UNDERSTANDINGS OF WHEN PEOPLE CITE AND HOW THEY CITE WILL PROVIDE EXTRA DIMENSION OF NAILSIS THAT CITATION ANALYSIS COULD BENEFIT IN THAT CITATION GRAPH ANALYSIS FOR EXAMPLE. METHOD, GENETIC ALGORITHMS, ISN'T THIS COOL TO CONNECT EVOLUTIONARY BIOLOGY WITH OPTIMIZATION APPROACHES. THAT WAS THE TOPIC OF THE PAPER, EVENTUALLY THAT TOPIC MOVED INTO METHODS SECTION. I'M TRYING TO COMPARE HOW WELL GENETIC ALGORITHMS WORKS AGAINST GRADIANT DESCENT ALGORITHMS AN IT BECAME SOMETHING PEOPLE REFERENCE. I USED GENETIC ALGORITHM TO OPTIMIZE THIS. THEY DON'T BOTHER TO DECIDE OR -- BECAUSE -- AND THE CASE IS RELEVANT FOR MANY OTHER PARTICULAR METHODS. SO THAT'S ANOTHER ONE. PARTICULAR APPLICATIONS BEING POSITED AND ACTUALLY SEEING THEM BECOME PART OF THE ACTUAL USE AND BEING TESTED IN THE LAB. AND EVENTUALLY IN THE FIELD. CONNECTING FEATURES IN CONTEXT. CREATING FEATURES AN PATTERNS. THIS IS TAKING FEATURES AN BEING ABLE TO UNDERSTAND WHAT THE INDICATORS ARE. GENERATING GROUPS AND DOCUMENTS THAT REPRESENT THE WHAT IS EMERGING. BECAUSE IT'S SOMETIMES FORMS A TERM BUT HOW THAT CONNECTS TO GROUPS OF DOCUMENTS. THOSE ARE TWO LEGS OF THE FOUR LEGGED STOOL. THIRD LEG IS NOMINATION EVIDENCE. SO NOW A SYSTEM NEEDS TO NOMINATE THIS TECHNICAL AREA, THIS IS HOW EMERGENT AND PROBABILITY OF BEING EMERGENT OR ANSWERING A PARTICULAR QUESTION. AND PROVIDING THE EVIDENCE THAT A HUMAN CAN UNDERSTAND. THAT'S THE THIRD PART. FOURTH PART IS SYSTEM INTEGRATION ENGINEERING BECAUSE WE'RE TRYING TO MAKE THIS WORK AS SCALE OVER LARGE AMOUNTS OF DOCUMENTS, THERE'S A SERIOUS AMOUNT OF ENGINEERING INTO. THIS SO HYPOTHESIS, WE HAVE GOT RIGHT NOW, THAT WE'RE TRYING TO TEST, SCIENTIFIC CAPABLE CAPABILITY EMERGENCE CAN BE DETECTED ACROSS MULTIPLE TYPES OF CASE STUDIES ACROSS MULTIPLE DISCIPLINES, FEATURES FROM FULL TEXT ARE CRITICAL TO DETECTING EMERGENCE. WE DONE KNOW IF THAT'S TRUE. DO WE HAVE TO DO THE WORK WHERE NOISE IS HIGHER WHERE IT'S MUCH MORE DANGEROUS BUT MESSY P. DO WE NEED TO DELVE INTO DOMAINS OR STAY AT THE HIGHER LEVEL METADATA CONTENT BY NATURE. CAN WE PROGRESS TO PHASE 2 WHICH IS NOW AT SCALE, MUCH MORE DOCUMENTS TRYING TO PRIORITIZE TENS OF THOUSANDS OF TECHNICAL AREAS. THAT'S THE QUEST RIGHT NOW. NOW GETTING TO SOME OF THE MEAT. YOU HAVE HEARD P MEAT OF PEOPLE WORKING IN THIS AREA. SO I'LL BRIEFLY GIVE YOU MORE CONTEXT. SO CLEAR DEVELOPMENT HAS EMERGENT HYPOTHESES POSTULATED INDICATORS THAT ARE NOW BEING SAYING I THINK THIS IS INDICATIVE BUT I WOULD LIKE THE TEST IT. AND CHALLENGE QUESTIONS WHICH I'LL TALK BRIEFLY IN A SENG, WAYS OF -- IN A SECOND. THOSE ARE WAYS TO PROBE THE EMERGENCE OF THE PROCESS. LOOKING AT THE REAL WORLD PROCESS OF TECHNICAL EMERGENCE SO THE HYPOTHESIS EMERGENCE HAPPENS WITH PEOPLE. THAT'S WHERE IDEAS COME, WHERE INNOVATIONS GET DEVELOPED. BUT THERE'S A TRACE THAT'S LEFT, THE SHADOW IF YOU MAY OF WHAT HUMANS DO IS LEFT IN THE SCIENTIFIC LITERATURE. THERE IS A LAG THERE. YES, IT'S IMPERFECT REPRESENTATION BUT IT IS THE AREA, DISCERNIBLE SHADOW WE CAN MEASURE AND IT'S OBSERVABLE. SO WE'RE TRYING TO MAKE SURE THIS THEORIES CONNECT THE REAL WORLD PROCESSES AN LEASE A CONNECTION BETWEEN A REAL PROCESS AND WHAT'S HAPPENING IN THE LITERATURE. THESE ARE CONCISE STATEMENTS I'LL GET TO YOU IN A BIT. THESE INDICATORS ARE QUANTITATIVE COMPARABLE MEASURES, SOME ASPECT OF THE HYPOTHESIS. ONE INTERESTING THING GOING BEYOND IS SO THERE'S GREAT PHILOSOPHY OF EMERGENCE. PAWPER, COON, LACATOS. THERE'S A LOT OF GREAT -- BUT HOW DO YOU OPERATIONALIZE THESE PHILOSOPHIES INTO REAL MEASURABLE INDICATORS? MEASURABILITY IS A KEY PART THAT'S WHAT'S GOING ON HERE. THE CHALLENGE QUESTIONS ARE MECHANISMS TO TEST THE -- AND EVALUATE THE RELATIONSHIPS BETWEEN THESE HYPOTHESES AND THE INDICATORS IN A WAY WE CAN ACTUALLY SEE HOW WELL IT PERFORMED. SO WHAT IS THE TECHNICAL EMERGENCE? THESE ARE FOUR HYPOTHESES THAT PEOPLE -- FOUR TEAMS ARE -- WERE FUNDED. ARE EXPLORING. COLUMBIA TEAM IS GETTING THIS CONCEPT OF ACCEPTANCE AS A MEASURE OF EMERGENCE. IS A COMMUNITY SCIENTIFIC COMMUNITY ACCEPTING THIS PARTICULAR TERMINOLOGY. OFTEN RELEVANT AND PATENTS ALSO BECAUSE A TERM< CLAIMS SEC HAS STRONG LEGAL MEANING AS OPPOSED TO JUST BEING A REFERENCE IN THE BACKGROUND CONTEXTUAL STATEMENT THERE. SO ONCE A TERM HAS MADE IT THE CLAIMS, THAT'S AN ACCEPTED CONCEPT THAT LAWYERS CAN LITIGATE ON AND NOT LET IT IN UNTIL THEY CAN DO THAT. SO THIS IS ACCEPTANCE EXPLORING THE SYSTEMS LOOKING AT THIS CONCEPT OF ACTIVE NET WORK AS RESEARCH E FUNDERRERS, APPLICATION INSTITUTIONS TURNING IT INTO SOMETHING PRACTICAL AND LOOKING AT THAT NETWORK, THE HYPERGRAPH, IF YOU MAY, THAT THEN ALLOWS YOU TO UNDERSTAND WHETHER ITS ROBUSTNESS, THAT'S A LOADED TERM, NOW MEASUREING WHAT THIS NETWORK, THE ROBUSTNESS OF THIS HYPERDIMENSIONAL NETWORK. THIS CONCEPT OF HAS APPEARED CONCEPT IS NEW, UNEXPECTED NOTICEABLE AND GROWING. THIS IS NOW PRACTICALLY IMPLEMENTED AND WE HAVEN'T TALKED TO SOME OF THAT A LITTLE BIT EARLIER. THEN THE SRI TEAM IS CONCEPT OF EMERGING, IDENTIFIABLE BY ITS OWN PRACTICERS. THE IMMUNITY OF PRACTICE NOT ONLY VIRTUALLY EXIST BUS RECOGNIZE THEY EXIST. THERE'S SELF-AWARENESS IN THIS GROUP. THIS ENABLES CAPABILITY NOT ACHIEVABLE PREVIOUSLY. THAT'S DISTINGUISHED. AND THESE ARE ALL HAVE INDICATORS MIND THEM TRYING TO TEST THOSE PARTICULAR QUESTIONSCH AND IT PERSISTS, THERE'S A PERSISTENCE QUESTION, IT CAN'T JUST DISAPPEAR. SO THESE ARE THE WAYS THEY'RE PROBING THE LITERATURE SPACE, UNDERSTANDING WHAT'S HAPPENING IN THE LABORATORY SPACE. SO HOW WE'RE PROBING THE CHALLENGE QUESTION, THE TECHNICAL EMERGENCE QUESTION WITH A SERIES OF QUESTIONS, WAS THERE A IMMUNITY OF PRACTICE AROUND A CONCEPT DURING TIME PERIOD, WERE THERE DEBATES, PRACTICAL APPLICATIONS, ALTERNATIVES, SOMEONES EXPLORING, COMMERCIAL APPLICATION, WAS THE INFRASTRUCTURE REQUIRED TO PERFORM READILY AVAILABLE? THESE ARE NOW DIMENSIONS OF EMERGENCE STARTING TO EXPLORE. EACH TEAM HAS PROPOSED SOME OF THEIR OWN IDEAS HOW TO GET AT THIS. THESE ARE DYNAMICS SO DON'T HOLD THEM TO THE FIRE THAT THIS IS EXACTLY THE WAY THEY'RE WORKING, RIGHT NOW IT'S A VERY DYNAMIC RESEARCH PROCESS. I TOLD YOU ACCEPTANCE BY THE COLUMBIA TEAM. THEN THE QUESTION OF INCREASING OR DECREASING, INTERDISCIPLINARITY, ALAN PORTER CAN TALK ABOUT THAT. WITHIN A PARTICULAR KNOWLEDGE BASE, DID -- DO USAGE OF NEW TERMINOLOGY BECAUSE THESE TERMINOLOGY AND EMERGING AREA IS GOING TO BY DEFINITION BE CHANGING AND PEOPLE WILL BE COALESCING ON THAT OR DISPUTING WHETHER IT'S THE DEWEY THEORY VERSUS THE JIM THEORY OR VARIOUS THINGS LIKE THAT, THAT HAPPEN, SOMETIMES JUST KIND OF JUST HAPPENED OR SOMETIMES IMPOSED BY A PARTICULAR INFLUENTIAL ACTOR IN THAT PARTICULAR GROUP. DID A -- THIS IS THE ROBUSTNESS I WAS TALKING ABOUT PREVIOUSLY. NOW THERE'S SOME PREDICTIVE NATURE ALSO STARTING TO WORK INTO THE PROGRAM A LITTLE BIT ABOUT CAN YOU ACTUALLY UNDERSTAND CITATIONS, WITH SOME OF THESE EXTRA DIMENSIONS CAN WE PREDICT MORE WITH HIGHER FIDELITY. DOES THIS CONCEPT DOMINATE A THREAD WHICH DWOAS TO THE THREAD CONCEPT WITH THE SMALL DIMENSIONS OF EMERGENCE VERY FINE RESOLUTION CONCEPT OF EMERGENCE WHICH IS MORE WHAT KEVIN TALKED ABOUT. SO THAT THE THEORY PART. THE OTHER LEG IS TURNING THE TEXT ELEMENTS INTO INDICATORS. THIS IS A REALLY INSTRUMENTAL APPROACH. SO NOW -- ALL THE TEAMS AREK LOG ATTENTION CHURL AN DOCUMENT CENTRIC PERSPECTIVES, DOCUMENT -- TOPIC MODELS AND TEXT-BASED MIGHT BE SOME MORE VERY NUANCED LANGUAGE WITHIN THE TEXT RHETORICAL ANALYSIS WIDE VARIETY OF APPROACHES YOU CAN TAKE ON THAT REGIME. EACH HAS THEIR OWN PARTICULAR SETS OF FOCUSES. I TALKED ABOUT THE CITATION. DI MINGSES UNDERSTANDING WHY SOMEONE IS SITE -- DIMENSIONS, THE GROUP OF AUTHORS IS TAKING TOWARDS CITATION. TALKED PATENTS. THE LANGUAGE OF INTRODUCING NEW TERMS, CITE -- CITATIONS IN THE USPCO A RELEVANT AREA, ALSO WORTH NOTING NOT ALL PATENT OFFICES USE CITATIONS. SO IF YOUR GOAL IS TO MOVE ACROSS -- UNDERSTAND WHAT'S HAPPENING WORLDWIDEh3GT LITERATURE, CITATIONS AREN'T GOING TO WORK BECAUSE YOU HAVE TO GET INTO MORE SEMANTIC ANALYSIS BECAUSE THEY DON'T PROVIDE THAT CITATION, INLINE CITATIONS LIKE THE U.S.PTO DOES. SO THERE'S A VARIETY OF FEATURES HE EXPLORED HERE, I WON'T GO TO ALL OF THEM. THIS NOMINATION STEP. IS AN INTERESTING ELEMENT SO THIS IS A QUESTION WAS THERE A IMMUNITY OF PRACTICE AROUND RNAI DURING 2006 TO 2010. THIS SYSTEM IS NOW SAYING YES WITH CONFIDENCE OF 72%. THIS IS COMPUTER GENERATEDCH THIS IS A MOCK UP BECAUSE PEOPLE ARE TRYING TO DEVELOP THIS BUT THIS IS THE TOPIC WE'RE TALKING ABOUT, QUICK SUMMARY FOR HUMAN CONSUMPTION. JUSTIFICATION TO THE ANSWER. AND THEY PROVIDE EVIDENCE YOU CAN DRILL DOWN INTO EXAMPLE OF A TIME SERIES, A WIDE VARIETY OF INDICATORS. THIS IS ONE TEAM MOCK UP HOW THEY'RE DOING IT, EVERY TEAM HAS DIFFERENT APPROACHES. THIS IS THE CONCEPT OF BEING ABLE TO TAKE THESE METRICS AND TRENT NEM A WAY HUMAN CAN UNDERSTAND THEM. ALSO A WAY VALIDATED. THAT IS KEY TO THE PROGRAM. THE FUSE PROGRAM IS FIVE YEAR FUNDAMENTAL RESEARCH PROGRAM, WE'RE IN THE RED LINE HERE IN THE FOURTH QUARTER OF FY 12. YOU GUYS SHOULD BE FAMILIAR WITH THAT. AND THAT'S FOUR TEAMS WORKING ON THIS PROCESS, WE HAVE A FORMAL TEST EVALUATION BEGINNING IN OKAY. AND SO THESE ARE PARTICULAR CASE STUDIES FUSION META MATERIALS DNA ALGORITHMS, RNAi, HORIZONTAL GENE TRANSFER ARE ALL CASE STUDIES WE HAVE DEVELOPED WITH THE REFERENCE BASELINE AMONG SIX DIMENSIONS OF EMERGENCE, WE'RE STARTING TO SEE CAN THE SYSTEMS MATCH THIS LEVEL OF EMERGENCE, CAN I ACTUALLY CORRECTLY FOLLOW A GROUP OF EXPERTS WOULD HAVE SAID WAS STATE OF EMERGENCE AT A PARTICULAR GIVEN TIME. SO NOW I MOVE TO SLIGHTLY HYPOTHETICAL PART OF TALK, WHAT WOULD YOU DO IF YOU HAD A SYSTEM THAT COULD ACTUALLY RELIABLY OBVIOUSLY NOT PERFECTLY, MAYBE 85% OF THE TIME THERE'S SOMETHING OR PARTICULAR GOAL. THAT COULD IDENTIFY WHAT TECH CAP CAPABILITIES ARE EMERGING AN PROVIDE A EXPLANATION, WHAT WOULD YOU USE. SO I TRIED MIGHTILY TO PUT ON AN NIH PROGRAM OFFICER HAT AND REALIZE DIDN'T QUITE FIT BECAUSE I DON'T UNDERSTAND HOW SCIENCE -- HEALTH SCIENCE LIKE I SHOULD. SO I DECIDED I'LL PUT ON A PROGRAM MANAGER HAT AND HOPEFULLY YOU CAN DRAW CONNECTS TO THAT. SO TWO MAJOR STAGES I PERCEIVE I AS A PROGRAM MANAGER WOULD BE INFLUENCED BY HAVING THIS KIND OF TOOL AT MY DISPOSAL. THIS IDEA DEVELOPMENT STAGE, WE'RE VERY MUCH CRAFTING A PROGRAM WHICH WE SOLICIT RESEARCH IDEAS AND GO FORWARD. WHETHER THAT DIRECTLY FITS WITH ALL NIH CATEGORIES OF RESEARCH, I UNDERSTAND IT PROBABLY WOULDN'T. BUT MY PERSPECTIVE HOW WOULD WE BE ABLE TO IDENTIFY A PLACE THAT IS REALLY WORTH INVESTMENT? THE SECOND PART IS, ANALYZING THE IMPACT OF THE PROGRAM, JUSTIFYING THE FACT THAT THIS MUCH MONEY WAS SPENT ON A PARTICULAR PROGRAM. THE IDEA IN THE DEVELOPMENT STAGE IS STARTING TO UNDERSTAND WHAT IS THE OPTIMAL AGE OF INVESTMENT IN THE PROGRAM WHERE IS THE APPROPRIATE POINT TO INSERT THAT RESEARCH FUNDING WITH A FOCUS PROBLEM STATEMENT WITH THE TEST EVALUATION CONSTRUCT SO THESE ARE QUESTIONS PEOPLE DON'T HAVE REALLY STRONG ANSWERS TO. I THINK PEOPLE WOULD BE ABLE TO ANSWER IT WITH EXPERIENCE AN OPINIONS, IT WILL BE VALUABLE ANSWERS BUT FROM THE PERSPECTIVE OF THE IMPACT ON THE PARTICULAR SCIENTIFIC COMMUNITY OR THE OUTPUT IMPACT, WHERE IS THE APPROPRIATE PLACE TO PUT THIS? I THINK A LOT OF INDICATORS WOULD GIVE INSIGHTS INTO HISTORICALLY WHERE PROGRAMS WERE APPROPRIATELY INVESTED IN. IT MIGHT HELP GUIDE WHERE OPTIMAL INVESTMENT STRATEGIES MIGHT LIE. HYPE VERSUS REALITY. WHEN THERE'S A LARGE AMOUNT OF ACTIVITY BURST OF ACTIVITY THAT STARTS OUT A PHASE, IS THAT A HEIGHT STAGE OR REALLY A VIABLE STAGE OF RESEARCH? BEING ABLE TO UNDERSTAND AND DISTINGUISH BETWEEN THESE ELEMENTS IS SOMETHING THAT'S VERY RELEVANT AN VERY MUCH NEEDED AND IF YOU CAN START TO CHARACTERIZE WHETHER A RESEARCH COMMUNITY WITH LOTS OF STRAP HANGERS AN EXCITED PEOPLE WHO ARE EXCITED ABOUT AN AREA OPPOSED THE TO MAKING FUNDAMENTAL ADVANCES IN THE AREA. IS A VERY RELEVANT AREA. DIMENSION OF ANALYSIS. NEW ENABLING COMPONENT CAPABILITIES. IS THIS A NEW CAPABILITY THAT TRANSFORMS A PARTICULAR AREA, TO BE DEVELOPED, I INDIANA KAYTORS OF WHETHER A CAPABILITY -- NEW CAPABILITY LOOKS LIKE AND HOW THAT MANIFESTS WILL BE RELEVANT BECAUSE IT'S A HUGE ELEMENT, MY PRE-PROGRAM ANALYSIS UNDERSTANDING WHERE THE CRITICAL COMPONENTS IS, NOW THE TIME TO BRING THE OM POINTS TOGETHER TO -- COMPONENTSING TO TO SEE SOMETHING RELEVANT TO THE MISSION SPACE. SCIENTIFIC POTENTIAL CONVERGENCE, RELEVANT TOO BECAUSE OF THE MOST INTERESTING EMERGENCE IS CONVERGING IDEAS THAT WEREN'T PREVIOUSLY CONNECTED. IF YOU WATCH COMMUNITY OF PRACTICE, FOR EXAMPLE, JUST PICKING COMMUNITY PRACTICE BECAUSE THE FIRST ONE ON MY MIND, THERE'S MANY OTHER DEBATES OR APPLICATION OR MATURITY. CAN YOU ACTUALLY SEE COMMUNITY OF PRACTICE AN UNDERSTAND WHEN THEY'RE GETTING READY FOR CONVERGENCE OR AMENABLE TO CONVERGENCE THERE'S ENOUGH CRITICAL MASS TO ENCOURAGE, I HAVE HAD THE EXPERIENCES IN RESEARCH PROGRAM MANAGEMENT WHERE I THOUGHT CONVERGENCE EXCITING. THEY JUST DIDN'T HAVE CRITICAL MASS, THEY DIDN'T HAVE THAT PARTICULAR SPACE, THEY WEREN'T READY FOR THAT YET. OVERLAP WITH PS AN EXISTING EFFORTS, THIS IS A RELEVANT QUESTION WHO ELSE THE GOVERNMENT PARTICULARLY THE WORLD OF GOVERNMENT FUNDED AREA IS A BIG PLACE, THERE'S LOTS OF FUNDING, DIFFERENT NATIONS FUNDING RESEARCH, IS THIS SOMETHING THAT'S TRIED BEFORE, HOW DOES IT FIT WITH THAT PERSPECTIVE. I THINK WE HAVE A LOT OF TOOLS THAT SPEAK TO THAT ALREADY. NOW WE'RE ADDING A NEW DIMENSION OF EMERGENCE AND NEWNESS AND NOVEL UNTESTED NOVEL NEW KIND OF QUESTIONS. WHY IS THIS INNOVATIVE? WHICH IS A KEY QUESTION I ALWAYS ASK, ARE YOU INNOVATING OR JUST PUTTING LE GOES TOGETHER IN A DIFFERENCE WAY? AND THAT'S A QUESTION YOU CAN JUSTIFY, THIS ACTUALLY IS BASED ON COMMUNITY PRACTICE WHICH DIDN'T EXIST TWO YEARS AGO OR FIVE YEARS AGO, IT IS RELEVANT AND THIS IS HOW YOU CAN JUSTIFY THAT. PROGRAM IMPACT ASSESSMENT, RETURN ON INVESTMENT IS ALWAYS A PERNICIOUS ANALYTIC TASK. IT'S VERY CHALLENGING, VERY HARD. CERTAINLY NOT SOLVE THIS PROBLEM, HOWEVER, IT WOULD ADD A LOT OF NEW DIMENSIONS ON BASICALLY SAYING OKAY, IT WAS EMERGENT WITH THESE CHARACTERISTICS, WE'RE INDICATIVE OF THE FOLLOWED AFTER MY INVESTMENT IN THIS PARTICULAR AREA. APPLICATION PO DEVELOPMENT POTENTIAL IS A CLASS OF INDICATORS FOR MATURATION OF TECHNOLOGY AND MOVING BEYOND. SO THIS IS A KIND OF -- TO PROVOKE YOUR THINKING. COMPONENTS OF CAPABILITY ENABLE -- THESE ARE THINGS WE'RE DOING -- IN FUSE LINEAGE PROVIDENCE OF DATA ANALYSIS, PDF TO XML CONVERSION, XML ENRICHMENT, WIDE VARIETY OF THOSE. CROSS-DOCUMENT LINKAGES, GENERATION OF RELATED DOCUMENT GROUPS, INDICATORS CONFIRMING NOMINATION PRIORITIZATION SERVICES, AND EVIDENCE OF A PARTICULAR NOMINATIONCH THESE ARE TYPES OF SERVICES THAT WE'RE APPROACHING. PERHAPS IN YOUR PERSPECTIVE YOU MIGHT ASSEMBLE IN A DIFFERENT WAY PROVIDE USEFUL CAPABILITY. SO IF YOU HAVE FEEDBACK THAT WOULD BE PREESH APPRECIATIVE. PROBABLY WON'T TALK TOO MUCH ABOUT THIS OTHER THAN POINT OUT MOVING FROM -- TOWARDS SCAN MOVING BEYOND SEARCH IS AN EXCITING AREA. THE CHALLENGING SCALE BUT IT IS TRACTABLE. EVIDENCE EXPLANATIONS ARE REALLY VERY HARD ASPECT AND WE CAN START TO CRACK THAT NUT IN A SYSTEMATIC WAY. I'M REALLY EXCITED ABOUT THAT. LOTS OF NLP WORK DONE ON NEWS WIRE, ACTUALLY CONVERTING THAT INTO TRANSITIONING INTO SCIENTIFIC TECHNICAL LITERATURE AND PATENT LITERATURE IS NON-TRIVIAL. AND THERE'S -- THAT'S A CHALLENGE THAT PERFORMERS ARE FACING RIGHT NOW, THAT GENRE SHIFT. SO THE GOAL, ANTICIPATED IMPACT IS TO MAKE BIG IMPACT ON THE ANALYSTS. BE ABLE TO GENERATE VALIDATED THEORIES OF EMERGENCE, FEATURE EXTRACTION, SIGN ACTIVIC -- SCIENTIFIC ENTITIES, IT'S EXCITING THE ENTITIES WE'RE GO AFG THE NAMED ENTITIES OF THE TIME THAT WE HAVE SO FAR. METHOD EXTRACTION. BEING ABLE TO PRESENT THIS FROM AN EVIDENCE PERSPECTIVE AND PERHAPS START TO IMPACT POLICY ASPECTS IF IT'S SUCCESSFUL. IF RESEARCH TEAM, THIS IS JUST TO GIVE YOU A SENSE OF WHO WE ARE, HOW AM I ON TIME? STILL HAVE TIME FOR QUESTIONS? THERE'S A LOT OF -- WHENEVER WE TALK TO CONGRESS THEY'RE EXCITED ABOUT SMALL BUSINESSES AND LARGE BUSINESSES AND ACADEMIC ORGANIZATIONS SO LET ME GIVE YOU THOSE NUMBERS FOR YOUR OWN PERSONAL JOY WHICH I'M SURE YOU'RE ENJOYING RIGHT NOW. I CAN SEW ON YOUR FACE YOU'RE ELATED BUT THESE ARE THE ORGANIZES WE WORK WITH. IT'S REALLY EXCITING TEAM, WE HOPE TO BUILD POTENTIAL SO HOPEFULLY THIS PARTICULAR TALK IS WET YOUR APPETITE FOR NEW CLASSES THAT ARE COMING OVER THE HORIZON. MEANTIME I HOPE THAT A WIDE VARIETY OF TECHNOLOGIES AS REPRESENTED IN SCIENTIFIC PUBLICATIONS AND PERFORMER TEAMS THAT YOU SEE UP HERE WILL START TO TRICKLE OUT INTO IMPROVING ANALYTIC CAPABILITIES. IF YOU HAVE ANY QUESTIONS I WOULD BE HAPPY TO ENTERTAIN THEM. [APPLAUSE] >> WE HAVE TIME FOR A FEW QUESTIONS. TO START THINGS OFF, OUR GROUP ACTUALLY LOOKED AT RNAi AS AN EXAMPLE OF A TECHNOLOGY AND EMERGENT FIELD THAT HAD A LAG BETWEEN WHEN IT EMERGED AND WHEN NIH FUNNED IT. SO CAN YOU GIVE US ANY MORE DETAILS ABOUT THE STRUCTURE OF THAT HOW THAT EMERGED? >> I'M SORRY, ACTUALLY I SHOULD HAVE BROUGHT THAT PARTICULAR CASE IN WITH ME BUT MY MEMORY IS A LITTLE FUZZY ON THAT PARTICULAR CASE. I'M SORRY. BE GLAD TO TALK TO YOU MORE ABOUT THAT. >> YEAH I'LL TOUCH BASE WITH YOU. >> WHAT IS INTERESTING JUST TO COMMENT SINCE I ALWAYS HAVE TO SAY SOMETHING. THE DIFFERENT CASE STUDIES HAVE DIFFERENCE CHARACTERISTICS OF EMERGENCE. FOR EXAMPLE, DNA MICROARRAYS IS ONE OF THEM. IT HAS VERY DIFFERENT INITIATION AND GROWTH PROCESS AS KIND OF A PATENT INITIATED OR CORPORATE INITIATED INNOVATION POINT MOVE FORWARD. DIFFERENT OTHER ONES STARTED AS A HORIZONTAL GENE TRANSFER, SOME STARTED IN THE EMERGE AN BUBBLED UP TO HUMAN SCIENCE AND I THINK RNAi FITS THAT CATEGORY ALSO. THESE -- WHAT WE'RE TRYING TO LOOK AT DIFFERENT PROFILES OF EMERGENCE AN UNDERSTAND, MAKING FIRST WE'RE COVERING THE WATER FRONT WIDE VARIETY OF PROFILES. THIS IS A LOT MORE THAN JUST TWO, THERE'S A LOT OF DIFFERENT WAYS, THAT'S THE REASON WE CHOSE RNAi BECAUSE IT HAD AN INTERESTING PROFILE OF EMERGENCE. UNFORTUNATELY I'M NOT SUFFICIENTLY ARTICULATE THE GIVE YOU MORE. >> ACTUALLY A QUICK FOLLOW-UP. DO YOU HAVE A SENSE THAT IT IS FINITE T NUMBER OF WAYS -- CAN YOU COMMENT ON THAT? >> I HAVE A PERSONAL SENSE, I BELIEVE IT IS A FINITE SET OF EMERGENT PROFILES, OBVIOUSLY ALWAYS RAREIATION, EVERYONE IS EWE -- VARIATION. EVERYONE IS UNIQUE WITH THEIR OWN PERSONALITY BUT IN GENERAL WE'RE STARTING -- I'M STARTING TO SEE WHETHER IT'S ACTUALLY THERE YET, CHARACTERISTICS THAT ARE STARTING TO CONNECT. >> (INDISCERNIBLE). I'M ACTUALLY VERY INTERESTED IN THE INCORPORATION OF FOREIGN LANGUAGE. SO IN PRINCIPLE YOUR PRICT WILL BE ABLE TO PREDICT SAY YOU LISTED CHINESE AND GERMAN WOULD BE ABLE TO PREDICT HOW SCIENCE OR SCIENTIFIC RESEARCH WILL BE GOING IN THOSE COUNTRIES OR IT COULD BE EASILY APPLIED TO OTHER LANGUAGES. THAT WOULD BE HUGELY HELPFUL TO US GOVERNMENT, IF NO ONE ELSE. >> THIS IS A MISSING DIMENSION IN MY PERSPECTIVE. THERE'S ENGLISH BIAS, THAT'S PRETTY HEAVY. AND I THINK IT'S UNFORTUNATE BECAUSE THERE'S A LOT OF GOOD WORK GOING OVER THERE, THAT ISN'T TO SAY THAT BECAUSE OF THE ENGLISH BIAS GOOD SCIENTISTS STILL GET THEIR WORK OUT THERE BUT THERE IS A LAG, A DELAY AND YOU DONE SEE THE FULL STRUCTURE OF A RESEARCH GROUP, A RESEARCH THAT'S MOTIVATED BY NATIONAL NEEDS, THERE AREN'T NECESSARILY INTERNATIONAL NEEDS. YOU SEE THIS IN RICE ENGINEERING AND OTHER AREAS THAT ARE CRITICAL TO A PARTICULAR NATION AND YOU DONE SEE THAT IN THE WORLD BECAUSE THE WORLD ISN'T AS INTERESTED IN RICE, JUST USE AN AN EXAMPLE AS A PARTICULAR NATION. >> JUST TO FOLLOW-UP, IN PRINCIPLE IF YOU WERE GETTING SAY DIFFERENT DATA YOU WOULD BE ABLE TO ANALYZE THE TREND OF WHATEVER IN THAT COUNTRY OR EVEN IN THE UNITED STATES, RIGHT? >> IN PRINCIPLE, YES. AT THIS POINT ALL WE'RE LOOKING AT THE IS TO FORMAL SCIENTIFIC LITERATURE AND DISCOURSE. THE INFORMAL IS SOMETHING TO TACKLE THIR PHASE, THERE WE ARE. WE'RE GOING TO BE STARTING THAT HERE AND BE LOOKING WE ARE TESTING NOW TO SEE IF WE'RE READY. THOSE QUESTIONS WILL BE MORE VALID. WE'LL TALK TO IT A LITTLE BIT MORE. YES, SIR. >> LET ME MENTION I HAVE A COUPLE OF STUDIES ON RNAi EMERGENCE AND ONE OF THEM IN TERMS OF MESH CATEGORIES YOU MAYBE INTERESTED FROM THAT WE HYPOTHESIZE A LITTLED NUMBER OF TRAJECTORIES, I'LL COME TO THE QUESTION, A LIMITED NUMBER OF TRAJECTORIES BECAUSE IT HAS TO DO -- IF YOU LOOK AT THE TRA SCREKTOR -- TRAJECTORY IT MOVES FROM ONE SELECTION ENVIRONMENT TO ANOTHER SELECTION ENVIRONMENT SO YOU GET SELECTION UPON SELECTION. THAT WILL CREATE SPECIFICITY IN THE -- SO WE ARE TALKING ABOUT SIGNATURES OF TRAJECTORIES. (INDISCERNIBLE) BY THE WAY. BRINGS ME TO MY QUESTION, MY QUESTION IS, IS ABOUT IS THIS NOT OVERAMBITIOUS? IN THE SENSE THAT WE HAVE THESE BIG QUESTIONS AND WE HAVE LOT OF DATA AND THERE'S AMBITION TO DO MULTIPLE LANGUAGES, WHEN WE HAVE LITTLE UNDERSTANDING ABOUT WHAT EMERGENCE IS. BECAUSE ME BEING HERE AT THE MICROPHONE IS EMERGENT. BUT IT IS NOT INNOVATIVE. [LAUGHTER] >> IN BETWEEN BIG DATA AND BIG QUESTION IS NARROWING DOWN BY THEORIZING, WE NEED A BIT OF THAT. >> I APPRECIATE THE ARE YOU CRAZY QUESTION. IN SOME WAYS I ENCOURAGE THESE, IT'S APPROPRIATE. ARE YOU ABSOLUTELY OUT OF YOUR MIND BECAUSE THIS IS SUCH A HARD PROBLEM. I ACCEPT THE POINT IT IS A HARD PROBLEM AND THE FACT THAT IARPA IS IN A POSITION TO TAKE THIS RISK ANDTRY TO STRUCTURE A PROBLEM TO MOVE US DOWN THIS ROAD IS INCREDIBLY EXCITING IN MY OPINION, WE HAVE MANAGED TO ASSEMBLE 75% OF WORLD SCIENTIFIC LITERATURE. WE'RE ACQUIRING THE WORLD'S SCIENTIFIC LITERATURE AND PROVIDING A PLAYGROUND, NOT PLAYGROUND, A WORK GROUND, WE DON'T PLAY, WE WORK. THAT IS ALLOWING A GROUP OF OVER 100 RESEARCHERS IN AN AREA THAT IS NOW TRY TO MARCH THROUGH THESE PROBLEMS, WILL WE BE SUCCESSFUL, WILL WE ARTICULATE COHERENT VALIDATED EMERGENCE THEORIES? I DON'T KNOW BUT WE'RE TRYING AND WE'LL PRODUCE A LOT OF PRODUCTIVE OUTPUT. IT IS POSSIBLE THAT AT THE END OF THIS PROGRAM, OR EVEN NEXT YEAR WE MIGHT NOT HAVE MADE AS MUCH PROGRESS AS WE WANTED TO BECAUSE OF SOME DEFICIENCY IN THEORY, FOR EXAMPLE. BUT AND I ACCEPT THAT. I MADE A CALCULATED RISK. WE'LL SEE. BUT I DO APPRECIATE YOUR CONCERN. >> LET'S TICK A PAUSE THERE, WE CAN DISCUSS THIS FURTHER IN THE PANEL AFTER LUNCH. WE MIGHT RETITLE THAT SESSION, HAVE WE LOST OUR MINDS? WE'LL RECONVENE AT 1:15.a WE HAVE ASSEMBLED OUR PANEL. WE HAVE TWO MEMBERS YET TO BE INTRODUCED. I WOULD LIKE TO INTRODUCE THEM. START WITH ON MY RIGHT DICK KLAVANS WHO PUBLISHED ON SCIENCE AND ART OF SCIENCE MAPPING. HE'S PLANNED INDUSTRY TO MANY GROUPS TO NAME, GOVERNMENT AGENCIES, DOE, NSF, NIH AND OVER 20 UNIVERSITY. THE RESEARCH INITIATIVES TO SIGN FIFFIC BREAK THROUGHS USING DYNAMIC MICROSTRUCTURES OF SCIENCE. TO HIS RIGHT ALAN PORTER, PROFESSOR EMERITUS OF INDUSTRIAL SYSTEMS ENGINEERING IN PUBLIC POLICY GEORGIA TECH, WHERE HE REMAINS CO-DIRECTOR OF THE TECHNOLOGY POLICY AND ASSESSMENT CENTER. HE'S ALSO DIRECTOR OF R&D, RESEARCH TECHNOLOGY AND NORTHWEST GEORGIA. HE'S AN ACTIVE COLLABORATOR OR SCIENCE OVERLAY MAPS TO VISUALIZE DISCIPLINARY ENGAGEMENT IN E RESEARCH ACTIVITIES AND KNOWLEDGE DIFFUSION PROCESSES. WITH THAT, WE CAN BEGIN THE DISCUSSION. FEEL FREE TO USE THE MICROPHONES. MAYBE I'LL START BY ASKING ONE OF MY QUESTIONS FROM YESTERDAY WHICH IS HOW DO YOU CHARACTERIZE OVERLAP, FROM AN NIH PERSPECTIVE PROMOTING BIOMEDICAL RESEARCH MAKING THE DISTINCTION BETWEEN OVERLAP THE IS APPROPRIATE AND OVERLAP THAT IS EXCESSIVE. PLEASE PUSH THE BUTTONS ON YOUR MOARK PHONES. -- MICROPHONE. ANYBODY WANT TO WEIGH IN ON THAT? >> FIRST I WOULD -- I TEN TO MAKE A DISTINCTION BETWEEN SCIENCE WHICH IS NOVEL, AND SCIENCE WHICH IS NOT NOVEL. SCIENCE WHICH IS NOT NOVEL USUALLY IS PERSISTING BECAUSE THERE'S EQUIPMENT IN HAND, PEOPLE HAVE TO DO DIFFERENT THINGS. OVERLAPS IN NON-NOVEL AREAS PROBABLY SHOULD BE LOOKED AT. CLOSELY. OVERLAPS IN NOVEL AREAS ARE APPROPRIATE IF YOU WANT TO HAVE EXPERIMENTATION TO SEE WHAT MIGHT WORK. THE FIRST TESTILY USE IS THE HARD QUESTION OF WHAT AREAS ARE NOT NOVEL. LOOK AT THIS IN TERMS OF POSSIBLY SAYING THESE ARE AREAS, NOT NOVEL, WE MAY WANT TO CONSOLIDATE THEM. >> RELATED TO MATURITY OF THE FIELD. OTHER THOUGHTS? EVERYBODY IS IS PUNTING ON THIS ONE, I GUESS. GO AHEAD. >> I THINK THIS PRINCIPLE YOUR POINT MAKES SENSE IN THE SENSE THAT AT LEAST FROM OUR PERSPECTIVE WE GENERALLY HAVE VERY HAPPY WITH DIFFICULT PROBLEMS THE COMPETITIVE PERSPECTIVE IS REALLY IMPORTANT, BECAUSE WE DONE KNOW WHICH ONES -- I THINK THE CHALLENGE THOUGH WHAT IS YOUR DEFINITION. AND THAT'S WHERE THE PEOPLE FISTFIGHTS MIGHT BEGIN. YOU POSITED SIMPLISTIC -- NOT TRYING TO DIMINISH WHAT YOU'RE DOING, SIMPLE, BASICALLY IT'S NOT NOVEL IF IT'S AROUND A LONG TIME. >> IT'S NOT NOVEL THE SOCIOCOG MITIVE STRUCTURE HASN'T CHANGED THE WAY OF THINKING BECAUSE WHAT WE'RE TRYING TO PICK UP IS SOCIOCOGNITIVE STRUCTURE. NOT THAT IT'S NOT AROUND BUT WAY PEOPLE ARE THINKING IS SELF-SET. THAT'S THE IDEA THE EQUIPMENT IN YOUR STUFF TENDS TO MAKE YOU DO THE SAME THINGS OVER AND OVER AGAIN. SO BUT I AGREE WITH YOU, IT'S HIGHLY SIMPLISTIC AS FIRST CUT, AS YOU SAID IT'S NOT GOLD OR SILVER OR BRONZE STANDARD, IT'S A LEAD STANDARD OR WOOD STANDARD. I AGREE WITH THAT. BUT THE BEGINNING OF THAT. THE SECOND ONE WHICH IS, THAT'S THE DIFFERENCE BETWEEN GOVERNMENT AND INDUSTRY. INITIATIVES SPEND A LOT OF TIME DECIDING WHAT TO SAY, WHEN THERE'S NOVELTY YOU TRY MULTIPLE APPROACHES BUZZ YOU DON'T KNOW WHICH WORKSCH PEOPLE GOING UP ALLIES TO SEE WHICH ONES ARE BLIND. AS SOON AS YOU FIND AN ALLEY IS BLIND, INDUSTRY SAYS LET'S STOP THAT. THE SPEED YOU ABANDON RESEARCH IS CRITICAL FOR RESEARCH IN INDUSTRY. WHETHER OR NOT YOU WANT TO LOOK AT THE SPEED WHICH YOU ABANDON THINGS IN GOVERNMENT I DON'T KNOW. THAT IS MUCH MORE FRIGHTENING TO PEOPLE BUT THAT'S PART OF THE PROBLEM SO NOT JUST THE NOVEL WHICH WE HAVE POOR ENERGY BUT THE SPEED YOU ABANDON THINGS WHICH SAID THAT'S COLD -- DATA ISSUE, PEOPLE WANT TO CONTINUE WORKING BECAUSE THEY HAVE BUDGETS ON IT. >> I WOULD ADD ONE THING, COUNTER PART TO WHAT DICK AN DEWEY HAVE SAID. LOOKING AT OVERLAP HAS A SCALE COMPONENT TO IT. SO IF YOU'RE LOOKING AT A HIGH LEVEL, AT DISCIPLINARY LEVEL OR SPECIALTY LEVEL IT MAY APPEAR YOU HAVE OVERLAPS IN PORTFOLIOS. IF YOU DRILL DOWN THE TOPIC LEVEL YOU MAY SEE OVERLAP AT HIGHER AGGREGATION IS NOT REALLY AN OVERLAP AT ALL. SO BE CAREFUL LOOKING AT OVERLAPS TO REALIZE THE SCALE LOOKING AT MAP BECAUSE THE HIGHER YOU AGGREGATE THE MORE (INAUDIBLE). >> JEFF (INAUDIBLE) FROM NHGRI. MY FIRST EXPOSURE TO THE FEEL SO THE QUESTIONS MIGHT BE NAIVE. ONE IS GENERAL. IT FEELS TO ME LIKE THE BIOMEDICAL LITERATURE AND DATA IS RELATIVELY DIRTY OR MAKES ME WORRY ABOUT THE GARBAGE IN PROBLEM. I WAS CURIOUS Y'ALL DEAL WITH MANY, MANY OTHER FIELDS OF MORE BASIC SCIENCE LIKE MATHEMATICS AN ENGINEERING AND SUCH. DOES IT -- DOES IT GIVE YOU CHILLS WHEN YOU HEAR ABOUT ANOTHER PROJECT RELATED TO BIOMEDICAL RESEARCH BECAUSE YOU JUST THE DATA IS MESSY OR IS IT JUST -- IS IT A SIMILAR SITUATION FOR OTHER FIELDS? THAT'S ONE QUESTION. THE OTHER IS I HAD A THOUGHT ABOUT REVIEW ARTICLES, ESPECIALLY IN THE CONTEXT OF WHAT'S NEW AND EMERGING AND WHETHER SOMETHING IS OLD OR STALE. I THINK I ONLY HEARD REVIEW ARTICLES TALKEDN'T YESTERDAY AS PART OF THE WAY THE JOURNAL TO PUMP UP IMPACT FACTOR. IS THERE A WAY TO USE REVIEW ARTICLES AS WHEN YOU SEE HIGH PROFILE JOURNALS THAT IDENTIFIES THE EMERGING FIELD AND ONCE YOU HAVE SEEN LOW LEVEL JOURNALS WITH REVIEWS, IT IS A DONE DEAL. >> >> AS FAR AS BIOMEDICAL DATA AND DIRTIER THAN ANYTHING ELSE, I DON'T WORRY ABOUT THAT. THE DIRTIEST DATA I HAVE SEEN IN PATENT DEALERS THEY'RE PAYING, THERE'S LOTS OF (INAUDIBLE). BUT THESAME PERSON YOU HAVE -- SAY YOU HAVE A PROLIFIC INVENTOR IN THOSE DATA. THE LAST NAME IS USUALLY THE SAME, SOMETIMES THERE'S MISSPELLINGS, THEY CAN USE INITIALS, THEY CAN USE FIRST NAME, MIDDLE INITIAL, THEY CAN HAVE PERIODS AND COMAS AND STUFF SO THERE'S -- SAME WITH THE INSTITUTION. I DON'T SEE A NASTY PROBLEM THERE COMPAREDDED TO OTHER THINGS. JOINING DATABASES WHERE YOU GET INTO CLEANING IN MY EXPERIENCE. NOW I HAVE FORGOTTEN THE SECOND QUESTION. REVIEW PAPERS. ONCE REVIEW PAPERS HAVE SHOWN UP IT'S MERGED PRETTY MUCH, IF SOMEBODY HAS GOTTEN THAT MUCH LOOK AT AND BUILD REVIEW ARTICLE THAT'S GOT 200 REFERENCES IN IT, IT'S ALREADY ONGOING PROVEN AREA. >> ONE MORE COMMENT ON THE CLEANLINESS OF DATA F. YOU USE INSTITUTIONAL REPOSITORY SUCH AS HUMAN DATABASES, FUNDING DATABASES WHICH VERY MOST UNIVERSITIES HAVE, THIS IS BEAUTIFULLY CLEAN DATA BECAUSE IT HAS TOUCHED MONEY. IF YOU COMPARE THIS DATA, THIS PUBLICATION DATA, (INAUDIBLE). IT KEEP IN MIND TRY TO ALSO USE VERY HIGH QUALITY DATA. >> I'M WAITING FOR YOU THE TALK ABOUT THE REVIEW PAPER. >> ONE THING ABOUT REVIEW PAPER, IN TERMS OF THEORY OF STRUCTURAL VARIATION, THEY HAVE THE SIMILAR PROPERTIES, SOMETIMES THEY PLANNED IT WITH TRANSFORMATIVE PAPERS. SO BASED ON THE FACT REVIEW PAPERS TALK ABOUT LOTS TOPICS, SOME ARE NOT REALLY HAVE BEEN DISCUSSED TOGETHER. AS ONE FEATURE. CHOWMY DID A NICE STUDY -- CHAOMEI DID A NICE STUDY ON TRANSFORMATIVE RESEARCH IN A COUPLE OF REVIEW AREAS, KEY AND EARLY INDICATORS AS I RECALL. THEN BUILDING ON THAT, YOU NOT ONLY LOOK AT REVIEW PAPERS BUT CHARACTERIZE REVIEW PAPERS THAT REINFORCE EXISTING SCIENCE OR CHALLENGING THE EXISTING STRUCTURE, IN TERMS OF CITATIONS IN THERE REINFORCING WHAT WERE THE EXISTING LINKS WITHIN ISLANDS OR BETWEEN ISLANDS USING THE LANGUAGE. IT APPEARS TO BE GOOD POTENTIAL INDICATOR, WE AGREE WITH THAT, THAT'S ONE THINK CHAOMEI THE IS FOLLOWING UP. WHICH IS INDICATIVE OF TRANSFORMATIVE AND ONE OF THE BETTER EARLY SIGNALS OF TRANSFORM BIG LOOKING ATS THE DEGREE IN WHICH THE REFERENCES IN THERE ARE REINFORCING YOUR PRIOR ASUNLSES ABOUT STRUCTURE OR CHALLENGING THE PRIOR -- SHOWING A LOT OF STRUCTURAL (INAUDIBLE). >> MY TWO CENTS. MY PERSPECTIVE IS CLEANER THAN OTHER SCIENTIFIC DISCOURSE AREAS BECAUSE OF AVAILABILITY OF WHAT YOU'RE NOT NATIONAL LIBRARY LOSS NLM HAS DONE BY MAKING RESOURCES AVAILABLE IT'S IMPROVED THE QUALITY OF DATA SO WHEN -- FROM THE TYPE OF ANALYTICS THAT I HAVE ENGAGED IN, THE USAGE OF THINGS LIKE MESH AND CONTENT IMPROVED THROUGH THE -- NIH, IT'S MADE THE INVESTMENT MUCH CLEANER FROM ANALYTICS SO MY INITIAL REACTION, THIS IS OPINION, IT IS MUCH BETTER THAN USUAL. REGARDING THE ROLE OF REVIEW ARTICLES, TO COMMENT ON KEVIN'S COMMENT, IT HAS EMERGED WHAT HE SAID WHEN REVIEW ARTICLE IS PUBLISHED. THAT'S A RELEVANT PERSPECTIVE FROM DEPENNING HOW YOU DEFINE EMERGENCE. THE EMERGENCE OF A CONCEPT, EMERGENCE OF A METHOD OR TECHNOLOGY REVIEW ARTICLES PLAY DIFFERENT ROLES IN THAT IN THOSE TYPES OF EMERGENT STATES. THEY'RE NOT LINEARLY, THERE'S REPRESENTATIONS IN ALL THREE OF THOSE. YOU CAN PROBABLY USE ALL THREE CLASSIFICATIONS. IT'S RELEVANT THAT WHAT WE ARE LEVERAGING IN THE FAWS PROGRAM A NUMBER OF RESEARCHERS USING REVIEW ARTICLES AND DIFFERENCE ROLES. AND DIFFERENT DISCIPLINES REVIEW ARTICLES PLAY DIFFERENCE ROLES IN SOME WAYS. AND SO THEY ARE LEVERAGED. >> MESH DATA BIOMEDICAL DATA WE DECIDED NOT TO GO FOR OVERLAY STRUCTURES USING EQUIPMENT DATA BUT TO FIRST TRANSFORM THEM INTO ISI DATA BECAUSE THE ISI DATA, THE FS FIELD IS MORE RELIABLE IN TERMS OF DISTRIBUTED (INAUDIBLE) FOR THE BIOMEDICAL DATA. >> (INAUDIBLE) NIMS. SO REALLY ENJOYED ALL OF YOUR FOUR TALKS. I THINK IT WOULD BE NICE THAT YOU COMPARE WHAT YOU CAN DO AMONG EACH OTHER, IT WOULD BE NICE, PUT YOURSELF ON THE MAP OR EVEN A SIMPLE CHART, A, B, C, D, E E F, G, YOU CAN DO THESE THINGS, MAYBE CHECK, IT WOULD BE NICE TO DO THAT. FOR US SIMPLE THINGS, WHEN WE HEAR INDIVIDUAL PRESENTATION THEY'RE ALL WONDERFUL, IT WOULD BE NICE TO SEE WHAT THE STRONG POINT, MAYBE A BIGGER DOT, SOCCER COLOR OR SOMETHING. ONE EXAMPLE I HAVE IN MIND, WE DO OUTCOME ANALYSIS OF A PROGRAM, FOR EXAMPLE. AND I WOULD LIKE TO HEAR YOUR PROGRAM CAPABILITY TO DO THAT. >> I CAN ADD TO THE FIFTH PART. I THINK IT'S VERY IMPORTANT THAT YOU HAVE COLLABORATION AND COMPETITION GOING ON. THE AREA OF SCIENCE METRICS IN THE U.S. RECEIVES LITTLE FUNDING. ONE WHICH IS MOST RELEVANT. IT FUNDS A LOT OF SOCIAL SCIENCE, SOCIOLOGY, ECONOMIC AND OTHER RESEARCH. SO THE IMMUNITY WHICH YOU CAN GROWTH THIS AMOUNT OF MONEY IS RELATIVELY SMALL. WHAT WE HAVE BEEN DOING IS TO COLLABORATE AND OTHERWISE ALSO TRY POTENTIALLY COMPETING APPROACHES. AND WHAT IS NEEDED RIGHT NOW IS TO VALIDATE SOME OF THOSE APPROACHES. THE BEST WAY TO VALIDATE IS TO WORK CLOSELY WITH ONE OF US, YOU WILL OF US, TIME FOR DOING. AND TO SEE IF THESE TOOLS MAKE A DIFFERENCE FOR YOU. I BELIEVE WITH SUFFICIENT FEEDBACK FROM THE USER COMMUNITY AND THE FUNDING IT TAKES TO IMPLEMENT TOOLS TO DOCUMENT THOSE TOOLS, TO TEACH THOSE TOOLS, AND TO ALSO THEORY DEVELOPMENT AT THE SAME TIME AND TO TRAIN NEW GRADUATE STUDENTS WHICH COME TO YOUR OFFICE AN WORK WITH YOU, IT WILL GO FORWARD. IF IT'S ONLY SCIENCE FUNDING THIS IS WHAT YOU GET. >> ANOTHER QUICK ANSWER. IN THE PRESENTATION THAT I'LL BE GIVING AFTERWARDS I LIKE ALAN'S AND CHAOMEI'S TECHNIQUES. WE'RE COLLEAGUES AND WE USE DIFFERENT TECHNIQUES AN WE'RE AIMING TO DO AS KEVIN SAID THE SAME THING BUT THE QUALITATIVE DIFFERENCES BETWEEN THEM IS PART OF THE EXPLORATION OF THAT SPACE. AND KATY IS ACTUALLY HAS MULTIPLE APPROACHES BECAUSE YOU'RE TRYING TO ADD MULTIPLE TOOLS. AND DEWEY IS TRYING TO FUND NEXT GENERATION. THAT NONE OF US HAVE USING FULL TEXT. SO THAT'S IN A NUTSHELL, WE CAN TALK MORE ABOUT IT AFTERWARDS. >> SO LOT OF YOU ARE USING TOOLS THAT INVOLVE TEXT MINING OF JOURNAL PUBLICATIONS. THAT INVOLVE TEXT MINING OF JOURNAL PUBLICATIONS. AS WE DISCUSSED A SECOND AGO NOT ALL PUBLICATIONS ARE EQUIVALENT, YOU HAVE REVIEW ARTICLES, YOU HAVE OPINION ARTICLES, BUT THERE'S METHODS PAPERS, THEORY PAPERS, EXPERIMENTAL PAIP E AND SEEMS TO ME THAT THE INFORMATION YOU CAN -- THAT COULD EMERGE, TECHNOLOGY THAT COULD EMERGE WOULD DIFFER DEPENDING IF YOU'RE TALKING THEORY PAPER OR EXPERIMENTAL PAPER. RETRACTIONS. THE THEORY IS TO VALIDATE A THEORY, THE THEORY IS WRONG, THIS COULD BE A BIG DIFFERENCE THEN SOME METHODS PAPER THAT IS REFERRING TO SOMETHING ELSE. SO YESTERDAY WE LEARNED ALL THESE ONE CREATES TOOLS THAT CATEGORIZE AND BEND DIFFERENCE TYPES OF DATA. IF IT DOESN'T REEXIST WE COULD GENERATE A TOOL THAT COULD CATEGORIZE JOURNAL PUBLICATIONS TO CATEGORIZE THEM AS METHODS OR EMPERIMENTALLIST TYPE OF PAPER. WOULD THAT TOOL BE USEFUL, IS IT WORTH THE EFFORT TO GENERATE THAT TOOL USEFUL IN NEXT GENERATION OF THESE TECHNOLOGY EMERGING TYPES OF TOOL? DOES THAT MAKE SENSE? >> PARSING OUT THE DIFFERENCE TYPES OF JOURNAL ARTICLE. >> JUST AS A START, THE DATABASES PROVIDE FIRST CUT WITH DOCUMENT IN TIME, SO FOR INSTANCE WHETHER THE SCIENCE, 8 OR 10 TYPES SO YOU CAN PARTITION THIS IS AN EXPERIMENTALIST PAPER VERSUS THEORY PAPER VERSUS DEVELOPING A MODEL. DO YOU USE THAT, IS THAT A USEFUL TYPE OF INFORMATION IN YOUR META ANALYSIS TOOLS? >> I THINK FOR US IT GIVES YOU DOZEN OR SO FIELDS THAT SOMETIMES YOU WANT TO CUT AND SAY LET'S LOOK AT THESE, NOT THOSE. BUT IS IT FULLY MINED TO PICK UP THOSE DIFFERENCES? I DOUBT IT. WOULD IT BE WORTHWHILE TO INVEST IN THAT TOOL? >> A QUICK COMMENT REGARDING THAT. I THINK IN GENERAL, IT'S NOT FULLY TESTED. BUT I THINK IT IS, THERE'S A LOT OF PROMISE. AN EXAMPLE WHERE IT MIGHT BE. TRYING TO IDENTIFY CAPABILITY OF AN ORGANIZATION, UNIVERSITY OR GROUPING OF PLACES, WHETHER THEY'RE CAPABLE OF PRODUCING IN THE AREA, UNDERSTANDING WHETHER THEY HAVE A THEORETICAL EFFORT VERSUS EXPERIMENTAL EFFORT OR SOME COMBINATION OR WHAT THE BALANCE IS BETWEEN THOSE CAN GIVE YOU INSIGHT OF WHAT ACTUAL CAPABILITY IS. SO DEPENDING ON THE USE CASE, I THINK JULIA WAS HARPING ON THIS YESTERDAY, ABOUT THE NEED TO BASICALLY UNDERSTAND WHAT YOU'RE TRYING TO DO. IF YOU'RE USE CASE REQUIRES THAT KIND OF DIFFERENTIATION, DIFFERENTIATION OF CAPABILITY THEN I THINK THAT KIND OF CHARACTERIZATION IS USEFUL. IN THE FUSE PROGRAM THERE ARE NOT A LOT OF TEAMS BUT TEAMS WORKING ON GENRE CLASSIFICATION TO DISTINGUISH SOME OF THOSE CHARACTERISTICS. IT'S STILL EARLY IN TERMS OF HOW USEFUL IT IS FOR A PARTICULAR USE CASE OR EMERGENCE USE CASE. I DON'T KNOW YET. I THINK IN PRINCIPLE IT MAKES SENSE. AND MANUALLY I HAVE HAD TO DO THAT CLASSIFICATION MYSELF. BECAUSE OF THE PARTICULAR NATURE OF ANALYTICS PAST. -- I WOULD HAVE TO SAY BACK UP WHAT DEWEY SAID, IN TERMS OF OFF THE SHELF SYSTEMS THAT PROVIDE DIFFERENTIATION BETWEEN ARTICLE TYPES, THE ONE IN PUBMED IS PROBABLY BEST. IT DOES DISTINGUISH SOME DEGREE OF CLINICAL VERSUS EMPERIMENTAL. THAT LEVEL DOES NOT EXIST AT SCOPIST OR OTHER. THEY SAY ARTICLE OR CONFERENCE PAPER OR NOTE FOR REVIEW. >> ONE LESSON I'M TAKING FROM THIS MEETING IS FROM A POLICY PERSPECTIVE THE SINGLE MOST USEFUL THING NIH MIGHT BE IEBL TO CONTRIBUTE PORTFOLIO ANALYSIS IS LITERATURE DISAM BIGGIZATION OR MAKING SURE UNIQUE IDENTIFIERS FOR EACH SCIENTIST AND WE KNOW WHAT THEIR OUTPUT IS. IN A WAY THAT CAN BE MOVED EASILY. ONE IS THAT A REASONABLE THING, I ASSUME THAT IT IS. TWO, ARE THERE ANY OTHER POLICY THINGS THAT YOU THINK WOULD REALLY HELP US DO A BETTER JOB ANALYZING THE PORTFOLIOS FROM A GENERAL POLICY PERSPECTIVE, NOT SU SUPPLYING EACH WITH GRANTS. >> I WANT TO MAKE A DIN BETWEEN THINGS THAT -- DISTINCTION BETWEEN THINGS HELP CLEAN UP THE DATA, NOISE REDUCTION AND THINGS THAT WILL HELP DECISION MAKING. MOST OF WHAT WE'RE TALKING ABOUT IS ACTUALLY DEVELOPING THE TOOL NOT KNOWING HOW THE HECK TO USE IT TO MAKE A DECISION. IF THAT DECISION OR FRAMEWORK THAT IS UNDERDEVELOPED AND GOING BACK TO KATY'S SUGGESTION, THAT'S THE PART THAT NEEDS MORE ATTENTION THAT'S THE GAP WE HAVE TO FIX. YOU ALWAYS WANT TO HAVE PEOPLE DEVELOPING TOOLS, WANTING TO SPEND MORE MONEY MAKING THEIR TOOLS THAT ARE FASTER BUT THE QUESTION IS DO YOU REALLY USE THEM OR WOULD THEY BE USEFUL. I THINK THAT IS THE GAP. >> I HAD THE COMMENT IN WHICH IS VERY RELATED TO YOURS. I DIDN'T KNOW YOU WERE GOING TO MAKE IT. I'M (INAUDIBLE) FROM NHLBI. SO I'M REPRESENTING THE SORT OF STARRY EYED USE POTENTIAL USERS, Y'ALL BLOWING US AWAY WITH WONDERFUL TOOLS BUT I FEEL LIKE WHEN I WALK INTO LOWE'S OR ANY OTHER HOME IMPROVEMENT STORE I SEE ALL THESE WONFUL TOOLS, I HAVE NO IDEA WHAT AM I GOING TO DO WITH THEM. I WOULD LOVE TO BE ABLE TO USE THEM. SO I THINK THAT I'M FEELING WE ARE IN THAT SITUATION WHERE WE HAVE TOOLS AND WE HAVE QUESTIONS. AND I'M NOT SEEING A GREAT CONNECTION BETWEEN THE TWO. I THINK THE TOOLS ARE WAY AHEAD OF OUR QUESTIONS. I THINK WE DIDN'T FORMULATE OUR QUESTIONS YET. I'M HOPING THAT WHAT NIH COULD DO IS COLLECT A SERIES ACROSS NIH OR VARIOUS INSTITUTES OR VARIOUS PROGRAMS. AND THEN MATCH THEM WITH THE TOOLS THAT OBVIOUS WE ARE WAY AHEAD OF US. SO I'M NOT SURE THE CORRECT -- COULD YOU DEVELOP THIS TOOL BECAUSE I'M GOING TO COME BACK AND SAY WHAT IS YOUR QUESTION. I THINK WE HAVE HAMMERS AN NAILS AND A LOT OF HAMMERS AND WE DON'T KNOW YET IF WE HAVE A SCREW OR A NAIL. OR SOME OTHER THING THAT WE NEED TO FIX. WONDERING IF WE CAN DO THAT, MATCH POTENTIAL USERS WITH TOOL DEVELOPERS AN WORK TOGETHER INDEVELOPPING THOSE TOOLS. >> I THINK WE SHOULD ALL STAND UP AND CLAP. >> I THINK YOU'RE RIGHT. TYPICALLY THIS KIND OF IDENTIFICATION OF SWEET SPOTS IS NOT ONLY DONE BY COMING FROM THE MEAT SIDE BUT ALSO FROM THE SIDE OF WHAT TECHNOLOGY CAN GIVE YOU TODAY. AND WHERE YOU WANT TO GO SO YOU CAN START COLLECTING THE DATA TO DO WHAT YOU WANT TO ACHIEVE. TYPICALLY WE BEST WHICH OF DOING IT IS PO DOCKS. POST-DOCS WHICH POLLINATE FROM ONE LAB TO THE NEXT. OUR DOMAIN MY Ph.D. STUDENTS ARE HIRED AWAY BEFORE THEY FINISH. MY STAFF MEMBERS GET PULLED AWAY FROM COOL COMPANIES, IT'S HARD FOR ME TO HAVE ANY KIND OF POST-DOC. BUT WHAT CAN BE DONE IS THAT STAFF MEMBERS FROM MY TEAM COME HERE AND WORK CLOSELY WITH SOMEBODY ESPECIALLY (INAUDIBLE) FOR THIS. WHAT CAN ALSO BE DONE IS IF YOU HAVE SOMETHING LIKE A SABBATICAL AND YOU WANT TO BE REHEARSED IN OUR RESEARCH ENVIRONMENT PLEASE COME. WE HAVE MANY VISITING RESEARCHERS, TYPICALLY FROM OTHER IVORY TOWERS NOT IMMEDIATELY GOVERNMENT BUT WE HAVE GOVERNMENTAL COLLEAGUES WHICH COME FROM MAYBE GIVING A TALK FOR A FEW HOURS TO GET THEM TO ANSWER THEIR QUESTIONS, ET CETERA. WE NEED THIS VERY CLOSE COLLABORATION ABOUT COMMUNICATION TO BUILD TOOLS ADDRESSING YOUR NEEDS. >> OUR OFFICE OF PORTFOLIO ANALYSIS ALSO CAN HELP WITH THAT. SO DON'T BE SHY ABOUT CONTACTING US AS WELL. >> I EACH RICHARD (INAUDIBLE) FROM THE NATIONAL HEART LUNG AND BLOOD INSTITUTE. I WOULD LIKE TOFER A TOOL AND -- TO OFFER A TOOL AND APPROACH THAT WOULD BE VERY HELPFUL IN FEELING WHAT I SEE IS INFORMATION ASYMMETRY I GUESS ECONOMISTS MIGHT CALL IT. THE YESTERDAY THE PAM TALKED ABOUT SO MANY HURDLES A GRANT HAS TO GO THROUGH BEFORE IT'S FUNDED THAT ELIMINATES THE CHANCE PS OF IT DOING THE SAME RESEARCH THAT'S ALREADY BEEN DONE OR BEING DONE. I GUESS I WOULD DISAGREE WITH THAT. I THINK APPLICANTS SOMETIMES DONE KNOW WHAT RESEARCH IS BEING FUNDED BY NH -- OR BY NIH. REVIEW GROUPS, PEER REVIEW GROUPS WE ASSUME THEY'RE KNOWLEDGEABLE ABOUT ALL FIELDS BUT CLEARLY THAT'S NOT TRUE. BOTH OF THEM AS WELL AS COUNCIL IN THEIR REVIEW BENEFIT FROM KNOWING WHAT IS CURRENTLY BEING FUNDED. THERE IS A NEW TECHNOLOGY NOW AVAILABLE THAT I HEARD THE PREVIOUS PORTFOLIO ANALYSIS SYMPOSIUM AND HAS BEEN DEVELOPED SINCE, THE FEATURE OF LIKE ON THE FLY OR LIKE THIS THAT ALLOWS AN INDIVIDUAL TO DO AN ANALYSIS OF THE CURRENT FUNDED PORTFOLIO AT NIH. IF EVERYONE WAS REQUIRED TO DO THAT, FOR INSTANCE IF THE APPLICANT WAS REQUIRED TO DO THAT BEFORE SUBMITTING THEIR GRANT THEY'D EITHER SEE THAT THEIR RESEARCH WAS REDUNDANT AND MODIFY THEIR APPROACH OR MODIFY THEIR ABSTRACT OR MODIFY THEIR SPECIFIC AIMS, TO TAKE THEIR RESEARCH FARTHER OUT TO THE EDGE OF WHERE WE ARE IN THE SCIENTIFIC COMMUNITY. THE SAME WAY WITH REVIEW GROUPS, ASKED TO FUND INNOVATIONCH UNLESS THEY KNOW WHAT'S CURRENTLY BEING DONE, THEY DONE REALLY KNOW WHAT'S INNOVATIVE. AND HAVING A STANDARDIZED APPROACH TO E EVALUATION TO WHAT'S IN THE PORTFOLIO TO ALLOW THEM TO SCORES INNOVATION THAT COME CLOSER TO REALITY THAN THEIR GENERAL GESTALT OF THE FIELD. SAME WITH COUNCIL. IF THEY HAD THIS AT THEIR FINGERTIPS, THEY COULD KNOW WHETHER THIS IS NEW RESEARCH OR WHETHER THIS IS SAME OLD SAME OLD. SO I PROPOSE THAT AS AN IDEA. >>Q.And:IT'S DOABLE. IT'S MATTER OF DOING POLICIES OR MAKING THESE SERVICES AVAILABLE AND KNOWN TO MANY PEOPLE AT NIH. SO I JUST LEARNED YESTERDAY ABOUT THIS -- COMMENTS YOU CAN GO IN AND PASTE YOUR ABSTRACT OR SUMMARY IN THERE AND YOU GET LIKE NO PROPOSALS BUT A SHORT YOU CAN CHECK SOMETHING I DID NOT KNOW ABOUT THIS. MAYBE THERE ARE MANY OTHERS WHICH ALSO DON'T KNOW IT AND I WONDER IF ALL THE STUDY SECTIONS KNOW ABOUT IT. IT'S ALSO IMPORTANT THAT CI PACE THE TEXT BECAUSE I MIGHT BE INTERESTED TO GET THIS FUNDING. OBVIOUSLY I WROTE THIS PROPOSAL, YES I WANT TO GET THIS FUND AND I MIGHT NOT KNOW WHAT MIGHT BE A GOOD FIT. SO BY SEEING WHAT SIMILAR PROPOSALS WERE FUNDED OR STUDY SECTIONS THIS GIVES ME A HINT WHERE TO SUBMIT IT. VERY IMPORTANT. >> THESE METHODS ARE IMPORTANT TO INTRODUCE AT EVERY STAGE FROM WHEN THE GRANTEE FIRST HAS THE IDEA ALL THE WAY THROUGH TO THE COUNCIL MEETING. >> PETER GUTHRIE, CSR. I WOULD LIKE TO POINT OUT THE QVR SYSTEM AND LIKE FUNCTION ARE BASED ON THE RCDC FINGERPRINTING WHICH IS BASED ON THE UML LIST KIND OF THESAURUS. UNFORTUNATELY, BY THE TIME A WORD GETS INTO A THESAURUS IT'S USUALLY NOT AN EMERGING FIELD. SO INNOVATION IS GOING TO BE A LITTLE DIFFICULT TO DETECT THAT WAY. THE LIKE THIS IS ONLY MATCHING AGAINST FUNDED PROJECTS. WHEREAS THE INTERNAL NIH PEOPLE CAN MATCH IT AGAINST ANY SUBMITTED PROJECTS SO WE CAN GET A MUCH BETTER FEEL FOR WHETHER THINGS HAVE BEEN SUBMITTED BUT IN TERMS OF OUTSIDE APPLICANT BEING ABLE TO SEE WHAT'S BEEN -- THEY CAN ONLY SEE WHAT'S FUNNED. THEY CAN'T NECESSARILY SEE WHETHER THERE'S A LOT OF INTEREST AND WHETHER THERE'S A LOT OF STUFF COMING MANY. SO GOOD IDEA BUT LIMITED BY LIMITATIONS ON THE THESAURUS AND THE LIMITATIONS ON THE EXTENT OF THE DATABASE. IN TERMS OF INTRAMURAL USE, IT'S A FANTASTIC TOOL, PARTLY BECAUSE WE HAVE ACCESS TO ENTIRE DATABASE AND NOT JUST FUNNED PROJECTS. >> THIS COULD BE ADDED TO THE NIH MAP.ORG WHICH I (INAUDIBLE) THIS MORNING SO BASED ON TOPIC MODELING, THERE YOU COULD ANALYZE IT PROPOSAL TEXT AND MAKE IT AVAILABLE THIS WAY ALSO. THEN AGAIN YOU HAVE TWO SYSTEMS THAT WILL DEPEND HOW OFTEN PEOPLE USE ANY OF THE SYSTEMS, WHICH ONE HAS BETTER SERVICE, BETTER FUNCTIONALITY, WILL WIN. >> SO INTERESTING TO COMPARE MAPPING WITH THE RCDC FINGERPRINT BECAUSE THE SCIENCE MAPPING IS FREE TEXT, CORRECT? WHEREAS THE THESAURUS IS LIMITED IN THAT SENSE. TO SEE HOW THEY MAP TO EACH OTHER. I LOVE THE VISUALIZATION OF THE SCIENCE MAP. WE DON'T HAVE THAT ARE THE RCDC FINGERPRINTING. WE'D LOVE TO HAVE SOMEONE TELL US HOW TO DO THAT. >> COME TO THE TUTORIAL. >> I THINK THE IDEA OF RESEARCH PROFILING IN MULTIPLE WAYS IS VERY ATTRACTIVE AND NOTION OF STANDARDIZING A PROCESS SO THAT EMPIRICAL INFORMATION WERE MADE AVAILABLE TO THE COMMITTEE, THE REVIEW PANEL AND SO ON. >> JOHN THOMAS, NHLBI. I'M INTERESTING HEARING COMMENTS ABOUT COMPARISON OF TOOLS AND VALIDATION OF TOOLS. HOW CAN WE CHOOSE WHICH TO USE? >> WE HAVE A NUMBER OF TOOL DEVELOPERS HERE IN THE ROOM. THE IF YOU GO TO THE THE SIGN OF SCIENCE TOOL YOU WILL SEE A LISTING OF 30 PLUS DIFFERENT TOOLS STUDY NETWORKS BECAUSE WE HAVE LOT OF NETWORKS IN SCIENCE BUT ALSO LISTED PUBLISH OR PERISH, VERY INTERESTING SOFTWARE, AND THERE'S MANY OTHER TOOLS WHICH ARE USED OUT THERE. RELATIVELY LIMITED FUNDING WHICH GOES TO THIS AREA, WE HAVE AN ECOLOGY OF DIFFERENCE TOOLS, OFTENTIMES YOU HAVE TO THE APPLY MULTIPLE TOOLS TO GET FROM YOUR DATA TO THE FINAL RESULT. THIS IS NOT SATISFYING BECAUSE IT TAKES TIME TO LEARN ALL THESE DIFFERENT TOOLS. THE SCIENCE OF SCIENCE TOOL IS AN ATTEMPT TO DO A PLUG AND PLAY SO THAT YOU CAN VERY EASILY SUCK NEW DATA READERS, YOU COULD DEVELOP A NEW CONNECTION TO THE INTERNALLY USED NIH DATA, PLUG THIS IN AN INSTEAD OF FILE LOAD YOU DO FILE CONNECT TO DATABASE AND YOU GET YOUR DATA FROM NORMAL DATA HOLDING. SIMILARLY YOU CAN ADD ANY NEW ALGORITHMS WHICH ARE DEVELOPED IN TEMS OF CLUSTER ORGANIZE DATA MINING, VISUALIZATION, IN AND YOU HAVE ALWAYS THE SAME INTERFACE, IT'S RELATIVELY EASY TO ROUGH COMPUTER CODE BE IT JAVA OR (INAUDIBLE) UP IN THESE PLUG INS SO AT SOME POINT YOU WILL HAVE AN ENVIRONMENT JUST LIKE ON YOUTUBE YOU HAVE A SIDE WHERE YOU DON'T SHARE PICTURES AND VIDEOS BUT ALGORITHMS. WHEN YOU COME IN YOUR OFFICE YOU WILL SEE NO NU DATA, THE NEXT GENERATION TWITTER STREAM. OR NEW ALGORITHMS, ANYONE USE THEM. THAT'S A LITTLE SCARY FOR THOSE THAT JUST HAVE TOOLS LIKE EXCEL OR PHOTO SHOP OR WORD. WHAT ELSE IS IN THE MENU SYSTEM AND HOW YOU SHALL USE THIS. THESE FLEXIBLE TOOLS WHERE YOU DECIDE WHAT YOUR PLUG INS NEED TO TO BE AND AND RESEARCHERS AND WHAT WORK REDESIGN AND WRITE UP PEERL REVIEWED ARTICLES MAKE AVAILABLE TO YOU OR YOU START DESIGNING YOUR OWN WORK FLOWS BECAUSE MY EXPERIENCE NOW IS THAT MANY DESIGN POLICY MAKERS THAT YOU DECIDE TO TUTOR RUN VERY DIFFERENT ANALYSIS THAN WHAT PEOPLE IN IVORY TOWERS ARE DOING THAT WANT TO PUBLISH PAPERS. QUITE A BIT DIFFERENCE BETWEEN WHAT THOSE PEOPLE DO AND HOWEVER STILL IT'S OUR DUTY TO ALSO GIVE YOU WORK FLOWS THAT ARE PEER REVIEWED AND VALIDATED AND GIVE YOU THE HIGHEST QUALITY RESULTS. THEN YOU CAN RUN THOSE IN THOSE TOOLS. SO VERY BRIEFLY, GIVEN THE DATA FUNDING, WE HAVE MANY DIFFERENT TOOLS, MANY DIFFERENT THINGS AND WE ARE TRYING TO INTERLINK THEM BY SHOWING HOW YOU CAN REALLY USE DIFFERENT TOOLS IN ONE WORK FLOW AND HOW YOU BENEFIT FROM THE BEST IN THOSE TOOLS SO IT'S NOT SUFFICIENT FUNDING TO DO ONE TOOL WHAT YOU NEED. >> I COULDN'T AGREE MORE WHAT YOU'RE SAYING. THERE'S ALSO MODULARITY, NOT JUST BECAUSE (INAUDIBLE) I THINK IT'S ACTUALLY A VERY VIABLE CONCEPT. I WANT TO ADD ONE BRIEFLY TO WHAT YOU WERE SAYING. YOU ASKED HOW CAN CAN WE VALIDATE THE EFFICACY OF TOOL OR PARTICULAR APPROACH. I THINK THERE ARE -- MY PERSONAL OPINION IS THIS IS ACTUALLY RARELY DONE. COUPLE OF REASONSCH ONE, HAS TO BE CONNECTED WITH USE CASE OR WORK FLOW FOR IT TO BE EFFECTIVE. I DON'T THINK WE HAVE GOT A GREAT CATALOG. SOUND LIKE YOU'RE WORKING ON IT BUT A CATLING OF WORK FLOWS FOR THIS THING. YOU HAVE TO CLEARLY ARTICULATE WHAT YOU WANT. BUT THERE HASN'T BEEN VALIDATION OR EVEN AGREEMENT WHAT METRICS SHOULD BE USED TO VALIDATE. FOR EXAMPLE, YOU'RE SHOWN A MAP, ARE YOU GIVEN A SUR RAY AFTERWARDS TO KNOW DID YOU ACTUALLY COME TO THE RIGHT CONCLUSION FROM THIS MAP. I'M NOT -- ARE YOU TOOL TO REACH CONCLUSION AT. OR SET OF TOOLS, VERY RIGHT. OFTEN A WHOLE VARIETY OF TOOLS YOU HAVE TO RUN IN YOUR DATA TO FORM TO MAYBE AN ANALYTIC CONCLUSION FROM. MY POINT TO SAY IT'S EARLY DAYS. IT HASN'T BEEN REALLY WORKED OUT YET. I THINK IT'S HARD TO COME TO A CONCLUSION REMOTELY TO YOU BECAUSE OF THOSE CHALLENGES. >> IT'S A LOT OF -- I'M IN FAVOR OF MODULARITY BUT THESE TOOLS WHICH ARE CURRENTLY AVAILABLE TO YOU, THEY WERE DEVELOPED BY SOCIAL SCIENTISTS, BY ECONOMISTS. BY BIOLOGISTS, BY PEOPLE LIKE ME FROM THE ENGINEERING DOMAIN. THEY HAVE DIFFERENT PHILOSOPHIES BEHIND AND LEARNING EACH OF THEM IS VERY TIME CONSUMING. MANY OF US USE SITE ESCAPE TO DO A NETWORK VISUALIZATION. IT'S A WONDERFUL TOOL BUT TAKES QUITE A BIT OF TIME TO GET FAMILIAR WITH IT IF YOU WANT TO USE IT PROFESSIONALLY. THEN DOWK THE WORK WITH IT. SO THERE ARE MANY TOOLS LIKE THIS AND IT'S OUR JOB TO GET YOU BETTER TOOLS. IT TAKES CLOSE COLLABORATION. >> I WANT TO MENTION TO ME IT'S A VERY IMPORTANT QUESTION, ONE KEVIN AND I, ONE OF THE REASONS WE COLLABORATE. WHEN WE CAME, WHEN WE'RE BOTH INVOLVED, PASSIONATELY CONCERNED THAT NOBODY TRIES TO MEASURE THE QUALITY OR PERFORMANCE OF ANY TOOLS, IT'S JUST THIS MASSIVE INDUSTRIES SAYING TRY ME, TRY ME. IT WAS NICE FINDING SOMEBODY WHO AGREES WITH YOU'RE WISE CRAZY. SO FOR THE LAST FIVE SIX YEARS WE HAVE BEEN WORKING ON THAT. NIH FUNDED A PROJECT TO COMPARE DIFFERENT QUALITY OR ACCURACY OF DIFFERENT TOOLS, DIFFERENT MEASURING TECHNIQUES THAT ARE USED IN THIS KIND OF DOCUMENT CLUSTERING. KATY CONTRIBUTED TO THAT. WE HAVE PEOPLE FROM THE TEXT, WE HAVE PEOPLE FROM CITATION, KEVIN WAS PRINCIPLE INVESTIGATOR. TWO ARTICLES THAT COMPARED ACCURACY OF SPECIFICS. THAT'S THE EXCEPTION TO THE RULE. THAT'S A COMPARISON OF ACCURACY, YOU RARELY SEE BECAUSE IT'S COSTLY AND NOBODY REALLY SEEMS TO CARE EXCEPT FEW OF US. IT'S EVEN WORSE WHEN TRYING TO FIGURE OUT THE ACCURACY OR IMPORTANCE WHEN IT'S USED IN ACTUAL DECISION MAKING. THAT'S WHAT YOU ARE TALKING ABOUT. THERE IT'S EVEN WORSE BECAUSE KEY QUESTION, DID IT MAKE A DIFFERENCE IN YOUR DECISION? THIS IS TO ME THE MOST IMPORTANT THING. IT MAY KNOW DIFFERENCE IN YOUR DECISION, IT WAS A WASTE OF TIME. IF YOU MAKE YOUR DECISION FASTER AN MORE COMFORTABLE WITH IT, THAT'S GOOD. BUT IF IT MACKS YOU SLEEP BETTER AT NIGHT, TAKE SOME MEDICATION FOR SLEEPING BETTER AT NIGHT AND SAVE YOURSELF HALF A MILLION DOLLARS. COME ON. SO THIS IS A REALLY HARD PROBLEM AND THAT'S ONE OF THE THINGS THE FUSE PROGRAM IS TRYING TO STRUGGLE WITH. NOT ONLY FOUR TEAMS DEVELOPING TOOLS BUT A TEAM TO FIGURE HOW TO EVALUATE THEM FAIRLY SO YOU -- THIS KIND OF QUESTION CAN BE ANSWERED SO IT'S A FRONTIER QUESTION, THAT'S WHY IT'S BEING FUNDED AS A FRONTIER QUESTION TO TRY CAN WE ANSWER THAT QUESTION. THERE IS NO ANSWER TO THE QUESTION. >> A THOUGHT ON HERE, THERE'S VALIDITY AND UTILITY ISSUES, AND MERGE AD BIT WHAT DICK SAID, I THINK ON THE UTILITY SIDE IS A KEY AND FEELS TO ME LIKE OFFICE OF PORTFOLIO ANALYSIS IS REALLY IN A NICE POSITION TO HELP Y'ALL COME TO DANCE. TO COME IN WITH ISSUE, YOU MAY NOT HAVE THEM ORCHSTRATED IN A NICE NEAT FORM WE CAN SAY TOO LATE WE'LL WORK HERE, BEAUTIFULLY. BUT TO BRING SOME ISSUE, TRY SOME TOOLS, DO SOME ITERATION, WITH AN OFFICE KEEPING TABS WHAT'S WORKING FOR WHOM, UNDER WHAT CONDITIONS. >> I AGREE, ALSO HAVING AN OFFICE TO GO TO IF YOU'RE STUCK WITH WORK FLOW OR WANT TO KNOW WHAT TOOLS ARE ACTUALLY BEST IN THIS PARTICULAR CASE, I THINK OUT'S GOOD. YOU MIGHT LIKE TO ULTIMATELY SET UP ENVIRONMENTS WHERE YOU HAVE PRINCE ONE STUDY GROUP USE A CERTAIN TOOL, THE OTHER STUDY GROUP DOESN'T GET IT. ONE GETS A CL QUEUE LAYTOR, THE OTHER ONE DOESN'T. AFTER A WHILE YOU CAN SEE HOW SUCKsFUL THOSE PROJECTS WERE OR WHAT OTHER SIDE BENEFITS THOSE PROJECTS BROUGHT OR HOW STUDY GROUPS FUND TO THOSE. YOU CAN SET UP IN VIVO EXPERIMENTS. (OFF MIC) >> A QUESTION OF WHETHER TOOL GIVES YOU THE BEST MAP IS A VERY INTERESTING ONE, IT IS IN SOME WAYS SUBJECTIVE. I RUN A REVIEW SECTION, I'M TRYING TO BID APPLICATIONS AND FIND REVIEWERS FOR THEM. IF I LOOK IN A PILE OF APPLICATIONS I SET THEM IN CERTAIN BINS SOMETIMES E DEPENDING ON REVIEWERS I HAVE, DEPENDING UPON HOW THEY SORT OUT. WE'RE TAKING A MULTI-DIMENSIONAL RELATIONSHIP IN A CASE LIKE THIS. 100 APPLICATIONS, 30 REVIEWSERS, TRYING TO MAP IN TWO DIMENSIONS. DEPENDING HOW YOU DECIDE TO SLICE IS WILL DETERMINE THE WAY THINGS CLUSTER. WHAT -- YOU ALREADY TOLD ME YOU DON'T HAVE AN ANSWER FOR THE BEST -- HOW TO DEFINE THE BEST BUT I'M WILLING TO LISTEN TO QUESTIONS OR IDEAS WHETHER THERE IS GOING TO BE A BEST ANSWER AND THE APPROACH YOU GUYS ARE TAKING TO FIND THAT. >> THE PURPOSE OF THE TWO IS TO OPEN UP YOUR MIND, >> I WOULD ADD TO THAT, ALTHOUGH WE ALL TALK ABOUT HAVING THE MOST ACCURATE TOOL OR THE MOST ACCURATE MAPS, WE'RE DEALING WITH VERY HEIDI MENTIONALS. THE CHANCES THERE'S A SINGLE OPTIMUM BEST ANSWER IS ALMOST NIL. LET ME GIVE YOU AN EXAMPLE. WE DID A STUDY, CSR OVER THE WINTER WHERE WE LOOKED AT THE STUDY SECTION. THE STUDY HAS EVOLVED OVER TIME, NOT STRICTLY SEGMENTED BY DISEASE, ORTY PLIN OR ANY OTHER STANDARD CATEGORY. IT'S SOMETHING THAT IS GROWN HISTORICALLY. IS THERE A BETTER WAY TO SLICE UP. WE DID NEXT ANALYSIS BASED ON GRANT APPLICATIONS, WHAT WE FOUND IS EXISTING STUDY SECTION STRUCTURE IS MORE COHERENT FROM A TEXTURAL STANDPOINT THAN THE ONE WE TRIED TO COME UP WITH, AD HOC BASED ON GRANT LANGUAGE ITSELF. THAT IS A VERY INTERESTING YOU COME. WE WOULD HAVE THOUGHT OKAY, IF WE CLUSTER THE GRANTS WE'RE GOING TO END UP WITH SOMETHING A LITTLE BETTER THAN WHAT EXISTS, IT WASN'T TRUE. DUE TO THE HIGHLY MULTI-DIMENSIONAL NATURE OF SCIENCE YOU GUYS ARE IN CHARGE OF THERE ARE MANY WAYS TO SLICE IT UP. THERE'S TEN OR 12 WAYS TO SLICE IT UP EQUALLY AS VALID. THE IDEA IS TO GET TOOLS TO ALLOW YOU TO TAKE A MEASURE OF THAT AND MAKE INFORMED JUDGMENTS BASED ON WHATEVER CLASSIFICATIONS TO USE. BUT THE STUDY SECTION RIGHT NOW IS VERY GOOD. >> MAY I RESPOND BRIEFLY? >> I'M NOT A MAP GUY SO THAT'S NOT WHAT I'M THINKING ABOUT WHEN I THINK OF EFFICACY I THINK IN TERMS OF METRICS. MY MEASURES OF THEIR OWN AND THE EFFICACY OR THE UTILITY OF THAT PARTICULAR METRIC, I DO BELIEVE, I THINK I'M AN OPTIMIST RIGHT NOW. MY PERSONAL CONVICTION IS WILL WE BE ABLE TO ESTABLISH FOUR GIVEN USE CASES, METRICS AND MEASURES OF UTILITY AS FAR AS I'M CIGGING AND SCREAMING IN THIS AREA, I WILL CONTINUE TO MOVE TOWARD THAT. JUST BECAUSE OF MY OWN (INAUDIBLE). SO I'M MOVING THAT WAY. SO I THINK IT IS POSSIBLE. HOWEVER IT E NOT GOING TO COME WITHOUT CONCERTED EFFORT AND CONCERTED EFFORT CAN MANIFEST IN MANY DIFFERENT WAYS. I THINK YOU HAVE TO BE AN INTENT TO ARTICULATE THE USE CASES OR THE ANALYST WORK FLOWS, VERY CLEARLY, AN FIGURE OUT WHAT IS THE FACTORS OF AN EFFECTIVE TOOL THAT HELPS THAT. I DON'T THINK WE'LL COME TO THE -- THERE WILL NEVER BE THIS IS THE TOOL YOU USE, SHUT UP, DON'T TALK TO ME AGAIN, BY DEFINITION IT SHOULD BE VERY FLUID. BUT I THINK FACTORS WE WILL MOVE FORWARD. >> I THINK AS AN OPTIMIZATION SERVICE. THERE ARE LOCAL MAXIMUM, LOCAL MINIMUM, THAT'S PROBABLY AN ABSOLUTE MAXIMUM SOMEWHERE. HARD TO KNOW EXACTLY BUT YOU DEFINITELY WANT TO BE MOVING (INAUDIBLE) TRYING TO DO. IT MAYBE AN EASIER SITUATION FOR THE STUDY SECTION WHERE YOU HAVE A LIMITED NUMBER OF APPLICATIONS. IT'S A RELATIVELY SMALL NUMBER COMPARED TO THE ENTIRE DATA SET IN THE WORLD OR 5,000 APPLICATIONS PRINTED BY NIH SO THE HUMAN BRAIN CAN FIT ITSELF AROUND SMALLER NUMBERS PROBABLY BETTER THAN THEY CAN THE ENTIRE DATABASE. THAT MAYBE A PLACE ANALYTICAL TOOLS ARE SUITED WHERE THERE'S SO MUCH DATA THE HUMAN MIND CAN'T DEAL WITH IT. >> IT'S VALIDATING A GOOGLE, YOU SEE THE EARTH, THEN YOU TURN IT TO WHERE YOU LIVE. IF THE MAP DOESN'T SHOW THE NEIGHBORS ARE YOUR NEIGHBORS, THERE'S SOMETHING WRONG WITH THE MAP. THAT'S MOST PEOPLE I THINK INTERNALLY VALIDATED. THEY VALIDATED AT LOCAL LEVEL BUT IT IS USEFUL TO KNOW YOUR NEIGHBORSND WHO YOUR NEIGHBORS NEIGHBORS ARE BECAUSE THAT'S WHAT WE COGNITIVELY DO NOT KNOW, THAT'S WHERE ONE OF THE VALUES ARE IN IT. PUT THE PEOPLE AND CONCEPT I EXPECTED IN GOOD ORDER. EVERY EXPERT TENDS TO LOOK AT THAT FIRST. THEN THEY FEEL IT'S GOOD ENOUGH. THEY KNOW ENOUGH TO KNOW HOW IT'S SLIGHTLY OFF AND BUT GOOD MUST HAVE TO HAVE CONFIDENCE GOING ONE LEVEL OUT AND TWO LEVELS OUT. IT'S THAT KIND OF LOCAL VALIDATION AN INSIGHTS GOING ONE OR TWO LEVELS OUT. AND CHAOMEI WAS EM PA SIZING IT'S USED IN MANY CASES FOR THAT INSIGHT CREATION ON -- OF YOUR BOUNDARY, OF YOUR NEIGHBORHOOD THAT YOU ARE BLIND TO. >> JUST TO TORTURE YOUR ANALOGY FURTHER. THIS CONCEPT OF VALIDATION OF MAPS THOUGH MAPS IS MY MAIN WAY OF VIEWING THE WORLD, TRY TAKINGING A TRIP WITH THAT MAP. IF IT GETS YOU WHERE YOU NEED TO GO, THIS IS HOW MY WIFE CHOSE BETWEEN GOOGLE AND MAPQUEST, ONE OF THEM INFURIATED HER ENOUGH TIMES SHE DECIDED TO USE THE OTHER ONE AND IT DIDN'T MAKE HER AS MAD IT DIDN'T LIE TO HER. THESE KINDS OF MEASURES ARE EXTREMELY EFFECTIVE. >> OKAY. I THINK THAT'S A GOOD STOPPING POINT. THIS HAS BEEN GREAT. LET'S THANK THE PANEL. [APPLAUSE] >> OKAY, BEFORE WE START THE REPORT OUT OF THE BREAK OUT SESSIONS. I WANT TO REMIND EVERYONE THAT SINCE THIS QUESTION CAME UP ACTUALLY EARLIER THAT OUR OFFICE IS--IN ADDITION TO BUILDING TOOLS, AND TRYING TO MAKE THEM EASIER TO USE, PROVIDING TRAINING AND SUPPORT, SO THAT WE HAVE A TRAINING LAB, WE'LL SO IN A SECOND AND WE'RE DEVELOPING CASE STUDIES THAT WILL BE LOCATED ON OUR WEB REPOSITORY, YOU'LL HAVE TOOLS AND CASE STUDIES AND THE POINT OF THAT IS TO HELP DIRECT WHAT TOOL TO USE FOR WHAT PORTFOLIO ANALYSIS ACTIVITY AND WE'LL TRY TO DIRECT TRAFFIC THAT WAY BUT OF COURSE WE'RE AVAILABLE FOR A CONSULTATION AND WE CAN HELP WITH QUESTIONS ABOUT WHAT TOOL TO USE, HOW TO USE IT IN ADDITION TO THE TRAINING WHICH WE WILL START IN THE FALL WHICH IS A FEW MONTHS IN THE TOOLS LAB. SO I WANT TO MAKE THAT POINT. AND NOW WE CAN GO ON TO THE REPORTS. AND IF YOU WOULD JUST SAY WHICH BREAK OUT SESSION YOU ATTENDED AND THEN TAKE ABOUT 5 MINUTES OR SO„i TO DESCRIBE WHAT HAPPENED AND YOUR THOUGHTS ABOUT PUTTING IN IT IN BROADER CONTEXT AND NIH CONTEXT. KEITH, I'LL START WITH--GEETHA I'LL START WITH YOU AND SHE'S WITH THE PORTFOLIO ANALYSIS. >> [INDISCERNIBLE]--HE PRESENTED ABOUT MAPPING, LOCATING THE ACTIVITY USING SCIENCE OF MAPS AND ALSO TRACKING AND ASSESSING RESEARCH KNOWLEDGE TRANSFER AND FORECASTING PATHWAYS, MEASURING AND MAPPING INTERDISCIPLINARY RESEARCH USING DIFFERENT STRATEGICAL MEASURES WHICH HAVE INTEGRATION SCORES, SPECIALIZATION SCORES, [INDISCERNIBLE] SCORES AND HE USED THIS MEASURE TO EVALUATE SOME OF THE NSF FUNDED PROGRAMS, SUCH AS RESEARCH COORDINATION THAT WORKS WHICH IS SET UP TO IMPROVE COORDINATION IS COLLABORATION BETWEEN RESEARCH NETWORKS AND HE ASSESSED NETWORK ENRICHMENT USING SOME OF THESE MEASURES AND ALSO USING INTERDISCIPLINARY OF THE RESEARCH. AND HE USED TO CALCULATE THESE MEASURES USED FOR SOFTWARE WHICH HE DOES FOR SCIENCE TECHNOLOGY WHICH IS PART OF--IT'S THE SOFTWARE NAME IS CALLED VANTAGE POINT AND IT'S A SOFTWARE, IT HAS COMMERCIALIZE DEVELOPERS AND IT HAS DEVELOPERS FOR GOVERNMENT, FREE OF COST AND IT HAS--ESSENTIALLY THE SAME. THERE ARE A FEW ADDITIONAL FEATURES THAT ARE IN THE COMMERCIAL VERSION AND YOU CAN CREATE SEVERAL SCIENCE LABS AND MATRIXES AND IT WORKS WITH DIFFERENT KINDS OF DATA FORMATS, BOTH KINDS, EXCELLENCE AND PUB MED AND SEVERAL OTHER DATA FORMATS. AND WE CAN CLEAN THE DATA USING THE SOFTWARE, AUTODISINTEGRATION IS ALSO WHAT THIS SOFTWARE CAN DO FOR YOU. THIS IS ALL I HAVE TO SAY ABOUT IT. >> AND BY THE WAY IF THERE ARE QUESTIONS OR COMMENTS, THE MICROPHONES ARE STILL AVAILABLE. NEXT WE GO TO MIKE LAUER FROM NHLBI. >> THANK YOU. SO I WAS IN THE SIDE 2 TOOL BREAK OUT GROUP, A TOOL FOR SCIENCE RESEARCH AND PRACTICE AND DR. BURNER LEAD THIS. THIS IS AN EXCITING TOOL. WHAT IS KIND OF COOL IS SHE SHOWED HOW YOU CAN TAKE HUNDREDS OF ALGORITHMS AND BOILS THEM DOWN TO 4 STEPS AND BY BOILING IT DOWN TO 4 STEPS, IT MAKE ITS POSSIBLE FOR US TO GET A HANDLE ON THIS. THE 4 STEPS ARE 1 LOADS DATA, AND THIS TOOL, CAN ACCEPT A VARIETY OF DATA FORMATS, THEN 1 PREPROCESSES DATA, IT MEANS THAT 1 CLEANS THE DATA SO THAT IT IS ESSENTIALLY A USE BELIEVE AND THEN THE THIRD STEP THAT'S 1 ANALYZE DATA, WHATEVER YOU WANT TO DO AND THEN FINALLY YOU VISUALIZE IT. AND EACH OF THESE--IN THE COURSE OF ANALYZING THE VISUAL DATA, THERE ARE A VARIETY OF TYPES OF ANALYSIS, VISUALIZATION WHAT YOU CAN DO, 1 CAN DO TEMPORAL ANALYSIS, LOOKING AT LIKE WHAT'S HAPPENING OVER TIME, FOR EXAMPLE, SHE WALKED THROUGH HOW HE COULD TAKE A WHOLE BUNCH OF NSFs PROJECT AND YOU COULD MAKE A MAP WHERE YOU SEE WHERE THEY ARE OCCURRING OVER CALENDAR YEAR AND THEN THE PICTURE WAS DELAYED TO HOW BIG THE PROJECTS WERE, HOW MUCH MONEY THEY WERE USING, SO 1 COULD EASILY IDENTIFY WHAT THE BIG PROJECTS WERE AND WHEN THE MONEY WOULD SPEAK TO THAT, SHE SHOWED US YOU COULD DO GEOFACIAL ANALYSIS, WHERE YOU COULD SEE WHERE IN THE COUNTRY OR WHERE IN THE WORLD, ACTUALLY THIS IS TAKING PLACE, SHE SHOWED US A TOPICAL ANALYSIS, SO, EXAMPLE OF THAT WAS A VARIETY OF JOURNAL ARTICLES FOR A GROUP OF SCIENTISTS AND YOU COULD MAP OUT WHAT TOPICS THESE PEOPLE WERE WORKING IN AND THEN PERHAPS MEN WITH HIGHER WHAT I FOUND MORE INTERESTING WAS NETWORK ANALYSIS. THIS RELATES TO COLLEAGUES IN MY DIVISION, WHO HAVE USED TO TOOL AND HAVE DONE TEAM WORK. THEY DID WHETHER OR NOT A PARTICULAR PROEPBLG ECTOMYOSIN IN COHORT STUDIES WE'RE FUNDING, 1 OF THE GOALS OF THE COHORT STUDY IS TO DEVELOP NETWORKS AND DEVELOP COLLABORATIONS AND PREVIOUSLY UNINVOLVED SCIENCE AND THIS TOOL CAN SHOW YOU HOW THAT--HOW THESE NETWORKS WERE EVOLVING OVER TIME AND THEY SHOWED AN EXAMPLE IF YOU COULD SEE HOW NETWORK EVOLVED OVER TIME. FINALLY I JUST MENTION THAT ANOTHER--ANOTHER ITEM I FOUND INTERESTING WAS, YOU CAN USE THIS TO TRACE THE FLOW OF KNOWLEDGE. SO, LET'S SAY THAT SOME PAPERS COME OUT AND READ THROUGH TIME AND THESE PAPERS WE THINK ARE VERY IMPORTANT AND YOU WANT TO SEE WHERE DOES THE KNOWLEDGE COME FROM THAT'S ULTIMATELY FOR THOSE PAPERS SO WHAT YOU CAN DO IS YOU CAN TRACK BACK, YOU TRACK BACK THROUGH THE REFERENCES, REFERENCES, YOU CAN TAKE THE CURRENT PAPER WHICH GOES BACK TO PREVIOUS PAPERS, GOES BACK TO PREVIOUS PAPERS AND I COULDN'T IMAGINE THAT WE COULD USE SOMETHING LIKE THIS IF WE WANTED TO SEE, FOR EXAMPLE, WE MIGHT WANT TO DO A CASE CONTROL STUDY, AND WHAT THE IMPACT OF OF NIH FUNDED BASIC OR EARLY TRANSLATIONAL RESEARCH ON HIGH IMPACT CLINICAL PAPERS. SO WE COULD POTENTIALLY USE SOMETHING LIKE THIS, WHERE WE DRAW MAPS GOING BACK IN TIME, SO WE COULD SEE WHERE THE ROLE OF NIH SUPPORTED RESEARCH PLAYS, WHERE IT WAS IN DEVELOPING THE KNOWLEDGE, LET ME SUM UP BY SAYING THAT I LIKE THE WAY SHE BOILS IT DOWN TO THESE 4 STEPS, OF LOAD, PROCESS, ANALYZE AND VISUALIZE BECAUSE HONESTLY I FELT LIKE I WAS DRINKING WATER FROM A FIRE HOSE AND I THINK 1 OF THE POINTS THAT DR. GARNER MADE UP WHICH WAS A GREAT PIECE OF ADVICE, WHICH WAS SPEND AN HOUR A WEEK IMMERSING YOURSELF IN INTO THIS AND I THINK IT'S A NEAT IDEA. MAYBE ACROSS OUR DIVISION, WE WERE ALL TO SPEND AN HOUR A WEEK WORKING ON VARIOUS ASPECTS OF THIS, WE COULD DEVELOP AN ENORMOUS AMOUNT OF EXPERTISE AND IT WOULD POTENTIALLY, DRAMATICALLY ENHANCE THE FOCUS OF OUR DIVISION TO KNOW WHERE WE ARE AND WHERE WE'RE GOING. THANKS. NEXT IS YANCY BODENSTEIN FROM NIMH. >> I ATTENTED THE APPLICATION AND PATTERN FOR KINDS OF LITERATURE BY DR. CHEN. HE FINISHED UP HIS PRESENTATION THIS MORNING, SO HE DOVE INTO A LOT MORE OF THE DIFFERENT ANALYTICS YOU COULD DO WITH THE SOFTWARE PACKAGE. SOME OF THE MAIN POINTS THAT I TOOK AWAY FROM HIS DEMONSTRATION THIS AFTERNOON IS THAT IT WOULD BE POSSIBLE TO OVERLY--OVERLAY DATA ON TOP OF THE DATA, SUCH AS--SUCH AS A TERM CLOUDS, TERMS ALSO DEALING WITH A PATENT AS WELL. AND OTHER INTERESTING THINGS IS THAT THE SOFTWARE COULD DO IS BE ABLE TO MOVE BACK IN TIME AND FIGURE OUT WHERE THE PIVOTAL POINT IS IN A SCIENTIFIC TRAJECTORY, SO YOU COULD SEE IF YOU CAN OVERLAW THE GRANT--EVERYLAY THE GRANT INFORMATION ON THAT AND YOU FIND THAT DISCOVERY OR NOT, I THINK THAT WOULD BE SOMETHING WORTH WHILE, TOO, BECAUSE HE RECEIVED [INDISCERNIBLE] TO THOSE QUESTIONS TOO. SOME OF THE INTERESTING QUESTIONS WE HAVE, IS HOW DOES THE SOFTWARE PACKAGE HANDLE THE CONFIGURATION? AND DR. CHEN HAD SAID THAT YOU CAN USE THE DATA AS IT IS DROPPED FOR CLEAN [INDISCERNIBLE] AND YOU GET SLIGHTLY BETTER. SO THAT'S 1 COMMON THREAD AND THAT AS WELL. SO IF WE COULD SOMEHOW GET CLEAN DATA TO GO THIS THE TOOLS, THEN WE'LL GET A WHOLE BUNCH AND ANOTHER INTERESTING THING, 1 OF THE QUESTIONS WAS THE SOFTWARE OPERATE EASIER AND IDENTIFY TOOLS. SO THE THOUGHT CAME UP ABOUT [INDISCERNIBLE]. [LOW AUDIO„i ] START GOING OUT TO OTHER FIELDS. BUT I THINK FROM MY PERSPECTIVE AND BE ABLE TO DO A VERY INTERESTING TOOL, AND FIND IT AND PLAY WITH IT, AND THIS IS A PAPER THAT WE HAVE IDENTIFIED VERY LITTLE AND WHAT HAPPENS BEFORE IT. WHAT HAS HAPPENED. I'M REAL EXCITED ABOUT THAT [INDISCERNIBLE] >> OKAY, RICHARD NAKAMURA, CSR. >> I SAT IN ON THE SESSION THAT RICHARD CLAVINS, SESSION WAS TITLED: USING SCIENCE MAPPING TO SHAPE THE NIH RESEARCH PORTFOLIO. I ACTUALLY THINK HE WAS REALLY TALKING ABOUT THINKING ABOUT INNOVATION IN THE NIH RESEARCH PORTFOLIO. FIRST DISCUSS SOME THEORY OF INNOVATION, PARTICULARLY 3 DIFFERENT STRANDS WITHIN INDUSTRY AND ACADEMIA, TALK ABOUT A THEORY OF TECHNOLOGY DEVELOPMENT FORT FOLIO ANALYSIS DONE IN INDUSTRY LARGELY PRACTICES BY GENERAL ELECTRIC, AS STARTED BY GENERAL ELECTRIC AND THEN LOOKAD STUDIES THIS TIME WITH TECHNOLOGY LITERATURE AND I GUESS, GROUP INCLUDED THAT INNOVATION WAS A COMMON THEME AND ESSENTIALLY THE KEY TO DEVELOPMENT OF SCIENCE AND TECHNOLOGY AND SO, COULD THEY DEVELOP A TOOL WHICH WOULD HELP EVALUATE THE EVOLUTION OF SCIENCE AND THE INNOVATION IN SCIENCE? I GOT THE FEELING THIS IS SOMETHING THAT'S VERY RECENT AND STILL IN THE PROCESS OF BEING SHAPED. BUT WITH THE EXTENT THAT I UNDERSTOOD IT, IT WAS--THEY ESSENTIALLY TOOK THE LITERATURE OF SCIENCE AND LOOKED AT THAT, WORDS AND PHRASES THAT WERE USED CROSS PUBLICATIONS AND THEN CLUSTERED THESE INTO AREAS IN WHICH THERE WERE COMMON USE OF LANGUAGE AND OF RESEARCHERS OF REFERENCES, BY LOOKING AT THESE CLUSTERS AND HOW THEY CHANGED OVERTIME, THAT IS OVER YEARS YOU COULD DEVELOP ESTIMATES OF WHETHER OR NOT PUB LIAISONALATIONS WERE DEPENDING ON THE SAME SET OF IDEAS AND REFERENCES OVER TIME OR IF THESE WERE REALLY CHANGING IN HOW RAPIDLY THEY WERE CHANGING. IT SOUNDED LOOK A REALLY INTERESTING TOOL TO EXPLORE. BUT WE WERE TOLD THAT THIS WAS VERY EARLY IN DEVELOPMENT, YET, AND WITH SOME INTERESTS BY NIH, IT MIGHT BE EXPLORED FURTHER. I THINK THE PEOPLE WHO WERE SITTING IN ON THE SESSION FELT THAT THIS SOUNDED VERY INTERESTING. IT HAD BEEN PARTIALLY VALIDATED BY HAVING GROUPS OF SCIENTISTS COME IN, AND LOOK AT THE CLUSTERS THAT WERE IDENTIFIED AND THE SCIENTIST COULD ESSENTIALLY SAID VALIDATED THAT THOSE IN THEIR OWN AREAS, THEY FELT THESE WERE I DENTIFYING REAL CLUSTERS AND REAL IMPORTANT CLUSTERS IN SCIENCE AND THAT THEY COULD ALSO RELATIVELY READILY SAY WHETHER OR NOT THE CLUSTERS REPRESENTED HOT SCIENCE OR COLD SCIENCE. AND PRESUMABLY THERE WAS SOME CORRELATION BETWEEN HOT SCIENCE AND SCIENCE THAT THEY IDENTIFIED AS [INDISCERNIBLE]. I THINK WE ALL FELT THAT THIS WAS THE START OF AN INTERESTING NEW TOOL THAT REQUIRED MORE WORK AND HOPEFULLY SOME INTERACTION BETWEEN PROGRAMS AND REDUCED STAFF HERE WITH RICHARD CLAIBORNE TO DEVELOP THE [INDISCERNIBLE] >> GREAT, THANKS. OKAY, WELL, I THINK I'M GOING TO DO SOMETHING A LITTLE UNUSUAL HERE. I THINK SINCE WE HAVE A LOT OF ACCUMULATED EXPERIENCE AND DIFFERENT PERSPECTIVES ON THIS PANEL, MAYBE IN SUMMARIZING THE MEETING, IT WOULD BE GOOD TO GET EVERYBODY'S TAKE ON THE ENTIRE MEETING, IF YOU'RE WILLING. AND I'LL START. SO, I THINK THAT IT'S VERY DIFFICULT TO SUMMARIZE WHAT--ALL OF WHAT WE'VE HEARD OVER THE LAST 2 DAYS. THIS IS A REALLY EXCITING AREA OF SCIENCE AND THE SPEAKERS HAVE BEEN FANTASTIC. I THINK DOING EXTRAORDINARY WORK IN CHARACTERIZING WHAT WE ALL KNOW IS A VERY DIFFICULT THING TO CHARACTERIZE WHICH IS HOW SCIENCE EVOLVES, HOW IT PROGRESSES, WHAT WORKS, WHAT DOESN'T, HOW THAT RELATES TO INVESTMENTS, ALL THESE THINGS ARE VERY MUCH ON OUR MINDS AND ARE VERY MUCH UPON--IT'S VERY IMPORTANT TO HAVE THIS KIND OF SCIENCE HELP US TO UNDERSTAND WHAT THESE PATTERNS ARE, HOW THEY DEVELOP AND MOST IMPORTANTLY MAKE USE OF THEM PROPERTILY WHEN THE TOOLS METHODOLOGIES ARE MATURE ENOUGH THAT WE CAN DO THAT CONFIDENTLY AND KNOW THAT WE'RE FOLLOWING THE COURSE--OF COURSE THE AGE OLD: FIRST DO NO HARM WITH SCIENCE AND PRACTICES. SO SCIENCE IS DIFFICULT, IT'S A DIFFICULT THING TO MAKE PREDICTIONS ABOUT WHERE THINGS ARE GOING TO BEAR FRUIT AND THIS SCIENCE AS EXEMPLIFIED BY THESE WONDERFUL SPEAKERS IS HELPING US TO GO IN THAT DIRECTION AND GET OUR BEARINGS IN THAT AREA. AND EVEN THOUGH IT'S--IT'S PERFECT, IT'S NEVER EVEN IF THE SCIENCE THAT WE HEARD ABOUT OVER THE LAUGH 2 DAYS, REMAINS JUST APPROACHES PERFECTION, WHICH IS THE BEST WE CAN HOPE FOR BECAUSE IT IN ITSELF IS A SCIENCE THAT WILL NEVER BE PERFECTED, WE--WE CAN STILL USE THESE TOOLS AND THESE APPROACHES TO IMPROVE OUR PRACTICES IN MAKING DECISIONS, MAKING INVESTMENTS, AND I HOPE THAT YOU'VE ALL LEARNED AS MUCH AS I HAVE AND FOUND IT AS INTERESTING AS I HAVE BECAUSE I THINK THAT THAT--IF THAT'S TRUE AND WE CAN GET EXCITED ABOUT STAY NOTHING TOUCH WITH THE SCIENCE AND FINDING NEW WAYS TO APPLY IT AND USE THESE TOOLS THEN I DO THINK WE'LL BE HEADED IN THE RIGHT DIRECTION AND INJECTING MORE DATA DRIVEN APPROACHES AND DECISION MAKING. >> [INDISCERNIBLE]--NOISY DATA SO WE WOULD MAKE THAT 2 VANTAGE POINTS, NAVIGATION IS A BIG PROBLEM WHEN WE ARE LOOKING AT COST. PUBLICATION AND WE SPEND MANUEL HOURS CLEANING UP, KNOWING IT IS A TOOL THAT IT WAS ANNOUNCED THAT THIS KIND OF [INDISCERNIBLE] AND OTHERS ALSO, SO I CAN DO IT, BUT THERE'S A CLEANING, PART FROM HAVING 2 CLEANING THE DATA, IT'S AN IMPORTANT ISSUE. >> I WANT TO THANK AND YOU YOUR COLLEAGUES FOR PUTTING TOGETHER AND OUTSTANDING CONFERENCE AND YOU'VE CERTAINLY GIVEN ALL OF US A LOT TO THINK ABOUT AND THE DEBATE--THE DEPAEUT WE'VE HEARD HERE HAS BEEN EXTRAORDINARILY INTERESTING AND IN 1 RESPECT, THEY'RE DISCOMFORTING IN THE SENSE THAT IT'S PRETTY SCARY WHERE WE HAVE LOTS OF MONEY AT OUR DISPOSAL, ALTHOUGH IT WASN'T WHERE IT USED TO BE AND BIGGER OTHER BIGGER BUDGETS IT USED TO BE AND YET, IT'S DIFFICULT FOR--EACH WITH ALL THE KNOWLEDGE AND DATA THAT ARE AVAILABLE, NOT ENTIRELY CLEAR HOW WE THAN WE'RE MAKING THE BEST POSSIBLE DECISION, NOW THAT RESPECT IS DISCOMFORTING, LET ME END WITH A QUICK STORY HERE. SO SOME WE'RES AGO, I WAS ASKED TO BE A JUDGE FOR A COMPLICATION FOR A CLINICAL RESEARCH AND THE WINNING PROJECT DEALT WITH CLINICAL DECISION, QUESTION IS HERE, YOU'RE A DOCTOR AND HAVE YOU A PATIENT AND IF YOU'RE TRYING TO EXPLAIN THE PATIENT, HOW TO HELP THEM MAKE A VERY DIFFICULT DECISION, DO I--DO I GET THIS KIND OF A TEST, OR THAT KIND OF A TEST, DO I NEED AN EVASIVE PROCEDURE OR NOT, THIS PROCEDURE HAS A LOT OF SIDE EFFECTS, DO I UNDERGO SURGERY AND THE THAT TRICIANAL WAY IS WE TALKED AND EXPRESSED OUR OPINIONS AND OPINIONS AND BIASES AND THAT'S NOT VERY GOOD, SO THERE HAVE BEEN EFFORTS THAT HAVE BEEN MADE TO TRY AND STANDARDIZE DECISION MAKING TOOLS, WHICH ARE REALLY VERY SIMILAR TO THE KINDS OF THINGS YOU'RE LOOKING AT OVER THE LAST COUPLE OF DAYS EXCEPT THAT THESE ARE TOOLS THAT HELP DOCUMENTATION TALK TO EACH OTHER. AND THEY OFTEN INVOLVE GRAPHICS AND PRETTY PICTURES AND WHAT THIS PARTICULAR PROJECT WAS, IS A RANDOMIZED TRIAL THAT LOOKED AT THE USE OF A CERTAIN KIND OF DECISION MAKING TOOL, VERSES STANDARD APPROACH. IT SHOWED THAT PATIENTS MADE DIFFERENT DECISIONS THEY PROBABLY MADE MORE EFFICIENT DECISIONS AND BOTH PATIENTS OF DOCTORS WERE MORE SATISFIED WITH THE RESULTS AND THE OVERALL SAFETY RESULTS WHERE ABOUT THE SAME. EMPLOY THIS IS WHERE POTENTIALLY, THIS FIELD COULD GO. WE ARE ALSO MAKING DIFFICULT DECISIONS. WE HAVE TO MAKE--WE HAVE LIMITED RESOURCES AVAILABLE TO US, WE HAVE LOTS OF BIASED PEOPLE OUT THERE WHO HAVE STRONG FEELINGS ABOUT WHAT'S THE RIGHT WAY FOR US TO GO AND THESE KINDS OF TOOLS, SENTIALLY WILL MAKE IT POSSIBLE FOR US TO TALK TO EACH OTHER IN A MORE INFORMED WAY, JUST LIKE THE DOCTORS AND PATIENTS TALK TO EACH OTHER IN A MORE INFORMED WAY AND I HOPE THAT AS THEY MATURE, WE'LL GET TO THE POINT WHERE WE SUBJECT OURSELVES TO THE KINDS OF METHODS WHERE WE'LL DO RANDOMIZED TRIALS AT NIH AND WE WILL BE RANDOMIZED TRIAL OHMIZING OURSELVES IN OUR PROGRAMS WITH DIFFERENT APPROACHES TO ANSWER DIFFICULT PROBLEMS. WE ALL AGREE WE DO NOT KNOW THE BEST WAY TO ANSWER THE DIFFICULT PROBLEMS AND WHEN WE DON'T KNOW WHAT TO DO, THE THING TO DO IS DO A TRIAL AND UP UNTIL NOW, THE TOOLS WEREN'T IN A POSITION THAT WE COULD EVEN THINK ABOUT DOING THAT. WE'RE NOW IN A POSITION WHERE WE THINK ABOUT DO TAG AND MAYBE WHEN YOU HOLD THIS SYMPOSIUM AGAIN IN A FEW YEARS WE WILL PRESENT THE RESULTS OF THE EARLY TRIAL. >> NEEDLESS TO SAY, I LIKE THAT STORY YOU TOLD THERE. SO YANCY? >> WELL, AS„i A USER FOR FOOTBALL ANALYSIS, I HAVE TO SAY I'M TOTALLY THRILLED AND HONORED TO BE HERE. I THINK THE INFORMATION AND THE DIFFERENT TOOLS THAT ARE PRESENTED, IS LIKE A KID IN A CANDY STORE. I DON'T KNOW WHICH 1S ARE THE SOUR APPLES AND WHICH 1 ARE THE GOOD TREATS BUT 1 THING I NEED TO MENTION, 1 OF THE PROJECTS WE'RE WORKING ON IS LOOKING AT PUBLICATIONS, MENTAL HEALTH FIELD, USING MULTIPLE DATA SOURCES FOR THAT, NOW THAT I KNOW THAT THERE'S A TOOL SYSTEM IN ANY RATE IN THAT DATA, 10S OF OF THOUSANDS OF DATA, AND THE NEXT THEY THINK I THINK IT WOULD REALLY, REALLY NEED, IT WOULD EMERGE THE MAPS FOR CITATION OF PATENTS. THE TERM CLOUDS, USAGE DATA, AND ALSO THROW IN NIH GRANT [INDISCERNIBLE]. I THINK THAT SORT OF TOOL CAN BE CONSIDERED [INDISCERNIBLE] FIGURE OUT WHERE THE OVERLAPS ARE, HOPEFULLY [INDISCERNIBLE] IMEXTRAMURAL--I'M EXCITED. >> RICHARD YOU GET THE LAST WORD. >> THE GOLD STANDARD HAS BEEN USED FOR A LONG TIME IN DEVELOPING NIH PEER REVIEW. AND I THINK WE LONG FELT THAT HAVING OUTSTANDING TIMES TO EVALUATE STUDIES, APPLICATIONS IS THE WAY TO GO BECAUSE THE U.S. HAS BUILT THE STRONGEST SIGNS OF ESTABLISHMENT IN THE WORLD ON THAT BASIS. EVERYONE NEEDS TO SELF-EXAMINE THEMSELVES. YOU CAN'T REST IN OUR LAURELS ON A RANGING WORLD AND IT'S IMPORTANT THAT WE HAVE--THINK ABOUT WAYS THAT WE CAN IMPROVE THE OVERALL QUALITY OF WORK PORTFOLIO AND I THINK YOU'VE HEARD A LOT OF IDEAS TODAY ABOUT HOW THAT CAN BE DONE AND ALSO HAVE TO BE CAUTIOUS ABOUT BEING FORCED TO ADAPT OR TO ADAPT TOOLS THAT ARE NOT POORLY CONTESTED, I COMPLETELY AGREE WITH THE NOTION, STUDY THE CASE AND WE CAN IN FACT IMPROVE THE QUALITY OF OUR PORTFOLIO, USING SOME OF THESE TOOLS. I THINK WE SHOULD DO THAT CAUTIOUSLY IN ORDER TO MAKE SURE THAT WE DON'T HARM OUR SYSTEM AT THE SAME TIME WHEN THERE'S A WONDERFUL THINGS ABOUT NIH IS THAT WE HAVE MANY DIFFERENT INSTITUTES THAT MAKE THAT POSSIBLE TO EXPLORE DIFFERENT WAYS TO IMPROVE PORTFOLIOS, WITH MANY OF THOSE WE HAVE PROGRAM STAFF ALOS ANGELES AS TREMENDOUS [INDISCERNIBLE] STAFF AND I THINK WE HAVE LOTS OF OPPORTUNITIES TO LOOK FORWARD TO THESE AND LOOK FORWARD TO THEM AND WE'LL HAVE A BRIGHT FUTURE. >> GREAT, THANKS. FINALLY, ARE THERE THE HILL GROUP PEOPLE HERE? THEY DID THIS TO ME AT THE WORKSHOP. THEY SAID THEY WERE GOING TO COME IN. WELL, I WANTED TO THANK THEM. WE CAN-- >> [INDISCERNIBLE]-- >> SORRY? >> ANY REBUTTALS? >> WE CAN AT LEAST THANK THE NIH EVENTS MANAGEMENT FOLKS, IF YOU COME OUT HERE AND WAVE TO PEOPLE SO WE CAN THANK YOU FOR A GREAT JOB. THANK YOU VERY MUCH. [ APPLAUSE ] AND WE'LL THANK THE HILL GROUP QUIETLY, BETWEEN OUR EARS AND AND ALSO--HERE THEY COME. OKAY, GOOD. THAT'S YOU, RIGHT? THANK YOU. HILL GROUP! [ APPLAUSE ] CAROL ARE YOU HERE? OKAY EMPLOY. >> ALL RIGHT, WELL, WITH THAT THANK YOU VERY MUCH AND WE'LL SEE YOU NEXT TIME! [ APPLAUSE ]