>>WELCOME, I'M JEFF REZNICK
IT IS MY PRIVILEGE TO BE PART OF
THIS PROGRAM AND TO INTRODUCE
TODAY'S SPEAKER.
DR. MICHELE WEIGLE IS A
PROFESSOR IN THE DEPARTMENT OF
COMPUTER SCIENCE IN OLD DOMINION
UNIVERSITY.
DR. WIGGLING IS PUB REQUIRED
OVER 150 ARTICLES AND SHE HAS
SERVED AS PI OR CO-PI ON GRANTS.
INCLUDING THE NATIONAL SCIENCE
FOUNDATION, THE NATIONAL ENDOW
ME.
FOR THE HUMANITIES, THE
INSTITUTE OF MUSEUM AND LIBRARY
SERVICES AND THE MELON
FOUNDATION.
SHE CURRENTLY SERVES ON THE
EDITORIAL BOARDS OF THE JOURNAL
OF ASSOCIATION FOR INFORMATION
AND SCIENCE TECHNOLOGY.
I'M THRILLED THAT SHE CAN JOIN
US TODAY TO SPEAK ON WHAT I IS N
WEB ARCHIVE COLLECTION.
SUMMATION AND DISCOVERY OF
ARCHIVED WEB PAGES.
WELCOME.
>>THANK YOU, CHRISTIE.
I'M HONORED TO GIVE A TALK IN
THIS GREAT SERIES.
BEFORE I GET INTO THE MAIN TOPIC
I WANT TO GIVE A BIT OF CONTEXT
ABOUT OUR RESEARCH GROUP AND
WHAT WE STUDY.
WE'RE COMPUTER SCIENTISTS AND
INSTEAD OF USING THE CONTENT OF
THE WEB OR WEB ARCHIVES FOR
RESEARCH OUR MAIN FOCUS IS IN
STUDYING THE REPRESENTATION OF
THE CONTENT ON THE WEB AND HOW
WELL, THE WEB AS IT CHANGES OVER
TIME IS AND CAN BE ARCHIVED.
AND WE BUILT PROOF OF CONCEPT
TOOLS TO HELP FACILITATE THE USE
OF WEB ARCHIVES BY RESEARCHERS
IN OTHER FIELDS.
TODAY I WILL BE GIVING AN
INTRODUCTION AND WHY IT'S SO
IMPORTANT AND WHY SOMETIMES IT'S
SO HARD.
AND I WILL BE HIGHLIGHTING WORK
THAT OUR GROUP HAS DONE TO STUDY
AND ADDRESS SOME OF THESE
CHALLENGES.
HOW WE CAN MAKE WEB ARCHIVES
MORE ACCESSIBLE TO RESEARCHERS
WHO WANT TO STUDY THE CONTENT
AND HOW TO MAKE THAT CONTENT
MORE EASILY DISCOVERABLE.
THESE SLIDES CONTAIN LOTS OF
REFERENCES AND LINKS SO, I'VE
MADE THEM AVAILABLE FOR THOSE
WHO ARE INTERESTED IN FINDING
MORE INFORMATION.
THE SHORT LINK IS ON THE SLIDE
HERE AND WILL APPEAR AGAIN ON MY
LAST SLIDE.
SO OUR WEBSITE AND DIGITAL
LIBRARIES WS-DL AS WE CALL IT --
AND THE PARTICULAR WORKS AND
PROJECTS THAT I'LL MENTION TODAY
WERE DONE IN COLLABORATION WITH
MY COLLEAGUE MICHAEL NELSON AND
THE STUDENTS MANY OF WHOM ARE
NOW ALUMNI WHO ARE PI -- ARE PI.
WE WOULD NOT BE ABLE TO DO ANY
OF THIS WORK WITHOUT OUR
GRADUATE STUDENTS.
AND I'D ALSO LIKE TO ACKNOWLEDGE
AND THANK THE AGENCIES THAT HAVE
FUNDED MUCH OF THIS WORK.
SO I THINK IT'S APPARENT TO
EVERYONE NOW THAT THE WEB IS
ESSENTIAL TO OUR DAILY LIVES.
WHERE MANY US GO TO FIND OUT
WHAT IS HAPPENING IN THE WORLD.
AND LOOKING AT WEB FROM THE
VANTAGE IMPORTANT OF SPRING 2020
CAN HELP US REMEMBER WHAT IT
FELT LIKE IN THE EARLY DAYS OF
THE PANDEMIC.
SO MUCH HAS HAPPENED IN THE TWO
AND A HALF YEARS SINCE THEN.
AND SOMETIMES HARD TO REMEMBER
SOME OF THE FEELINGS OF ALARM,
ANXIETY, UNCERTAINTY AND EVEN
LOWLINESS AS WE HAD TO CURTAIL
OUR ACTIVITIES.
FOR ME AS A COLLEGE BASKETBALL
FAN THINGS REALLY HIT HOME.
THEY GOT REAL WHEN THE NCAA
CANCELED THE MARCH MADNESS
BASKETBALL TOURNAMENT.
AND AS YOU CAN SEE IN THE SCREEN
SHOT FROM MARCH 13, 2020 THE
BANNER AT ESPN.com HAS THIS
LONG LIST OF BASKETBALL GAMES
THAT HAD BEEN CANCELED.
AND THERE IS A SCREEN SHOT FROM
CNN .COM WITH A PICTURE FROM A
GROCERY STORE AND NOTHING QUITE
CONVEYS GENERAL PUBLIC PANIC
LIKE LONG LINES AND FULL GROCERY
CARTS.
BUT WEB PAGES THAT EVOKE THESE
FEELINGS CAN CHANGE QUICKLY AND
THIS IS ESPECIALLY TRUE FOR WEB
PAGES THAT ARE BUILT TO STAY
CURRENT LIKE THE MAIN PAGES OF
ESPN.com AND CNN.com.
SO THE REASON THAT WE NOW IN
LATE 2022 HAVE ACCESS TO THESE
WEB PAGES FROM SPRING 2020 IS
BECAUSE OF WEB ARCHIVES AND THE
IMPORTANCE OF CAPTURING FULL
CONTENT OF WEB PAGES INSTEAD OF
JUST CAPTURING SCREEN SHOTS IS
WE CAN GO BACK AND INTERACT.
WE CAN CLICK LINKS, SCROLL
AROUND.
THE JAVASCRIPT CAN EXECUTE AND
WE CAN GET A FULLER EXPERIENCE
OF WHAT THAT WEB PAGE WAS LIKE
THAN WITH JUST A STATIC IMAGE.
SO A FEW YEARS AGO I WROTE AN
ARTICLE ABOUT WEB ARCHIVING AND
SOME OF OUR GROUP OTHER WORK IN
THE AREA.
I WON'T BE DISCUSSING THAT
ARTICLE TODAY BUT IT WAS WRITTEN
FOR A GENERAL ACADEMIC AUDIENCE
SO I WANTED TO SHARE THE LINK
WITH YOU HERE TODAY.
SO, I'VE BEEN STUDYING WEB
ARCHIVES FOR ABOUT 10 YEARS NOW
AND WHEN I FIRST STARTED WE HAD
TO SELL THIS IDEA THAT PEOPLE
WOULD WANT TO LOOK AT "OLD WEB
PAGES."
RESEARCHERS AND SCHOLARS HAVE
GOTTEN THAT POINT NOW BUT I
WOULD LIKE TO INCLUDE THIS
QUOTE.
HE SAID YOU CANNOT STUDY THE
1990s WITHOUT WEB ARCHIVES.
I SHUDDER TO THINK THIS COULD BE
HISTORY.
BUT WHAT HE SAYS IS VERY TRUE.
YOU MAY BE FAMILIAR WITH THE
INTERNET ARCHIVE AND THEIR WAY
BACK MACHINE.
BUT THERE ARE SEVERAL OTHER
PUBLIC ARCHIVES.
SOME HAVE PARTICULAR GOALS FOR
WHAT THEY CAPTURE SUCH AS THE UK
WEB ARCHIVES AN ARE -- ARCHIVES.
LET'S LOOK AT HOW WEB PAGES GET
ARCHIVED.
FIRST WE NEED TO LOOK AT HOW WEB
PAGES ARE CONSTRUCTED.
MOST WEB PAGES ARE MADE UP OF
HTML SOURCE AND EMBEDDED
RESOURCES LIKE CSS STYLE SHEETS
AND JAVASCRIPTS.
EACH MUST BE DOWNLOADED
SEPARATELY.
THE CSS AND JAVASCRIPT ARE
DISPLAYED DIRECTLY ON THE PAGE
BUT THEY CAN MODIFY HOW THE
CONTENT APPEARS.
AND THE EXAMPLE I'VE SHOWN HERE
THE THE EMBEDDED RESOURCES SUCH
AS IMAGES AND OTHER THINGS
NEEDED TO BUILD THE PAGE ARE HOW
LIGHTED IN YELLOW AND LINKS TO
OTHER WEB PAGES ARE HIGHLIGHTED
IN GREEN.
AND BOTH OF THESE WILL BE
NECESSARY WHEN WE THINK ABOUT
ARCHIVING.
AND THIS IS A NECESSARILY
SIMPLIFIED VIEW OF WEB PAGES.
MOST ARE A COMPLEX MIX OF
RESOURCES THAT ARE DRIVEN
LARGELY BY JAVASCRIPT AND
RESOURCES REQUESTED AFTER YOUR
PAGE HAS BEEN RENDERED IN YOUR
WEB BROWSER.
SO WEB ARCHIVES EMPLOY WEB
CRAWLERS.
AND THE PROCESS IS STARTED WITH
A SEED LIST OF URLs THAT GO
INTO A CUE OR A FRONTIER.
AND THE CRAWLER GRABS A URL AND
DOWNLOADS THE RESOURCE FROM THE
WEB AND SAVES THE HTTP HEADERS
AND CONTENT IN THE WORK FILE
FORMAT.
AND IN THE CRAWLER CHECKS IT
LOOKS TO SEE IF THE RESOURCE CON
TWINS URL EMBEDDED RESOURCES.
AND IF SO THOSE ARE PLACED BACK
ON TO THE FRONTIER ON SO THEY
CAN BE DOWNLOADED AS WELL AND IF
THE CRAWLER IS CONFIGURED TO
CRAWL OUT LINKS MEANING THOSE
PAGES THAT ARE LINKED TO FROM
THE CURRENT PAGE THEN THE URL
FOR EACH WILL ALSO BE PLACED ON
TO THE FRONTIER.
CRAWLERS TYPICALLY HAVE A DEPTH
SETTING TO LIMIT THE NUMBER OF
SUCCESSIVE LINKS THAT WILL BE
FOLLOWED AND DOWNLOADED.
SO WARC IS THE STANDARD FORMAT
USED.
AND THE RESOURCES THAT A WEB
ARCHIVE CRAWLER DOWNLOADS ARE
STORED.
THE KEY IS THAT IN THE WARC FILE
THE CRAWLER RESTORES THE
REQUESTS AND RESPONSES INCLUDING
THE RESOURCE CONTENT FOR EVERY
RESOURCE THAT YOU NEED TO BUILD
A WEB PAGE AND THESE WERE THE
FILES THAT ARE ARE ACCESSED WHEN
IT'S TIME TO REPLAY.
SO WHEN IT IS REPLAYED ONLY THE
RESOURCES STORED IN WARC FILES
ARE USED AND NOTHING FROM THE
LIVE WEB IS ACCESSED.
THIS IS SHOWING THAT THE HTML
SOURCE CODE AND THE EMBEDDED
IMAGE ARE BOTH STORED IN THE
SAME WARC FILE.
MANY WEB ARCHIVES SET THEIR OWN
SEED LIST BASED ON THEIR
ORGANIZATION'S PRIORITIES.
OTHER ARCHIVES ALLOW USERS TO
SPEAK FEW ITEMS.
WHICH WE CALL ON DEMAND
ARCHIVING.
THIS IS THE INTERNET ARCHIVE
SUBSCRIPTION SERVICE FOR
DEVELOPING WEB ARCHIVE
COLLECTIONS.
IT'S LARGELY USED BY MUSEUMS AND
LIBRARIES AND GOVERNMENT
AGENCIES INCLUDING NNLM.
THE COLLECTING ORGANIZATION
CONTROLS WHAT GETS CRAWLED AND
WHO HAS ACCESS TO THE ARCHIVED
WEB PAGES WHICH WE CALL
MOMENTOS.
AND PUBLIC MOMENTOS ARE ALSO
ADDED TO THE INTERNET ARCHIVE
HOLDINGS.
IT HAS A SAVE PAGE NOW FEATURE
THAT ALLOWS ANY USER TO REQUEST
THAT THEIR WEB CRAWLER ARCHIVE
WEB PAGE.
THE MEMENTO WILL BE AVAILABLE IN
THE PUBLIC WAY BACK MACHINE AND
THE INTERNET ARCHIVE FREQUENTLY
HAS A MOTTO THAT SAYS IF YOU SEE
SOMETHING SAVE SOMETHING.
SO THEY ENCOURAGE THE USE OF
THIS FEATURE.
ARCHIVE.TODAY IS SIMILAR.
IN THAT ANYONE CAN SUBMIT A
REQUEST FOR A PAGE TO BE CRAWLED
AND THAT IS MADE AVAILABLE IN
THEIR PUBLIC ARCHIVE.
AND THEN CONFER WHICH I'LL TALK
ABOUT NEXT ALLOWS REGISTERED
USERS TO COLLECT WEB PAGES.
WHEN SPECIFIES THE URL TO
CAPTURE USERS CAN ALSO SPECIFY
WHICH WEB BROWSER THEY WANT TO
USE AND USERS GET FIVE GIGABYTES
OF FREE STORAGE AND THEY CONTROL
THE ACCESS TO THE MOMENTOS BUT
THERE IS NO PUBLIC SEARCH OR
LOOK UP MECHANISM AS THERE IS
WITH OTHER WEB ARCHIVES.
CONFER STARTED OUT AS THE WEB
RECORDER PROJECT AND SOME PEOPLE
NIGHT KNOW THE TOOL BY EITHER OF
THOSE NAMES AND THERE ARE SOME
UNIQUE THINGS ABOUT IT.
CONFER USES A WEB BROWSER BASED
CRAWLER.
ANY JAVASCRIPT ON THE PAGE CAN
BE EXECUTED BEFORE THE PAGE IS
ARCHIVED AND THIS IS IMPORTANT
BECAUSE JAVASCRIPT CAN CHANGE
THE LAYOUT OF THE PAGE AND CAN
ALSO REQUEST ADDITIONAL
RESOURCES THAT MIGHT BE MISSED
BY A TRADITIONAL WEB CRAWLER.
SO EVEN THOUGH THIS CAN MORE
FAITHFULLY CAPTURE DYNAMIC WEB
PAGES MOST CRAWLERS DON'T USE
THIS TECHNIQUE BECAUSE IT TAKES
MUCH LONGER.
THERE IS A TRADE-OFF BETWEEN
SPEED AND FIDELITY.
WITH CONFER THE ARCHIVING IS
DONE WHILE YOU VISIT THE PAGE.
YOU CAN INTERACT WITH THE PAGE
AND SOMETIMES THAT CAUSES MORE
RESOURCES TO BE LOADED WHICH
THEN THEY CAN BE ARCHIVED AS
WELL.
AND YOU CAN ALSO ARCHIVE PAGES
WHERE YOU WOULD NEED A LOG IN TO
ACCESS.
SO I WANT TO MENTION SOME OF OUR
PREVIOUS WORK RELATED TO THIS.
BACK IN 2012 WITH FUNDING FROM
THE NEH WE DEVELOPED A TOOL
CALLED WARK CREATE.
THAT PROVIDED A BROWSER BASE
ARCHIVING USING A BROWSER
EXTENSION AND THIS IS
ACKNOWLEDGED AS AN INSPIRATION
FOR SOME OF THESE LATER EFFORTS
SUCH AS WEB REPORTER AND CONFER.
SO NOW THAT WE'VE LOOKED BRIEFLY
AS HOW WEB PAGES ARE ARCHIVED
THE OTHER END IS HOW WE CAN
ACCESS THE WEB PAGES THAT HAVE
ALREADY BEEN ARCHIVED.
SO WEB ARCHIVES ARE NOT LIKE
GOOGLE.
YOU CANNOT OFTEN DO FREE TEXT TO
SEARCH.
MOST ARE INDEXED BY URL AND DATE
TIME.
THE TIME AND DATE THAT THE PAGE
WAS CAPTURED.
SO THE FIRST INPUT IS THE URL
AND FROM THERE YOU CAN USUALLY
GET A LIST OF DAYTIMES WHEN THAT
WEB PAGE WAS CAPTURED.
SO HERE I'M SHOWING THE QUERIES
FOR CAPTURES OR MOMENTOS OF A
HISTORY OF MEDICINE DIVISIONS
MAIN WEB PAGE FROM THE WAY BACK
MACHINE AND FROM ARCHIVE.TODAY.
SO THE WAY BACK MACHINE REPORTS
ALMOST 1500 MOMENTOS OF THE WEB
PAGE BETWEEN 1998 AND 2022.
ARCHIVE.TODD ONLY HAS TWO, ONE
FROM 2012 AND ONE FROM 2014.
REMEMBER THAT ARCHIVE.TODD IS AN
ON DEMAND ARCHIVING SERVICE SO
IT DOESN'T DO CRAWLS ON ITS OWN
SO IF YOU WOULD LIKE TO ADD A
MEMENTO TO THE ARCHIVE.TODD
ARCHIVE YOU CAN * SUBMIT A
REQUEST THAT IT BE CRAWLED.
SO BECAUSE THE WAY BACK MACHINE
OFTEN HAS THOUSANDS OF MOMENTOS
FOR A WEB PAGE THEY EMPLOY A
CALENDAR BASED VIEW FOR ACCESS.
SO FIRST YOU SELECT THE YEAR AND
IN THIS CASE I'VE SELECTED 1999
AND THEN ON THE CALENDAR THERE
IS A BUMBLE FOR EACH DAY THE
PAGE WAS ARCHIVED.
LARGER BUBBLES MEAN THERE WERE
MORE MOMENTOS CAPTURED ON THAT
DAY.
MARCH 2, 1999 I'M GOING TO DROP
DOWN AND SELECT A PARTICULAR
DAYTIME AND THESE TIMES ARE IN
24 HOUR FORMAT BASED ON GMT.
SO I CLICKED ON 10:07.
AND SO HERE IS WHAT THE HISTORY
OF MEDICINE DIVISION WEB PAGE
LOOKED LIKE IN MARCH 1999.
SO WHILE WE'RE HERE LET ME
UNPACK THIS URL AT THE BOTTOM.
THIS IS THE PREFIX FOR ALL
INTERNET ARCHIVE WAY BACK
MACHINE MOMENTOS.
THE NEXT 16 DIGITS REPRESENT THE
DAYTIME.
1999, 03 FOR MARCH AND 02 AND
THEN THE LAST SIX DIGITS ARE THE
TIME.
100723 AND AT THE END IS THE URL
OF THAT ORIGINAL RESOURCE.
SO IF YOU WANTED TO SHARE THIS
PARTICULAR MEMENTO THIS WHOLE
URL IS WHAT YOU WOULD USE TO
ACCESS THAT FROM THE WAY BACK
MACHINE.
BUT WHAT IF THE ARCHIVE YOU
TRIED DOESN'T HAVE THE WEB PAGE
AT THE TIME THAT YOU'RE
INTERESTED IN.
WHAT IF YOU WANTED A MEMENTO OF
THE HISTORY OF MEDICINE DIVISION
FROM 2020 AND YOU TRIED
ARCHIVE.TODD.
THEY ONLY HAVE 2012 AND 2014 SO
YOU CAN TRY THE QUERY AT ANOTHER
ARCHIVE.
BUT WITH MORE AND MORE WEB
ARCHIVES THIS PROCESS CAN GET
TEDIOUS SO TO HELP WE DEVELOPED
THE MEMENTO PROTOCOL.
THIS IS AN EXTENSION THAT AMONG
OTHER THINGS ALLOWS A CLIENT TO
OBTAIN A TIME MAP OR LIST OF ALL
MOMENTOS FROM ALL URLs.
AND THERE ARE SEVERAL BROWSER
EXTENSIONS BUILT ON TOP OF THE
PRO MOAN TOE PROTOCOL THAT YOU
CAN USE IT WITHOUT A BROWSER
EXTENSION.
SO I'M SHOWING YOU AN EXAMPLE
HERE FOR QUERYING FOR THE
HISTORY OF MEDICINE DIVISION
PAGE FROM JUNE 2020.
AND THE TOP THREE RESULTS COME
FROM THE PORTUGUESE ARCHIVE.
AND THEY TELL YOU EXACTLY HOW
DIFFERENT THE VERSION THAT THEY
HAVE IS FROM THE PAGE THAT YOU
REQUESTED.
SO OUR RESEARCH GROUP HAS BUILT
A MEMENTO AGGRAVATOR.
ALL OF THE MOMENTOS FOR THAT URL
AND THIS IS BUILT AS AN A PI.
THE EXAMPLE HERE SHOWS A QUERY
FOR ALL OF THE MOMENTOS FOR THE
HISTORIC OF MEDICINE DIVISION
PAGE.
SO I'LL MAKE THIS LARGER.
YOU CAN SEE IN THE SHORTENED
LIST THAT THERE ARE MOMENTOS
FROM BOTH THE INTERNET ARCHIVE,
WEB.ARCHIVE. ORG AND THE
LIBRARY OF CONGRESS.
AND YOU CAN ALSO SEE THAT THE
FORMAT OF THE LIBRARY OF
CONGRESS ARCHIVE IS SIMILAR TO
THAT OF THE INTERNET ARCHIVE.
NOW NOT ALL ARCHIVES USE THIS
FORMAT.
FOR THOSE THAT DO IT CAN PROVIDE
HELPFUL INFORMATION ON THE
MEMENTO THAT YOU ARE VIEWING.
ONE TOOL THAT WE HAVE TO HELP
USERS DISCOVER RELEVANT MOMENTOS
IS TO CREATE CURATED COLLECTIONS
BASED ON A TOPIC.
THEY CAN PROVIDE SEED URLs AND
PROVIDE ADDITION METADATA.
THE EXAMPLE I'M SHOWING HERE IS
NLMs GLOBAL HEALTH EVENTS WEB
ARCHIVE COLLECTION WHICH IS
HOSTED AT ARCHIVE.
THIS IS STARTED IN 2014 AND
CONTAINS OVER 19,000 SEED
URLs.
THE MEMBERS OF THE WEB
COLLECTING SPEND A LOT OF TIME
EVALUATING SEEDS TO BE ADDED TO
THIS COLLECTION.
ONE OF OUR GROUP'S PROJECTS
INVOLVES SEEING IF WE CAN MINE
QUALITY SEEDS FROM SOCIAL MEDIA
POST TO HELP BOOT STRAP SOME OF
THESE TASKS AND THIS PROJECT WAS
GREATLY HELPED WITH DISCUSSIONS
BY CHRISTIE MOFFETT.
WE LOOKED AT LINKS THAT WERE
POSTED IN SOCIAL MEDIA ABOUT
SPECIFIC TOPICS AND WE CONSIDER
THE DOMAIN OF THE LINK, THE
POPULARITY OF THE POST AUTHOR,
IF IT WAS A LOCAL MEDIA SOURCE
AND SEVERAL OTHER FACTORS.
ALTHOUGH THIS IS A PROJECT THAT
FOCUSED ON THE COLLECTION SIDE
MOST OF OUR OTHER RESEARCH
FOCUSES ON WHAT HAPPENS ONCE A
WEB PAGE HAS BEEN ARCHIVED.
AS I MENTIONED EARLIER ARCHIVE
IT IS A SERVICE USED BY MANY
ORGANIZATIONS.
THIS ALLOWS A BIT MORE FILTERING
AND BROWSING.
BUT COLLECTIONS CAN STILL GROW
QUITE LARGE SO BROWSING AT TIMES
CAN STILL BE DIFFICULT.
ONE THING THAT CAN MAKE LARGE
COLLECTIONS HARD TO NAVIGATE IS
THAT FOR EVERY SEED URL THERE
ARE ONE OR MORE MOMENTOS.
CURATORS CAN CHOOSE TO HAVE THE
SEEDS CRAWL AT CERTAIN
INTERVALS.
SO EVEN IF THE SEED LIST DOESN'T
GROW THE NUMBER OF MOMENTOS IN
THE COLLECTION CAN STILL GROW.
IN 2018 WE DID A STUDY OF THE
SIZE OF OUR COLLECTIONS IN THE
TERMS OF SEEDS AND MOMENTOS AND
HOW THEY GROW OVER TIME.
WE IDENTIFIED NINE DIFFERENT
GROWTH PATTERNS.
THE EXAMPLE HERE IS WHERE ALMOST
80% OF THE SEED URLs WERE
ADDED FAIRLY EARLY IN THE
COLLECTION SLIDE TIME AND A
CRAWLING STRATEGY THAT HAD
MOMENTOS BEING ADDED
CONTINUOUSLY.
THE RED LINE SO WHEN THERE IS A
HORIZONTAL GREEN LINE THAT MEANS
THAT NOTHING IS BEING ADDED TO
THE SEED LIST AND AT THE SAME
TIME WE SEE THE RED LINE
REPRESENTING THE NUMBER OF
MOMENTOS GROWING LINEARLY.
THOSE SEEDS WERE CONTINUING TO
BE CRAWLED AND NEW MOMENTOS
ADDED.
THE INTERNET ARCHIVES WAY BACK
MACHINE IS TOO LARGE TO OFFER
FULL TEXT SEARCH ON THE ENTIRE
ARCHIVE BUT SOME SMALLER
ARCHIVES INCLUDING ARCHIVE IT DO
OFFER FULL TEXT SEARCH.
SEARCHING THROUGH TIME CAN BE
TRICE.
WHAT IF THE SEARCH TERM APPEARS
IN ONLY SOME ITEMS.
HOW SHOULD THAT BE INDICATED?
IN THIS EXAMPLE I SEARCH FOR THE
PHRASE MASK REQUIRED.
ONE OF THE RESULTS WAS THE HOME
PAGE FOR THE NEW YORK STATE'S
COVID RESPONSE.
THE MEMENTO RETURNED IN THE
SEARCH RUNS IS FROM DECEMBER 23,
2021.
BUT LOOKING AT OTHER MOMENTOS
AROUND THAT DATE REVEALS THAT
THE PHRASE FIRST APPEARED IN THE
MOMENTOS OF THE WEB PAGE ON
DECEMBER 16 AND FURTHER ALTHOUGH
THE PHRASE NO LONGER APPEARS IN
THE HEADLINE ON DECEMBER 30 IT'S
STILL PRESENT LOWER ON THE WEB
PAGE SO WHICH IS THE MOST
APPROPRIATE DATE TIME TO RETURN.
ONE OF OUR MASTER STUDENTS IS
EXPLORING DIFFERENT WAYS TO
PRESENT SEARCH RESULTS FOR WEB
ARCHIVES INCLUDING ADDING THE
ABILITY TO SEARCH FOR ADDED OR
DELETED TERMS OR PHRASES AND
THIS WOULD ALLOW RESEARCHERS TO
SEARCH FOR WHEN PAGES HAVE
CHANGED.
SO I'M SHOWING A PROTOTYPE OF A
SEARCH INTERFACE THAT SHE
DEVELOPED THAT ANIMATES THE
SEARCH CHANGE.
WE SEARCH FOR THE ADDITION OF
THE PHRASE NONCLOTH AND THIS
EXAMPLE REFLECTS UPDATED
GUIDANCE.
ANOTHER WAY TO FILTER DOWN A
COLLECTION IS TO USE FACETS.
THESE ARE ALONG THE SIDE.
IN ARCHIVE IT THIS IS PROVIDED
IN THE PANEL ON THE LEFT HERE.
7 HOWEVER THE PROBLEM WITH
FACETS IS THEY DEPEND ON CURATED
DATA. WE FOUND THAT --
TYPICALLY THE MORE SEEDS A
COLLECTION HAS THE FEWER
METADATA FIELDS IT HAS AND THAT
IS WHAT WE FOUND IN THIS GRAPH
ON THE SIDE.
THE SEEDS ARE ON THE X AXIS AND
THE METADATA ON THE Y.
THIS MAKES SENSE BECAUSE IT CAN
BE A TIME-CONSUMING PROCESS.
HOWEVER I HAVE TO RECOGNIZE THE
WORKING GROUP.
BECAUSE HERE IS A COLLECTION
WITH THOUSANDS OF SEEDS THAT
ALSO HAS TONS OF GREAT METADATA
WHICH ALLOWS FOR DETAIL
FACETTING.
SO, I'VE FILTERED THE GLOBAL
HEALTH EVENTS COLLECTION TO
FOCUS ON THE COVID-19 OUTBREAK
LOOKING FOR ENGLISH LANGUAGE
ARMS THAT REFERENCES VULNERABLE
POPULATIONS' AND THIS BRINGS IT
DOWN TO JUST OVER A THOUSAND
RESULTS.
YOU CAN ADD MORE FACETS OR
PERFORM A FULL TEXT SEARCH ON
THIS FILTERED DATA.
WHAT I REFER TO AS THE COVID-19
COLLECTION IS THE CONTENTS OF
THIS GROUP LABELED CORONAVIRUS
DISEASE COVID-19 OUTBREAK.
I WANT TO POINT OUT A COUPLE OF
NICE ARTICLES.
IT'S ONE THING TO COLLECT A LOT
OF WEB PAGES BUT IT'S ANOTHER
EFFORT TO ENSURE THAT QUALITY
METADATA IS APPLIED TO THE SEEDS
SO THAT RESEARCHERS CAN FIND THE
INFORMATION THAT THEY NEED.
TO HIGHLIGHT THE IMPORTANCE
LET'S LOOK AT A COUPLE OF THE
SEEDS.
THE NORTH CAR -- THE NEW MEXICOF
HEALTH -- SHOWED A TOTAL OF 10
POSITIVE CASES.
BY SEPTEMBER 2020 THE THERE WERE
OVER 27,000.
ONE YEAR LATER IN SEPTEMBER 2021
ALMOST 250,000 POSITIVE CASES.
AND BY MARCH 15 TWO YEARS INTO
THE PANDEMIC NEW MEXICO HAD OVER
HALF A MILLION POSITIVE CASES.
NOW AS AN A SIDE THIS MEMENTO
SHOWS SOMETHING THAT YOU'LL FIND
IF YOU SPEND ENOUGH TIME
EXPLORING WEB ARCHIVES.
THERE WERE FOUR CSS STYLE FILES
THAT WERE USED TO BUILD THIS
PAGE THAT HAPPENED TO NOT BE
ARCHIVED SO THE REPLAY OF THE
PAGE IS AFFECTED.
THE INFORMATION IS STILL THERE
BUT IT DOES NOT LOOK LIKE WHAT
APPEARS ON THE LIVE WEB.
AND NOW TODAY NEW MEXICO'S
DEPARTMENT OF HEALTH COVID PAGE
HAS TRANSITIONED AWAY FROM
REPORTING POSITIVE CASE NUMBERS.
THEY ARE STILL HERE IN THIS
BANNER UP AT THE TOP BUT THEY
ARE REALLY SMALL.
BY LOOKING AT OTHER MOMENTOS
DURING THIS TIME PERIOD I FOUND
THAT THIS REDESIGN HAPPENED
BETWEEN MARCH AND MAY 2022.
THIS WEB PAGE WAS CAPTURED IN
NNLM's -- YOU WOULD HAVE TO GO
TO THE CALENDAR INTERFACE AND
SELECT THE YEAR AND HOVER ON THE
DATE.
AND FOR A LARGE SET OF MOMENTOS
THIS CAN GET TEDIOUS.
SO A FEW YEARS AGO WE HAD A
FUNDED PROJECT TO VISUALIZE HOW
INDIVIDUAL WEB PAGES CHANGE OVER
TIME.
IN THIS TOOL A USER COULD SUBMIT
A URL AND BE PROVIDED OF SCREEN
SHOTS OF SELECTED MOMENTOS
THROUGH TIME.
FOR SOME SEEDS THERE ARE VERY
LARGE NUMBER OF MOMENTOS TOO
MANY TO DISPLAY AT ONCE SO WE
USE THE AMOUNT OF DIFFERENCE IN
THE HTML SOURCE AS A PROXY FOR
LARGE CHANGES IN THE WEB PAGE.
AND THEN WE RENDERED THOSE
UNIQUE MOMENTOS AND TOOK SCREEN
SHOTS FOR THE DISPLAY.
WE EXPERIMENTED WITH A COUPLE OF
DIFFERENT DISPLAY TYPES AND I
FOUND THE GRID HERE TO BE USEFUL
FOR AIR LARGE SCALE OVERVIEW AND
THE ANIMATED GIF THAT CAN BE
INSERTED IN WEB PAGES OR EVEN IN
SLIDES AS I'VE DONE HERE.
SO HERE IS ANOTHER MEMENTO FROM
THE COVID COLLECTION.
THIS IS AN ARTICLE ABOUT VACCINE
HESITANCY IN AFRICAN-AMERICAN
COMMUNITIES FROM A LOCAL NEWS
OUTLET IN MICHIGAN.
PUBLISHED IN JULY 2020 BEFORE
VACCINES WERE AVAILABLE AND IT
PROVIDES US WITH A SNAPSHOT IN
TIME OF WHAT PEOPLE WERE
CONCERNED ABOUT.
AND THERE ARE A COUPLE THINGS
THAT I WANT TO NOTE ABOUT THIS
MEMENTO.
IT WAS PUBLISHED IN JULY BUT
ARCHIVE COLLECTION IN SEPTEMBER
SO CAPTURE DATE DOES NOT MEAN
PUBLISH DATE.
IT'S RECORDING THAT THE PAGE WAS
STILL LIVE IN SEPTEMBER.
SECOND, ONE NICE THING ABOUT
THIS ARTICLE THOUGH IT'S NOT
SHOWN IN MY SCREEN SHOT IS THAT
IT CONTAINS LINKS TO OUTSIDE
RESOURCES.
MANY OF WHICH ARE ALSO ARCHIVED
BY THE COLLECTION.
SO YOU CAN REVIEW SOURCES AS
THEY EXISTED WHEN THE CAPTURE
WAS MADE.
THIS PAGE IS STILL ON THE LIVE
WEB BUT WE ARE NOT GUARANTEED
THAT IT ALWAYS WILL BE.
THEY ALLOW US TO HOLD THIS
SNAPSHOT IN TIME IN CASE THE
LIVE WEB PAGE DISAPPEARS.
AND HERE IS AN EXAMPLE OF THAT.
THIS IS THE PAGE FROM THE SOUTH
CAROLINA DEPARTMENT OF EDUCATION
CAPTURED ON JANUARY 14, 2021.
AND THIS WAS THE SOUTH CAROLINA
SCHOOL REOPENING PLAN SUMMARY.
TODAY THAT URL DEDICATION TO
EDUCATION.com JUST REDIRECTS
TO THE SOUTH CAROLINA DEPARTMENT
OF EDUCATION'S COVID NEWSROOM.
SO, ITS ONLY BECAUSE OF THE WEB
ARCHIVE THAT WE KNOW WHAT THIS
PAGE LOOKED LIKE.
NOW THE NLM COVID COLLECTION
CONTAINS MANY CAPTURES OF SOCIAL
MEDIA DURING THE PANDEMIC.
I FOUND THESE BY SEARCHING FOR
THE TERM HASHTAG.
SOME OF THESE ARE WHAT YOU WOULD
EXPECT.
LIKE COVID-19.
MY COVID VACCINE AND WEAR A MASK
BUT OTHERS ARE TERMS THAT YOU
MIGHT ONLY KNOW BECAUSE YOU
LIVED THROUGH THIS TIME LIKE
FLATTEN THE CURVE.
I STAYED HOME FOR.
AND SONGS OF COMFORT.
SO THERE WAS THE I STAYED HOME
FOR A CHALLENGE WHICH BROUGHT
CELEBRITIES LIKE DOLLY PARTON.
AND IN MARCH THE SONGS OF
COMFORT HASHTAG.
PLAYING FROM HIS HOME.
BOTH OF THESE EXAMPLES ARE A
REMINDER OF THE GENERAL FEELING
IN MARCH AND APRIL OF 2020 OF
UNITY.
-- YOU WILL JUST SEE A STILL
IMAGE AND THE VIDEO DOESN'T
PLAY.
MEDIA LIKE VIDEOS ON SOCIAL
MEDIA ARE TRICE TO CAPTURE SO
SOMETIMES THE CRAWLER WON'T HAVE
CAPTURED EVERYTHING AND FOR
REPLAY FROM A COLLECTION IN
ARCHIVE IT THE PLAYBACK ENGINE
WILL ONLY REQUEST RESOURCE
THAT'S WERE CAPTURED INSIDE THAT
COLLECTION.
HOWEVER THEY ARE INCLUDED IN THE
INTERNET ARCHIVES WAY BACK
MACHINE AND REPLACE HAVE ACCESS
TO EMBEDDED RESOURCES THAT WERE
MAYBE PART OF OTHER CRAWLS.
7 SO IF YOU JUST CHANGE A
PORTION OF THE MEMENTO URL TO
LOAD THE SAME MEMENTO THROUGH
THE WAY BACK MACHINE SO YOU JUST
REPLACE WAY BACK.ARCHIVE IT.ORG
AND IN THIS INSTANCE 4887 WITH
WEB.ARCHIVE.WEB.
HERE THE VIDEO PLAYING THE SONG
WAS CAPTURED LIKELY BY A
DIFFERENT CRAWL FROM AROUND THE
SAME TIME.
AND SO SINCE THE WAY BACK
MACHINE HAS ACCESS TO USE THE
EMBEDDED RESOURCE IT CAN PLAY
THE VIDEO.
THE BROADER HASHTAGS THAT WE CAN
SEE MAY HAVE MORE MEANING OVER A
LONGER TIME PERIOD.
#COVID-19 IS STILL BEING USED
TODAY.
ONE IS FOR MARCH 4, 2020 AND THE
OTHER FROM MAY 3, 2022.
AND BECAUSE THESE ARE ARCHIVED
WEB PAGES WHEN YOU LOAD THEM YOU
CAN SCROLL THE PAGES AND SEE
MORE TWEETS.
SO ONE THING TO NOTE IS BETWEEN
THIS TIME Twitter CHANGED
THEIR USER INTERFACE AND IN JUNE
OF 2020 FORCED EVERYONE TO
MIGRATE TO THEIR NEW USER
INTERFACE WHICH LOOKED MORE LIKE
THEIR MOBILE APP AND THIS CAUSED
SOME ISSUES FOR WEB ARCHIVES.
IF YOU LOOK THROUGH MOMENTOS
FOR Twitter SEEDS IN THE WAY
BACK MACHINE YOU'LL FIND BETWEEN
JUNE 2020 AND SOME TIME IN 2021
YOU WOULD FIND A LOT OF THESE
SOMETHING WENT WRONG CAPTURES.
AND DURING THIS TIME WE HAPPENED
TO BE INVESTIGATING SOME
ARCHIVED TWEETS AND NOTICED THIS
ISSUE.
AND IN OUR BLOG POST WE DETAILED
SOME OF THE PROBLEMS THAT
VARIOUS WEB ARCHIVES HAD WITH
ARCHIVING THE NEW Twitter --
AND IDENTIFIED A WORK AROUND FOR
SOME WEB ARCHIVES AND THIS WAS
HAVING THE ARCHIVES WEB CRAWLERS
PRETEND TO BE GOOGLE'S CRAWLER.
BUT THIS POINTS TO SOME OF THE
DIFFICULTIES THAT WEB ARCHIVES
CAN ENCOUNTER WHEN TRYING TO
ARCHIVE SOCIAL MEDIA.
USER INTERFACE AND DELIVERY
CHANGES ARE MADE FOR THE BENEFIT
OF THE PLATFORM.
AND NOT FOR THE BENEFIT OF OR
EVEN WITHOUT REGARD FOR WEB
ARCHIVES.
AND THIS IS ISSUE HAS BECOME
EVEN MORE IMPORTANT WITH THE
TURBULENCE CURRENTLY HAPPENING
AT Twitter.
SO WE'VE LOOKED A LITTLE BIT AT
THE COVID COLLECTION BUT AS YOU
MIGHT SUSPECT THERE ARE MANY
OTHERS.
ARCHIVE IT RECENTLY PULLED
TOGETHER 154 COVID COLLECTIONS
INCLUDING THE GLOBAL HEALTH
EVENTS WEB ARCHIVE INTO ONE
PLACE AND THIS ALLOWS USERS TO
SEARCH ACROSS VARIOUS
COLLECTIONS AND ALTHOUGH THE
INTERFACE LOOKS DIFFERENT IT'S A
FRONT END TO THE SAME
COLLECTIONS.
SO WHAT IF A RESEARCHER WAS
INTERESTED IN STUDYING SOME
ASPECT OF COVID AND WANTED TO DO
FURTHER ANALYSIS.
WITH SO MANY DIFFERENT
COLLECTIONS ALL ABOUT COVID HOW
WOULD THEY DISCOVER WHAT TYPES
OF PAGES ARE INSIDE EACH TO KNOW
WHICH ONE THEY WANTED TO
INVESTIGATE FURTHER.
OUR APPROACH WAS TO PRODUCE A
SOCIAL MEDIA STYLE STORY TO GIVE
A HIGH LEVEL O -- OVERVIEW.
A STORY AS AN ARRANGED SAMPLE
FROM THAT COLLECTION.
WE WERE INSPIRED BY THE STORE
FEW SERVICE *.
AND IT ALLOWED USERS TO CREATE
AND CONSTRUCT STORIES BY
COMBINING TEXT AND IMAGES AND
PREVIEWS OF WEB PAGES AND SOCIAL
MEDIA OF POSTS.
BEFORE WE STARTED CREATING
STORIES OURSELVES WE ANALYZED
THE MOST POPULAR STORE FEW
STORIES TO FIND OUT WHAT MAKES A
GOOD STORY.
WE FOUND THAT POPULAR STORIES
HAD ABOUT * 28 ELEMENTS ON
AVERAGE AND WE USED THIS AS A
ROUGH BOUND ON THE STORIES THAT
WE GENERATE.
TO PROVIDE AN OVERVIEW OF A
LARGE COLLECTION LIKE THIS ONE
WITH OVER 23,000 MOMENTOS WE
DEVELOPED A SUITE OF TOOLS THAT
CAN TAKE AN AND KENTUCKY IT
COLLECTION AND PRODUCE A STORE
FIND STORY *.
METADATA ABOUT THE COLLECTION
AND A STRIKING IMAGE AND WE CALL
THIS THE DARK AND STORMY
ARCHIVES TOOLKIT WITH A NOD TO
IT WAS A DARK AND STORMY NIGHT.
THE GENERAL PROCESS THAT WE
DEVELOPED FOR STORYTELLING IS TO
SELECT MOMENTOS THAT CAN BE
CONSIDERED AND THEN GENERATE THE
STORY METADATA USING CONTENT
FROM THE COLLECTION AND THE
SELECTED MOMENTOS.
GENERATE METADATA AND THEN
VISUALIZE AND DISTRIBUTE THE
STORY.
WE SPLIT THESE FUNCTIONS INTO
THREE DIFFERENT TOOLS.
TO MAKE THE PROCESS MODULAR SO
THAT THE TOOLS COULD BE USED FOR
VARIOUS PURPOSES.
SO HERE IS A CLOSE-UP OF THE
GENERATED STORY THAT I SHOWED
EARLIER.
THIS WAS GENERATED IN APRIL 2020
USING THE II PCs COVID-19
COLLECTION.
WHICH ALREADY HAD OVER 10,000
SEEDS AND OVER 20,000 MOMENTOS.
SO WE USED HYPER CANE TO ANALYZE
AND IT SELECTED 36 MOMENTOS.
FROM THOSE HYPER CANE SHOWS A
STRIKING IMAGE AND THEN FROM THE
CONTENT IT COMPUTED THE MOST
FREQUENT ENTITIES AND THE MOST
FREQUENT PHRASES AND THEN WE
USED MEMENTO IN BED TO GENERATE
A SOCIAL CARD FOR EACH AND THEN
THE STORY WEB PAGE WAS BUILT
USING RAIN TAIL USING THE INPUTS
FROM THOSE PREVIOUS TOOLS.
SO IN SELECTING MOMENTOS FROM AN
ARCHIVE IT COLLECTION HYPER CAIN
RELIES ON THE LIBRARY THAT WE
DEVELOPED.
THIS IS A PYTHON LIBRARY THAT
CONTAINS THE METADATA AND SEED
LIST AND SEED LIST METADATA.
HYPER CANE PROVIDES A SET OF
OPERATORS * FOR SELECTING THE
EXEMPLARS FROM A COLLECTION.
* NOW THE SAMPLE OPERATOR
DIRECTLY PRODUCES A SAMPLE BASED
ON EXHIBITING ALGORITHMS BUT
THEY CAN BE COMPOSED USING OTHER
OPERATORS LIKE FILTER AND
CLUSTER.
AND IN ADDITION TO USING AN
ALGORITHMS FOR SAMPLING *
MOMENTOS THEY CAN TAKE A
FILTERED LIST AS INPUT AND THEN
PRODUCE OUTPUT SUITABLE FOR USE
IN RAIN TAIL AND IN THIS WAY THE
DSA TOOLKIT CAN BE USED TO
CREATE A HUMAN EXPERT CURATED
STORY.
SO DURING THIS PROCESS WHERE WE
WERE TRYING TO BUILD THESE
SUMMARIES OF COLLECTIONS WE
FOUND THAT THE STANDARD SOCIAL
CARD CREATORS LIKE WHAT TWITTER
DOES DON'T HANDLE MOMENTOS WELL.
SO WE BUILT THE TOOL TO MAKE
SURE THE ORIGINAL SOURCE AND THE
ARCHIVE WERE NOTED ON THE CARD.
MEMENTO IN BED TAKES THE URL AND
THEN GENERATES A CARD THAT CAN
BE EMBEDDED IN A WEB PAGE SO IT
EXTRACTS THE TITLE OF THE WEB
PAGE, A STRIKING IMAGE AND A
SNIPPET FROM THE MEMENTO.
AND IT INCLUDES INFORMATION
ABOUT THE ARCHIVE THAT IT CAME
FROM.
AND SPECIFIES THE ORIGINAL
SOURCE ALONG WITH THE DAYTIME
THAT THE MEMENTO WAS CAPTURED
AND IT INCLUDES OTHER LINKS TO
OTHER MOMENTOS OF THAT SAME WEB
PAGE.
AND ALSO A LINK TO THE LIVE AND
CAN ALSO INDICATE IF THE PAGE IS
NO LONGER AVAILABLE ON THE LIVE
WEB.
SO YOU CAN TRY THIS YOURSELF.
YOU PROVIDE A URL OF THE MEMENTO
OR AS WE CALL IT A URILM.
AND IT WILL GENERATE A SOCIAL
CARD THAT YOU CAN EMBED INTO A
WEB PAGE.
RAIN TALE WAS THE FINAL PIECE
FOR CREATING THIS STORY.
IT TAKES INPUT AND ALLOWS THE
USE OF A TEMPLATE TO ARRANGE THE
METADATA AND SOCIAL CARDS INTO A
STORY AND THESE ARE THREE
EXAMPLES OF DIFFERENT TYPES OF
STORIES THAT CAN BE CREATED.
YOU CAN PRODUCE AN HTML WEB
PAGE, OUTPUT AND MARK DOWN OR AS
A Twitter THREAD AND THE WEB
PAGE SHOWN HERE WAS GENERATED
WITH A TEMPLATE BUILT FOR RAIN
TALE BY THE NATIONAL LIBRARY OF
AUSTRALIA WHO WE WORKED WITH
LAST YEAR TO DEVELOP SOME WEB
PAGED USER INTERFACES FOR THIS
WEB CHAIN.
AS WE'VE BEEN DEVELOPING THE
TOOLS AND ADDING FEATURES WE
COLLECTED THE STORIES ON TO THE
DSA PUDDLES WEBSITE AND THE URL
IS DOWN HERE.
SO MOST OF THE STORIES YOU'LL
SEE ARE NEWS RELATED BECAUSE WE
HAVE A SEPARATE PROCESS THAT
GENERATES A STORY FOR THE
BIGGEST NEWS STORY EACH DAY AND
THIS IS BASED ON ANOTHER TOOL
AND IT MONITORS ARTICLES FROM A
RANGE OF NEWS SITES.
IF YOU SCROLL DOWN TO THE BOTTOM
OF THE DSA POST PAGE YOU CAN
SELECT THE CATEGORY OF STORIES
TO VIEW.
HERE ARE SOME OF THE STORIES
GENERATED BY ALGORITHMS AND THIS
INCLUDES STORIES GENERATED BASED
ON THE ARCHIVE IT COLLECTIONS.
HERE ARE THREE OF THE STORIES
THAT WE GENERATED FROM NLM WEB
ARCHIVE COLLECTIONS.
WE APPLIED THE DSA TOOLKIT TO
THE OPIOID ARCHIVE.
AND THE ENVIRONMENTAL HEALTH
COLLECTION AND EACH WAS
GENERATED WITH THE GOAL OF
SELECTING MOMENTOS ON DIFFERENT
TOPICS FROM A RANGE OF DAYTIMES
CROSS THE COLLECTION AND * AND
THE LINKS WILL TAKE YOU TO THE
FULL STORY.
AGAIN THE GOAL OF THE DSA
STORIES IS TO PROVIDE A QUICK
OVERVIEW OF THE CONTENTS OF A
COLLECTION SO THAT USERS CAN
BROWSE THE STORIES TO DETERMINE
WHICH COLLECTION THEY MIGHT WANT
TO EXPLORE FURTHER.
SO OUR DSA TOOLKIT IS ONE
APPROACH FOR COLLECTION OWNERS
TO HIGHLIGHT SOME OF WHAT IS
CONTAINED.
BUT IT IS NOT THE ONLY WAY.
THE COVID-19 ARCHIVE THAT I
SHOWED EARLIER WHICH COMBINES
THE 150 PLUS COVID COLLECTIONS
AT ARCHIVE IT HAS A FEATURED
COLLECTIONS AREA THAT GROUPS
RELATED COLLECTIONS TOGETHER FOR
EASIER DISCOVERY.
THE INTERNET ARCHIVE HAS BEEN
WORKING ON GATHERING TOPICAL
COLLECTIONS OF ITS HOLDINGS AS
WELL.
IN CONJUNCTION WITH THE RECENT
RELEASE OF DEMOCRACY'S LIBRARY
THEY PULLED TOGETHER THE
GOVERNMENT WEBSITES OF THE WORLD
WHERE THEY FILTERED AND GROUPED
THEIR HOLDINGS BY GOVERNMENT
COUNTRY CODE.
ALSO PROVIDING A KEY WORD SEARCH
FOR ABOUT A DOZEN COLLECTIONS
AND THIS IS AVAILABLE FROM THE
MAIN WAY BACK MACHINE WEB PAGE.
THESE COLLECTIONS INCLUDE PDFs
FROM.GOV WEBSITES.
PowerPoint PRESENTATIONS AND
TWO OF THE REGULAR PRESIDENTIAL
CRAWLS.
ALL ARE HELPING WITH DISCOVERY
BUT WE NEED MORE.
ONE TOOL OR ONE APPROACH WON'T
FIT EVERY SITUATION.
ANOTHER CHALLENGE IS THAT ONCE A
RESEARCHER IDENTIFIES A
COLLECTION HOW CAN THEY ACTUALLY
WORK WITH THE CONTENT?
SO THE ARCHIVES PROJECT HAS BEEN
DEVELOPING TOOLS FOR ANALYSIS OF
THE CONTENT FOR SEVERAL YEARS.
JUST TO NOTE THIS IS NOT OUR
GROUP'S WORK BUT I HAVE SUNDAY
OF ON THIS PROJECT'S ADVISORY
BOARD.
THE TOOLKIT IS A SET OF TOOLS
LITTLE BIT ON APACHE SPARK TO
GENERATE DERIVATIVES.
AND THIS CAN INCLUDE THINGS LIKE
CRAWL FREQUENCY, HOW MANY TIMES
PARTICULAR PAGES WERE CRAWLED.
LINK DIAGRAMS WHICH VISUALIZE
WHICH PAGES LINK TO OTHER PAGES
IN THE COLLECTION.
DOMAIN FREQUENCY.
WHAT ARE THE DOMAINS THAT ARE
CONTAINED IN THE COLLECTION.
NAMED ENTITY EXTRACTIONS AND
FULL TEXT EXTRACTIONS BUT IT
REQUIRES PROGRAMMING KNOWLEDGE
AND THAT THE COLLECTIONS WORK
FILES BE LOCAL TO WHERE THE
PROGRAM IS RUN SO THE USER MUST
HAVE ENOUGH STORAGE TO BE ABLE
TO DOWNLOAD THE COLLECTION.
HOWEVER IN CONJUNCTION WITH THE
INTERNET ARCHIVE THE ARCHIVES
UNLEASHED TEAM IS DEVELOPING
ARCH.
AND THIS WOULD ALLOW THE
GENERATION OF THESE DERIVATIVES
TO BE PERFORMED AT THE INTEREST
NIGHT ARCHIVE.
AND THEN USERS COULD THEN
DOWNLOAD THE MUCH SMALLER
DERIVATIVES FOR USE IN THEIR OWN
ANALYSIS TOOLS.
BOTH ARCHIVES UNLEASHED AND THE
PROJECT HAVE SEVERAL EXAMPLES
THAT CAN BE DONE WITH WEB
ARCHIVES.
WHAT PEOPLE CAN DO WITH THE
DERIVATIVES THAT THEY'VE
DOWNLOADED.
SO I WANTED TO INCLUDE A COUPLE
OF LINKS TO RECAPS AND VIDEOS OF
PRESENTATIONS THAT RESEARCHERS
HAVE MADE ABOUT BOTH OF THESE
PROJECTS AND THIS IS PART OF THE
INTERNET ARCHIVES LIBRARY AS
LABORATORY PROGRAM.
SO IF YOU'RE INTERESTED IN SOME
OF THE WAYS THAT RESEARCHERS
HAVE BEEN USING AND ANALYZING
DATA I WOULD ENCOURAGE YOU TO
CHECK THESE OUT.
SO THE TOOLS I JUST MENTIONED
ALLOW USERS TO WORK ON THE DATA
FROM A COLLECTION IN AGGREGATE.
BUT WHAT IF YOU WANTED TO
IDENTIFY FEATURES IN PARTICULAR
MOMENTOS OR FIND THOSE THAT HAVE
CERTAIN QUALITIES.
HOW CAN WE HELP USERS FIND
MOMENTOS WITH THIS SPECIFIC
CRITERIA WHEN THE EXACT DATE
TIME OR THE EXACT URL IS NOT
KNOWN.
THEY CLASSIFIED THE USER TASKS
OF JOURNALISTS.
THE MOST COMMON WAS TO FIND A
PAGE THAT HAD BEEN DELETED FROM
THE LIVE WEB.
AND ALTHOUGH THESE ARE DIFFERENT
THEY MAY BE SIMILAR TO THOSE
TASKS THAT RESEARCHERS WOULD
WANT TO PERFORM ONCE THEY HAD
IDENTIFIED A SUITABLE
COLLECTION.
SO WEB ARCHIVES STORES AN
IMMENSE AMOUNT OF DATA BUT THE
BARRIERS ARE SUCH THAT
RESEARCHERS AND SCHOLARS MAY NOT
KNOW WHAT IS AVAILABLE.
COLLECTIONS CAN HELP BUT FOR
LARGE COLLECTIONS EVEN THE
CURATORS MAY NOT KNOW EVERYTHING
THAT IS CONTAINED IN THEIR
COLLECTIONS.
7 SO WE NEED TOOLS THAT CAN HELP
PEOPLE NOT ONLY FIND WHAT THEY
ARE LOOKING FOR BUT TO BROWSE
WHAT IS AVAILABLE.
SOMETIMES YOU DON'T KNOW WHAT
YOU'RE LOOKING FOR OR EVEN HOW
TO ASK FOR WHAT YOU WANT UNTIL
YOU CAN SEE WHAT QUESTIONS MIGHT
BE POSSIBLE.
NOW, WE'VE COME A LONG WAY IN
HELPING RESEARCHERS ACCESS AND
USE WEB ARCHIVES BUT WE STILL
HAVE A LOT OF WORK AHEAD OF US
IN MAKING THESE TOOLS EASIER TO
USE AND HELPING RESEARCHERS WITH
DISCOVERY.
SO TO WRAP UP AND RECAP MY MAIN
POINTS.
WEB ARCHIVES ARE IMPORTANT TO
UNDERSTANDING THE RECENT PAST
AND COLLECTIONS ARE AN IMPORTANT
PART OF ALLOWING RESEARCHERS TO
DISCOVER ARCHIVED CONTENT.
OUR GROUP HAS BEEN STUDYING THE
PROBLEMS ASSOCIATED WITH ACCESS
TO WEB ARCHIVES AND CHALLENGES
TO ARCHIVING WEB PAGES FOR OVER
A DECADE AND WE HOPE SOME OF OUR
TOOLS CAN HELP PROVIDE
UNDERSTANDING OF LARGE WEB
ARCHIVE COLLECTIONS.
CHALLENGES AND DISCOVERY AND
RESEARCHER ACCESS REMAINS.
AND WE'RE CONTINUING TO THINK OF
WAYS THAT WE CAN SUPPORT
RESEARCHERS DESIRED TASKS AND TO
HELP PROMOTE THE USE.
AND FOR MORE INFORMATION ON OUR
GROUP'S WORK I'VE PUT OUR
GROUP'S WEB PAGE AND OUR
Twitter HANDLE IN THE FOOTER OF
THESE SLIDES AND I'VE INCLUDED
THE LINK TO OUR GitHub
ORGANIZATION.
SO THANK YOU FOR YOUR ATTENTION
TODAY.
>>DR. WEIGLE THANK YOU FOR THAT
GREAT TALK.
I'VE LEARNED A LOT AND I'M SURE
MANY WHO ARE WATCHING DID AS
WELL.
AND APPRECIATE WHAT YOU
PRESENTED AND THEIR
CONTRIBUTIONS TO ALL THAT YOU
PRESENTED AND MUCH MORE OF
COURSE.
WE HAVE TIME FOR Q&A.
SO I WOULD LIKE TO REMIND THOSE
WHO ARE WATCHING PLEASE USE THE
LIVE FEEDBACK BUTTON UNDERNEATH
THE VIDEO STREAM TO SEND
QUESTIONS TO DR. WEIGLE.
WE HAVE A COUPLE OF QUESTIONS TO
BEGIN.
THANK YOU FOR THIS INFORMATIVE
AND FASCINATING TALK.
YOU NOTED EARLY THAT IT WOULD BE
DIFFICULT TO STUDY THE HISTORY
OF THE 1990s WITHOUT AN
INTERNET ARCHIVE BUT WEB
ARCHIVING WAS MUCH MORE LIMITED
IN THE EARLY DAYS.
IF SOMEONE HAS THE EARLY CODING
IS IT STILL POSSIBLE TO ADD THAT
TO AN ARCHIVE LIKE ARCHIVE IT?
>>THAT IS A REALLY GOOD
QUESTION.
SO WE HAD ONE OF OUR FORMER
GRADUATES HAD, WAS WORKING AT
STANFORD AND HELPED THEM TO
DISCOVER THE VERY FIRST WEB PAGE
THAT WAS BUILT AT I THINK IT WAS
SLAC A STANFORD LABORATORY AND
THEY HAD JUST THAT.
THEY HAD THE COMPONENTS OF THE
WEB PAGE.
AND THEY WERE ABLE TO
ESSENTIALLY FORMULATE IT INTO
WARK FILES SO THEY COULD THEN
REPLAY THAT BACK.
IT CAN BE DONE.
BUT I DON'T KNOW NECESSARILY
THAT THERE IS A TOOL BUILT RIGHT
NOW THAT WOULD ALLOW TO YOU
SUBMIT ALL OF THE FILES THAT
WOULD BUILD A WEB PAGE AND MAKE
IT AVAILABLE.
BUT -- THE FOLKS AT INTERNET
ARCHIVE MAY BE ABLE TO HELP WITH
THAT.
>>THANK YOU.
ANOTHER QUESTION, KNOWING AND
DOING WHAT YOU AND YOUR RESEARCH
TEAM YOUR COLLEAGUES ARE DOING
NOW WHAT IS ON THE HORIZON FOR
WEB ARCHIVING?
WHERE ARE INNOVATIONS HAPPENING
TO KEEP UP WITH COLLECTING THIS
DIGITAL CONTENT?
>>THAT IS A GREAT QUESTION.
AVENUE ONE OF THE THINGS THAT WE
HAVE TO DO IS WE HAVE TO BE
AWARE OF THE CHANGING TECHNOLOGY
ON THE WEB ITSELF.
BECAUSE YES, WE'RE ALWAYS
CHASING WHAT ARE THE NEW THINGS
THAT ARE BEING DEVELOPED.
BECAUSE AGAIN THE PLATFORMS AND
THE TECHNOLOGIES USED TO BUILD
WEB PAGES ARE NOT BUILT IN MIND
WITH WEB ARCHIVING IN MIND.
THEY ARE NOT MAKING IT EASY FOR
THESE THINGS TO BE ARCHIVED
NECESSARILY.
AND IT'S NOT THAT THEY DON'T
WANT THINGS TO BE ARCHIVED BUT
THAT IS NOT THEIR MAIN GOAL.
THEIR MAIN GOAL IS SPEED AND
DELIVERY.
AND SO WE HAVE TO CONTINUE TO BE
UP ON HOW WEB PAGES ARE BEING
BUILT TO BE ABLE TO REACT TO
CHANGES IN THE WEB.
OTHER THINGS THAT -- SOME OF THE
THINGS THAT I MENTIONED HERE --
THERE IS THE ISSUE OF THE
CAPTURE.
WHICH IS WE HAVE TO KEEP UP WITH
THE TECHNOLOGY.
ALSO THE ISSUE OF WE HAVE SO
MUCH STUFF ALREADY.
HOW CAN WE HELP PEOPLE DISCOVER
AND USE IT.
SO I MEAN BOTH OF THOSE HAVE
LOTS OF CHALLENGES.
>>THANK YOU.
SO WE HAVE ANOTHER QUESTION
ABOUT REALLY ABOUT PRESERVATION.
THIS PERSON WRITES IN, "SINCE
THE INTERNET ARCHIVE IS A
PRIVATE BASED SYSTEM IT EXISTS
ON WHAT PEOPLE CALL SOFT MOONEY
AND COULD BE SEEN AS VULNERABLE
TO FUNDING SHORTAGES THAT COULD
MAKE IT DISAPPEAR.
WHAT ARE THE CONCERNS ABOUT THIS
POSSIBILITY AND HOW ARE THEY
MITIGATED?
>>HONESTLY I DON'T HAVE A GOOD
ANSWER FOR THAT.
THOSE ARE REAL CONCERNS.
AND I KNOW THAT TYPICALLY
LIBRARIES ARE UNDER FUNDED.
AND THESE ARCHIVING EFFORTS THAT
ARE TAKING PLACE IT TAKES A LOT
OF MONEY TO RUN THESE SERVERS
FOR THE STORAGE AND FOR ALSO THE
ACCESS FOR THE PUBLIC TO BE ABLE
TO REPLAY THESE WEB PAGES.
SO I MEAN IT WOULD BE GREAT IF
THERE WERE A LARGE INFUSION OF
MONEY INTO THESE EFFORTS.
BUT I DON'T KNOW THE ANSWER TO
THAT.
BUT I THINK IT'S A REAL PROBLEM.
>>THANK YOU.
SOMEONE WRITES IN THANKS
DR. WEIGLE FOR THIS GREAT TALK.
THIS WAS AN EXCELLENT OVERVIEW.
I WOULD LIKE TO REVIEW THE
SLIDES AGAIN ESPECIALLY THE
VALUABLE LINKS THAT YOU PROVIDED
AND WILL NOTE THE URL IS AT THE
BOTTOM OF YOUR CONCLUDING SLIDE
AND OF COURSE IN THE TWEET THAT
YOU KINDLY SENT.
SO THANK YOU VERY MUCH.
>>YOU'RE WELCOMED
>>CAN YOU COMMENT -- TO WHAT
EXTENT ARE THESE GROUP WORKING
TOGETHER TO NOT ONLY PRESERVE
HISTORY BUT TO WRITE ABOUT IT?
>>THAT IS A GOOD QUESTION.
SO I WOULD HAVE TO DEFER TO MY
COLLEAGUES WHO ARE HISTORIANS
LIKE DR. IAN MILLIGAN.
WE MET HIM SEVERAL YEARS AGO AND
HE WAS ON THIS FOREFRONT AS A
HISTORIAN STUDYING AND USING WEB
ARCHIVES IN HIS RESEARCH IN
WRITING ABOUT HISTORY.
AND SO I WOULD DEFER TO SOMEONE
LIKE HIM FOR THAT QUESTION.
AND HE IS ACTUALLY WRITTEN AND
BEEN AN EDITOR ON A COUPLE OF
VERY NICE BOOKS.
I BELIEVE THE WEB IS HISTORY IS
ONE OF THEM.
BUT THERE IS MORE AND MORE BEING
WRITTEN ABOUT IN USING WEB
ARCHIVES FOR HISTORICAL
RESEARCH.
78
>>THANK YOU.
*.
ANOTHER QUESTION IS WHEN DID IT
OCCUR TO PEOPLE TO BEGIN TO
ARCHIVE THE WORLDWIDE WEB?
>>SO THE INTERNET ARCHIVE
ITSELF WAS STARTED IN 1996.
SO IT WAS PRETTY EARLY IN KIND
OF THE HISTORY OF THE WEB.
8 AND BREWSTER KEEL STARTED THE
INTERNET ARCHIVE AND * HE WANTED
TO BUILD A NEW ALEXANDRIA
LIBRARY.
CAN YOU CONTAIN ALL OF THE
WORLD'S KNOWLEDGE?
SO PART OF THAT AND THE INTERNET
ARCHIVE CONTAINS MANY THINGS
OTHER THAN JUST WEB ARCHIVES.
THEY HAVE MOVIES AND BOOKS AND
RECORDINGS.
BUT THE WEB WAS A LARGE PART OF
THAT.
SO HE HAD THAT VISION EARLY ON
THAT THIS WAS GOING TO BE AN
IMPORTANT RESOURCE FOR US.
AND WE'RE VERY GLAD THAT HE DID
THAT.
>>CERTAINLY.
THANK YOU.
CAN YOU SAY MORE ABOUT HOW THE
COVID PANDEMIC CHANGED THINKING
IN PRACTICE AROUND WEB
ARCHIVING.
ARE THERE ASPECTS THAT YOU THINK
WOULD NOT BE DONE NOW IF IT
WASN'T FOR THE PANDEMIC
EXPERIENCE OF THE PANDEMIC?
>>I HAVEN'T THOUGHT ABOUT IT
THAT WAY BUT I THINK THERE ARE
DEFINITELY SOME OF THINGS THAT
EXPERIENCES THAT MAYBE WERE
MISSING BECAUSE SOME OF THE,
MANY OF THE THINGS THAT WE DO
NOW JUST LIKE WHAT WE'RE
CURRENTLY DOING ARE THINGS THAT
WE DIDN'T DO VERY OFTEN BEFORE
COVID IN TERMS OF THIS VIDEO
CAST AND THE ZOOM MEETINGS THAT
WE'VE HAD.
AND IT FEELS LIKE THIS IS NOT
NECESSARILY WEB ARCHIVING BUT
OUR COLLECTIVE EXPERIENCE
ARCHIVING AND HOW ARE PEOPLE
CAPTURING THINGS LIKE ZOOM
MEETINGS THAT HISTORIANS MAY
WANT TO GO BACK AND TALK ABOUT
THIS.
BECAUSE WE DEFINITELY ADDED SOME
NEW TERMINOLOGY TO OUR LEXICON
THAT WE'RE ALL FAMILIAR WITH
NOW.
>>THANK YOU.
SO WE HAVE TIME FOR ONE MORE
QUESTION, DR. WEIGLE.
IF I WANT TO DO WEB ARCHIVING
AND IN SIMILAR WAYS TO WHAT YOU
PRESENTED TODAY WHAT KIND OF
TRAINING DO I NEED?
WHERE DO I NEED TO GO TO SCHOOL?
WHAT KIND OF BACKGROUND DO I
NEED?
>>COME TO ODU.
I'LL MAKE A PITCH FOR US.
WE -- THERE ARE IN BOTH LIBRARY
SCHOOLS AND COMPUTER SCIENCE
SCHOOLS, WE COMBINE A LITTLE BIT
OF LIBRARY SCIENCE AND COMPUTER
SCIENCE IN WHAT WE DO.
WE HAVE TO BE AWARE OF TOPICS IN
BOTH AREAS.
AND HONESTLY AS A PITCHER WITH'S
DEVELOPING A WEB ARCHIVING
CERTIFICATE JOINT WITH OUR
LIBRARY PROGRAM HERE AT ODU.
SO WE'RE ACTIVELY THINKING ABOUT
IN TERMS OF WHAT DO WE NEED TO
GIVE TO STUDENTS TO GIVE THEM
THE BACKGROUND IN BOTH OF THOSE
ASPECTS SO THAT THEY ARE AWARE
OF ISSUES IN TERMS OF METADATA
IN COLLECTING AND ACCESS AND
THEN ALSO HAVE THE COMPUTER
SCIENCE SIDE TO BE ABLE TO
UNDERSTAND HOW THE PAGES ARE
ACTUALLY ARCHIVED AND REPLAYED.
>>THANK YOU.
AND CERTAINLY ALL OF THE BEST
WITH THE CERTIFICATE PROGRAM.
AND WE CERTAINLY WISH THIS
PERSON WHO WROTE THIS QUESTION
ALL OF THE BEST IN SEEKING A
CAREER IN WEB ARCHIVING AND
ARCHIVAL SCIENCE AND COMPUTER
SCIENCE AS WELL.
WITH THAT -- I WANT TO THANK YOU
VERY MUCH FOR OUR ENGAGEMENT FOR
THE TIME THAT YOU TOOK TO
PREPARE YOUR PRESENTATION.
AND WE LOOK FORWARD TO KEEPING
IN TOUCH AND WE REALLY
APPRECIATE YOU TALKING ABOUT OUR
WEB ARCHIVING COLLECTION
ALONGSIDE SO MANY OTHERS THAT
HAVE BEEN DEVELOPED AND SO WELL
CURATED BY YOU AND YOUR
COLLEAGUES AND OF COURSE OUR
COLLEAGUES HERE AT THE NATIONAL
LIBRARY OF MEDICINE.
I WANT TO THANK CHRISTIE MOFF
MOFFETT.
THANK YOU VERY MUCH.
WE WISH YOU AND YOUR COLLEAGUES
ALL THE BEST.
THANK YOU.