MAKING THE BLACK HOLE GRAY: WEB ARCHIVING AT THE NEW YORK ART RESOURCES CONSORTIUM (NYARC)

Dr. Stephen Bury,
Andrew W. Mellon Chief Librarian,
Frick Art Reference Library

To address the potential loss of important art historical materials contained in websites, the New York Art Resources Consortium (NYARC) was awarded at the end of 2013 a two-year grant by the Andrew W. Mellon Foundation. Our intention is to implement a specialized and sustainable web archiving program. This covers the whole life cycle of web archiving, including the selection, harvesting, resource discovery (including the selection and implementation of a discovery layer) and preservation of art-rich websites.
This is a report 8-months into our program.
This program is based on previous pilot projects on online auction houses and on linking to already archived websites in the Internet Archive where the live site had been linked to in the shared NYARC catalog, ARCADE as well as a previous feasibility study, funded by the Andrew W. Mellon Foundation in 2012. Walter Schlect will report on a survey and analysis of the websites selected for archival capture. I am grateful to the consultants and interns who have worked on the building blocks of our program.
Web archiving is probably one of the most expensive activities in collection development and management. For example, a website may be of insufficient quality at one moment but at another deserving of selection (and vice versa).

Why are we even doing it? Isn’t the Internet Archive sufficient?
In the print world the partners of NYARC – the libraries of Brooklyn Museum, the Museum of Modern Art and the Frick Art Reference Library – have very specialized collections – a 2008 study by OCLC Research revealed that 83% of the total NYARC collective collection was held by just one library. (SLIDE 2) A further OCLC Research investigation of so-called ‘unique items’ in 2013 disclosed that 27% of the Frick’s collection was not held by another library in WorldCat. It is this specialization that we want to continue into the born-digital era.
Some national libraries e.g. Bibliotheque national de France or the British Library have Legal Deposit for websites in their domains but these are only accessible in Reading Rooms and under strict conditions. They do not necessarily capture all the layers of a website and the timing of the harvesting of an individual site may not capture what we are looking for. They also do not usually capture password-protected sites or those protected by robots.txt.
The Internet Archive has these limitations of timing and depth. It also difficult to locate content: the only way to search is via the URL  (SLIDE 3)
So for a variety of reasons – access (via the Web and not via the Reading Room), depth and timing of harvesting and not least discoverability – we think it is necessary to do our own web archiving of specialist art history materials. But where there is a good enough web-accessible copy elsewhere we will make do with that.
We are using the Internet Archive’s Archive-It service supplemented by an on-demand service, Hanzo Archives for more technically challenging content – we are their first non-profit customer. They are working together to ensure that Hanzo-captured websites will be properly rendered from the Archive-It repository. (SLIDE 4)
Target websites include selected auction houses, catalogues raisonnés, artists’ websites, websites selected for or already catalogued as live websites in the shared catalog ARCADE, as well as the websites of the three partner institutions, Brooklyn Museum, Frick Art Reference Library and the Museum of Modern Art. We are already thinking that this scope is too wide (in terms of storage space alone) and that we should focus the ARCADE catalogue candidates on NYC galleries and exhibitions and provenance sites. This argues that we need a collaborative network of libraries.

We also had always thought that catalogues raisonné may be a special case, and that educating their publishers on the importance of archiving and long-term access might work: this is what we have done with Artifex Press. But we have permission to archive the Noguchi catalogue raisonné site. (SLIDE 5).
We concluded from the legal advice from the pilot study that the permissions route was the way we forward. But of course this is not necessarily simple: who has the authority/right to give permission, and most sites have third-party content too. Initially we developed an online form which would feed directly into our permissions database, but we decided to be informal so as not to frighten off the website owner. We also decided to build out from our own sites, then organizations that we had built good relations with.
Our priority is to give web access to the archived instances and we will only dark-archive as the very last resort.
And we want to make sure that what we archive is discoverable, making sure we can give access through our catalog, Arcade, WorldCat/Art Discovery. Part of our program is the adoption of a discovery layer, which will pull from our Archive-It files at the Internet Archive. This has proven problematic for some discovery layer vendors/suppliers but we have persuaded Primo (Ex Libris), a hybrid central index and API approach, (SLIDE 6, 7) and ExplorIt (Deep Web Technologies), a federated search approach (SLIDE 8). A late candidate is AquaBrowser but this is at a very early stage in its development.
Finally, we will preserve copies of the websites in DuraCloud, who have been working with Archive-It already on an automatic backup.

What we are trying to do is create a program that will be a model for others interested in archiving  art-rich websites. NYARC cannot do everything.

It is my pleasure now to hand over to Walter Schlect who will describe the “webarchivability” project he was involved in.

Walter Schlect,
Pratt Institute Library & Information Science student and NYARC intern

I worked on the web archiving project at the Frick Art Reference Library in the Fall of 2013 and the Spring of 2014 under a IMLS-funded partnership between my school, Pratt Institute, and NYARC (SLIDE 9).
 In each semester my fellow intern, Janet Burka, and I analyzed art-related websites as possible “seeds” to include in NYARC’s future web archive. In web archiving jargon a “seed” is a URL that is given to a web archiving program (typically called a “web crawler”) so that it can capture all content contained within that particular website.  As Stephen has pointed out, a single institution cannot collect and curate every art-related resource on the web.  Selection of seeds is thus a critical step in building a meaningful web archive.

But what does one decide to select (SLIDE 10)?  
The more one thinks about it, the more agonizing the decision becomes.  With digital materials, scholarly or research value is not the only factor in selection; a wide array of technical considerations comes in to play.  
Web Archiving software creates WARC files, which are captures of website content rendered in a form that replicates the appearance and functionality of the original website at a particular period of time.  Yet the constant development of new web technologies makes it difficult for web archiving software to keep up with the times, and more dynamic content is often difficult or nearly impossible to capture (SLIDE 11).  
Furthermore, websites often lack the metadata for a crawler to easily capture it, and sometimes they even contain “traps” that confuse crawlers or robots.txt files that block them (SLIDE 12).
 Crawls of such websites can look unrecognizable (SLIDE 13).

In the Fall, Janet and I started to go through a backlog of websites that librarians at the Frick had collected to be cataloged and included in ARCADE. This turned out to be a ready made list of seeds for our future web archive This list was augmented by selections made by librarians at MoMA and Brooklyn Museum libraries.  There were 72 seeds in all. We first weeded through the list to exclude what was out of scope of the web archiving project, such websites that were merely lists of links to other websites, websites with limited information, and websites that required paid subscription.  We also looked to the other web archives when possible to avoid potential duplication. Nine websites were thrown out for being out of scope

The remaining seeds were then compared the live versions of these sites with their captures in the Wayback Machine (SLIDE 14).  The Wayback Machine is the portal to the Internet Archive, and automated crawler of web content which casts a very wide net across the web.  Nearly all of the potential seeds had some kind of capture in Wayback.  

Though the Wayback machine is automated and and does not curate and perform QA in the way that NYARC does with Archive-It software, it provides a good impression of potential crawl problems of the seed.  Archive-It itself is the software developed by the Internet Archive to crawl websites, so the quality of captures in the Wayback machine is very informative.  In a spreadsheet we tracked the number of crawls in the Wayback Machine, the wayback URL, any use of dynamic content that is difficult to archive (e.g. Flash), and notes about the quality of capture in Wayback.  Comparing the websites was an iterative process and each site presented different concerns.  
 With the project coordinator, Sumitra Duncan, we developed a ranking system to quantify the “webarchivability” (i.e. how well the structure and content  of a site lends itself to being crawled by Archive-it) and the value of the resource to NYARC’s collection (Slide 15).

Typical impediments to “webarchivability” are outlined in this chart (SLIDE 16).

 Unsurprisingly, the most frequent problem in crawls was a lack of depth.  Because the Wayback Machine is automated, there is no control over the scope of a crawl, and often the crawl is too shallow.  The next most frequent barrier to successful captures was dynamic content like Flash. Only seven websites had problems with image capture, which is not bad considering that all of the potential seeds were image-heavy resources.  Very few of the sites had video, but those that had video seemed to be difficult to capture.  Other typical problems included the CSS not capturing accurately and messing up the layout of the page, robots.txt files blocking web crawlers, and cms-based websites capturing poorly
Rating the “webarchivability” of a seed is easier said than done.  Though we found the Wayback machine a useful tool, it also gives an incomplete picture of “webarchivability”.
 A lack of depth may not show that a website is imposingly large or has crawl traps, but manually adjusting the scope of a crawl can alleviate some of those problems.  With proper permissions, robots.txt files can be circumvented.

 Dynamic content is not impossible to capture, and NYARC’s captures of MoMA’s websites have demonstrated that Archive-it is capable of capturing certain dynamic content. But the Wayback machine does give a sense of the relative ease or difficulty of archiving a site, and the potential pitfalls that a site has in store.
In spring Sumitra introduced a tool that was helpful to visually organize potential seeds called Dropmark (Slide 16).  We were able to add the rankings that we applied to everything, so that selectors could have a view of all websites at a glance in a visually more engaging way than a cluttered spreadsheet.  The rankings give a sense of what to prioritize and what will likely be an uncomplicated seed.
 
Before our internships ended, Janet and I started applying the same selection process to sites already cataloged in ARCADE. It is a daunting task but it will help a clean up catalog (and prevent user frustration about link rot) and unearth more possible seeds. Selection may theoretically be first part of digital preservation, but the process is cyclical not linear. New seeds will continually be added as old seeds will be continually re-evaluated.
Near the end of my internships, other NYARC interns and I toured the conservation labs of the Frick Art Reference Library. (SLIDE 17)
We learned there that conservation of ephemeral materials, whether print or born-digital, is integral to the library’s mission, and a permanent part of the library’s workflow.  I am certain that this dedication will continue in the future at NYARC, and I share with Stephen the hope that other libraries will follow this model and implement their own web archiving programs.