About Archive-It

Created in 2006 by the Internet Archive, Archive-It is a subscription web archiving service that has long been used by individuals and organizations practicing web archiving around the world. As of 2014, the service was a partner to “over 400 . . . organizations in forty-eight . . . states and sixteen countries worldwide” made up of “college and university libraries; state archives, libraries, and historical societies; federal institutions and NGOs; museums and art libraries; and public libraries, cities, and counties.”² Archive-It differs from the Wayback Machine, a public archives of websites run and maintained by Internet Archive staff, in that Archive-It allows subscribing institutions to curate collections of websites relevant to their institutions and/or aligned with their collection development practices. Websites that are captured and preserved exist as another form of archival material, equal in importance to traditional formats. They are considered part of each institution's archival holdings and may be used in much the same way as analog or digitized archival collections.

Websites are captured (or “crawled”) with software designed by the Internet Archive, then saved on Archive-It servers. Archive-It offers a back-end interface for a subscribing institution's archival or library staff to perform data management and metadata creation. Completed crawls can be made available and accessed through an end-user interface, as Figure 1 shows. Our study addresses only this end-user interface.

View Figure

• Download as Powerpoint
• Download Figure

Through the end-user interface, users can explore collections of websites, or “seed” URLs, maintained by the Archive-It collection institution. Users can browse and search collections, and use facets provided to filter the displayed results based on a certain field's contents, as seen in Figure 2.

View Figure

• Download as Powerpoint
• Download Figure

Once a user selects a specific seed, he or she is able to see how many times and on which dates that website was captured by Archive-It software, as shown in Figure 3.

View Figure

• Download as Powerpoint
• Download Figure

After selecting a date, users are shown the archived website view, made clear by an informational banner, as Figure 4 shows.

View Figure

• Download as Powerpoint
• Download Figure

Literature Review

In 2017, more than a hundred individuals participated in the National Digital Stewardship Alliance (NDSA) survey “Web Archiving in the United States.” The survey, also administered in 2011, 2013, and 2016, was developed to “better understand the landscape of web archiving in the United States by investigating the organizations involved, the tools and services being used, access and discovery services being provided, and overall policies related to web archiving programs.”³ “Web Archiving in the United States” not only interrogates the field of web archiving as it stands, but acts as a barometer and “tracks the evolution of the field over the last few years, and points to future opportunities and developments.”⁴ The survey asked questions like “How much full-time equivalent staff time does your organization dedicate to web archiving?” and “What types of content do you have concerns about your capacity to archive?”

Most of the results from the 2017 survey changed little from previous years. One area, though, saw an increase: Archive-It use. As the authors write, “94% . . . of institutions . . . capture content with Archive-It” and “Archive-It has been the favored external service provider since NDSA began recording these statistics in 2011, when 72% of responding institutions were using it.”⁵

Archive-It's high use makes sense, as the subscription-based web archiving service “helps organizations to harvest, build, and manage born digital collections” and “collect, catalog, and manage their collections of archived content” with round-the-clock access and full-text search capabilities without organizations having to devote significant internal resources to harvesting, preserving, and providing access to web archives.⁶ An Archive-It subscription also includes the ability to create and manage metadata, and, though the platform has no required metadata fields, it does support the use of fifteen standard Dublin Core elements and the ability to create custom metadata fields, all of which are “displayed and hyperlinked for browsing on public collection pages hosted at Archive-It.org.”⁷ In the end, it may be Archive-It's ability to “provide the viewer a wealth of information”⁸ that makes the service so appealing to archivists. As researchers at Old Dominion University note, “Archive-It collections, in particular, are interesting because a single organization is responsible for each collection, meaning that the curation strategy for a collection is guided by humans rather than automated crawling operations.”⁹ As professionals, we can infer that Archive-It's popularity has much to do with the ability to carry out web archiving using a platform that handles much of the technical work, while allowing archivists to provide some intuitive level of access to researchers.

And yet, despite its status as the most utilized web archiving service, professionals in the field know little about how researchers use and interact with collections in Archive-It. In fact, the field knows very little about how researchers use and interact with any web archives access tools. This contrasts with the more substantial literature on the usability of other types of archival discovery systems. Beginning in 2003, with Elizabeth Yakel's study of an Encoded Archival Description (EAD) interface, archivists have published numerous studies about how users interact with online finding aid discovery systems. Research includes studying social interaction in finding aids, how search functionality affects discoverability, and the general usability of several different finding aid sites, either individually or comparatively.¹⁰ The usability of web archive discovery systems has been explored by archivists at national libraries with their own custom discoverability systems, and several explorations of search functionality, context, and researcher needs have been conducted. This research will be summarized in the ensuing pages. However, to date, no study exists of how users interact with Archive-It, even though it is by far the most widely used web archiving discovery platform.

As Helen Hockx-Yu stated in 2014: “Web archiving institutions have traditionally focused their attention on developing the processes and technology for data collection” while the “use of web archives has not been a key element of the strategy.”¹¹ Others agree. As David Cruz and Daniel Gomes found in 2013, researchers and “most web users are not acquainted with web archives and accessing . . . archived web data is a significantly different user experience than accessing the live web.” They also report that “although there [are] a significant number of web archives available, only preliminary research has been done about the design of user interfaces to gain access to temporal web data.”¹² Additionally, as is mentioned in “Researcher Engagement with Web Archives: Challenges and Opportunities for Investment”: “We know some of what researchers want from web archives and we are just now beginning to know what web archives can provide, but web archives of use to researchers have not been available long enough yet to develop a comprehensive understanding of how researchers are engaging with web archives.”¹³

What do we know, then? The study conducted by Cruz and Gomes revealed that users of web archives like having the ability to search—as long as a bit of guidance accompanies that ability. “Searching historical web content was an awkward concept for most users,” they found, and “the existence of a website that provides access to pages that are no longer available on their original websites is a perception that requires technical knowledge about the functioning of the Internet that is far beyond the skills of common web users.”¹⁴ Cruz and Gomes made these observations when looking at the Portuguese Web Archive and noted that an explanation of, and context surrounding, the archive was necessary. They point out that, “unlike most live-web search engines, our web search homepage also includes a fat footer. The objective was to provide additional links to information that clarifies users about the context of web archiving and web archive search.”¹⁵ In fact, for users of the Portuguese Web Archive to be successful, Cruz and Gomes found that “search interfaces for web archives must be similar to live-web search engines to facilitate the adoption of web archives by new users” alongside ample context, including a “homepage for a web archive search service” that presents “contextual information such as examples of archived pages, that enable new users to gain awareness about what is, and the potential of searching, a web archive.”¹⁶

This request for context is a popular one. In “Web Archives for Researchers, Representations, Expectations, and Potential Uses,” the authors write that “Regarding services and promotion, the decisions and criteria that are used in the selection policy must be justified, documented, and made visible.”¹⁷ The article also indicates that “as web archiving is a new concept to most researchers, to improve communication and promotion it is necessary to explore different metaphors that will allow different user groups to represent and imagine the web archives.”¹⁸ Still, for all that we know about what researchers might want in a web archives, what do we know about how they use what functionality already exists?

As mentioned in “Researcher Engagement with Web Archives: State of the Art,” too little is known about users' behaviors in relation to web archives. This article was written in 2010, and since then, few articles have directly added to the literature on how researchers use web archives. This dearth of research means that

Most archiving institutions therefore rely on semi-hypothetical use cases to refine and expand their usability and interfaces. One particularly detailed study was conducted at the National Library of the Netherlands. This structured experiment, run similarly to a task-oriented usability study, evaluated user comfort level with search and access tools and attempted to determine user satisfaction with archive contents.¹⁹

The National Library of the Netherlands launched its web archiving program in 2006, the same year Archive-It debuted. The library's collections are both domain based (collecting much of the .nl domain) and thematic in nature, and consist of collections and websites with searching that can be “carried out by URL (via the Internet Archive's Wayback Machine) and by full-text (NutchWax).”²⁰ In 2007, the National Library of the Netherlands conducted a user survey of its web archive to form a “definitive list of user preferences, on the basis of which existing search functionality and interface can be further developed for a web archive.”²¹ During the survey, “A number of persons . . . were invited to conduct search operations in a test version of the National Library of the Netherlands web archive on the basis of concrete operations” during which it was “deliberately decided to let the test person use a full-text search option as well as a URL-based search.”²² Following the test, “users were asked about their experience of the Archive and about their preferences with regard to the selection of websites, access to the archive, and search functionality.”²³ The feedback the group received was scattered, with “a few participants” indicating that their preference would be for a full-text search option and two participants who said “they missed an advanced search function.”²⁴

In recent years, a handful of studies have looked at how historians and scholars interact with archived web pages in a variety of formats, though—to date—the field has yet to study how researchers interact specifically with Archive-It. In 2017, Susanne Belovari, assistant professor at University Archives, University of Illinois, Urbana-Champaign, carried out a thought experiment: in the year 2050, with access to only web collections from 2015, how might a historian fare if attempting a web-based history? Belovari's imagined historian ultimately fails due to problems encountered on numerous web archiving platforms, many of which have poor usability design. While using the Wayback Machine, for instance, Belovari's imaginary user is unable to execute a keyword search, and, when using smaller collections in Archive-It, is hampered by a lack of metadata. The frustrations are fictionalized, yet speak to key questions of usability. Belovari concludes that a lack of “basic search tools and possibilities with which to scope out and identify topics, individuals, institutions, and sources”²⁵ make web research nearly impossible; furthermore, “good metadata and authority control are critical in order to find particular” web pages.²⁶ The challenges Belovari highlights do not just belong to a distant 2050 future—researchers today already attempting to use web archives encounter these same problems. Belovari's imaginary 2050 study casts light on the very real need for improvements today.

In a similar study carried out in 2007, Marilyn E. Rackley asked participants working at the North Carolina State Archives and State Library to locate specific records in either their “State Library Catalog” or the “State Government Web Site Archives Portal and the Wayback Machine.”²⁷ Ultimately, Rackley's study was about context in web collections—specifically, “how easy it was to locate contextual information about the records”²⁸—but her findings, like Belovari's, shroud the task in the question of usability. In her tests, Rackley found that, often, archived websites “did not . . . contextualize [records] well enough for . . . researchers to quickly understand the record” and that “perhaps if they had more time to spend on the site as a whole, they would have demonstrated a more complete understanding” of the materials.²⁹ Providing context to help users better understand archived web pages is, of course, not a requirement of Archive-It (or of web archiving in general); however, both Belovari and Rackley illustrate various users who looked for, but could not find, information related to once-live web pages.

Though rare, usability studies focused on web archives reveal important truths about the way researchers access—or would like to access—contemporary web collections. In studying Archive-It, we aimed to address the lack of knowledge about the usability of the most widely used web archiving platform, while also contributing to the larger literature on web archives use. Our study confirms results from other web archives usability studies, contributes new findings about the importance of metadata in the usability of web archives search interfaces, and provides the first in-depth research into how users experience the Archive-It search interface in particular.

Methodology and Limitations

Our study focused on the usability of Archive-It, a tool used by approximately 94 percent of libraries and archives with web archiving programs.³⁰ Some libraries and archival institutions use other tools for archiving the web, such as Webrecorder, but they do not provide the same level of access and discovery that Archive-It affords at this time.

Each institution represented here had a subscription to Archive-It prior to the start of this study. We conducted our testing between September and October of 2017. Our analysis does not reflect any changes made to Archive-It or the Wayback Machine since that time.

Participants

A variety of testing institutions are represented in this study, including a small college, large research universities, a governmental archives, and a special library (an art museum). We identified a range of patron profiles likely to use web archives and chose to perform user testing with representatives of those profiles. These included students, faculty members of academic institutions, members of the public, library staff members, and nonlibrarian members of institutional staff (see Figure 5 for a breakdown of participant and institution types). All of these categories were represented but not in equal proportion: library staff constituted 5 of the 14 (36%) participants, faculty 3 of 14 (21%), and nonlibrarian staff 3 of 14 (21%), with 2 students (14%) and only 1 member of the general public (7%).

View Figure

• Download as Powerpoint
• Download Figure

Participants were recruited through email solicitations to personal networks and faculty listservs, and by word of mouth, as opposed to random sampling, because this simplified and accelerated the testing process. Each institution selected different types of users (student, faculty member, member of the public, library staff, and/or nonlibrarian institutional staff) on which to focus, and then recruited 2 or 3 users by reaching out directly to individuals via email.

These participants had varying backgrounds in web and research activity, as reported in their responses to the second question in the study (see Appendix). Based on their self-assessment, we labeled participants as “experienced researchers” or as having “limited or no archives experience.” Slightly more than half of the participants were categorized as experienced (8/14, 57%), while the remainder (6/14, 42%) were classified as having little or no archives research experience.

Future studies might benefit from different, more diverse participant pools. These could include a more balanced representation of user group segments; participants recruited in a more random way (versus through institutional and personal networks); users with a greater range of background experience, including exposure to archives generally and web archives specifically; participants from different fields of study, such as faculty in different research areas; participants from a wider public audience; and/or participants unaffiliated with the testing institution(s).

Testing Process

To ensure consistency across institutions, all researchers followed the same script (see Appendix: Script & Questions), which was based on a template from Princeton University Libraries website usability testing. Sessions were not recorded, but investigators (or their confederates) took notes on user behavior and transcribed select comments. The script had 3 sections.

Analysis Process

Because we sought a tool that would allow easy exchange between our 7 collaborators at geographically distinct locations with different capacities to purchase software, we decided to do all of our analysis using Google Sheets.

Initially, we planned to analyze quantitative results of each task based on whether or not the participant completed the task. However, we did find some gray areas while testing and developed 3 categories for results: completed easily, completed with difficulty, or did not complete. “Completed easily” indicates the user was able to navigate the site and successfully complete the task without any assistance. “Completed with difficulty” means that the user did eventually complete the task, but encountered significant issues and confusion, and perhaps received a minor hint from the test administrator. “Did not complete” indicates the user could not complete the task within approximately 10 minutes, or became frustrated and gave up. We did not collect data on task completion time, but estimate that based on the 30 to 45 minutes we told participants the testing would take, if a participant could not complete the task in less than 10 minutes, the test facilitator moved on to the next task and considered the previous task a “Did not complete.”

Through an iterative and collaborative process of qualitative analysis, we created 24 codes recognizing the most common behaviors or themes, such as sorting search results successfully or unsuccessfully, using the search bar appropriately or incorrectly, and understanding versus being confused by faceted filtering of results. Each test administrator coded her own notes from the testing she conducted at her institution. We created charts and graphs to analyze and visualize certain types of data. We also compiled suggestions for the Internet Archive and for archivists using the Archive-It service, based on users' direct feedback or our observations based on their behavior.

Concerns

We recognize that bias is inherent in every research design, and we tried to be aware of bias during this study.

Our test instrument reflects a particular understanding of archives and digital libraries, influenced by the papers referenced in our literature review and characterized by certain shared ideas: the concept of archival collections and how they are organized via Archive-It, familiarity with metadata and description practices in archives and libraries, and the idea of websites as archival artifacts. Investigators with different backgrounds who came to the study without these same preconceptions about archives and digital libraries may have formulated aspects of the study differently.

In our analysis, we did not account for the computer or information literacy of participants, variation in user profile (faculty, students, public, or staff), or variation in the type of institution (academic, state archives, or museums). Accounting or controlling for these variables might have changed our analysis.

Findings: Task Descriptions and Results

The following section reports task completion rates and participants' interface ratings and analyzes these results as well as the frequency of the codes investigators applied to the data.

Overall, participants performed well and completed most tasks, as shown in Figure 6. Nearly half of the participants (6 of 14) completed all of the tasks. An equal number completed half of the tasks. One participant completed 3 of the 4 tasks. Only 1 participant was unable to complete 3 tasks, and no participants were unable to complete all 4 tasks.

View Figure

• Download as Powerpoint
• Download Figure

Task 1: Find a Capture in Archive-It

In the first task, test facilitators asked participants to find a web page captured on a certain date in Archive-It (e.g., “Can you find the Swarthmore College homepage in March 2011?”). Testers used a variety of methods to find the websites they were asked to locate. Figure 7 shows some of the strategies employed: the Archive-It collection search bar at the top of the screen, the facet filters on the left panel, and browsing seed metadata toward the bottom right.

View Figure

• Download as Powerpoint
• Download Figure

Of the 14 testers, 12 successfully located the exact version of the site that they were asked to find (85%). Most used the Archive-It collection search bar to locate a seed; this approach was successful 6 times out of 8. Testers attempted to use facets to identify the desired seed and date 6 times, but were only successful twice. In most cases, using faceted filters did not lead to the results the testers wanted or expected: either the facets did not narrow the results enough, or they eliminated the desired result because the participant did not realize a secondary filter was still applied, or the participant did not apply the archivist-provided metadata as expected (for example, dates added by an archivist in the “date” field did not necessarily reflect the capture dates). Six participants looked at the dates in the metadata entries for a website in Archive-It (either the descriptive metadata entered by an archivist, or the “Captured” dates automatically generated by Archive-It and added to the metadata section) to find the range for the desired date. These participants had a higher success rate, leading to a desired outcome half of the time. Other strategies included leaving the Archive-It site to utilize the Internet Archive's Wayback Machine. Three participants successfully used this strategy 4 times; 1 participant was unsuccessful in trying this strategy. The least-used strategy was searching via methods other than the Archive-It search engine. Two participants were successful with CTRL+F, and one was unsuccessful with a Google search. The results of this task illustrate that participants relied on a variety of search strategies in assorted combinations.

Task 2: Find a Capture in the Wayback Machine

The second task asked participants to find captures from dates that were not available through the repositories' Archive-It captures for that website, but were available in the general Internet Archive through the Wayback Machine (i.e., https://archive.org/web, Figure 8). This task tested whether people were aware of the Wayback Machine and how to access it. This was the most difficult task for participants in all groups. Half of them failed, while 4 completed it with difficulty. Only 3 participants easily completed the task.

View Figure

• Download as Powerpoint
• Download Figure

Using the search bar for an Archive-It collection was the most commonly used strategy, and it was unsuccessful most of the time. Only 4 times did it result in participants finding information that assisted them in navigating to the website they were asked to find. The second most-used strategy was to go directly to the Wayback Machine. Used 8 times, it was only successful for participants in 3 cases. Other strategies were used much less frequently, although 4 participants employed Google search and CTRL+F with some success in getting to a desired result or to the Wayback Machine.

One participant, familiar with finding aids, followed a link in an Archive-It seed's descriptive metadata to an associated finding aid (see Figure 9). However, the finding aid did not include a link to the Wayback Machine, so, ultimately, referring to the finding aid did not assist the participant in completing this task.

View Figure

• Download as Powerpoint
• Download Figure

Four participants tried strategies such as faceting (3 times) and sorting (1 time), but these methods, although used correctly, were not helpful in completing the task. The low task completion rate illustrates that many participants may not have been aware of web archive discovery resources (such as the Wayback Machine) beyond what they learned from participating in this study.

Task 4: Find a Video

The last task asked participants to find a video within an Archive-It collection. Half of the participants completed it easily, while the other half failed completely. Following the link in seed metadata, as shown in Figure 10, was the most popular strategy (21 times). Participants also used the main search bar (15 times), navigating through Wayback (1), a Ctrl+F search on the collection page, and facets (3). Four participants were able to locate where videos would have been on the live site by browsing the archived site, but discovered that Archive-It had not captured them. Those who used metadata to find videos were successful 15 times out of 21. Searching was successful 11 out of 15 times.

View Figure

• Download as Powerpoint
• Download Figure

Some of the difficulty with this task was due to the videos not being successfully captured by Archive-It, although the participants otherwise succeeded in locating where the videos would have been on the live site. Results of this task show that Archive-It does not do an adequate job of situating videos within its website context, and a significant number of participants, possibly 50%, missed otherwise viewable video content. Several participants specifically complained that the metadata for the videos was confusing or inadequate.

View Figure

• Download as Powerpoint
• Download Figure

Following the seed metadata was the most popular strategy for finding videos. This is in contrast to tasks 2 and 3, where search was the most popular strategy, or task 1, where strategies were evenly distributed. In this particular case, this may have been because the metadata for each seed described the content requested by the task, limiting the need for extensive use of the search function.

Participants' Interface Ratings

At the conclusion of testing, participants were asked to rate their overall impression of the Archive-It end-user interface in terms of its user-friendliness on a scale of 1 to 10 (see Figure 12). The average rating was a 6.5, with 1 participant giving a rating of 3, 1 participant giving a rating of 9, and the rest of the participants clustered in the 5 to 8 range. The mode (most frequently given) score was a 7.

View Figure

• Download as Powerpoint
• Download Figure

The outlier low score of 3 may indicate that this participant especially struggled with the tasks. While all of the other participants in our study were able to complete between 2 and 4 tasks, the participant who gave a rating of 3 was the only one unable to complete 3 out of 4 tasks.

Analysis

In addition to assessing each task, the research team considered the overall strategies that participants used when attempting to complete each task. This analysis provides an overview of the strategies that were generally successful or unsuccessful, independent of each task.

Nine of the participants were confused when using the search bar. Unsure of the scope and precision of the search, they nonetheless expressed expectations for an intuitive and comprehensive search. Without taking the time to formulate a search strategy, participants tried entering many different types of information into the search bar: numerical dates and even analytic terms like “main site” or “homepage” in addition to keywords. A comprehensive search algorithm integrating both metadata and full-text search with appropriately weighted relevancy, or more obvious differentiation between the 2, would benefit these participants. However, this criticism is softened by the observation that successful use of the search bar was tagged nearly as often: 31 times. So, while the search bar caused confusion for some participants, in nearly as many occurrences, participants used it successfully. Making the change suggested above, if done properly, should not negatively affect participants who used the search bar successfully.

We also coded the following in our analysis: misleading or unsatisfactory metadata display, confusion regarding the meaning of collection (a group of websites) versus seed (a single website), and misuse of facets or expressing the view that the facets were unhelpful or confusing. It is worth noting that for the number of times each attribute or tool was unsuccessful in helping a participant complete a task, it sometimes helped different participants, or even the same participant in a different moment. Metadata (reading, understanding, and clicking on the descriptive metadata displayed with each seed on the collections page) was helpful 3 times as often (38) as it was unhelpful or confusing. Facets (using the faceted terms along the left side of the collections page as filters) was helpful about the same number of times (14) as it was unhelpful or confusing. All references to collections in the data caused confusion. Participants expressed confusion as to the meaning of an Archive-It collection versus a repository's holdings or a traditional analog collection—demonstrating that the organizational structure of collections in Archive-It tends to make users feel more, rather than less, disoriented.

Participants raised half of the metadata complaints on the video task, including multiple complaints about the lack of information, or even a title, for each video, and confusion as to which link to click on in results. Several participants also expressed confusion over the meaning of some metadata terms—“creator” versus “publisher,” and the meaning of “coverage.” Some participants did not notice metadata in Archive-It that would have led them directly to videos. In some cases, facets caused frustration for participants who struggled with the search because they did not realize a filter was still applied, and their results were unknowingly restricted.

A common point of confusion was the terminology used by Archive-It to label the organization of the websites. Nine participants explicitly voiced confusion over the meaning of the terms “capture,” “coverage,” and “creator,” as well as “collection” versus “site” or “archival collection,” and “title” versus “URL.” That some of these terms come directly from Dublin Core standards does not change the fact that the terms confused many participants. It would be more helpful to users if archivists hid or changed the labels, or defined certain metadata fields.

Five participants were confused about the concept of a “collection” in Archive-It. Four also expressed the view that the arrangement of the site was confusing, and had difficulty navigating from a seed page to a collection page to an organization page and understanding the differences between these entry points.

In 10 occurrences, participants used an alternative search, such as Google or the CTRL+F in-browser search function. Six times this strategy was unsuccessful, and the other four times, the users succeeded using alternative searches.

On 6 occasions, participants encountered difficulty navigating between the subscription Archive-It archive and the general Internet Archive collection. Nearly as many times, a total of 4, participants were successful. In only 1 instance was the participant unable to navigate the dates in the Wayback Machine, where 4 attempts were successful.

Only Swarthmore's 3 participants encountered at least a moment when they accidentally navigated to the live site without realizing they had left the archive. This may have been because the Swarthmore tasks used the college's homepage, a website all test participants were familiar with; other institutions' tasks used web pages that test participants likely had rarely or never seen before.

Participants made general positive remarks about site design and function. One participant liked the relaxing color scheme and felt the site was welcoming and clean; a second echoed that the design was clean and well laid out. Several remarked that it could be a useful reference tool or that they never thought about websites being archived but were pleased to know that they were.

There were also general negative remarks. As many participants commented that the general design was too busy, too filled, or overwhelming, as had commented that the general design was clean. Participants remarked that the site was disorganized; one wished the search bar was higher and another wished the research content and collections came before any information about the partner organization. Another participant felt that too many click-throughs were required to get to the pertinent information.

A recurring theme of this user study is that Archive-It is initially unintuitive, but that once a user becomes acclimated to its layout, logic, and terminology, it becomes easy to use. Eight of the participants in the study expressed this sentiment, which we characterized as “learning curve.” One person stated that a librarian may find it easy to navigate the site, but that a general user of the Internet may struggle. Another felt he or she would have to try certain things first to figure it out. However, as one participant commented, “After a day of using it, I wouldn't have trouble.” Someone else noted, “once you get into it, it works well.” A different user similarly stated, “once [he or she] understood how the site worked it was easy to navigate.”

Recommendations

The resulting data set of this study identified 6 key areas for improvement to the Archive-It interface: the search bar, video access, terminology glossary, date facet, published metadata options, and metadata labels. Our recommendations for clarifying these issues follow.

Conclusion

Comprehensive and intuitive web archiving tools are essential to support the ongoing work of archivists who increasingly rely on this technology to preserve and access websites. Without them, important historical information would be unavailable after websites are updated, edited, or removed from the Internet. More robust data about the usability of web archiving will support the development of tools that meet the needs of both archivists and researchers.

In our study, participants' overall impressions of Archive-It were positive-leaning, with an average score of 6.5 out of 10. Archive-It was not initially intuitive for most participants, though many noted growing ease as they overcame the learning curve and became acclimated to the site. The study found that individuals who self-reported as being experienced in doing archival research were more successful in completing the tasks overall compared to participants lacking experience with researching in archives. We discovered that some of the terminology employed by Archive-It confused participants, which we suggest could be mitigated with a glossary, tooltip definitions, or the ability to customize metadata labels. We also noticed that videos were difficult for participants to find and access through the Archive-It interface due to unsuccessful capture or insufficient metadata. Half of all metadata complaints were associated with the video task; the addition of video metadata could improve users' experience. In addition to analysis of Archive-It, the data collected in this study offer insight into the behaviors and expectations of internet users. For example, we found significant search bar confusion as participants expected a comprehensive single-search algorithm. They also expected date expressions, currently indexed as strings in Archive-It, to be normalized, indexed as dates, and searchable.

While the data compiled in this study were instructive, future studies using larger, more diverse pools of participants and institutions are needed to fully explore the usability of Archive-It and other web archiving tools.