Knowledge Concepts
What is web archiving? A clear explanation of how the Internet Archive, Wayback Machine, and personal archiving tools preserve web content — and why researchers should care.
Web archiving is the process of collecting, preserving, and providing access to web content over time — capturing snapshots of web pages as they exist at a specific moment, so that the content remains accessible even after the original page changes or disappears.
Web archiving is conducted at institutional scale (the Internet Archive, national libraries, universities) and at personal scale (individuals archiving sources for their own research). It addresses the fundamental problem of link rot — the decay of URLs that makes 25–50% of web links unreachable within 10 years.
The web is not a permanent record. Research by the Pew Research Center (2021) found that 25% of web pages that existed between 2013 and 2023 were no longer accessible. Studies by Harvard Law Library have documented that 49% of links cited in Supreme Court opinions no longer work.
Web archiving matters because:
Research integrity: Scientific, legal, and journalistic citations depend on verifiable sources. A broken link in a published paper is a reproducibility failure — no one can verify the claim the citation was supposed to support.
Historical record: News articles are updated or deleted without notice. Government pages change to reflect current policy. Corporate announcements are removed after leadership changes. The web's first draft of history disappears without archiving.
Personal research: For individual researchers and knowledge workers, web archiving means their sources remain accessible for the duration of a project — not just until the source decides to restructure its website.
At the institutional level, web archiving typically uses web crawlers:
Automated crawlers visit URLs, download the page content (HTML, CSS, JavaScript, images), follow links, and repeat — systematically working through large portions of the web. The Internet Archive's Heritrix crawler is the leading open-source tool.
WARC format: Web content is saved in WARC (Web ARChive) format — a standardized container format that stores HTTP response headers, page content, and metadata. WARC files are the archival standard used by national libraries and the Internet Archive.
Playback systems: Tools like the Wayback Machine (web.archive.org) provide interface for navigating archived content: enter a URL, select a date, view the page as it appeared on that date. OpenWayback and PyWB are open-source playback systems.
At the personal level, web archiving works through simpler tools:
The Internet Archive (archive.org) is the largest web archive in the world, founded by Brewster Kahle in 1996. It has archived over 800 billion web pages and holds petabytes of web content.
The Wayback Machine (web.archive.org) is the public interface: enter any URL and see snapshots of that page across its history. Most crawled URLs have multiple snapshots per year; high-traffic pages have multiple per day.
The Internet Archive crawls the web continuously but doesn't capture everything:
robots.txt are excludedSave Page Now: Anyone can submit a URL to web.archive.org/save/[URL] to request an immediate crawl and archival. This is the most reliable way to ensure a specific page is archived.
| Aspect | Institutional (Internet Archive) | Personal |
|---|---|---|
| Scale | Billions of pages | Hundreds to thousands |
| Coverage | Broad but incomplete | Targeted and complete |
| Reliability | Dependent on crawl cadence | Immediate on save |
| Citability | Yes (stable archive.org URLs) | Tool-dependent |
| Access | Public | Private |
| Link rot protection | Best-effort | Strong for saved items |
For researchers who need citable archival links (published papers, legal briefs): use Perma.cc (Harvard Law Library) or Wayback Machine's Save Page Now, then cite the archive.org or perma.cc URL.
For researchers who need reliable personal access to sources: use a tool that saves full content locally or to a personal library.
| Tool | What it saves | Searchable | Citable | Price |
|---|---|---|---|---|
| Internet Archive (Save Page Now) | Page snapshot | No (external search) | Yes (stable URL) | Free |
| Perma.cc | Page snapshot | No | Yes (legal/academic use) | Free (limited) / institutional |
| WebSnips | Full article text | Yes (personal library) | No | Free / $49–99 lifetime |
| Zotero (with snapshot) | Page snapshot + citation | Yes | Locally only | Free |
| ArchiveBox (self-hosted) | Full page + screenshots | Yes | No (local only) | Free |
| SingleFile browser extension | Full HTML locally | No (file search only) | No | Free |
For research that needs formal citation: Wayback Machine or Perma.cc. For personal research that needs persistent access: WebSnips or Zotero (depending on whether you're doing web research or academic literature).
A PhD student researching corporate communications during a financial crisis:
Problem: She's citing press releases, news articles, and company blog posts that could change or disappear at any time. She needs these sources to remain accessible through the two years she'll be working on her thesis.
Institutional archiving: For each key source she cites formally, she submits the URL to Wayback Machine's "Save Page Now" and notes the archive.org URL as her stable citation link.
Personal archiving: For the dozens of background sources she's reading for context (not formal citation), she saves them with WebSnips. Full content is preserved in her library regardless of what happens to the source URLs.
Result: Two years later, when her committee asks about a broken URL in her reference list, she has both: a formal archive.org citation and her own saved copy for her research notes.
"The Wayback Machine has everything." It has a lot — over 800 billion pages — but coverage is uneven. Pages blocked by robots.txt are excluded. New pages aren't immediately archived. Content behind logins, dynamic content, and paywalled pages are rarely captured. Don't assume a page is archived; verify, and submit to Save Page Now for critical sources.
"Saving a URL in my bookmarks is archiving." Bookmarks save URLs, not content. When the source page changes or disappears, the bookmark still points to the dead URL. Archiving means saving the content — not the pointer.
"Web archiving is only for academics." Journalists use web archiving to document online statements that are later retracted. Lawyers use it to preserve evidence of website content in disputes. Competitive intelligence professionals archive competitor pages to track changes over time. The use cases extend well beyond academia.
"Archived content is always legally accessible." Web archiving operates in a complex legal space. The Internet Archive has been involved in multiple legal disputes about whether archiving constitutes copyright infringement. For most personal research purposes, archiving for private use is well within fair use; publicly redistributing archived commercial content is more legally complex.
WebSnips addresses the personal research dimension of web archiving:
WebSnips is not an institutional archiving tool — it doesn't create publicly citable stable URLs. For citations in published work, combine WebSnips (personal research access) with Wayback Machine or Perma.cc (formal, citable archival records).
How do I cite archived web pages?
Use the Wayback Machine URL format: https://web.archive.org/web/[timestamp]/[original-url]. Include both the original URL and the archive URL in your citation. For legal and academic work requiring permanent links, Perma.cc is purpose-built and provides perma.cc/[code] URLs.
Can I archive paywalled content for my research? If you have a subscription, you can use personal archiving tools (WebSnips, Zotero, SingleFile) to save content you have legitimate access to, for your own personal research use. Distributing archived paywalled content publicly would be a copyright violation. Saving for personal research is generally protected as fair use.
Is the Internet Archive legal? The Internet Archive has been involved in litigation from publishers who argue its lending library and archiving activities infringe copyright. A 2023 court ruling found against the Archive on its "Controlled Digital Lending" program (lending digital copies of books). Web page archiving for research and historical preservation operates in a different legal context and has stronger precedent. The Archive continues to operate and the web archiving function remains active.
How often should I archive important sources? For sources you'll cite formally: archive before you publish (not when you first read). For active research: archive at the point of saving, since link rot can occur faster than you expect — government pages and startup websites in particular can change quickly.
Web archiving is the practice of preserving web content against the inevitable decay of URLs — conducted at institutional scale by the Internet Archive and at personal scale by individual researchers using web clipping tools, browser extensions, and local archiving software.
For academic researchers: use Perma.cc or Wayback Machine's Save Page Now for citable archival links. For personal research that needs persistent access: use full-content-saving tools like WebSnips or Zotero (with snapshot). Combine both for serious research.
The fundamental principle: cite URLs that might change, and preserve a copy of the content when the research depends on it.
More WebSnips articles that pair well with this topic.
What is citation management? A clear explanation for researchers and academics — how citation managers work, the best tools, and how to avoid the most common mistakes.
What is tacit knowledge? A clear explanation for team leads and ops people — what tacit knowledge is, why it's the hardest knowledge to transfer, and practical methods to make it explicit.
What is a commonplace book? A clear explanation of the centuries-old knowledge collection practice, famous examples, and how to build a modern digital version for writers and researchers.