“Archiving URLs § Preemptive Local Archiving”, Gwern2011-03-10

Archiving the Web, because nothing lasts forever: statistics, online archive services, extracting URLs automatically from browsers, and creating a daemon to regularly back up URLs to multiple sources.

In 2020-02, because of the increasing difficulty of repairing old links, I switched Gwern.net’s primary linkrot defense to preemptive local archiving: automatically mirroring locally all PDFs & web pages using manually-reviewed (and edited) SingleFile snapshots.

While it costs more time upfront (and presented some subtle UX problems like the “Arxiv problem”), it reduces total linkrot work.

• Link Rot
• Linkrot Quantities
• Detection
• Prevention
  • Remote Caching
  • Local Caching
    • Caching Proxy
    • Batch Job Downloads
    • Daemon
    • Cryptographic Timestamping Local Archives
    • Resource Consumption
    • URL Sources
      • Browser History
      • Document Links
      • Website Spidering
  • Fixing Redirects
    • Gwern.net Redirect Fixing
  • Preemptive Local Archiving
    • Local Snapshots
    • Workflow
    • The Arxiv Problem
      • Every CS Problem…
• Reacting To Broken Links
  • Automatic Internet Archive Repairs
    • Why Not Internet Archive?
• External Links
• Appendices
  • filter-urls
  • sort Key Compression Trick
  • Cryptographic Timestamping