Caching Proxy

The most ambitious & total approach to local caching is to set up a proxy to do your browsing through, and record literally all your web traffic; for example, using Live Archiving Proxy (LAP) or WarcProxy which will save as WARC files every page you visit through it. (Zachary Vance explains how to set up a local HTTPS certificate to MITM your HTTPS browsing as well.)

One may be reluctant to go this far, and prefer something lighter-weight, such as periodically extracting a list of visited URLs from one’s web browser and then attempting to archive them.

Batch Job Downloads

For a while, I used a shell script named, imaginatively enough, local-archiver:

#!/bin/sh
set -euo pipefail

cp `find ~/.mozilla/ -name "places.sqlite"` ~/
sqlite3 places.sqlite "SELECT url FROM moz_places, moz_historyvisits \
                       WHERE moz_places.id = moz_historyvisits.place_id \
                             and visit_date > strftime('%s','now','-1.5 month')*1000000 ORDER by \
                       visit_date;" | filter-urls >> ~/.tmp
rm ~/places.sqlite
split -l500 ~/.tmp ~/.tmp-urls
rm ~/.tmp

cd ~/www/
for file in ~/.tmp-urls*;
 do (wget --unlink --continue --page-requisites --timestamping --input-file $file && rm $file &);
done

find ~/www -size +4M -delete

The code is not the prettiest, but it’s fairly straightforward:

1. the script grabs my Firefox browsing history by extracting it from the history SQL database file¹⁴, and feeds the URLs into wget.

   wget is not the best tool for archiving as it will not run JavaScript or Flash or download videos etc. It will download included JS files but the JS will be obsolete when run in the future and any dynamic content will be long gone. To do better would require a headless browser like PhantomJS which saves to MHT/MHTML, but PhantomJS refuses to support it and I’m not aware of an existing package to do this. In practice, static content is what is most important to archive, most JS is of highly questionable value in the first place, and any important YouTube videos can be archived manually with youtube-dl, so wget’s limitations haven’t been so bad.

2. The script splits the long list of URLs into a bunch of files and runs that many wgets in parallel because wget apparently has no way of simultaneously downloading from multiple domains. There’s also the chance of wget hanging indefinitely, so parallel downloads continues to make progress.

3. The filter-urls command is another shell script, which removes URLs I don’t want archived. This script is a hack which looks like this:

   #!/bin/sh
   set -euo pipefail
   cat /dev/stdin | sed -e "s/#.*//" | sed -e "s/&sid=.*$//" | sed -e "s/\/$//" | grep -v -e 4chan -e reddit ...

4. delete any particularly large (>4MB) files which might be media files like videos or audios (podcasts are particular offenders)

A local copy is not the best resource—what if a link goes dead in a way your tool cannot detect so you don’t know to put up your copy somewhere? But it solves the problem decisively.

The downside of this script’s batch approach soon became apparent to me:

1. not automatic: you have to remember to invoke it and it only provides a single local archive, or if you invoke it regularly as a cron job, you may create lots of duplicates.

2. unreliable: wget may hang, URLs may be archived too late, it may not be invoked frequently enough, >4MB non-video/audio files are increasingly common…

3. I wanted copies in the Internet Archive & elsewhere as well to let other people benefit and provide redundancy to my local archive

It was to fix these problems that I began working on archiver—which would run constantly archiving URLs in the background, archive them into the IA as well, and be smarter about media file downloads. It has been much more satisfactory.

Daemon

archiver has an extra feature where any third argument is treated as an arbitrary sh command to run after each URL is archived, to which is appended said URL. You might use this feature if you wanted to load each URL into Firefox, or append them to a log file, or simply download or archive the URL in some other way.

For example, instead of a big local-archiver run, I have archiver run wget on each individual URL: screen -d -m -S "archiver" sh -c 'while true; do archiver ~/.urls.txt gwern@gwern.net "cd ~/www && wget --unlink --continue --page-requisites --timestamping -e robots=off --reject .iso,.exe,.gz,.xz,.rar,.7z,.tar,.bin,.zip,.jar,.flv,.mp4,.avi,.webm --user-agent='Firefox/3.6' 120"; done'. (For private URLs which require logins, such as darknet markets, wget can still grab them with some help: installing the Firefox extension Export Cookies, logging into the site in Firefox like usual, exporting one’s cookies.txt, and adding the option --load-cookies cookies.txt to give it access to the cookies.)

Alternately, you might use curl or a specialized archive downloader like the Internet Archive’s crawler Heritrix.

Cryptographic Timestamping Local Archives

We may want cryptographic timestamping to prove that we created a file or archive at a particular date and have not since altered it. Using a timestamping service’s API, I’ve written 2 shell scripts which implement downloading (wget-archive) and timestamping strings or files (timestamp). With these scripts, extending the archive bot is as simple as changing the shell command:

@reboot sleep 4m && screen -d -m -S "archiver" sh -c 'while true; do archiver ~/.urls.txt gwern2@gwern.net \
        "wget-archive" 200; done'

Now every URL we download is automatically cryptographically timestamped with ~1-day resolution for free.

Resource Consumption

The space consumed by such a backup is not that bad; only 30–50 gigabytes for a year of browsing, and less depending on how hard you prune the downloads. (More, of course, if you use linkchecker to archive entire sites and not just the pages you visit.) Storing this is quite viable in the long term; while page sizes have increased 7× between 2003_22ya and 2011_14ya and pages average around 400kb¹⁵, Kryder’s law has also been operating and has increased disk capacity by ~128×—in 2011_14ya, $123.76^$80₂₀₁₁ will buy you at least 2 terabytes, that works out to 4 cents a gigabyte or 80 cents for the low estimate for downloads; that is much better than the annual fee that somewhere like Pinboard charges. Of course, you need to back this up yourself. We’re relatively fortunate here—most Internet documents are ‘born digital’ and easy to migrate to new formats or inspect in the future. We can download them and worry about how to view them only when we need a particular document, and Web browser backwards-compatibility already stretches back to files written in the early 1990s. (Of course, we’re probably screwed if we discover the content we wanted was dynamically presented only in Adobe Flash or as an inaccessible ‘cloud’ service.) In contrast, if we were trying to preserve programs or software libraries instead, we would face a much more formidable task in keeping a working ladder of binary-compatible virtual machines or interpreters¹⁶. The situation with digital movie preservation hardly bears thinking on.

There are ways to cut down on the size; if you tar it all up and run 7-Zip with maximum compression options, you could probably compact it to 1⁄5^th the size. I found that the uncompressed files could be reduced by around 10% by using fdupes to look for duplicate files and turning the duplicates into a space-saving hard link to the original with a command like fdupes --recurse --hardlink ~/www/. (Apparently there are a lot of bit-identical JavaScript (eg. JQuery) and images out there.)

Good filtering of URL sources can help reduce URL archiving count by a large amount. Examining my manual backups of Firefox browsing history, over the 1153 days from 2014-02-25 to 2017-04-22, I visited 2,370,111 URLs or 2055 URLs per day; after passing through my filtering script, that leaves 171,446 URLs, which after de-duplication yields 39,523 URLs or ~34 unique URLs per day or 12,520 unique URLs per year to archive.

This shrunk my archive by 9GB from 65GB to 56GB, although at the cost of some archiving fidelity by removing many filetypes like CSS or JavaScript or GIF images. As of 2017-04-22, after ~6 years of archiving, between xz compression (at the cost of easy searchability), aggressive filtering, occasional manual deletion of overly bulky domains I feel are probably adequately covered in the IA etc, my full WWW archives weigh 55GB.

URL Sources

#!/bin/sh
set -euo pipefail

cp --force `find ~/.mozilla/firefox/ -name "places.sqlite"|sort|head -1` ~/
sqlite3 -batch places.sqlite "SELECT url FROM moz_places, moz_historyvisits \
                       WHERE moz_places.id = moz_historyvisits.place_id and \
                       visit_date > strftime('%s','now','-1 day')*1000000 ORDER by \
                       visit_date;" | filter-urls
rm ~/places.sqlite

@hourly firefox-urls >> ~/.urls.txt

linkchecker --check-extern -odot --complete -v --ignore-url=^mailto --no-warnings http://www.longbets.org
    | grep -F http #
    | grep -F -v -e "label=" -e "->" -e '" [' -e '" ]' -e "/ "
    | sed -e "s/href=\"//" -e "s/\",//" -e "s/ //"
    | filter-urls
    | sort --unique >> ~/.urls.txt

Local Snapshots

So after mulling over the design for a while, I pulled the trigger in 2020-02, creating LinkArchive.hs, which maintains a database of URL/archive/status tuples, and calls linkArchive.sh to lookup or generate the snapshots using SingleFile CLI. The snapshots are reviewed by hand to verify that they work & are readable; if they are not, I make a new one, and often invest some time in making a bunch of uBlock Origin rules to clean a page up. It is worthwhile to do so because the rules will make future archiving easier.

Workflow

1. Database+directories: A simple database of URLs is maintained, with their status: either the first-seen date, or their on-disk path.

   The URL’s snapshot lives at /doc/www/$DOMAIN/SHA1($URL).html; the hash makes it simple to encode the name without escaping issues or incurring the difficulties of imitating the target URL’s directory structure²⁰, and the separate prefixed domain helped compatibility with the link-icon rules.²¹

2. Compile-time archiving:

   When compiling a page, each link is checked to see if it is on the blacklist (because it is a link which cannot or should not be archived). If not, then if it has an archive, it is rewritten to point to the archive instead. (The original URL is stored in an HTML attribute for any other tools that need to know what that was, such as the popups.) If it doesn’t, then the date is checked: if it was first-seen >60²² days ago, then it is time to archive the URL. (I limit the number of archives per site compilation to 12, to spread out the burden.)

   The archiving script checks the URL. If the URL returns a PDF MIME type, then it downloads the PDF and (like my manually-hosted PDFs) runs ocrmypdf on it to ensure that it has decent OCR, is a standard PDF/A archival-quality PDF, and throws in JBIG2 compression which will often save 2–3× space. If it’s a regular HTML URL, it instead runs SingleFile CLI, which opens the URL in a local Chrome installation, loads all the JS and lazy images etc, and writes out a static snapshot. (In theory, this Chrome instance should have all my usual extensions enabled like uBlock & cookies enabling logins²³, but in practice I’ve run into problems and I’m not sure what exactly is getting run because it is impossible to debug.)

   Large snapshots will be rewritten into a multiple-file form using a custom PHP script, deconstruct_singlefile.php. This helps reduce the bandwidth burden of Gwern.net readers previewing pages, and lets us run optimization on the individual files. (‘Wild’ JPG, PNG, and GIF files on the Internet can often be shrunk a lot, and this also helps expose invalid files.)

3. Manual review: The URL & its new snapshot (whether PDF or HTML) are then both opened by the archiving script in Firefox for manual review.

   A bad snapshot may require me to add the domain or URL to the blacklist. It may also just be broken (in many different ways, from broken media to sudden paywalls to reloading to a nonexistent URL!), in which case I have to find a replacement URL & do a global rewrite (and the replacement URL will eventually be archived).

   Alternately, I may need to make a new snapshot manually, after deleting sticky elements with Always Kill Sticky, scrolling the page to load lazy elements, or spending a while with the uBlock element-picker to delete the most obnoxious elements. (I tried initially to edit the raw HTML of the SingleFile snapshot, but the data-URI encoding of images, and the spaghetti architecture of the pages most in need of de-cruddification, meant that it was far harder to edit it in a text editor than to simply use uBlock or the built-in web tool ‘inspector’ to select & delete. This can also be done repeatedly: take a SingleFile snapshot, delete stuff with inspector, and save with SingleFile again.) Sometimes a page will fail in Chromium SingleFile, but then succeed in Firefox SingleFile.

   Since SingleFile saves the final DOM, I can edit the live page: C-I to open the browser webdev tools, then C-Shift-c to select an element with the mouse, then Del to delete it. (One can also do fancier tricks, like opening multiple pages, and then copy-pasting a particular element. This is good for turning multi-page articles into a single page easily and reliably, which makes reading & archiving easier: you just find the ‘content’ node in each page, and copy it into the first ‘master’ page, and save that page.)

   Taking the time to remove all the ads & bric-a-brac is useful because it makes the page less likely to break in the future, smaller & faster, more readable as a ‘live’ popup, easier to see if anything is missing, and amortizes across future snapshots (so the manual review gets a little easier every time as the blacklist & blocklist expand).

   While it did not solve existing broken links or breakage of links where SingleFile can’t make a good snapshot, and it is a bit annoying to curate snapshots, it is a dramatic improvement over the status quo and I expect the benefits to only increase with time as web pages become ever grosser.

   Overall, this review process tends to take about 10–60s per page, and I do ~10⧸day to keep up with my writing & bookmarking, which is an acceptable burden.

4. Metadata/hiding: The local archives are treated mostly like regular files, while trying to minimize search engine or user exposure; this is because various entities might not appreciate the mirrors²⁴, and I don’t want to have to maintain them indefinitely (particularly by setting up redirects to avoid breaking old hash-URLs due to churn). People will still link to them, but there’s only so much one can do.

   The Gwern.net web server (nginx) is instructed to set appropriate headers to block indexing (beyond just noindex):

   location /doc/www/               { add_header X-Robots-Tag "none, nosnippet, noarchive, nocache"; }

   And for good measure, robots.txt:

   User-agent: *
   Disallow: /doc/www/*

5. Results: As of 2023-03-09, there are 14,352 targeted links weighing 47GB (including now-unused archives which will be removed at some point), which is no problem.

The Arxiv Problem

One subtlety that came up after a while was the issue of ‘URL transforms’. In some cases, like BioRxiv, it’s best to simply rewrite URLs to the full-text HTML version (just append .full to the URL); this is simple & one can ensure it happens by including a global search-and-replace in the compilation process, and forget about it while linking BioRxiv normally. Unfortunately, sometimes a global search-and-replace is inadequate: there is a set of URLs where you link to one URL, but for archiving purposes, you want to archive an alternative, a transformation of that URL. The motivating example was ‘the Arxiv problem’, one of the most heavily-linked domains on Gwern.net, and a difficult one to solve: I wound up needing 3 separate URLs to handle browsing as conveniently as possible for both desktop & mobile users.

‘The Arxiv problem’ is the difficulty in presenting Arxiv links to both desktop & mobile users which is not (1) slow, (2) redundant, or (3) foisting a PDF on readers (especially mobile readers). It is polite to link to the Arxiv HTML landing page like /abs/$ID rather than its fulltext PDF under /pdf/$ID, but that would be useless to archive because I already extract most of the relevant information to present as the annotation popup. It would be a waste of reader time to hover over an Arxiv annotated link, read the annotation with the title/author/date/abstract, and then hover over the ‘live’ Arxiv link to… read the same thing, presumably wasting their time until they can navigate to the fulltext PDF which they must have decided is worth their time. Obviously, the ‘live’ link should be the PDF, which can then be a local optimized PDF.²⁵ Straightforward, except Arxiv has an additional wrinkle: whether HTML landing page or PDF, it’s bad for mobile readers on smartphones, who really need a responsive dark mode-compatible HTML version of the paper—which often exists, as there is yet a third set of Arxiv URLs located at the experimental LaTeX → HTML service which covers most (but not all) Arxiv papers, ‘Ar5iv’ (ie. https://arxiv.org/html/$ID)!²⁶

Arxiv is the main problem, but there are a few other cases worth handling. The preprint website OpenReview operates similarly to Arxiv, with an HTML landing page & PDF. And Reddit turns out to archive very poorly by default even using the ‘old’ Reddit interface which is good for regular browsing; but I discovered that while the mobile Reddit (i.reddit.com) didn’t look great, it at least preserved the content reliably. (Particularly acute readers may note that LessWrong → GreaterWrong is not mentioned, because that is treated like a Wikipedia popup in that the JS code calls a specialized API.)