Implementation Details

There are some tricks & details that web designers might find interesting.

Efficiency:

• fonts:

  • Adobe Source Serif⁠/​Sans⁠/​Code Pro: originally Gwern.net used Baskerville⁠, but system fonts often don’t have true small caps (requiring browsers to synthesize ugly downscaled capitals as ‘small caps’). Some Baskerville digital fonts do, but if one is going to bite the bullet of needing to download a font, one might as well consider other options.

    Adobe’s open-source “Source” font family of screen serifs⁠, however, is high quality and actively-maintained, comes with good small caps and multiple sets of numerals (‘old-style’ numbers for the body text and different numbers for tables), and looks particularly nice on Macs. (It is also subsetted to cut down the load time.) Small caps CSS is automatically added to abbreviations/​acronyms/​initials by a Hakyll/​Pandoc plugin, to avoid manual annotation.

  • efficient drop caps by subsetting: 1 drop cap is used on every page, but a typical drop cap font will slowly download as much as a megabyte in order to render 1 single letter.

    CSS font loads avoid downloading font files which are entirely unused, but they must download the entire font file if anything in it is used, so it doesn’t matter that only one letter gets used. To avoid this, we split each drop cap font up into a single font file per letter and use CSS to load all the font files; since only 1 font file is used at all, only 1 gets downloaded, and it will be ~4kb rather than 168kb. This has been done for all the drop cap fonts used (yinit⁠, Cheshire Initials⁠, Deutsche Zierschrift⁠, Goudy Initialen⁠, Kanzlei Initialen), and the necessary CSS can be seen in fonts.css⁠. To specify the drop cap for each page, a Hakyll metadata field is used to pick the class and substituted into the HTML template.

• lazy JavaScript loading by IntersectionObserver: several JS features are used rarely or not at all on many pages, but are responsible for much network activity. For example, most pages have no tables but tablesorter must be loaded anyway.

  To avoid this, IntersectionObserver can be used to write a small JS function which fires only when particular page elements are visible to the reader. The JS then loads the library which does its thing. So an IntersectionObserver can be defined to fire only when an actual <table> element becomes visible, and on pages with no tables, this never happens. Similarly for image-focus.js. This trick is a little dangerous if a library depends on another library because the loading might cause race conditions; fortunately, only 1 library (tablesorter), has a prerequisite (jQuery), so I simply prepend jQuery to tablesorter and load tablesorter.

  Other libraries, like sidenotes or WP popups, aren’t lazy-loaded because sidenotes need to be rendered as fast as possible or the page will jump around & be laggy, and WP links are so universal it’s a waste of time making them lazy since they will be in the first screen on every page & be loaded immediately anyway, so they are simply loaded asynchronously with the async/​defer JS keywords.

• image optimization: PNGs are optimized by pngnq/​advpng, JPEGs with mozjpeg, SVGs are minified, PDFs are compressed with ocrmypdf’s JBIG2 support. (GIFs are not used at all in favor of WebM/​MP4 <video>s.)

• JS/​CSS minification: because Cloudflare does Brotli compression, minification of JS/​CSS has little advantage and makes development harder, so no minification is done; the font files don’t need any special compression either.

• MathJax: getting well-rendered mathematical equations requires MathJax or a similar heavyweight JS library; worse, even after disabling features, the load & render time is extremely high—a page like the embryo selection page which is both large & has a lot of equations can visibly take >5s (as a progress bar that helpfully pops up informs the reader).

  The solution here is to prerender MathJax locally after Hakyll compilation⁠, using the local tool mathjax-node-page to load the final HTML files, parse the page to find all the math, compile the expressions, define the necessary CSS, and write the HTML back out. Pages still need to download the fonts but the overall speed goes from >5s to <0.5s, and JS is not necessary at all.

• Automatic Link-Ification Regexps: I wrote LinkAuto.hs⁠, a Pandoc library for automatically turning user-defined regexp-matching strings into links, to automatically turn all the scientific jargon into Wikipedia or paper links. (There are too many to annotate by hand, especially as new terms are added to the list or abstracts are generated for popups.)

  “Test all strings against a list of regexps and rewrite if they match” may sound simple and easy, but the naive approach is exponential: n strings, r regexps tested on each, so 𝒪 (n^r) matches total. With >600 regexps initially & millions of words on Gwern.net… Regexp matching is fast, but it’s not that fast. Getting this into the range of ‘acceptable’ (~3× slowdown) required a few tricks.

  The major trick is that each document is converted to a simple plain text format, and the regexps are run against the entire document; in the average case (think of short pages or popup annotations), there will be zero matches, and the document can be skipped entirely. Only the matching regexps get used in the full-strength AST traversal. While it is expensive to check a regexp against an entire document, it is an order of magnitude or two less expensive than checking that regexp against every string node inside that document!

Correctness:

• Dark mode: our dark mode is custom, and tries to make dark mode a first-class citizen.

  1. Avoiding Flashing & Laggy Scrolling: it is implemented in the standard best-practice way of creating two color palettes (associating a set of color variables for every element, for a light-mode and then automatically-generating dark mode colors by inverting & gamma-correcting), and using JS to toggle the media-query to instantly enable that color.

     This avoids the ‘flash of white’ on page loads which regular JS-based approaches incur (because the CSS media-queries can only implement auto-dark-mode, and the dark mode widget requires JS; however, the JS, when it decides to inject dark mode CSS into the page, is too late and that CSS will be rendered last after the reader has already been exposed to the flash). The separate color palette approach also avoids the lag & jank of using invert CSS filters (one would think that invert(100%) would be free from a performance standpoint—but it is not).

  2. Native Dark Mode Color Scheme: we modify the color scheme as necessary: some blue breaks the monochrome coloring, and the source code syntax highlighting is tweaked for dark mode visibility.

     Because of the changes in contrast, inverting the color scheme only mostly works. In particular, inline & code blocks tend to disappear.

  3. Inverted Images: color images are desaturated & grayscaled by default to reduce their brightness; grayscale/​monochrome images, are automatically inverted by an ImageMagick heuristic, falling back to manual annotations.

     This avoids the common failure mode where a blog uses a dark mode library which implements the class approach correctly… but then all of their images still have blinding bright white backgrounds or overall coloration, defeating the point! However, one cannot just blindly invert images because many images, photographs of people especially, are garbage as ‘photo-negatives’.

  4. Three-Way Dark Mode Toggle: Many dark modes are implemented with a simple binary on/​off logic stored in a cookie, ignoring browser/​OS preferences, or simply defining ‘dark mode’ as the negation of the current browser/​OS preference.

     This is incorrect, and leads to odd situations like a website enabling dark mode during the day, and then light mode during the night! Using an auto/​dark/​light three-way toggle means that users can force dark/​light mode but also leave it on ‘auto’ to follow the browser/​OS preference over the course of the day. This requires a UI widget and it still incurs some of the problems of an auto-only dark mode⁠, but overall strikes the best balance between enabling dark mode unasked, user control/​confusion, and avoiding dark mode at the wrong time.

• collapsible sections: managing complexity of pages is a balancing act. It is good to provide all necessary code to reproduce results, but does the reader really want to look at a big block of code? Sometimes they always would, sometimes only a few readers interested in the gory details will want to read the code. Similarly, a section might go into detail on a tangential topic or provide additional justification, which most readers don’t want to plow through to continue with the main theme. Should the code or section be deleted? No. But relegating it to an appendix, or another page entirely is not satisfactory either—for code blocks particularly, one loses the literate programming aspect if code blocks are being shuffled around out of order.

  A nice solution is to simply use a little JS to implement code folding approach where sections or code blocks can be visually shrunk or collapsed, and expanded on demand by a mouse click. Collapsed sections are specified by a HTML class (eg. <div class="collapse"></div>), and summaries of a collapsed section can be displayed, defined by another class (<div class="abstract-collapse">). This allows code blocks to be collapse by default where they are lengthy or distracting, and for entire regions to be collapsed & summarized, without resorting to many appendices or forcing the reader to an entirely separate page.

• Sidenotes: one might wonder why sidenotes.js is necessary when most sidenote uses are like Tufte-CSS and use a static HTML/​CSS approach, which would avoid a JS library entirely and visibly repainting the page after load?

  The problem is that Tufte-CSS-style sidenotes do not reflow and are solely on the right margin (wasting the considerable whitespace on the left), and depending on the implementation, may overlap, be pushed far down the page away from their, break when the browser window is too narrow or not work on smartphones/​tablets at all. (This is fixable⁠, Tufte-CSS’s maintainers just haven’t.) The JS library is able to handle all these and can handle the most difficult cases like my annotated edition of Radiance⁠. (Tufte-CSS-style epigraphs⁠, however, pose no such problems and we take the same approach of defining an HTML class & styling with CSS.)

• Link icons: icons are defined for all filetypes used in Gwern.net and many commonly-linked websites such as Wikipedia, or Gwern.net (within-page section links and between-page get ‘§’ & logo icons respectively).

  They are implemented in a scalable compile-time approach when the standard approach failed⁠.

• Redirects: static sites have trouble with redirects, as they are just static files. AWS 3S does not support a .htaccess-like mechanism for rewriting URLs. To allowing moving pages & fix broken links, I wrote Hakyll.Web.Redirect for generating simple HTML pages with redirect metadata+JS, which simply redirect from URL 1 to URL 2. After moving to Nginx hosting, I converted all the redirects to regular Nginx rewrite rules.

  In addition to page renames, I monitor 404 hits in Google Analytics to fix errors where possible, and Nginx logs. There are an astonishing number of ways to misspell Gwern.net URLs, it turns out, and I have defined >10k redirects so far (in addition to generic regexp rewrites to fix patterns of errors).

Returns To Design?

Particularly with typography, there seems to be an infinite number of finicky details one could spend time on (much of which appears to be for novelty’s sake⁠, while vastly more important things like advertising harms go ignored by so-called designers). One’s initial guess is that it’d be diminishing returns like most things: it’d look something like a log curve, where every additional tweak costs more effort as one approaches the Platonic ideal. A more sophisticated guess would be that it’d look like a sigmoid: at first, something is so awful that any fixes are irrelevant to the user because that just means they suffer from a different problem (it doesn’t matter much if a website doesn’t render because of a JS bug if the text when it does render is so light-shaded that one can’t read it); then each improvements makes a difference to some users as it approaches a respectable mediocrity; and after that, it’s back to diminishing returns.

My experience with improving the design of Gwern.net & reading about design has made me wonder if either of those is right. The shape may resemble more of a parabola: the sigmoid, at some point, spikes up and returns increase rather than diminish?

I noticed that for the first half-decade or so, no one paid much attention to the tweaks I made, as it was an ordinary Markdown-based static site. As I kept tinkering, a comment would be made once in a while. When Said Achmiz lent his talents to adding features & enhancements and exploring novel tweaks, comments cropped up much more frequently (consistent with the enormous increase in time spent on it); by 2019, the redesign had mostly stabilized and most of the signature features had been implemented, and 2020 was more about bug fixes than adding pizzazz. Under the intuitive theories, the rate of comments would be about the same: while the bug fixes may involve huge effort—the dark mode rewrite was a 3-month agony—the improvements are ever smaller—said rewrite had no user-visible change other than removing slowness. But while site traffic remained steady, 2020 attracted more compliments than ever!

Similarly, the LW team put an unusual amount of effort into designing a 2018 essay compilation⁠, making it stylish (even redrawing all the images to match the color themes), and they were surprised by unusually large the preorders were: not a few percentage points, but many times. (There are many books on data visualization, but I suspect Edward Tufte’s books outsell them, even the best, by similar magnitudes.) And what should we make of Apple & design⁠, whose devices & software have glaring flaws and yet, by making more of an attempt, command a premium and are regarded well by the public? Or Stripe?⁸

If the sigmoid were right, just how much more effort would be necessary to elicit such jumps? Orders of magnitude more? I & Said have invested effort, certainly, but there are countless sites (even confining the comparison to just personal websites and excluding sites with professional full-time developers/​designers), whose creators have surely invested more time; millions of books are self-published every year; and Apple is certainly not the only tech company which tries to design things well.

What might be going on is related to the “aesthetic-usability effect”: at a certain level, the design itself becomes noticeable to the user for its esthetic effect and the esthetics itself becomes a feature adding to the experience. That is, at the bottom of the sigmoid, on a website strewn with typos and broken links and confusing colors, the user thinks “this website sucks!”, while in the middle, the user ceases to think of the website at all and just gets on with using it, only occasionally irritated by design flaws; finally, at a certain level, when all the flaws have been removed and the site itself is genuinely unironically beautiful, both the beauty & absence of flaws themselves become noticeable, and the reader thinks, “this website, it is—pretty awesome!” The spike is where suddenly the design itself is perceived as a distinct thing, not merely how the thing happens to be. Designers often aspire to an end-state of sprezzatura or the “crystal goblet”, where they do their job so well the user doesn’t realize there was a job to be done at all—but in this fallen world, where excellence seems so rare, the better one does the job, the more the contrast with all the botched jobs inevitably draws attention.

It is difficult for even the reader least interested in the topic to open a Tufte book, or walk into an Apple store, and not be struck by first impressions of elegance and careful design—which is not necessarily a good thing if that cannot be lived up to. (Any person struck by this must also realize that other people will be similarly impressed, using their own response as a proxy for the general reaction⁹⁠, and will take it as a model for aspiration; liking Apple or Tufte signals your good taste, and that makes them luxury products as much as anything.)

This suggests a dangerous idea (dangerous because a good excuse for complacency & mediocrity, especially for those who do not manage even mediocrity but believe otherwise): if you are going to invest in design, half-measures yield less than half-results. If the design is terrible, then one should continue; but if the design is already reasonable, then instead of there being substantial returns, the diminishing returns have already set in, and it may be a too-long slog from where you are to the point where people are impressed enough by the design for the esthetic effect to kick in. Those moderate improvements may not be worthwhile if one can only modestly improve on mediocrity; and a sufficiently-flawed design may not be able to reach the esthetic level at all, requiring a radical new design.