Changelog

2013-09-152023-01-31 in progress certainty: log similar bibliography

Following my writing can be a little difficult because it is often so incremental. So every month, in addition to my regular subreddit submissions, I write up reasonably-interesting changes and send it out to the mailing list in addition to a compilation of links & reviews (archives).

2022

February 2022

• Gwern.net: backlinks & similar-links now recursively pop up; experimental Ar5iv use; wrote GPT-3-using paragraphizer.py tool & used to add line-breaks to ~1k annotations; link icon system rewrite (faster, better+test-suite, >220 new link icons); live link popup system rewrite (similarly, >850 domains to popup);

2021

October 2021

• compiled 40 subject-area tags

June 2021

• Refactored 60 pages
• LinkAuto.hs: a Pandoc library for automatically turning user-defined regexp-matching strings into links (discussion)

• Proposals:

March 2021

• Gwern.net: mobile “popins” are finally enabled! (example); new Wikipedia popups (this 7th implementation enables recursive WP popups)

February 2021

• Gwern.net: popups: can now be moved, stickied, and full-screened (another step towards our ambition of Windows-95-in-the-browser!)

2020

December 2020

• Gwern.net: recursive link annotations (memoized for efficiency); cross-page/​document popups (demo); syntax highlighting of source code popups (demo); directory indexes

November 2020

• On Development Hell
• Gwern.net: dark-mode rewrite complete (fixes page load flash & laggy scrolling) arabesque navigation bar in footer with JS keyboard shortcuts; IBM Plex Mono font & custom ALGOL-like syntax highlighting for code blocks; new sun⁠/​moon icons for horizontal rulers; images in Wikipedia popups; new internal link/​citation convention for multiple citations

October 2020

• more Gwern.net fixes of infelicities & outdated infrastructure: major ones: line-breaking on slashes, hyphenation & justified text on Chrome desktop browsers using Edward Kmett’s hyphenation at compile-time1⁠; width-full mobile images; rewrote GoodReads conversion for much more readable book review page⁠; browsable directories; popup annotations now render more like body.

Dark mode users should have noticed it’s improved considerably; most of the other fixes either are more subtle (consistency, correct formatting, edge-cases, bold rather than italics), or rewriting old hacks to pay off technical debt. In general, it should just look nicer and more pleasant, and be a little easier for me to write.

Minor ones:

• I wrote up things we’ve tried & abandoned

• rewrote GoodReads conversion for much more readable book review page⁠; began compiling all movie/​TV/​opera reviews into a single page⁠; began copying all MAL anime/​manga reviews & localizing into a single page⁠; all GR reviews have been moved to Gwern.net permanently

• dark mode:

• popup whiteness fixed
• image handling fixed (images are inverted & colorspace-rotated based on automated heuristic + manual tagging); tweaks like highlighting to make inline code blocks readable
• brightness decreased to enable a pure black background without the white foreground being blinding
• new test /  ​demo page: revealed a number of minor bugs, makes it much easier to test changes for correctness

• popup annotations:

• popup annotations now render more like body: they have link icons, paragraphs, properly indented lists, multi-column support etc

• annotations (titles/​authors/​abstracts) are auto-smallcaps like the body, making them more consistent and reducing the annotation toil in adding smallcaps spans

• automatic annotations get some rewrite rules like ‘N=’ → ‘n =’

• annotations can now be ‘definitions’ (popup annotations for non-link text)

• popups can now be arbitrary IDs, not just sections or annotations (same as the reverse-footnote popups)

• links pop up with a ‘context’, to support my new citation convention: every citation is either a fulltext hyperlink with annotation, or a link back to another use of that citation with more context/​discussion. (This exploits the new automatic ID generation so a link like [Foo et al 2020](/docs/ai/2020-foo.pdf) gets an HTML ID like #foo-et-al-2020, making it extremely easy to link elsewhere like for more details on the scaling laws, see the earlier discussion of scaling papers like [Kaplan et al 2020](#kaplan-et-al-2020).)
• tooltips are shorted versions of link annotations, as a fallback for non-JS or mobile users

• bolded abstracts (manual & automatically)

• Arxiv abstracts now parsed as LaTeX first (fixing a lot of weirdness); wrong publication dates bug fixed & reported upstream (upstream refuses to fix, so forked)

• PLOS/​PubMed⁠/​Wikipedia abstracts formatting greatly improved (and fixed upstream)

• MedRxiv support

• <video> tag support

• removed broken reverse search link

• better author truncation

• author initials get periods (“John H Smith” → “John H. Smith”; surprisingly common in a lot of bibliographic metadata)

• rebuilt all Wikipedia & Arxiv link annotations to bring up to date and with new formatting

• Wikipedia annotations now include image thumbnails, which get auto-inverted like regular images for dark mode viewers
• automatic annotations stored as YAML; more consistent, simpler implementation, and makes a lot easier to copy automatic annotations into the manual annotations for improvement

• annotations can now be selected/​copy-pasted (this was disabled previously, for unclear reasons)

• refactored the site infrastructure into a separate repo

• mobile:

• simplified mobile appearance, reduced italics over-usage
• width-full (edge-to-edge) mobile images (uses ~20% more horizontal space)
• typography: rewrite pass to insert <wbr> to enable line-breaking on slashes; rewrite pass to insert soft hyphens for hyphenation & justified text on Chrome desktop browsers using Edward Kmett’s hyphenation

• fixed Safari double hyphen bug, reported upstream
• revised list hierarchy to use bold (ie. bold → smallcaps → italics; smallcaps → italic → bold or smallcaps → bold → italic made no visual-emphasis sense, and I think it resulted in smallcaps overuse); switched ordered lists to uppercase Roman numerals to avoid confusion with lowercase alphabetic numerals; added a third-level icon, and made ordered/​unordered lists & blockquotes cycle through 3 styles based on depth

• exploiting that new metadata about page sequencing, a JS widget will automatically load the next/​previous page if you are at the bottom/​top of a page & hit a down/​up key twice within 0.5s
• simplified sidenotes appearance (removed double-lining, merged the number box with the main box)

• auto-smallcaps: catches more acronyms/​initialisms, and rewritten to avoid deeply nested duplicate spans (an unfortunate consequence of substitution + Pandoc AST walking, requiring a switch to bruteforce string substitution); headers no longer smallcaps (which looked weird)

• metadata blocks are compiled to HTML; before, they were inlined as raw Markdown⁠, which was unfortunate, but tricky to fix

• index is now a generated page rather than a pile of manually-edited HTML, so it will no longer be perpetually out of sync in style/​content

• custom asterism based on the list icons instead of horizontal ruler

• inflation adjusters: tooltips were wrong; Bitcoin adjuster now interpolates or extrapolates for missing exchange-rate dates instead of crashing

• browsable directories on Gwern.net & TWDNE (eg.  /  ​docs /  ​)

• changed all dates to YYYY-MM-DD for consistency

• cut down wasted whitespace margin in lists/​blockquotes (which looked especially ridiculous on mobile), and headers

• added missing link icons for 10+ domains, fixed spacing & overlapping, and harmonized opacity of all link icons; fixed Substack background, OpenAI size

• Pandoc now uses Mathjax directly, enabling colored equations

• collapse sections have ellipses

• ‘abstract’ class generalized, so pages can have multiple abstracts, simplifying formatting (the manual drop-caps+smallcaps are then redundant)

• internal links to a previous or later link ID (usually a section) now have an arrow icon pointing up or down (respectively) as a navigation aid

• experimental use of instant.page prefetching

• tag pages now in columns

• improved admonition icons (ugly or didn’t show up in dark mode, or at all on Android) & spacing

• the <code> syntax highlighting (outlined box) looks better inside links

• removed dead CSS & code

• shortened page descriptions & titles

• updated last-modified metadata on most pages

• checked all pages for MIME types and status, revealing a number of infinite redirect errors

• WWW archive: no crawl headers; force UTF-8 charset header for Markdown sources (for Chrome)

• all Twitter.com → Nitter (static, more reliable)

• converted simple LaTeX formulas to HTML where possible; added spaces to various equalities/​inequalities

• fixed blurriness on highlighting/​selecting text

• margin notes are now selectable/​clickable (z-index error)

• set up /r/​mlscaling as a more convenient way to browse “Matters of Scale” topic links

Known outstanding bugs:

• no popup annotations for mobile

• some users report popups ‘cycle’ rapidly in and out of existence; we have been unable to replicate this and don’t know why it happens

• dark mode is slow, because browsers implement filters like inversion very slowly. Sorry.

• dark mode is slow to load, yielding a ‘flash of white’

Injecting CSS doesn’t work because browsers seem to ignore it until after the page loads, with the exception of setting a class, which however breaks all the existing dark mode CSS.

• margin notes overlap with sidenotes⁠, and sidenotes may be overlap as well (desktop)

• sidenotes are laid out even when they’re in collapsed sections

• popup annotations may not correctly display LaTeX/​math

• compiled all movie & anime reviews

September 2020

• Gwern.net: dark mode image handling fixed (images are inverted based on automated heuristic + manual tagging); annotations can now be ‘definitions’ (popup annotations for non-link text); expanded tooltips as fallback for link annotations; bolded abstracts & revised list hierarchy to use bold; changed all dates to YYYY-MM-DD for consistency; added missing link icons for 7 domains, fixed spacing & overlapping, and harmonized opacity of all link icons; Pandoc now uses MathJax directly, enabling colored equations; internal links to a previous or later section now point up or down (respectively) as a navigation aid; experimental use of instant.page prefetching; asterism instead of horizontal ruler; simplified mobile appearance; many miscellaneous bug fixes

2019

February 2019

• “Origin of ‘Littlewood’s Law of Miracles’”

• Gwern.net CSS/​HTML/​JS changes: click-to-zoom images (using image-focus.js); headers are now self-links; Tufte CSS-style epigraph support; Table of Contents: Wikipedia-style section numbering, margin & size tweaks, lightweight subset of Source Sans Pro (for Mac users); nicer diamond list icons; sleeker sidebar (especially nice on mobile); PDF/​internal/​section links are now annotated with icons; borders on tables, image figures, and blockquotes; old-style numerals in text & tabular numerals in tables; justified text (but not in Chrome due to decade-old lack of hyphenation); narrowed maximum body-width in characters & made line-height responsive to body-width (hopefully addresses the perennial complaints that pages are always too wide/​too narrow/​lines too close); quote highlighting disabled by default; collapsible code-blocks; inline smallcaps support; optimized SVG logo & favicon; page-specific CSS overrides enabled; list paragraph bugs in Pandoc fixed; compressed JPEGs; changed code syntax-highlighting scheme to match overall esthetics better; miscellaneous responsive design/​mobile improvements

• image-focus.js (JS⁠/​CSS): release of new, correct, lightweight, dependency-free JS library written by Obormot for implementing “click to zoom” on images (useful for large images/​graphs)

2018

March 2018

• moved newsletter from MailChimp to TinyLetter due to exorbitant MailChimp fees

2017

September 2017

• Gwern.net updates: Google AdSense banner ads removed (due to initial analysis of A/​B test results); URL scheme changed to replace spaces by hyphens & delete commas/​apostrophes (due to persistent user error); footer moved to sidebar; re-tagging pages; font/​CSS tweaked for faster loads & wider margins; hosted documents reorganized & expanded with personal archives; began buying & scanning all cited books to provide fulltexts; added sitemap generation to expose fulltexts to search engines

March 2017

• Gwern.net: HTTPS now mandatory; HTML sections rewritten using HTML5 semantic markup for hopefully better compatibility with screen readers & offline browsers like Pocket/​ReadItLater; “importance” metadata added to all pages to rank them; “belief” metadata renamed more intuitively as “confidence”; finish adding 301 Redirects for all broken links & common typos; renamed & all darknet market pages for current terminology; rewrote Patreon profile

2015

June 2015

• rewrote Gwern.net CSS to be mobile-friendly; should now be readable in an iPhone 6 browser
• wrote two summer poems⁠, on earthworms and the rain

2014

December 2014

Nothing finished.

October 2014

QS:

• Redshift self-experiment: screen-reddening software shifts bedtime forward by 20 minutes
• LLLT re-analysis: no change in sleep as hypothesized by another LLLT user
• analysis of sceaduwe’s spirulina/​allergies self-experiment (no reduction in allergies)

September 2014

QS:

Stats:

Technology:

• My window tracker/​time-logger of choice is arbtt which records X window info for later classification and analysis; but one of the challenges is you don’t know how to set up arbtt or improve your environment or write classifications rules. So I wrote a tutorial.

• Time-lock crypto: wrote a Bash implementation of serial hashing time-lock crypto, link to all known implementations of hash time-lock crypto

July 2014

QS/​self-experiments:

Statistics:

Debunking:

Misc:

Site:

Bitcoin:

Tech:

January 2014

Darknet markets:

• I have begun systematically spidering all operational English darknet markets

General:

Statistical:

Site:

• rewrote configuration for Hakyll 4
• converted site’s Darcs repository to Git & hosted on GitHub

2013

Darknet markets:

Statistical:

Personal:

Site:

November 2013

Another busy month:

October 2013

My focus for October was coping with the fallout from the bust of Silk Road—dealing with the revelations, copying the SR forums, tracking down leads, talking to various people, recording the ensuing arrests, tracking the darknet markets popping up in its wake… I still have much material to work through, but some things I managed to do included:

I decided to post my most extensive self-experiment yet, on LSD microdosing⁠. While there was a lot of criticism, I still regard it as worthwhile and setting a new benchmark for any future research in that area.

My anti-linkrot system benefited from comments on Hacker News telling me how to use archive.today⁠; this may help me out quite a bit in the future.

A/​B testing has been active since Hacker News traffic furnished large sample sizes:

August 2013

• A/​B testing: the line-height test found no difference, so I did a quick one where I tested an empty test to check the A / ​B testing tool I’m using⁠; I successfully failed to reject the null. The next test is whether underlining hyperlinks annoys people or not.

• Book reviews: I wrote a Haskell program to parse my GoodReads ratings & reviews into flat Pandoc Markdown; it works somewhat well, it seems to be eating blockquotes & neutering hyperlinks, I’m not sure why. Was also an opportunity to clean up some reviews: inline some of them, spellcheck, expand references & links, which was a lot of work. But it’s nice to have my reviews gathered somewhere with a readable interface. kiba thinks the work may pay for itself in affiliate revenue with Amazon, but I’m skeptical.

• Scholz’s Radiance: Added a hundred pages or so. Annotating some of it is quite difficult; Scholz’s familiarity with Wagner’s operas is a challenge, since I’ve only ever read his Ring Cycle.

• I’ve started two new self-experiments:

• Touhou music growth rate: made a little more progress on Touhou music, with an analysis # of releases vs time: seems like we may’ve reached peak Touhou in 2009.

• Silk Road mirrors: I’ve started hosting public copies of subsets of the darknet markets; these are backups for particular incidents or timeseries

• Spaced repetition statistics: I’ve been analyzing my Mnemosyne data and the giant public database for time of day effects. While my results aren’t conclusive, my analysis of 48m flashcard reviews from the public database finds that the best time to study your flashcards seem to be noon. A little surprising, you’d think that late at night, before bedtime, would be the best time.

• My forgotten cleaning methods like Sand polls aren’t done because I got a much lower rate of responses to the polls than I was hoping for and only got enough responses in the final poll the other day.

July 2013

Interesting things I wrote during July:

• 2013 Lewis meditation quasi-experiment: A Quantified Selfer and a few other guys did some meditation while doing an arithmetic game; turned out to be a perfect application for multilevel modeling
• Sleep and lunar phases: A recent paper claimed that there’s a phase-of-the-moon effect on circadian rhythms; since I have so much sleep data on myself, I thought I’d see if there’s any effect…
• Sand: continues to progress; I closed the LW poll and set up 3 new polls on Gwern.net to test problems with the original poll.
• Betting Made a list of things I’ve bet on or at least tried to bet on with people (as opposed to prediction market use). Disappointingly short.
• Scholz’s Radiance: I’ve started transcribing and annotating one of my favorite tech/​lit novels. It’s mostly done.
• Cicadas for dinner: I finally got around to eating the cicadas I caught during the most recent Maryland emergence; so of course I had to write up this outré dining.

Right now, I’m listening through my Reitaisai 10 downloads (more Touhou music work); and working with this coach who is interested in predicting triple-jump performance by college athletes, and has collected a bunch of data about triple-jumpers.

June 2013

Hm, what did I get done this June…

It was a little boring, honestly; jury duty was a mental distraction where I couldn’t plan to do anything but I ultimately wound up going in once and not being picked up for the jury! So I spent a lot of time simply digging up fulltext papers for citations, so at least there’s now something like another 100 papers available online for melatonin/​nicotine/​modafinil etc…

• I was thinking of trying to meta-analyze the correlation of lithium in drinking-water with suicides/​murders/​mental-illness, but after I got copies of all the citations I knew of⁠, I’m not sure the data is homogenous to do that, which is disappointing, and the meta-analysis papers/​textbooks I’ve checked don’t seem to be very encouraging about the utility of doing it with epidemiology stuff.
• And reorganizing & fixing broken links & updating various pages (I figured out how to make a fun forest plot showing the active/​passive split in dual n-back studies)
• My long-running font A/​B test finished but with the most boring possible results of close to zero difference between the 4 fonts
• I researched an old family friend in his 90s who has never been willing to talk about his government work during the Cold War and found some stuff using released Census records, but that’s not really of interest to other people, and I decided to not make it public. Likewise when I added ~40 book reviews from my old notes to my Goodreads account.
• I managed to trace an arrested drug dealer back to his Silk Road account, which was somewhat interesting: Reddit discussion
• Is one familiar with Fukuyama & ‘the end of history’? I think he’s right but no one seems to agree with me, so I wrote a short essay defending him Like most political essays, it’s probably worse than I realize.
• From my perspective, probably the most interesting thing I wrote all month was some criticism of early SF⁠, pointing out the obsolete science behind some stuff written off as fantasy.

May 2013

• I’ve made a little more progress on the Touhou project

• Added 2 new studies to the DNB meta-analysis and also a new covariate (whether payment reduces gains: it doesn’t)

• updated my analysis of SDr’s sleep data

• oh wait, I did do a new project: applied survival analysis to modeling fiction reviews

And in my Google analysis (which I credit to last month and not May, even if it went viral in early May), I added in random survival forests⁠, fixing that gap in evaluating prediction methods, which was a little burden of guilt off my mind.

While I was at it, I reproduced that recent paper analyzing Bitcoin exchange shutdown or theft risk. The dude’s since given me his source code. (What can I say? Survival analysis is a great hammer, and it cost me enough tears and sweat to learn how to use the R library that I plan to use it everywhere I can.)

• I started a little Noopept self-experiment using the Noopept someone gave me, but unfortunately they gave me too little for the results to be very meaningful (see the power analysis); but maybe they will have a trend and I can try a bigger experiment later.

• TruBrain sent me a month’s supply of their all-in-one nootropic⁠, but I haven’t tried it yet because it would interfere with the Noopept. I also purchased magnesium l-threonate, which is a disappointment so far, and some nicotine patches, which I haven’t used yet. A small sample order from a new modafinil website selling the usual Indian Modalert is in progress but hasn’t arrived yet. (It ultimately did not arrive as the delivery required a signature.)

• I have a weird little literature / ​historical / ​survey article in progress; we’ll see where that goes

• I posted an analysis I wrote a few months ago in private of whether a particular vendor on Silk Road is a federal mole (probably not)

Actually, I guess it was overall a pretty productive month. Probably helped that jury duty has so far turned out to be a bust: I’ve been on call since 21 May but have yet to actually go into the courthouse to do anything.

2012–2013

Original

“But where shall wisdom be found?
And where is the place of understanding?
Man knoweth not the price thereof;
neither is it found in the land of the living
…for the price of wisdom is above rubies.”

Job 28:12 (KJV translation)

Here is material I’ve worked on in the 477 days since my last update. In roughly chronological & topical order, here are the major additions to Gwern.net:

Transcribed or translated:

More technical:

Personal:

2011

Original

“I’d like to poison your mind
With wrong ideas that appeal to you
Though I am not unkind…”

Per my past practice of linking stuff I think LWers will find interesting, here is what I’ve been up to lately:

1. “Politicians are Ethical”: applying base-rate neglect
2. Modafinil: a dual-pronged argument for low-balling estimates of modafinil harm by pointing to both temporal and quality discounting of health in old age. (If you missed it the first time around, I still think my mini-tutorial on drug ordering with Bayes’s theorem is worth reading.)
3. “The Narrowing Circle”: argument that the usual belief of ‘moral progress’ and the ‘expanding circle’ assume many of their conclusions by pointing to the beliefs and classes of entities discarded along the way. (As many LWers share those assumptions and will be unsympathetic, the interesting parts may be the appendices on perpetuities and waqfs, inasmuch as those bear directly on cryonics⁠.)
4. Both Modafinil and Spaced repetition have been expanded with scores more links to studies & PDFs. (Nicotine and Melatonin are next.)
5. “Worldbuilding: The Lights in the Sky are Sacs” is a silly bit of SF/​alternate history speculation involving floating hydrogen-sac organisms.
6. “Wikipedia and Knol” has been completed, as the 7 predictions I made on the matter have been judged thanks to Google’s recent announcements; I blew one⁠.
7. Stuff which is incomplete or which is just a pile of notes:
8. I stuck a link in the footer of every page to a Google spreadsheet form, borrowing the idea from Luke Muehlhauser - I’ve only gotten 1 feedback so far, IIRC, but that was before I put it in the footer and updated all the pages a few hours ago. (As of 2015, there are hundreds of responses and I consider the feedback form to have paid its way; see my later writeup detailing the benefits.)

1. Since removed in April 2021 when Chrome desktop support for hyphenation became sufficiently widespread.↩︎