This post replaces my old post about benchmarking and I added it to my compendium of posts/articles about FSRS. You do not need to read the old post, and I will not link it anywhere anymore.

First of all, every "honest" spaced repetition algorithm must be able to predict the probability of recalling a card at a given point in time, given the card's review history. Let's call that R.

If a "dishonest" algorithm doesn't calculate probabilities and just outputs an interval, it's still possible to convert that interval into a probability under certain assumptions. It's better than nothing, since it allows us to perform at least some sort of comparison. That's what we did for SM-2, the only "dishonest" algorithm in the entire benchmark. We decided not to include Memrise because we are unsure if the assumptions required to convert its intervals to probabilities hold. Well, it wouldn't perform great anyway, it's about as inflexible as you can get and barely deserves to be called an algorithm.

Once we have an algorithm that predicts R, we can run it on some users' review histories to see how much predicted R deviates from measured R. If we do that using hundreds of millions of reviews, we will get a very good idea of which algorithm performs better on average. RMSE, or root mean square error, can be interpreted as "the average difference between predicted and measured probability of recall". It's not quite the same as the arithmetic average that you are used to. MAE, or mean absolute error, has some undesirable properties, so RMSE is used instead. RMSE>=MAE, the root mean square error is always greater than or equal to the mean absolute error.

The calculation of RMSE has been recently reworked to prevent cheating. If you want to know the nitty-gritty mathematical details, you can read this article by LMSherlock and me. TLDR: there was a specific way to decrease RMSE without actually improving the algorithm's ability to predict R, which is why the calculation method has been changed. The new method is our own invention, and you won't find it in any paper. The newest version of Anki, 24.04, also uses the new method.

Now, let's introduce our contestants. The roster is much larger than before.

FSRS family

​1​)​ ​FSRS v3. It was the first version of FSRS that people actually used, it was released in October 2022. It wasn't terrible, but it had issues. LMSherlock, I, and several other users have proposed and tested several dozens of ideas (only a handful of them proved to be effective) to improve the algorithm.

​2​) ​FSRS v4. It came out in July 2023, and at the beginning of November 2023, it was integrated into Anki. It's a significant improvement over v3.

​3​) ​FSRS-4.5. It's a slightly improved version of FSRS v4, the shape of the forgetting curve has been changed. It is now used in all of the latest versions of Anki: desktop, AnkiDroid, AnkiMobile, and AnkiWeb.

General-purpose machine learning algorithms family

4) Transformer. This neural network architecture has become popular in recent years because of its superior performance in natural language processing. ChatGPT uses this architecture.

5) GRU, Gated Recurrent Unit. This neural network architecture is commonly used for time series analysis, such as predicting stock market trends or recognizing human speech. Originally, we used a more complex architecture called LSTM, but GRU performed better with fewer parameters.

Here is a simple layman explanation of the differences between a GRU and a Transformer.

DASH family

6) DASH, Difficulty, Ability and Study History. This is an actual bona fide model of human memory based on neuroscience. Well, kind of. The issue with it is that the forgetting curve looks like a ladder aka a step function.

7) DASH[MCM]. A hybrid model, it addresses some of the issues with DASH's forgetting curve.

8) DASH[ACT-R]. Another hybrid model, it finally achieves a nicely-looking forgetting curve.

Here is another relevant paper. No layman explanation, sorry.

Other algorithms

9) ACT-R, Adaptive Control of Thought - Rational (I've also seen "Character" instead of "Control" in some papers). It's a model of human memory that makes one very strange assumption: whether you have successfully recalled your material or not doesn't affect the magnitude of the spacing effect, only the interval length matters. Simply put, this algorithm doesn't differentiate between Again/Hard/Good/Easy.

10) HLR, Half-Life Regression. It's an algorithm developed by Duolingo for Duolingo. The memory half-life in HLR is conceptually very similar to the memory stability in FSRS, but it's calculated using an overly simplistic formula.

11) SM-2. It's a 35+ year old algorithm that is still used by Anki, Mnemosyne, and possibly other apps as well. It's main advantage is simplicity. Note that in our benchmark it is implemented the way it was originally designed. It's not the Anki version of SM-2, it's the original SM-2.

We thought that SuperMemo API would be released this year, which would allow LMSherlock to benchmark SuperMemo on Anki data, for a price. But it seems that the CEO of SuperMemo World has changed his mind. There is a good chance that we will never know which is better, FSRS or
SM-17/18/some future version. So as a consolation prize we added something that kind of resembles SM-17.

12) NN-17. It's a neural network approximation of SM-17. The SuperMemo wiki page about SM-17 may appear very detailed at first, but it actually obfuscates all of the important details that are necessary to implement SM-17. It tells you what the algorithm is doing, but not how. Our approximation relies on the limited information available on the formulas of SM-17, while utilizing neural networks to fill in any gaps.

​

Here is a diagram (well, 7 diagrams + a graph) that will help you understand how all these algorithms fundamentally differ from one another. No complex math, don't worry. But there's a lot of text and images that I didn't want to include in the post itself because it's already very long.

Here's one of the diagrams:

SM-2 is not included because it wasn't designed to predict the probability of recall.

Now it's time for the benchmark results. Below is a table showing the average RMSE of each algorithm:

https://preview.redd.it/szfdss44e0uc1.png?width=547&format=png&auto=webp&s=a4b903d93c6374b0013a8019ab4ae066d2ea2b47

I didn't include the confidence intervals because it would make the table too cluttered. You can go to the Github repository of the benchmark if you want to see more details, such as confidence intervals and p-values.

The averages are weighted by the number of reviews in each user's collection, meaning that users with more reviews have a greater impact on the value of the average. If someone has 100 thousand reviews, they will affect the average 100 times more than someone with only 1 thousand reviews. This benchmark is based on 19,993 collections and 728,883,020 reviews, excluding same-day reviews; only 1 review per day is used by each algorithm. The table also shows the number of optimizable parameters of each algorithm.

And here's a bar chart (and an imgur version):

Lower is better.

Black bars represent 99% confidence intervals, indicating the level of uncertainty around these averages. Taller bars = more uncertainty.

Unsurprisingly, HLR performed poorly. To be fair, there are several variants of HLR, other variants use information (lexeme tags) that only Duolingo has, and those variants cannot be used on this dataset. Perhaps those variants are a bit more accurate. But again, as I've mentioned before, HLR uses a very primitive formula to calculate the memory half-life. To HLR, it doesn't matter whether you pressed Again yesterday and Good today or the other way around, it will predict the same value of memory half-life either way.

The Transformer seems to be poorly suited for this task as it requires significantly more parameters than GRU or NN-17, yet performs worse. Though perhaps there is some modification of the Transformer architecture that is more suitable for spaced repetition. Also, LMSherlock gave up on the Transformer a bit too quickly, so we didn't fine-tune it. The issue with neural networks is that the choice of the number of parameters/layers is arbitrary. Other models in this benchmark have limits on the number of parameters.

The fact that FSRS-4.5 outperforms NN-17 isn't conclusive proof that FSRS outperforms SM-17, of course. NN-17 is included just because it would be interesting to see how something similar to SM-17 would perform. Unfortunately, it is unlikely that the contest between FSRS and SuperMemo algorithms will ever reach a conclusion. It would require either hundreds of SuperMemo users sharing their data or the developers of SuperMemo offering an API; neither of these things is likely to happen at any point.

​

Caveats:

1. We cannot benchmark proprietary algorithms, such as SuperMemo algorithms.
2. There are algorithms that require extra features, such as HLR with Duolingo's lexeme tags or KAR³L, which uses not only interval lengths and grades but also the text of the card and mildly outperforms FSRS v4 (though it's unknown whether it outperforms FSRS-4.5), according to the paper. Such algorithms can be more accurate than FSRS when given the necessary information, but they cannot be benchmarked on our dataset. Only algorithms that use interval lengths and grades can be benchmarked since no other features are available.

​

References to academic papers:

1. https://scholar.colorado.edu/concern/graduate_thesis_or_dissertations/zp38wc97m (DASH is first mentioned on page 68)
2. https://www.politesi.polimi.it/retrieve/b39227dd-0963-40f2-a44b-624f205cb224/2022_4_Randazzo_01.pdf
3. http://act-r.psy.cmu.edu/wordpress/wp-content/themes/ACT-R/workshops/2003/proceedings/46.pdf
4. https://github.com/duolingo/halflife-regression/blob/master/settles.acl16.pdf
5. https://arxiv.org/pdf/2402.12291.pdf

References to things that aren't academic papers:

1. https://github.com/open-spaced-repetition/fsrs-benchmark?tab=readme-ov-file#fsrs-benchmark
2. https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Metric
3. https://supermemo.guru/wiki/Algorithm_SM-17

Imgur links:

1. https://imgur.com/a/ZhsXaZi
2. https://imgur.com/a/V8u0wcD
3. https://imgur.com/a/fVxiJvx