[Enhancement] Improving the function for calculating difficulty

[user1823]

The current formula for D in FSRS is just a heuristic one. We have already tried a lot of approaches to improve the calculation of difficulty. But all of them failed.

While developing a better function for D, we should keep in mind the following:

• Calculation of difficulty after review should depend upon the retrievability, just like in SuperMemo algorithms. If the R at the time of review is not considered, the model effectively learns Anki's scheduler and not the actual patterns of forgetting. (Past discussion about this can be found here: Calculation of difficulty after review should depend upon the retrievability #248)
• There shouldn't be clustering of heterogeneous reviews at maximum difficulty. Based on the current formula, many of my reviews would have a difficulty of more than 10. Clamping of the difficulty caused the difficulty of heterogeneous reviews to become the same (i.e., 10). (Past discussion about this can be found here: Classification of difficulty is not good enough #262)
• It should try to incorporate the concept of mean reversion within the main function of D.
• If Again is the first rating, it should not have as much impact on the D as the lapses have. Reason: You might have created the card several weeks ago and you have almost no remaining memory of it when you review it for the first time. The extreme case would be where the card was created by someone else, and the first review is the first time you encounter the information. So, an Again rating here doesn't mean that the card is difficult.
• Variants of the easy and hard factors might also need to be included in the formula for D.

[Expertium]

• Calculation of difficulty after review should depend upon the retrievability, just like in SuperMemo algorithms.

While that would be great, all my attempts to integrate R into D have failed, and my attempts to change the formula completely and use a formula based on B-W (like in SM-18) have also failed.

• There shouldn't be clustering of heterogeneous reviews at maximum difficulty.

That depends on the material though. If you are learning something that is very hard, why wouldn't most values of D be close to 10?

• It should try to incorporate the concept of mean reversion within the main function of D.

Honestly, I'm not convinced that reversion to a fixed value is a good idea. I've experimented with reversing to a value that changes, making so that instead of all cards reversing to the same "mean" value, each card would have it's own "mean", but that also didn't work very well.

• If Again is the first rating, it should not have as much impact on the D as the lapses have. Reason: You might have created the card several weeks ago and you have almost no remaining memory of it when you review it for the first time. The extreme case would be where the card was created by someone else, and the first review is the first time you encounter the information. So, an Again rating here doesn't mean that the card is difficult.

That's an interesting idea.

• Variants of the easy and hard factors might also need to be included in the formula for D.

Since optimizable grades didn't work, I'm willing to bet this won't work either, since it's basically the same idea.

Honestly, I'm pretty tired of trying to improve D, there are other things I want to work on, like the matrices.

[hydrogs]

I guess a good way to take $R$ into account is through a surprise function. The less predictable the grade $G$ is, the bigger should be the change in difficulty. This way, when retrievability is low, a lapse should no have as much effect on $D$ as when $R$ is high, since the lapse was not expected. We can say that if the algorithm was right, $D$ shouldn't be updated as it reflected the intrinsic difficulty of the card and vice versa.

A surprise function I've come up with is $A(R, G) = e^{-(R-0.5) \cdot (G-2)}$ ($A$ for astonishment, since $S$ is already taken). When the two parts of the exponent are of the same sign (low surprise), the function is small; and when the parts are different (what happened and what was expected differed), the function will be high.

The way I think this new function should be used is as a new weight in the weighted average that the $D$ function is already. Instead of two parts —the reversion to avoid "ease hell" $D' = D_0(3)$ and the computed $D' = D - w_6 \cdot (G-3)$ —, I would add a third part, keeping the current difficulty as a good one $D' = D$.

In order to use the surprise function as a weight it should be normalize $ke^{-(R-0.5) \cdot (G-2)}$ where $k = e^{-1} = \frac{1}{e^{-(0-0.5)\cdot(4-2)}}$, the largest value the function can take. So it finally remains as $A(R, G) = e^{-1-(R-0.5) \cdot (G-2)}$.

Finally, we can connect the three posibilities this way:

$$D'(D, G, R) = w_7 \cdot D_0(3) + (1-w_7) \cdot \left(A \cdot (D - w_6 \cdot (G-3)) + (1-A) \cdot D \right)$$

I have chosen 2 as the midpoint grade of the surprise function as it is usually not seen as a success nor as a lapse.
Another value, such as 2.5 could be chosen. It is also a good place to insert a new learned weight in order to accomodate to the use of Hard of the different users $A(R, G) = e^{-1-(R-0.5) \cdot (G-2-w_a)}$.

[Expertium]

@L-M-Sherlock please make a version of the optimizer with visible code so I can test this idea? I need it for other testing as well, I can't test much if I can't change anything about the code.

[Expertium]

I tested it using this file (I made a copy): https://colab.research.google.com/github/open-spaced-repetition/fsrs4anki/blob/main/archive/candidate/fsrs4anki_optimizer_beta.ipynb
I added 2 more parameters, like this:
torch.exp(-(retrievability - self.w[15]) * (grade - self.w[16]))
Also, I had to modify how k is calculated, with 2 more parameters it's not just e^-1.
The results weren't statistically significant. The table below shows the average RMSE (normalized so that the RMSE of the original is 100%) and the p-value based on 15 decks/collections.

I'm getting more and more perplexed as to why nothing we do to modify the formula for D improves RMSE. I could make an entire list of ideas related to D, all of which failed.
I still like the idea of using a complex approach to calculate D and then, if it works, trying to find a simple approximation for it.
@L-M-Sherlock how about this: make a new file, but let a neural network (LSTM) calculate the values of D, see if it improves RMSE. If even a neural network can't find better values of D, then we might as well give up.

[L-M-Sherlock]

Maybe we can change our reasoning thread to another way. Why not use a similar method like fitting initial stability to determine the initial difficulty. Is it possible to find out the initial difficulty via the current data and formula? If we make progress in fitting the initial difficulty, we are more likely to find out the formula to update the initial difficulty.

[Expertium]

Initial D doesn't matter as much as initial S. I would really like to see a hybrid model, with all FSRS formulas, except that D is given by an LSTM. Though I'm not sure if it's possible in pytorch, you would have to optimize the parameters of FSRS and of LSTM simultaneously.

[hydrogs]

I have simplified the use of the surprise function and added some more weights.

$$ A(R, G) = e^{- w_{18} \cdot (R - w_{19}) \cdot (G - w_{20})}$$

$$ D'(D, G, R) = D - w_6 \cdot A(R, G) \cdot (G - w_{17}) $$

The surprise function is now just a companion to $w_6$ in order to determine the change rate of $D$.
The $w_18$ is added in order to learn how steep should the surprise function be. It has never been close to $1$ in my optimizations, this may be why previous attemps where no succesful.
In respect of the normalization with $k$, it is not longer needed since it can be done with the learned $w_6$.

I got a decrease in RMSE of 2% with my dataset. Specially a decrease of 5% with a deck that I stopped studying for some time and then returned, so the effect of different retrievabilities is bigger.

I'm not sure if the improvements are big enough. Actually, they should not be big for a dataset in which most reviews are done in time, since there is no variability in retrievability. This change becomes relevant in datasets where there have been a period long enough without studying in order to provide low retrievabilities.

This is the code I used if you want to test it with your datasets:

 def step(self, X: Tensor, state: Tensor) -> Tensor:
        '''
        :param X: shape[batch_size, 2], X[:,0] is elapsed time, X[:,1] is rating
        :param state: shape[batch_size, 2], state[:,0] is stability, state[:,1] is difficulty
        :return state:
        '''
        if torch.equal(state, torch.zeros_like(state)):
            keys = torch.tensor([1, 2, 3, 4])
            keys = keys.view(1, -1).expand(X[:,1].long().size(0), -1)
            index = (X[:,1].long().unsqueeze(1) == keys).nonzero(as_tuple=True)
            # first learn, init memory states
            new_s = torch.ones_like(state[:,0])
            new_s[index[0]] = self.w[index[1]]
            new_d = self.w[4] - self.w[5] * (X[:,1] - 3)
            new_d = new_d.clamp(1, 10)
        else:
            r = power_forgetting_curve(X[:,0], state[:,0])
            a = torch.exp( - self.w[18] * (r - self.w[19]) * (X[:,1] - self.w[20]))
            new_d = state[:,1] - self.w[6] * a * (X[:,1] - self.w[17])
            new_d = self.mean_reversion(self.w[4], new_d)
            new_d = new_d.clamp(1, 10)
            condition = X[:,1] > 1
            new_s = torch.where(condition, self.stability_after_success(state, new_d, r, X[:,1]), self.stability_after_failure(state, new_d, r))
        new_s = new_s.clamp(0.1, 36500)
        return torch.stack([new_s, new_d], dim=1)

[user1823]

Actually, they should not be big for a dataset in which most reviews are done in time, since there is no variability in retrievability. This change becomes relevant in datasets where there have been a period long enough without studying in order to provide low retrievabilities.

Not exactly because three different algorithms are involved here, i.e., Anki SM-2, FSRS v3 and FSRS v4. The reviews were done according to the algorithm in use at that time. Now, all the reviews are analysed using FSRS v4. So, the R at the time of review would vary.

This is the code I used if you want to test it with your datasets:

Can you pull the latest commits from the main branch on open-spaced-repetition/fsrs-optimizer? It would make the comparisons fairer.

[hydrogs]

Can you pull the latest commits from the main branch on open-spaced-repetition/fsrs-optimizer? It would make the comparisons fairer.

Sorry, I forgot to update the branch. I guess I mixed it up in my mind with another fork i I was also updating.

Anyway, the improvement still holds, if you guys want to check with your datasets.

[user1823]

I am not a dev and I am not very familiar with how GitHub works. But, I think that your changes are in the feat/difficulty branch and it is not up-to-date with the main branch at open-spaced-repetition/fsrs-optimizer.

[hydrogs]

omg, my bad 😥. I was pulling into the main branch instead of the feat/dificulty.
Sorry for the mistake, I am not a dev either.

The new commits don't change the result though. So the improvement still holds if you want to check.
I don't think my dataset is representative enough.

[Expertium]

I don't know how to test it the "proper" way, so I tested it my way again: I made another copy of https://colab.research.google.com/github/open-spaced-repetition/fsrs4anki/blob/main/archive/candidate/fsrs4anki_optimizer_beta.ipynb and implemented the code there.
It performed about the same as before, the difference between RMSE of the modified version and the one above was not statistically significant. Again, I'm showing normalized average RMSE, stat. significance tests are done using raw RMSE. I'm mostly using my own decks (I have all kinds of material, both easy and hard) + a few collections from other users.

@L-M-Sherlock I am once again asking you to make a hybrid LSTM-FSRS version where all formulas (such as S for lapses and S for successful reviews) are the same, except that the value of D is given by a neural network (assuming it's possible to optimize such a hybrid in pytorch). This isn't just to satisfy my curiosity; it will answer an important question: "Can we change how the value of D is calculated to improve the accuracy of FSRS without completely changing the formulas themselves?". If RMSE is the same as before, that would indicate that rather than adding more factors and terms to the already existing formula for D, we should completely redefine D. In other words, if even a neural network cannot provide a value of D (between 1 and 10) that improves RMSE, then our attempts won't either.

[L-M-Sherlock]

Why should the neural network be LSTM? I don't know what the input and output are for that LSTM.