Adding target normalization in addition to feature normalization in `BaseOptimizer`

[llfung]

Currently, BaseOptimizer handles the normalization of feature library, before the sparse regression take place. It then renormalize the learnt parameters according to the saved norm of each feature in the library as a post-processing step after the sparse regression is called.

However, in some optimizer, such as EvidenceGreedy in BINDy, the optimizer also require the target to be normalized. This is currently being handled inside the specific optimizer. However, it is easy to see that normalizing both feature library and targets have some mathematical advantage, especially in making hyperparameters such as the threshold in STLSQ, as well as the regulariser scaling parameters, more problem and scale independent.

However, it should be stressed that normalizing target will lead to breaking changes. For example, when applying classic SINDy a multi-dimensional dynamical system (say Lorenz63), normalizing target means the effective (scaled) threshold will be different for each target dimension, as different target has different scale. Hence, we recommend that this should be an optional change.

Given the breaking change, we believe this should be discussed in the community before implementation. Welcome to chip in your opinion on this change.

[Jacob-Stevens-Haas]

Thanks Lloyd, this was a great discovery. I think I ran into this last night working on the weak form. In that case, the norms of different LHS targets and RHS features depend upon subdomain size (a weak hyperparam), so only normalizing one side exacerbates the optimizer hyperparameter scale problem.

If we do this inside BaseOptimizer_, would EvidenceGreedy need that info internally? If so we could set tgt_scale_ and feature_scale_ on the optimizer. Otherwise it could be handled transparently.

[llfung]

Right now, EvidenceGreedy do use the norm to normalise sigma2 correspondingly. However, we can surely implement it in such a way that the scaling is done at either the TemporalNoisePropagation or at the normalisation level. I don't think it's a hard thing to do to move some of the code around.