Proposal: stable result interface or result-builder hook for downstream domain packages

[Harumaru58]

Hi Ronny,
I hope you had a peaceful easter break!
While working on stabilizing our package, I have a suggestion to make since you previously encouraged. :)

Here goes. Our package exposes custom public result types such as CPDResult, BTDResult, and ApproxResult. These wrap Manopt solver outputs into decomposition-friendly results with fields like components, factors, reconstruction helpers, and solver metadata.

At the moment, from upstream, downstream packages need to call solve(...), inspect the returned result object, pull out fields like point / cost / grad norm / iterations / converged / solver info, and then convert that into their own result type. This works, but it means downstream packages end up depending on the concrete shape of Manopt result values rather than on a stable interface.

Would you be open to providing a stable extension point for custom result construction?

What we need is not tensor-specific result types in Manopt, but a stable way for downstream packages to build their own typed results without depending on internal result layout.

A few possible shapes that would help:

• stable accessors like solution_point(result), solution_cost(result), solution_grad_norm(result), solution_iterations(result), solution_converged(result), solution_solver(result)
• a result_builder hook in solve(...)
• or a small stable summary result type with a guaranteed schema for those common fields

Any of those would make it much easier for downstream domain packages to wrap Manopt results safely and consistently.

Thanks!

[kellertuer]

Hi,

thanks for the ideas!
I did indeed not have any pause for easter, since I am still in Tokyo (travelling back this week) – and here there is no Easter. But when I am back home, I make sure to take a break.

Some of the things you want to have are already there:

Accessing “Things from the state” like the minimiser p*

Since for the high-level interfaces (so not calling solve! but using gradient_descent!) you can “activate” to get more information instead of just the minimiser (setting return_state = true) there is also the stable and documented get_solver_result(state) – which (a) for a state accesses the solver result (thought that is oftenn really just state.p) – and (b) for everything else than the state it is the identity. This way it is stable independent of whether you had return_state = true or not. See https://manoptjl.org/stable/plans/state/

Further things to access (also documented on the state page)

• get_iterate sounds nearly synonymous with the get_solver_result (and is called by the result thingy), but acts a bit more low level
• get_gradient to access a last evaluated gradient if the solver has this (often this is state.X)
• has_converged to indicate whether the reason the solver stopped was a convergence certificate

Further notes:

• the gradient norm can not that easily be accessed here, since we are missing the manifold, but you can compute it yourself

• solution_iterations(result) what do you want to get here? The final number of iterations? That can be recorded, see https://manoptjl.org/stable/plans/record/, but by default that is not stored

• solution_solver(result)

  I am not sure what this refers to, so what are your trying to get access to?

• `a result_builder hook in solve(...)

  I am in general not a fan of “hooks” in the sense of “here is a magic place where a function is called you can override”. What do you want to accomplish here?

• or a small stable summary result type with a guaranteed schema for those common fields

We could do an access function (I am not creative enough for a name before first coffee), but it would either (a) only contain the iterate, gradient and convergence and hence be very dull – or be the whole state anyways. So I am unsure what to do here.
There is status_summary(state) (and also a lot of other types from Manopt) which is an internal function – it will change its signature soon, you can sneak into the dev-0.6 branch; this prints a human readable summary of the solver run; basically also what you see when you call a high-level interface with return_state = true(or solve! on a state) in REPL,

If you look at the whole set of solvers, there is not so many common things to access. Beyond the ones I already explained here, I do not have one in mind; even the gradient might not be present and errors then (for gradient free methods like LTMADS, NelderMead or particle swarm)

Accessing “Things from the problem”

In general a “task” in Manopt (something you want to solve and a solver) is split into the AbstractManoptProblem (see https://manoptjl.org/stable/plans/problem/) and the state from before. In the problem you can access the manifold and the objective; with the objective you can of course call all the get_cost(M, objective p) ... listed for the objectives at https://manoptjl.org/stable/plans/objective/#Specific-Objective-typed-and-their-access-functions

Summary

I hope this helps a bit, but for now I am unsure where the interface is missing something. If we miss something, then sure, we can add those.
If for example one of the mentioned functions is not exported, we should do so. I do consider these functions stable interface, i.e. do not plan (for the time being) to change their names or signatures.

[kellertuer]

Ah, for the iteration number, there probably exists something easier: The stopping criteria started storing an at_iteration a while back, which is 0 or negative if they have not yet indicated to stop; and I think by now that is consistently available. so one could write a function that

• returns this value for a simple stopping criterion
• returns the maximum for the StopWhenAll (because only when all indicate, this indicates)
• returns the minumum (filtering nonpositive first) if it exists for the StopWhenAny (since then one of them fired)

I can think about a name for such a function.
For all other points from above I would first need more feedback on what you actually need, but most accessors should be there – since with DebugState and RecordState such accessors where necessary anyways.

[Harumaru58]

Thank you for the detailed information.
As we are approaching a pre-alpha release of our package, things have been a bit hectic, so I did not follow up earlier.

I think, on my side, I had not fully realized that these accessors exist. I should read more from Manopt documentations.

However, to clarify the downstream use case a bit more concretely: in our package, as you already know, roughly, we are already happily using Manopt as an optimization backend and wrapping solver runs into our own typed tensor-decomposition results such as CPDResult and BTDResult.

Concretely, we already do the following downstream:

• call high-level Manopt solvers with return_state = true,
• extract the final iterate,
• extract / compute convergence-related information such as gradient norm and iteration count,
• and convert that into tensor-specific result objects with factors / blocks / reconstruction helpers / metadata.
  Up to now, I have partly been handling these pieces on our side, rather than relying consistently on the existing upstream state/problem accessors.

Looking ahead, we also want to use Manopt for more advanced models, in particular BTD on Grassmann/Stiefel manifolds, where we would like to experiment with Gauss-Newton, quasi-Newton, and trust-region methods.

Regarding your second thread, yes, that would already cover what we need on the iteration side. What we want is exactly the final iteration index at which the stopping criterion triggered, so the semantics you describe for simple criteria, StopWhenAll, and StopWhenAny sound right to us.

Thank you again.

[kellertuer]

That sounds like an exiting time for you! But no worries, this is all not urgent.

For “finding” the accessors, it might be that the documentation is – while probably quite complete – maybe not in the best of organisation. The whole docs grew slowly since the start (oh nearly 10 years ago now), so maybe it is time to reorganise them. I can think about how to make the features like accessors “more findable”.

For GN – that one we currently do not have I think. But I saw Paul has a paper about it. Maybe that would be a neat project for a master student to look into Riemannian GN in Julia?

[Harumaru58]

For Riemannian GN, I will still look into it. :)

[mateuszbaran]

We already have Riemannian Levenberg-Marquardt, so GN can be obtained by just setting penalty to 0? One thing definitely worth exploring there would be adding a Riemannian dog leg method.

BTW, we have some major improvements to quasi-Newton in #554 , I think they are worth taking into account in experiments.

[Harumaru58]

Wow, nice! Thank you. I will closely look into quasi-Newton now. There will be definitely sth to learn for me. (Since I am a new into the field of RO, even nonlinear algebra in general, so for LM and GN, I need to get familiarized with all.)

First, I will adapt your quasi-Newton for our BTD and check for the improvement of the performance.
I will let you know how it goes if you want.

[Harumaru58]

We already have Riemannian Levenberg-Marquardt, so GN can be obtained by just setting penalty to 0? One thing definitely worth exploring there would be adding a Riemannian dog leg method.

BTW, we have some major improvements to quasi-Newton in #554 , I think they are worth taking into account in experiments.

#554 is still open, right?

[mateuszbaran]

Yes, it's still open because the PR is slightly breaking (mostly different default values) and we want to make it a part of a larger breaking release. It should be safe to try out though (you can dev Manopt in Pkg and then check out that branch). The big improvements are 1) better management of (s, y) pairs thanks to incorporating Byrd's discard rule, 2) adding a state-of-the-art line search (a slightly better variant of Hager-Zhang from LineSearches.jl) and 3) a nice way of handling mixed Euclidean bounds + Riemannian constraints -- almost on par with LBFGSB.jl in pure Euclidean problems.

[Harumaru58]

Thx for the info. I've quickly checked the doc and I think all the ingredients for our BTD to work with your L-BFGS (M, f(x), grad_f, and initial_point) are there. But, since BTD will depend heavily on the initialization, I will combine the Alternating Least Square warm-initialization step to find the best basin, then apply :lbfgs solver to check for the improvement of the relative error.

2. adding a state-of-the-art line search (a slightly better variant of Hager-Zhang from LineSearches.jl)

btw, Manopt.WolfePowellLinesearch: is this the state-of-the-art line search? I might change the previous Armijo line search to this for all other solvers and see!

Thanks for the good improvement.

[mateuszbaran]

btw, Manopt.WolfePowellLinesearch: is this the state-of-the-art line search? I might change the previous Armijo line search to this for all other solvers and see!

Maybe in some niche cases it's the best one but generally it's not. If you can quickly evaluate directional derivatives (inner products of gradient and a custom direction), which which should be the case most of the time, the best one currently available in Manopt.jl is CubicBracketingLinesearchStepsize. It can be about 2-3x times faster than Wolfe-Powell.

[kellertuer]

There we are, v0.5.36 introduces https://manoptjl.org/stable/plans/state/#Manopt.stopped_at