Manopt does not work with Float32 data

[rsnumerics]

Manopt does not work with Float32 data. This has to do with default function arguments all specified as Float64 in the code. The default argument should be converted to the correct data type, say R(1.0), where R is the type parameter. Machine learning models tend to use Float32...

Explicitly specifying default function arguments is not a workaround, because this still causes problems internally.

For example the modified trust regions test fails straight away (this is what I am mainly interested in)

julia> include("test_trust_regions.jl")
Riemannian Trust-Region: Test Failed at ???/Tests/test_trust_regions.jl:15
  Expression: trust_regions(M, f, rgrad, rhess, p; max_trust_region_radius = -0.1f0)
    Expected: ErrorException
      Thrown: MethodError
      MethodError: no method matching var"#trust_regions!#1049"(::ExponentialRetraction, ::StopWhenAny{Tuple{StopAfterIteration, StopWhenGradientNormLess{typeof(norm), Float64, Missing}}}, ::Float32, ::Float32, ::Bool, ::typeof(copyto!), ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::Float64, ::@NamedTuple{}, ::TrustRegionModelObjective{AllocatingEvaluation, ManifoldHessianObjective{AllocatingEvaluation, typeof(f), typeof(rgrad), typeof(rhess), Manopt.var"#trust_regions##6#trust_regions##7"}}, ::DefaultManoptProblem{Fiber{ℝ, ManifoldsBase.TangentSpaceType, PowerManifold{ℝ, Grassmann{ℝ, ManifoldsBase.TypeParameter{Tuple{3, 2}}}, Tuple{Int64}, ArrayPowerRepresentation}, Array{Float32, 3}}, TrustRegionModelObjective{AllocatingEvaluation, ManifoldHessianObjective{AllocatingEvaluation, typeof(f), typeof(rgrad), typeof(rhess), Manopt.var"#trust_regions##6#trust_regions##7"}}}, ::StopWhenAny{Tuple{StopAfterIteration, StopWhenResidualIsReducedByFactorOrPower{Float64}, StopWhenTrustRegionIsExceeded{Float64}, StopWhenCurvatureIsNegative{Float64}, StopWhenModelIncreased{Float64}}}, ::TruncatedConjugateGradientState{Array{Float32, 3}, Float32, StopWhenAny{Tuple{StopAfterIteration, StopWhenResidualIsReducedByFactorOrPower{Float64}, StopWhenTrustRegionIsExceeded{Float64}, StopWhenCurvatureIsNegative{Float64}, StopWhenModelIncreased{Float64}}}, typeof(copyto!)}, ::Base.Pairs{Symbol, AllocatingEvaluation, Nothing, @NamedTuple{evaluation::AllocatingEvaluation}}, ::typeof(trust_regions!), ::PowerManifold{ℝ, Grassmann{ℝ, ManifoldsBase.TypeParameter{Tuple{3, 2}}}, Tuple{Int64}, ArrayPowerRepresentation}, ::ManifoldHessianObjective{AllocatingEvaluation, typeof(f), typeof(rgrad), typeof(rhess), Manopt.var"#trust_regions##6#trust_regions##7"}, ::Array{Float32, 3})
      The function `#trust_regions!#1049` exists, but no method is defined for this combination of argument types.
      
      Closest candidates are:
        var"#trust_regions!#1049"(::AbstractRetractionMethod, ::StoppingCriterion, ::R, ::R, ::Bool, ::Proj, ::R, ::R, ::Any, ::R, ::R, ::Any, ::R, ::R, ::R, ::R, ::Any, ::Any, ::Any, ::StoppingCriterion, ::AbstractManoptSolverState, ::Base.Pairs{Symbol, V, Nothing, NT} where {V, NT<:NamedTuple}, ::typeof(trust_regions!), ::AbstractManifold, ::O, ::Any) where {Proj, O<:Union{AbstractDecoratedManifoldObjective, ManifoldHessianObjective}, R}
         @ Manopt ~/.julia/packages/Manopt/vVYNl/src/solvers/trust_regions.jl:432
      
      Stacktrace:
       [1] trust_regions(M::PowerManifold{ℝ, Grassmann{ℝ, ManifoldsBase.TypeParameter{Tuple{3, 2}}}, Tuple{Int64}, ArrayPowerRepresentation}, mho::ManifoldHessianObjective{AllocatingEvaluation, typeof(f), typeof(rgrad), typeof(rhess), Manopt.var"#trust_regions##6#trust_regions##7"}, p::Array{Float32, 3}; kwargs::@Kwargs{evaluation::AllocatingEvaluation, max_trust_region_radius::Float32})
         @ Manopt ~/.julia/packages/Manopt/vVYNl/src/solvers/trust_regions.jl:382
       [2] trust_regions(M::PowerManifold{ℝ, Grassmann{ℝ, ManifoldsBase.TypeParameter{Tuple{3, 2}}}, Tuple{Int64}, ArrayPowerRepresentation}, f::Function, grad_f::Function, Hess_f::typeof(rhess), p::Array{Float32, 3}; evaluation::AllocatingEvaluation, preconditioner::Function, kwargs::@Kwargs{max_trust_region_radius::Float32})
         @ Manopt ~/.julia/packages/Manopt/vVYNl/src/solvers/trust_regions.jl:345
       [3] top-level scope
         @ ~/Documents/Tests/test_trust_regions.jl:6
       [4] macro expansion
         @ ~/.julia/juliaup/julia-1.12.5+0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined]
       [5] macro expansion
         @ ~/Documents/Tests/test_trust_regions.jl:15 [inlined]
       [6] macro expansion
         @ ~/.julia/juliaup/julia-1.12.5+0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:774 [inlined]
       [7] macro expansion
         @ ~/Documents/Tests/test_trust_regions.jl:15 [inlined]

All code is here: Archive.zip

[kellertuer]

Hi.
Thanks for using Manopt.

with default function arguments all specified as Float64 in the code.

That is a bit imprecise, there might be a few older functions, where some arguments are still Float64, but newer ones are very often a generic type R<:Real.
Just that with 1-2 people developing here actively in their free time, about 10 years of grown code (trust regions is about 7 years old) and >10k lines – it is not so trivial to have all those places in mind.

If you see how to fix your case, a PR would be super helpful, since I am for now still busy with a few PRs (about 4-5 throughout the JuliaManifolds ecosystem) where I fix other reported issues.
If not, I will come back to this once time permits :)

If you could narrow the example down to an MWE that would also be helpful.

[rsnumerics]

Yes, it is a tedious thing to fix. I'll try to find out how to create PRs...

The minimal example is

using Manifolds, Manopt
M = Euclidean(3, 3)
p = randn(Float32, 3, 3)
f(M::Euclidean, p) = sum(p .^ 2) / 2
grad(M::Euclidean, p) = p
hess(M::Euclidean, p, X) = X
trust_regions(M, f, grad, hess, p; max_trust_region_radius = 0.1f0)

I just thought that your own test would be the best start: https://github.com/JuliaManifolds/Manopt.jl/blob/master/test/solvers/test_trust_regions.jl and that is what I have attached.

[mateuszbaran]

Hi! We are definitely interested in improving Float32 (and also CUDA) support, though currently Manopt is a bit rough there. I will make a PR with a fix, I think it's enough if you make minimal examples that fail.

[mateuszbaran]

OK, with #583 trust_regions works with Float32 on both CPU and CUDA 🙂

[rsnumerics]

Many thanks, it works. I figured out how to test: ] add https://github.com/JuliaManifolds/Manopt.jl #mbaran/fix-rtr-float32

I will soon test cubic regularisation...

[kellertuer]

Yes, it is a tedious thing to fix. I'll try to find out how to create PRs...

It is a bit of “back then I (maybe we) did not know better”, now I do, so for new parts I am more careful, but then there is still legacy code.

The MWE is great! As you see Mateusz could directly use it in the tests so we make sure it also works in the future.
And yes, if you even fix code and do PRs that would be absolutely great!
But we also are here for questions and guidance on that :)