Machine Learning Support

[Marco-Congedo]

Hello,

following this discussion, the steps followed by PosDefManifold.jl for doing classification in the tangent space are:

1. parallel transport of all points to the center of the manifold (identity for the positive definite matrices manifold). This involves the computation of a center of mass of the points in the manifold and a function for parallel transport.

2. Exponential map (projection on the tangent space)

3. vectorization (a special one, for example, for the pos def manifold a weight √2 is given to the off-diagonal elements)

I am not familiar with other manifolds. Are all these operations possible with all currently implemented manifolds?

[kellertuer]

Thanks for you idea! I would like to shortly comment

1. I don't understand this point. Do you mean you compute mean of your given data points to choose a tangent plane? then center would refer to the Riemannian Center of mass? This however would not require a parallel transport, really just the computation of the mean (see for example our current implementation here; actually we currently provide 2 methods a third is in one of the PRs: GreadientDescent, GeodesicInterpolation and CyclicProximalPoint.

2. I think you mean the inverse exponential map (usually referred to as logarithmic map)? The exponential map maps from the tangent bundle to the manifold. More generally it might be usefull to phrase this in terms of (inverse) retractions.

3. I don't follow this one. Why do you have to do this and what does it actually do?

The first two should already be widely available – but an implementation would anyways just use the interfaces from ManifoldsBase. And a manifold that wants to use the ML methods should implement the necessary.

Finally I would like to discuss whether this is in scope of the package. I don't want to say, that it's not nice stuff, just that ManifoldsML.jl might in some time also be a good idea to start. But of course we can first start here and with enough stuff to cover (maybe even combining with Manopt.jl for the optimization, though Manopt has still to be ported to ManifoldsBase.jl), we can do a spin of.

[Marco-Congedo]

Hi,

1. so, we compute a mean as a base-point for projecting onto the tangent space. This is just an educated guess, not necessarely the optimal base-point, but it works well in practice.

Important, we do not project on the tangent space directly at this point, rather, we first parallel transport all points to the center of the manifold (identity). In the pos def manifold this can be done directly on the manifold (see here and reference therein). This step is crucial to achieve good machine learning performance and also instrumental to operate transfer learning. For a general manifold, what we need is to

• compute the mean
• project onto the tangent space
• paraller transport to the center of the manifold

2. Yeah, sorry, i meant the logarithmic map.

3. In the pos def manifold, tangent 'vectors' are symmetric matrices. Machine learning models take as input feature vectors, that is why tangent 'vectors' are vectorized so as to get feature vectors. Since those matrices are symmetric, only one triangle is vectorized and to compensate for this a weight equal to √2 is given to the off-diagonal elements.

In summary, what is needed is

• an algorithm to estimate a mean
• the logaritmic map
• the parallel transport
• an appropriate definition of vectorization, which will depend on the manifold

Are all ingredients (besides the last one) available for all manifolds in Manifold.jl?

As per the scope, i would keep it in a separate package, something like ManifoldLearn.jl.
Actually, PosDefManifold.jl is an appropriate skeleton (it is Python-free, differently from sciKitLearn.jl).
All it is needed as far as i see is to implement an interface to Manifold.jl for these operations (currently this is done by PosDefManifold.jl for the pos def manifold only.)

PosDefManifoldML.jl also implements machine learning models acting directly on the manifold. For those, all is needed is a function for computing the mean of several points and a distance function.
Thus, to be complete, all is needed is:

• a distance function
• an algorithm to estimate a mean
• the logaritmic map
• the parallel transport

[kellertuer]

Thanks for the clarifications. I still don't follow the “center”, if its not the mean. So your “center” is always the identity? As the neutral element of the Lie group it has nice properties for SPDs, however, it is in general not available. I am a little stuck on your terms – you can only parallel transport tangent vectors not points. So to clarify: With m the mean and e the center, do you
log into the tangent space at m and then parallel_transport to e? I am not yet sure why, but if so a general algorithm needs maybe a replacement for e, since such a point is not necessarily available (in its prominence as the unit matrix), for example on the sphere. Still an algorithm could have a center which indicates your favourite point (and its tangent space), that could be set to the mean (if nothing prominent is available) and would be set to center=e on SPD.

Concerning the functions (distance, mean, log, parallel transport), they are all available for nearly all manifolds that we have currently implemented (where I would still suggest to to log more general and use an inverse_retraction that defaults to using the log, but that's a technical detail. For the vectorisation I also have a rough idea what to do, but that would be something quite ML-specific maybe.

Concerning the scope: I think it's a great idea to start a package like ManifoldML.jl or ManifoldLearn.jl (I would prefer the first name actually). It would implement algorithms using ManiofldsBase.jl (which provides all necessary function interfaces) and maybe Manopt for optimizations in between – and use Manifolds in its concrete examples/tutorials. I'd happily help and provide input to that!

[Marco-Congedo]

Thats great Ronny,

 i don't have time now but i will start working on that in January/February.

I will definitely need your input!

To clarify, yes, of course i am talking about transporting tangent vectors in the expected way as you describe, however one nice property of the pos def manifold is that the same transformation applies to points on the manifold, as explained here and proved in the reference therein. This is just a tachnical point, forget about it.

It is good enough to have a preferred point for parallel transporting the tangent vectors, which would be the identity for the pos def manifold and something else for the others. By the way, the reason why this works well is that when you have another set of points, independent from the first, hence with a different mean, by parallel transporting it to the identity as well makes the model you already computed a workable model. There are several works showing this and i can point to them if someone is interested.

All the best

[kellertuer]

Ah, I missed that detail. Thanks for clarification.

If you start working, just give me a note and I'll hapily help.

Concerning the working best – I trust you, but it would also be fine to “parallel transport the model” from one tangent space to another in order to be able to compare. We should collect such literature in the documentation, too.

[mateuszbaran]

Great, I'd definitely like to see support for ML on manifold-valued data 👍 . I can help with this as well 🙂 .

Regarding that vectorization, there are a few different ways this can be done. We have the vee function that does that (but most manifolds don't have this function), Another option is through an orthonormal basis of the tangent space at a point. We have a few ideas about providing this functionality (see JuliaManifolds/Manifolds.jl#66 ).

[Marco-Congedo]

Great Mateusz, i will let you know in this thread when i find some time for this. Looking forward to hearing your and Ronny's ideas on the vectorization when the time comes. And, by the way, thanks for the CovarianceEstimation.jl package. It's very handy!

[mateuszbaran]

Nice to hear that you like CovarianceEstimation.jl.

With JuliaManifolds/Manifolds.jl#86 merged, we have vectorization, so Manifolds.jl is ready now 🙂 .

[kellertuer]

Hi @Marco-Congedo,
I hope you're doing fine these days.

We have for a while a get_coordinates which vectorises a tangent vector in any tangent space on any manifold with respect to a basis, so the vector is even of the same size as the manifold dimension.

I haven't had the time to look closely at PosDefManifoldML.jl, but if you have time we could start a ManifoldML, I think? As far as I see we now have all tools you need available within ManifoldsBase.jl.

[Marco-Congedo]

Hello Ronny,
i have been thinking about this and i meant to write you my thoughts. Thanks for sparkling the discussion again then. Here is what i have been thinking: once you have vectorized tangent vectors, STANDARD machine learning tools applies. Therefore, PosDefManifoldML.jl is not the best way to empower Manifold.jl with ML capabilities, although it can also be used since it also supports vectorized tangent vectors as input. However, many more ML models are available in general ML packages. Right now in Julia it is hot MLJ.jl. My guess is that linking Manfold.jl to MLJ.jl would be a much more valuable endeavor for the community.

[kellertuer]

Hallo Marco,
ah, then I am not sure what PosDefManifoldML.jl does in total (but I also did not check too closely), since I hoped that it would roughly do what you write, but yes, if you decide for a tangent space and coordinates it's classical ML, though only locally of course! It heavily depends on the chosen tangent space.

But yes, for the pure linear part, coupling it to the best ML package is an approach I would've taken, too. Currently, I am a little busy (and not too familiar with MLJ neither).

[Marco-Congedo]

Hello, i am not familiar with MLJ.jl either. PosDefManifoldML.jl handles directly positive definite matrices, with (so far) a classifier in the PD manifold and two classifiers in the tangent space (handling the tangent space and vectorization step automatically). However, this only works for the PD manifold.

[kellertuer]

Then it would also be interesting, whether the classifier on the PD manifold can begeneralized to other manifolds – does it mainly use geodesics? “Just” the Euclidean case is – I think – not a package, since it would just require the steps above, logs/transports/coordinates, and MJL. The three steps before are all just one line each.