Support ECMP route configuration

[yushoyamaguchi]

Background

The current CNI Static IPAM plugin does not explicitly define or document support for ECMP (Equal-Cost Multi-Path) routing when multiple routes with the same destination prefix are provided.
I would like to clarify and formally support ECMP route configuration as part of the plugin behavior.

Proposal

I think two possible approaches:

Option 1: Implicit ECMP via multiple routes

Allow users to define multiple routes with the same destination prefix but different gateways, and install them as ECMP routes inside the Pod.

      "routes": [
        { "dst": "0.0.0.0/0", "gw": "10.0.0.1" },
        { "dst": "0.0.0.0/0", "gw": "10.0.0.2" }
      ]

Note:

• This is closer to a bug fix for the current inconsistency between IPv4 and IPv6 behavior than to a completely new feature.
• This behavior already works for IPv6 in practice. (ref: https://github.com/torvalds/linux/blob/v6.19/net/ipv6/ip6_fib.c#L1152-L1162) This was misleading. Sorry. Although this behavior currently works in IPv6, it does not appear to be intentionally supported.
• We propose to extend and standardize this behavior for IPv4 as well. This was misleading. Sorry. I want to implement that plugin handles this way of writing and intentionally inserts ECMP route in both v4 and v6.
• I think this should be clearly documented as supported behavior.

Option 2: Explicit ECMP syntax

Introduce a dedicated syntax in the routes field to explicitly define ECMP routes.
This could improve clarity and avoid ambiguity in interpretation.
However, this approach would require extending the current API/schema.

Use case

Our use case is as follows:

• In an OpenStack environment, we create multiple routers connected to the same L2 network.
• This network is attached to worker VMs.
• Inside the VM, we attach an additional interface to Pods using ipvlan (as a secondary interface).
• For this interface, we want to configure ECMP routes where multiple OpenStack routers act as gateways.
  This requires the ability to define multiple equal-cost default routes in the CNI configuration.

[yushoyamaguchi]

#1248
Previously, I implemented a test assuming that ECMP would be achieved with a configuration like the following:

      "routes": [
        { "dst": "0.0.0.0/0", "gw": "10.0.0.1" },
        { "dst": "0.0.0.0/0", "gw": "10.0.0.2" }
      ]

However, after further verification, I found that for IPv4 this does not actually result in ECMP routes being installed.

default via 10.0.0.1 dev eth1 
default via 10.0.0.2 dev eth1

What I initially observed as ECMP behavior was in fact caused by a different mechanism, and I mistakenly interpreted it as ECMP.
In addition, the logic used to determine whether an ECMP route had been installed in the FIB(in my previous unit-test) was also incorrect.

Given this, I would like to revert the previous PR(#1248) that introduced the test based on this incorrect assumption.

I'm so sorry.

[yushoyamaguchi]

cc @squeed

I'm sorry for mentioning.

[yushoyamaguchi]

After plan is fixed in the community, I'll implement it by myself.

[s1061123]

Thank you for the issue.

I'd like to clarify following items:

• In 'Option 1: Implicit ECMP via multiple routes', your comment seems to mention IPv4/v6 linux kernel configuration. Does this option include to modify Linux network kernel stack to change? (unfortunately CNI project is not directly connected Linux kernel community, so if you want to change IPv4 kernel network stack, it should be discussed in Linux kernel community).
• Is there any specific reason why you want to implement in static IPAM plugin only? It could be better to implement this feature outside of static CNI plugin, such as more common side because ECMP configuration could be applied not only static IPAM, also other IPAM plugins (e.g. host-local or whereabouts)

[yushoyamaguchi]

Answer to question1

I'm sorry. what I wrote about Option1 was misleading. (I've fixed the way of writing.)

Currently, ECMP works unintentionally for IPv6 due to Linux kernel behavior, while it does not work for IPv4. I plan to stop handling routes to the same prefix by simply inserting them into the FIB one by one.
Instead, when multiple routes to the same destination prefix are specified in routes, the plugin itself will handle them so that an ECMP route is intentionally installed into the FIB.

Concretely, I plan to modify this part of the code:
https://github.com/containernetworking/plugins/blob/v1.9.1/pkg/ipam/ipam_linux.go#L123-L154

[yushoyamaguchi]

Answer to question2

There was no intention to limit this idea to the static IPAM plugin specifically.
I mentioned the static IPAM plugin simply because that is what our current use case uses.
I agree that, conceptually, ECMP route handling could be implemented in a more common/shared layer so that it can also work with other IPAM plugins.

[yushoyamaguchi]

In addition

Maybe, Option2 requires change of CNI spec, so I want to choose Option1 is it is possible.

[yushoyamaguchi]

@s1061123
#601
#615
76ef07eb
I found that @mccv1r0 maybe intended to add multiple routes for the same prefix.
From the wording of the PR, I felt that he had the similar motivation as I do. (@mccv1r0 Is that right?)

[s1061123]

Hi,

First, I understand your ECMP use-case and ECMP support could be done in CNI, but your option1 has several concerns.
These are two major concerns at least.

Concern 1. Option1 breaks current static IPAM behavior and hence it introduces regression.

Currently static IPAM accepts two routes with same dst, as follow:

      "routes": [
        { "dst": "0.0.0.0/0", "gw": "10.0.0.1" },
        { "dst": "0.0.0.0/0", "gw": "10.0.0.2" }
      ]

In this case, CNI configures the routes in container like that:

#ip -4 route
default via 10.1.1.2 dev eth0
default via 10.1.1.1 dev eth0

This means 'configure two routes with same destination, with different gw'. This is 'valid linux network configuration' as kernel accepts.

(here's the ECMP configuration case. This output is different from above.)

# ip route
default
        nexthop via 10.1.1.1 dev eth0 weight 1
        nexthop via 10.1.1.2 dev eth0 weight 1
# ip -6 route
(snip)
default metric 1024 pref medium
        nexthop via fe80::f0a7:81ff:fea3:2d04 dev eth0 weight 1
        nexthop via fe80::f0a7:81ff:fea3:2d06 dev eth0 weight 1

So Option1 changes IPAM plugins' behavior to change ECMP, hence it causes regression. In addition, previous use-cases (two routes with same dest, with different gw) is no longer supported.
In addition, this data structure will be explicitly different from Linux kernel network stack (i.e. CNI's ECMP data structure is different from Linux kernel). This may cause unnecessary big misunderstanding from user side.
At least to me, 'adding a new feature' should not change current functionality. So it should be a big concern.

Concern 2. CNI Result object does not have a way to represent ECMP as well.

The CNI plugin returns 'CNI result' object in ADD command. This result object contains the configured interface status, including IP routes as well as IP address.
Currently CNI spec does not define how ECMP route is represented. This result object is also used in STATUS command, so it is also a concern. At least, once CNI supports ECMP, CNI Spec should mention how to define ECMP in Result object. Actually, there is no document in CNI Specification as well as CONVENSTIONS doc.

As mentioned above, I fully understand ECMP functionality is required/valid hence CNI support it. It should be implemented appropreately.
Introducing new argument, 'gws' in routes solves 1st concerns (gw and gws should be exclusive), for example. In this case, keeping current functionality CNI introduces ECMP support.

      "routes": [
        { "dst": "0.0.0.0/0", "gws": [ "10.0.0.1", "10.0.0.2" }
      ]

[yushoyamaguchi]

We are also considering implementing the same functionality on openshift/route-override-cni#68.