[Feature]: Centrally assign indices to daemons and update the Compute Domain Clique state.

[sakura-3]

Component

compute-domain-controller

Problem Statement

After a training workload is submitted under a compute domain, all compute-domain-associated daemons must negotiate a unique index for each daemon. This index is used to generate DNS configuration. Only after that can the imex process start in each daemon, and the daemon pod becomes Ready.

Only when all daemons associated with the compute domain are Ready can the PrepareResourceClaims phase of the compute-domain-kubelet-plugin complete, allowing the training pods to start.

In large-scale training scenarios, we have observed slow pod startup times. The main bottleneck is the negotiation process described above. The negotiation is based on optimistic locking: each daemon updates the status of the same compute domain object, which leads to a large number of conflicts and update requests.

This high request volume triggers API server APF throttling (code 429) and Retry-After: 30s, while update conflicts (code 409) further amplify the problem.

Related Progress

• Move CD status updates to centralized controller #810

  This PR moved the logic for negotiating indexes and updating the compute domain from distributed daemons to a centralized compute domain controller. The expectation was to avoid conflicts and reduce request volume.

• If the number of pod replicas for the job associated with ComputeDomain is large, the pod startup will be very slow. #816

  However, this issue points out implementation problems in Move CD status updates to centralized controller #810. The controller workqueue suffered from backlog accumulation. In testing, a 256-pod training job required 8 minutes to start.

• CD daemon: use SSA for conflict-free nodes list updates #822

  This PR still used distributed negotiation, but replaced update with server-side apply (SSA). During implementation, we found that naive SSA could result in two nodes being assigned the same index, which is invalid.

  As a result, we had to re-check index assignment after SSA and re-patch any conflicting nodes. In other words, retries were still required.

• Maintain CD daemon info in a new per-CD, per-clique ComputeDomainClique object #826

  This PR introduced the ComputeDomainClique(CDC) object, dividing one compute domain into multiple isolated domains. Index negotiation was then scoped to a single clique (at most 18 nodes).

  This proved to be highly effective. We ported the relevant DRA logic into fake-kubelet for stress testing. The results showed that at a scale of 5k+ nodes, DRA time was reduced to approximately 1 minute.

  However, during stress testing, we still observed 429 responses even after increasing some APF-related parameters. This indicates that the request volume is still too high and there is further room for optimization.

  In addition, fake-kubelet has a simpler control flow than the real kubelet, so the measured latency is optimistic. Our goal is to reduce DRA latency to the second level.

• CDClique: SSA per-node status, conflict retry on cleanup, scoped pod selection #1101

  This draft PR proposes using SSA to update clique objects. Unfortunately, we ran into the same problem as in CD daemon: use SSA for conflict-free nodes list updates #822. To mitigate it, we again had to apply the same workaround: re-check after SSA and re-patch if conflicts were detected.

Proposed Solution

Based on the lessons learned so far, we can draw the following conclusions:

• The CDC is a great idea. Further hierarchical partitioning may also be worth considering, for example: [Feature]: Hierarchy CDs(Divide and Conquer) or a new boundary beyond one NVLink partition to improve the performance for large scale workload #1093. However, more hierarchy also means more complexity.
• Compared with update, SSA merely shifts conflict detection from the API server to the daemon. It does not eliminate redundant update requests, so it provides limited benefit for our problem.
• Centralized index allocation and status updates are expected to reduce the number of update requests and avoid APF issues. In addition, after introducing CDC, we can parallelize across CDCs, which means multiple workqueues and may help avoid the backlog issue seen in Move CD status updates to centralized controller #810.

I suggest revisiting the centralized index allocation approach, with the help of CDC.

Component Changes

Compute Domain Kubelet Plugin

The DRA plugin needs to add an additional cliqueID label to each node, which helps the controller build clique domains.

Compute Domain Daemon

The daemon no longer needs to manage CDC lifecycle operations such as creation, update, deletion, or index registration logic (currently handled by DaemonInfoManager in the code).

The daemon only needs to watch the CDC object associated with itself, read the assigned index from it, generate DNS configuration, and start the imex process. When an ipset change is observed, it should regenerate the configuration and restart imex.

Compute Domain Controller

The controller will now be responsible for allocating indexes and updating CDC and CD status. Concretely, the controller will watch cd, node, daemonset, and pods.

When the controller starts, it watches nodes and builds both the clique domains and the node -> index mapping. The node -> index mapping is mostly stable and is only updated when nodes are added or removed.

cliqueID	nodeName	index
clique-1	node-1	0
clique-1	node-2	1
clique-1	node-3	2
clique-2	node-4	0
clique-2	node-5	1

When allocating mappings, a simple first-fit strategy is used. On restart, the controller first reconstructs the clique domains and mappings from existing CDC objects.

The controller spawns one cdcWorker per CDC. Workers across different CDCs run in parallel, while within each cdcWorker, events are processed serially via a workqueue.

Taking the lessons from #810 into account, daemon pod events are dispatched to the corresponding cdcWorker based on a nodeName → clique mapping. This is a fast, lightweight operation and therefore will not cause backlog accumulation at the controller.

Tasks handled by cdcWorker include:

• Updating the daemon pod IP received from events into the CDC
• Watching the corresponding daemonset, and when Ready = DESIRED, updating the CDC status to Ready

After all CDCs are Ready, the controller updates the CD status to Ready.

Batch Update

In the centralized update design, we can further reduce request volume by using batch updates. Given the structure of CDC, each individual update request is essentially updating or appending one entry in the daemons slice. We can merge multiple update requests inside the controller and flush them in batches.

Estimated Optimization Impact

Using a 5k-node scenario as an example, and consider using a batch size of 5 in the new design.:

Scenario	Step	Request Count	Total
Current	Update CDC to register DNS index	5000 * 3~5 retries = 15000~25000	25000~40000
Current	Update CDC status = Ready	5000 * 2~3 retries = 10000~15000	25000~40000
Optimized	Update CDC to register DNS index	5000 * 0.2~1 = 1000~5000	2000~10000
Optimized	Update CDC status = Ready	5000 * 0.2~1 = 1000~5000	2000~10000

Alternatives Considered

No response

Scope

Large: New feature gate, API change, or multi-component change

Upstream Kubernetes Dependencies

No response

Additional Context

No response

[shivamerla]

Thanks @sakura-3 for nicely summarizing different approaches to support ComputeDomains at scale. This proposal aligns with my thinking as well as i have commented here. Worth to hash out more details and get the initial implementation out to verify with virtual nodes at scale.

[sakura-3]

@shivamerla
I've drawn a sequence diagram to illustrate the workload startup process under the new proposal. It's a complex process, and I'm not particularly good at diagramming, so please let me know if anything is unclear.

I have discarded a previous flawed idea: building the topology based on cliques and nodes and assigning indexes to nodes upon the controller startup and before processing any pod events. I originally assumed this topology would be stable and could facilitate index allocation for daemon pods. However, it turns out that this topology is actually dynamic, and we inevitably have to deal with a dynamic world.

I read your comment.

Each daemon can continue using SSA for its own row (nodeName, ipAddress, status), but stop assigning index locally (getNextAvailableIndex). The controller, as a single writer with the full clique view, computes and assigns unique indices centrally.

There is a minor difference between our proposals. In your design, (nodeName, ipAddress, status) is still updated by the daemon via SSA. However, I propose that the controller handles all of these updates. Fundamentally, the difference lies in who is responsible for merging the cdc.Daemons array: the API server or the controller. I believe the controller is the better choice because we want to avoid overwhelming the API server.

[herb-duan]

Thanks for the sequence diagram, it helps clarify the proposed startup path.

I have two clarification questions:

1. In step 4.2 ensure CDC object exists, is the expectation that each node-local DRA plugin ensures/creates the CDC object, or should this be owned by the compute-domain-controller?

Is there a specific reason this needs to be done by the DRA plugin?

2. In step 4.2 wait for CDC Ready, does “CDC Ready” mean the whole CDC is Ready, or only the current node’s daemon row is Ready?

If it means the whole CDC, that may introduce unnecessary clique-wide blocking: one slow daemon pod could delay workload startup on other nodes in the same clique. I think the more precise condition for the workload prepare path should be: wait until the current node’s daemon row in the CDC becomes Ready.

Minor wording point: since CDC does not currently have a separate status subresource, maybe “CDC status” here means the readiness state stored in each per-node daemon row.

[shivamerla]

I like this direction much better since it avoids making distributed daemon coordination the primary scaling mechanism.

Right now the design depends on many daemon-controllers concurrently updating shared objects and relying on optimistic locking/retries to converge. That works at smaller scale, but at larger scale it starts becoming “designing around 409s” instead of reducing contention.

Moving index allocation + CDC coordination into a central controller gives a much cleaner ownership model:

• single allocator per CDC
• deterministic update sequencing
• fewer concurrent writers
• simpler daemon logic

The per-CDC worker model also feels like the right scalability boundary since it preserves parallelism across cliques without uncontrolled distributed updates within a clique. I also think the controller should own CDC object creation directly instead of depending on daemon-side coordination. On readiness semantics, I agree with @herb-duan point as well, the plugin should just watch for each daemon status in the CDC to be updated and marked ready before allowing the workloads to run.

[klueska]

If you do this you might as well get rid of the CDC altogether. The whole point of introducing it was to share the load, so that a single component was not responsible for making all the updates. It feels like a step-backwards to be considering something like this.

Independently, these two comments in the description of this issue are incorrect:

Only when all daemons associated with the compute domain are Ready can the PrepareResourceClaims phase of the compute-domain-kubelet-plugin complete, allowing the training pods to start.

The PrepareResourceClaims call on each node is unblocked as soon as its local IMEX daemon is marked as ready. We used to have to wait for all daemons to become ready, but this was relaxed in v25.8.0 with the introduction of the IMEXDaemonsWithDNSNames feature gate.

In large-scale training scenarios, we have observed slow pod startup times. The main bottleneck is the negotiation process described above. The negotiation is based on optimistic locking: each daemon updates the status of the same compute domain object, which leads to a large number of conflicts and update requests.

Individual daemons never touch the compute domain object. They only touch their local ComputeDomainClique object. The centralized controller is then responsible for syncing this info back to the ComputeDomain object for (1) backwards compatibility, and (2) to mark the whole compute domain as ready. This "readiness" marker is not consumed anywhere by the system and is only kept-around for human users to quickly look and see if all daemons that have attempted to be started have come online until now.

[sakura-3]

I'm working on it and have made substantial initial progress. During development, I identified and corrected several design flaws. @shivamerla @herb-duan You are right — the controller should be responsible for ensuring the CDC object exists and for updating the CDC status once the daemon is ready.

Previously, I mocked the project's NVIDIA-dependent interfaces, allowing the project to run in a non-GPU environment and simulate most of the workflow. Through this approach, I will now move into the testing phase. After that, stress testing will be conducted to evaluate the effectiveness of this improvement. Hopefully everything goes smoothly.

@klueska I still believe the CDC object is necessary. It provides a natural concurrency boundary. Daemon events across the cluster will be dispatched to their corresponding ComputeDomainCliqueWorker, with different ComputeDomainCliqueWorker instances running in parallel. I believe this can prevent the message backlog issue observed in #810.

[jgehrcke]

Great discussion!

About few general remarks about potential changes we make to the central controller.

We can merge multiple update requests inside the controller and flush them in batches

Absolutely! This will be a highly effective strategy. Kevin and I also discussed this in January. Let's make use of this idea wherever it makes sense, please. Here's a little trick how I like to think about this: a batching strategy after all should introduce a new time constant and guarantees an upper bound for the request mutation frequency. When built correctly, it does not delay any work to be done when there's little overall work to do. Two extremes: if basically no work is happening, a mutation request is emitted right away. If other mutations were made in a small time window leading up to "now" then anticipate more incoming work and wait before sending the next batch. Can be thought of as customized rate-limiting; and can be much smarter than the throttling that's built into kube client. Goal would be to never trigger the self-throttling of the client.

per-CDC worker model

Not exactly sure what these workers would do -- but any "per-CDC worker model" does AFAIU result in W concurrent workers, with W typically being just one order of magnitude smaller than O(N) (with N being the number of nodes).

For the O(10**4) nodes challenge, we'd have O(10**3) workers, and if I have followed correctly we'd also have as many queues.

Especially when there is a lot of work to do per worker (i.e., a lot of work overall) then this architecture may create more problems than it solves. I think we need smart batching and a hence opportunity for serialization to ensure we optimize for throughput and make wise use of API server resources (we must find ways to minimize any kind of resource starving).

It's really important that we learn from other systems here, and I've made the following argument argument before: a controllable and small-ish concurrency level by virtue of a introducing the concept of a worker pool (with a given size) is often used with success in these cases.

So, in summary I think we should make wise use of both of these two strategies whenever applicable, maybe in combination:

1. Merging mutations into batch requests (effectively custom rate-limiting)
2. Use a worker pool of configurable size. Again, Kevin and I discussed this here and I think that sub thread is still a good read.

[kangclzjc]

Previously, I mocked the project's NVIDIA-dependent interfaces, allowing the project to run in a non-GPU environment and simulate most of the workflow. Through this approach, I will now move into the testing phase. After that, stress testing will be conducted to evaluate the effectiveness of this improvement. Hopefully everything goes smoothly.

@sakura-3, You can refer to k8s-test-infra to do the mock test.

[sakura-3]

A tough week. I implemented the centralized index allocation approach in the controller and ran stress tests, hoping to resolve the API server throttling (429) caused by massive update conflicts during distributed index negotiation, and to reduce DRA latency for training jobs at the scale of tens of thousands of GPUs.

However, the stress test results revealed that after eliminating the 429 issue, the startup of the DaemonSet responsible for managing the imex process became the new bottleneck.

We simulated a training scenario with 20,000 GPUs, consisting of 280 cliques with 18 nodes each. We submitted a Deployment with 5,040 replicas as the training workload, expecting each node to host one workload pod and one daemon pod for managing the imex process. For each workload pod and its corresponding daemon pod, we collected timestamps at several key milestones.

Below are two sample data entries. The second column, workload pod scheduled, represents an absolute timestamp, while all subsequent columns are relative to the second column, in seconds. T0 = 02:30:33.745, T1 = 02:30:36.239. The scheduled timestamps of all workload pods fall within the range [02:30:33.732, 02:30:44.968].

	workload pod scheduled	node labeled	daemon pod created	daemon pod scheduled	daemon pod running	workload pod running
case 1	T0	0.441	1.379	16.568	19.090	19.153
case 2	T1	7.276	27.503	38.498	41.938	42.151