Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

When using K8sJob submitter mode, when the RayCluster head pod is evicted the behavior is something like:

1. Ray head pod is evicted
2. Submitter pod fails to watch the job submission, so it also fails
3. Kuberay marks the RayJob failed and performs any cleanup the user has configured (e.g. deleting the RayCluster, etc.)

However, the behavior when using the HTTP submitter mode is different:

1. Ray head pod is evicted
2. The RayCluster operator creates a new head pod
3. The pod becomes healthy
4. The ray operator hits this case and resubmits the job to the new head

Imo this is a bug. The docs about submissionMode here imply that HTTPMode, K8sJobMode, and SidecarMode are mostly interchangeable, aside from some nuances around kubernetes networking.

Reproduction script

Make a rayjob script with this code

def entrypoint_fn() -> None:
    time.sleep(1000)

and create a RayJob CRD with it. Replicate eviction behavior with kubectl delete pod <pod-name> --grace-period=0. Notice that a new head pod comes up and another ray job is submitted to it.

Anything else

I think it reproduces consistently with the repro steps I gave above.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!