Search before asking
Description
When RayJob.spec.submitterPodTemplate is set, GetSubmitterTemplate returns the user's template verbatim (*rayJobSpec.SubmitterPodTemplate.DeepCopy()) without merging in any of the computed defaults — image inherited from the head container, default resource requests/limits, container name ray-job-submitter, restartPolicy: Never, etc.
Users who only want to add or override a small piece of the submitter pod (most commonly metadata.annotations or metadata.labels for cluster-autoscaler hints, scheduler classes, do-not-disrupt markers, etc.) are forced to restate the entire pod spec, which:
- Adds significant boilerplate to RayJob manifests for what is conceptually a one-line tweak.
- Creates silent drift when KubeRay's defaults change (e.g. resource defaults bumped in a new release): the user's restated spec keeps the old values without any warning.
- Forces users to read KubeRay source to know what defaults to copy, since the defaults aren't declarative anywhere visible to manifest authors.
Use case
Common scenarios where users only need to tweak metadata, not the spec:
- Annotations for cluster-autoscaler / scheduler / preemption protection — e.g. mark the submitter pod as do-not-disrupt so it isn't evicted mid-job, breaking
ray job logs --follow and dropping driver-log visibility.
- Labels for log routing or cost allocation — propagate a job-id label to the submitter pod for log indexing.
- Tolerations or nodeSelectors when the worker cluster's nodes mostly carry taints (GPU-tainted nodepools) and the submitter pod would otherwise be confined to a small set of untainted nodes.
In each of these the user wants to override one or two fields while keeping KubeRay's choice of image / resources / container name / restartPolicy. Today they have to restate everything.
Proposal
Two possible shapes — either would solve the use case:
Option A — strategic merge of submitterPodTemplate. Keep the existing field; change GetSubmitterTemplate to overlay the user's template onto the computed default rather than replacing it. This matches K8s convention for PodTemplateSpec (StatefulSet/Deployment use strategic merge patch semantics for nested updates).
Option B — narrower submitterPodMetadata field. Add a new optional metav1.ObjectMeta field that accepts only annotations + labels (and possibly a few other metadata-only knobs). The submitter pod's spec always comes from defaults; metadata is overlaid from this field. This is a smaller API surface and avoids the question of "what does it mean to merge containers?".
Related issues
Are you willing to submit a PR?
Search before asking
Description
When
RayJob.spec.submitterPodTemplateis set,GetSubmitterTemplatereturns the user's template verbatim (*rayJobSpec.SubmitterPodTemplate.DeepCopy()) without merging in any of the computed defaults — image inherited from the head container, default resource requests/limits, container nameray-job-submitter,restartPolicy: Never, etc.Users who only want to add or override a small piece of the submitter pod (most commonly
metadata.annotationsormetadata.labelsfor cluster-autoscaler hints, scheduler classes, do-not-disrupt markers, etc.) are forced to restate the entire pod spec, which:Use case
Common scenarios where users only need to tweak metadata, not the spec:
ray job logs --followand dropping driver-log visibility.In each of these the user wants to override one or two fields while keeping KubeRay's choice of image / resources / container name / restartPolicy. Today they have to restate everything.
Proposal
Two possible shapes — either would solve the use case:
Option A — strategic merge of
submitterPodTemplate. Keep the existing field; changeGetSubmitterTemplateto overlay the user's template onto the computed default rather than replacing it. This matches K8s convention forPodTemplateSpec(StatefulSet/Deployment use strategic merge patch semantics for nested updates).Option B — narrower
submitterPodMetadatafield. Add a new optionalmetav1.ObjectMetafield that accepts only annotations + labels (and possibly a few other metadata-only knobs). The submitter pod's spec always comes from defaults; metadata is overlaid from this field. This is a smaller API surface and avoids the question of "what does it mean to mergecontainers?".Related issues
Resourcesfield for job submitter pod #1198 — RayJob: Set Resources field for job submitter pod (related: also a "hard to override defaults" pain point).Are you willing to submit a PR?