[OCTRL-1081] Wrap kubectl into Mesos executor task by justonedev1 · Pull Request #805 · AliceO2Group/Control

justonedev1 · 2026-03-18T15:46:52Z

The task description is in the documentation docs/kubernetes_ecs.md

knopers8 · 2026-03-19T15:41:11Z

+	// 	aux.Control.Mode = "direct"
+	// }
+
+	aux.Control.Mode = c.Control.Mode.String()


There is a change in behaviour for hooks though, no? Before they were getting direct instead of hook, which actually smells like a bug, but perhaps something is relying on it?

This is probably my misunderstanding, as I thought that it is a bug to implicitly change hook to direct.. especially when we have hooktask that is created only if controlmode.HOOK is present.
see

knopers8 · 2026-03-19T15:47:43Z

+While prototyping we used many Kubernetes clusters, namely [`kind`](https://kind.sigs.k8s.io/), [`minikube`](https://minikube.sigs.k8s.io/docs/) and [`k3s`](https://k3s.io/)
+in both local and remote cluster deployment. We used Openstack for remote deployment.
+Follow the guides at the individual distributions in order to create the desired cluster setup.
+For now we chose `k3s` for most of the activities performed because it is lightweight


That might give an impression that there was some kind of decision process in ALICE or FLP. Perhaps rephrase to:

k3s is recommended to run this prototype, as it is a lightweight and...

knopers8 · 2026-03-19T15:50:58Z

+All settings of `k3s` were used as default except one: locked-in-memory size. Use `ulimit -l` to learn
+what is the limit for the current user and `LimitMEMLOCK` inside the k3s systemd service config
+to set it for correct value. Right now the `flp` user has unlimited size (`LimitMEMLOCK=infinity`).
+This config is necessary because even if you are running PODs with the privileged security context


Suggested change

This config is necessary because even if you are running PODs with the privileged security context

This config is necessary because even if you are running Pods with the privileged security context

It's a detail, but "Pod" and "pod" are likely the correct spellings, since it's a proper English word, not an abbreviation, like "Plain Old Data". I mention this, so that we avoid having someone in ALICE read our documentation and then learn to read it "pee-o-dee".

fair enough

knopers8 · 2026-04-13T13:43:43Z

+
+* add path to the kubectl manifest as the first argument in `.command.arguments` field
+* change `.control.mode` to either `kubectl_direct` or `kubectl_fairmq`
+You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml`


Suggested change

You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml`

You can find working template inside `control-operator/ecs-manifests/control-workflows/*-kube.yaml`

knopers8 · 2026-04-14T11:43:30Z

+		log.WithFields(logrus.Fields{
+			"controlmode": task.Tci.ControlMode,
+			"name":        task.ti.Name,
+		}).WithError(err).Errorf("kubectl apply failed stderr: %s , stdin: %s", stderrBuf.String(), stdoutBuf.String())


Suggested change

}).WithError(err).Errorf("kubectl apply failed stderr: %s , stdin: %s", stderrBuf.String(), stdoutBuf.String())

}).WithError(err).Errorf("kubectl apply failed stderr: %s , stdout: %s", stderrBuf.String(), stdoutBuf.String())

knopers8 · 2026-04-14T11:59:23Z

+		}
+		supplementalString := "[" + strings.Join(strIds, ", ") + "]"
+
+		os.Setenv("FLP_SUPPLEMENTAL_GORUPS", supplementalString)


Suggested change

os.Setenv("FLP_SUPPLEMENTAL_GORUPS", supplementalString)

os.Setenv("FLP_SUPPLEMENTAL_GROUPS", supplementalString)

typo?

knopers8 · 2026-04-17T11:44:10Z

+> ⚠️ **Warning**
+> All Kubernetes work done is in a stage of prototype.
+
+## Kubernetes Cluster


My notes on what was generally missing here (or I just didn't see it):

make install to register the task CRD

make sure executor has access to a ~/.kube/config

kubectl does not react well to stfsender env var http_proxy="", which is surrounded by additional quotes in its manifest, making kubectl apply fail. Easy fix - http_proxy=.

ah, and one has to remember to create an image pull secret for the controller after make deploy, because only then the namespace already exists.

I think that some of those of those comments belong into kubernetse_ecs.md, but to the docs in control-operator. But those are fair comments. The reason why I didn't add them at the time of creation was that I didn't encounter those bcs k3s behaved a bit differently (can automatically read ~/.docker/config.json and other things) But I will add those comments into the control-operator with maybe a link pointing from kubernetes_ecs.md there.

I don't know what you mean by http_proxy comment. Can you tell me what exactly should I add and what you did, I don't think that I had to do anything about that.

I think that some of those of those comments belong into ...

Yes, very likely, at some point I became a bit lost which PR is which :D

I don't know what you mean by http_proxy comment. Can you tell me what exactly should I add and what you did, I don't think that I had to do anything about that.

I mean this line: https://github.com/AliceO2Group/ControlWorkflows/blob/15b05c6ea90cf61b322a921c9640300ac981ff4e/tasks/stfsender.yaml#L28

These env vars are somehow passed to the manifest and causing some quoting/escaping issue, I don't remember exactly.

user infor setup properly... kubectl passes arguments properly to the kubernetes attempt for fairmq bla

justonedev1 requested a review from knopers8 as a code owner March 18, 2026 15:46

knopers8 reviewed Mar 19, 2026

View reviewed changes

justonedev1 requested a review from knopers8 April 10, 2026 10:41

knopers8 reviewed Apr 17, 2026

View reviewed changes

justonedev1 added 4 commits April 28, 2026 17:26

OCTRL-1081 kubectl task to create rudimentary bridge to kubernetes

08e9a80

user infor setup properly... kubectl passes arguments properly to the kubernetes attempt for fairmq bla

fixed documentation

a8890fd

adding kubeconfig to kubectl calls so it works even without $HOME set

d756d80

fixing documentation and typos

da01241

justonedev1 force-pushed the OCTRL-1081 branch from ba80df7 to da01241 Compare April 28, 2026 15:45

	This config is necessary because even if you are running PODs with the privileged security context
	This config is necessary because even if you are running Pods with the privileged security context

	You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml`
	You can find working template inside `control-operator/ecs-manifests/control-workflows/*-kube.yaml`

	}).WithError(err).Errorf("kubectl apply failed stderr: %s , stdin: %s", stderrBuf.String(), stdoutBuf.String())
	}).WithError(err).Errorf("kubectl apply failed stderr: %s , stdout: %s", stderrBuf.String(), stdoutBuf.String())

	os.Setenv("FLP_SUPPLEMENTAL_GORUPS", supplementalString)
	os.Setenv("FLP_SUPPLEMENTAL_GROUPS", supplementalString)

Conversation

justonedev1 commented Mar 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justonedev1 Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justonedev1 Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justonedev1 Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

justonedev1 Mar 20, 2026 •

edited

Loading

justonedev1 Apr 28, 2026 •

edited

Loading

justonedev1 Apr 28, 2026 •

edited

Loading