[opentelemetry-kube-stack] Add missing namespace to hook and instrumentation resources #2370

Current working solution

We use the following setup to deploy opentelemetry operators on our kubernetes clusters.

1. Create inflated manifest with helm template. Example:

   # Render (inflate) the helm chart into a manifests file
   helm template opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
    --namespace opentelemetry-operator-system \
    --values services/opentelemetry-operator/helm/values.yaml \
    --version '0.16.0' \
    --include-crds \
    > services/opentelemetry-operator/base/helm/inflated-manifests.yaml

   Note: Some resources like the ones of the hook and instrumentation charts don't have namespaces.

2. Kustomize the inflated manifest with kustomize. Example:

   ---
   # kustomization.yaml    
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   namespace: opentelemetry-operator-system
   
   resources:
   - helm/inflated-manifests.yaml
   
   ...

   Note: Since kustomization files use the namespace directive, all created resources contain namespaces. AFAIK, helm install --namespace <ns> would have the same effect, i.e. add namespaces to inflated manifests where missing.

3. Deploy manifests via ArgoCD

Problem / Not working solution

Our custom configs are growing and managing them with kustomize is getting hard to maintain. Thus we started to migrate our configs from kustomize to helm values.

Our first approach was to use the kustomize built-in helmChart directive. Example:

---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: opentelemetry-operator-system

helmCharts:
- name: opentelemetry-kube-stack
  namespace: opentelemetry-operator-system
  includeCRDs: true
  valuesFile: values.yaml
  version: 0.16.0
  repo: open-telemetry/opentelemetry-kube-stack
...

Due to a change in kustomize v5.8.0, the created hook (and instrumentation) resources do not have a namespace anymore.
See github issue.

Even this is a kustomize bug, a more reliable solution would be to fix this a the root, i.e. add the missing namespaces to the chart directly. Note that if we wanted to deploy the operator with helm directly using ArgoCD, we would have to rely on ArgoCD "magic" as well, since ArgoCD uses helm template and not helm install to deploy applications.

I already created a PR with a fix.

If you need more information, please let me know.