[kube-stack] Declaring custom receivers or processors in pipelines discards some but not all built-in ones

[cyrille-leclerc]

Problem description

Declaring custom receivers or processors like a prometheus receiver to scrape a few metrics or a cumulativetodelta processor in telemetry pipelines discards some built-in ones but not all, creating confusion.

Precisely, the built-in receivers and processors defined in telemetry pipelines defined in the kube-stack original values.yaml (here) get discarded when the ones defined through presets are preserved:

• Receivers:
  • Discarded: otlp
  • Preserved: hostmetrics, k8s_cluster, k8sobjects, kubeletstats, receiver_creator/metrics, receiver_creator/logs
• Processors:
  • Discarded: resourcedetection/env, resource/hostname, batch
  • Preserved: k8sattributes

The root cause is that the processors & receivers that are discarded are initially defined in values.yaml while the preserved ones are defined through the helm rendering templates according to the values of the presets

Example with a custom cumulativetodelta processor

When declaring a custom metrics processor (eg cumulativetodelta), some built-in processors don't get added (eg resourcedetection/env, resource/hostname, batch) when others are added as expected (eg k8sattributes).

    presets:
      kubernetesAttributes:
        enabled: true
    config:
      exporters:
        otlp_http:
          # ...
      processors:
        cumulativetodelta:
          # ...
      service:
        pipelines:
          traces:
            exporters:
            - otlp_http
          metrics:
            exporters:
            - otlp_http
            processors:
            - cumulativetodelta
          logs:
            exporters:
            - otlp_http

Actual service metrics pipeline when declaring a cumulativetodelta processor

      pipelines:
        metrics:
          exporters:
          - otlp_http
          processors:
          - k8sattributes
          - cumulativetodelta
          receivers:
          # ...

Expected service metrics pipeline

      pipelines:
        metrics:
          exporters:
          - otlp_http
          processors:
          - k8sattributes
          - resourcedetection/env
          - resource/hostname
          - cumulativetodelta
          - batch
          receivers:
          # ...

[cyrille-leclerc]

@jaronoff97 @TylerHelmuth Would love to dig into this with you both. One approach worth exploring: exposing all built-in receivers and processors as preset-driven configuration. Something along these lines:

presets:
   resourceDetection:
      env:
         enabled: true
      mapKubernetesNodeNametoHostName:
         enabled: true
   otlpReceiver:
      enabled: true

One concern is that this might inadvertently introduce a new configuration language for the collector ; though I'm not entirely sure if that's actually a risk here.

[Pittu-Sharma]

Hi, I would like to work on this issue. Can you please assign it to me!

[TylerHelmuth]

Ya this is a result of providing a default config that uses lists and go templating not merging lists.

[cyrille-leclerc]

@TylerHelmuth would it make sense to convert some of these default config as presets? I'm typically thinking of the otlp receiver or the processor to derive hostname from k8s.node.name.

[TylerHelmuth]

We've historically tried to keep presets pretty separate from any receivers they end up installing. They should be more targeted to enabling capabilities than components. This way a user doesnt need to worry about which component is doing the work for the feature they want.

I'd be more in favor of adding our own merging logic. The collector has a feature flag for merging list when using multiple config files. We could look into a similar solution for our pipeline lists.