Skip to content

feat(kube-stack): add presets.prometheus.{nodeExporter,cadvisor,podAnnotations} - #2206

Merged
jaronoff97 merged 20 commits into
open-telemetry:mainfrom
cyrille-leclerc:feat/kube-stack-annotation-discovery-prometheus-prom
Jun 12, 2026
Merged

feat(kube-stack): add presets.prometheus.{nodeExporter,cadvisor,podAnnotations}#2206
jaronoff97 merged 20 commits into
open-telemetry:mainfrom
cyrille-leclerc:feat/kube-stack-annotation-discovery-prometheus-prom

Conversation

@cyrille-leclerc

@cyrille-leclerc cyrille-leclerc commented May 19, 2026

Copy link
Copy Markdown
Member

Summary

Adds a new top-level presets.prometheus.* namespace to the kube-stack chart — a daemonset-local replacement for scrape_configs_file: daemon_scrape_configs.yaml. Three independently-toggleable presets, each adding a dedicated named instance of the prometheus receiver to the daemonset collector's metrics pipeline:

  • prometheus.nodeExporter — scrapes a Prometheus Node Exporter on the local node via static_configs targeting ${OTEL_K8S_NODE_IP}:9100.
  • prometheus.cadvisor — scrapes cAdvisor from the local kubelet at https://${OTEL_K8S_NODE_IP}:10250/metrics/cadvisor with bearer-token auth. Drops the high-cardinality / low-value series by default (same drop list as daemon_scrape_configs.yaml).
  • prometheus.podAnnotations — scrapes pods on the local node carrying classic prometheus.io/scrape=true annotations via kubernetes_sd_configs with a node field selector.

Sharded by design: each daemonset pod scrapes only its own node, no central Prometheus bottleneck.

UX

collectors:
  daemon:
    scrape_configs_file: ""   # required when enabling any prometheus/* preset
    presets:
      prometheus:
        nodeExporter:
          enabled: true
        cadvisor:
          enabled: true
        podAnnotations:
          enabled: true

Per-preset knobs: scrapeInterval (default 30s), scrapeTimeout (default 10s), plus port for nodeExporter / cadvisor (podAnnotations reads port per-pod from the prometheus.io/port annotation).

Time-series alignment with the daemon_scrape_configs.yaml jobs

All three presets produce the same identifying labels as daemon_scrape_configs.yamljob / instance / Kubernetes label sets, so existing queries, dashboards, alerts, and recording rules relying on these labels keep working unchanged:

Preset Maps to daemon_scrape_configs.yaml job Compatibility
nodeExporter node-exporter Drop-in + bonus node label
cadvisor kubelet (/metrics/cadvisor) Drop-in — job_name: kubelet kept for KPS + daemon_scrape_configs.yaml parity
podAnnotations kubernetes-pods Drop-in + bonus node label + collector self-scrape filter

Documentation

  • README.md: new subsection presets.prometheus.* under the existing Kube-Prometheus-Stack compatability heading. Covers the constraints, discovery model, per-preset label set, alignment with daemon_scrape_configs.yaml, and remaining duplicate-scrape risks (e.g. the prometheus-node-exporter subchart's own ServiceMonitor).
  • values.yaml: each preset documented inline; constraints called out at the umbrella block.
  • values.schema.json: full schema entries with descriptions; scrape_configs_file description notes the mutual exclusivity.

Constraints (chart-enforced)

Template-render assertions fail with a clear error if either is violated:

  • Mutually exclusive with scrape_configs_file — the presets are a replacement, not an addition. The default daemon_scrape_configs.yaml already has node-exporter, kubelet/cAdvisor, and pod-annotation jobs; running both would cause duplicate scrapes.
  • Require mode: daemonset — the scrape configs use ${OTEL_K8S_NODE_IP} / ${OTEL_K8S_NODE_NAME} and node field selectors that only make sense on daemonset collectors.

Errors name the specific presets enabled, e.g. presets.prometheus.{nodeExporter, cadvisor} are a replacement for scrape_configs_file....

Why a standalone Prometheus receiver rather than receiver_creator?

The standalone Prometheus receiver reuses the native Prometheus discovery & scrape code paths and config, so users can lift-and-shift existing Prometheus jobs and relabel chains. The receiver_creator path is still available via the existing presets.annotationDiscovery.metrics.enabled.

Test plan

  • make generate-examples CHARTS=opentelemetry-kube-stack produces no further diff.
  • helm lint charts/opentelemetry-kube-stack clean.
  • New example examples/daemonset-prometheus-presets/ demonstrates enabling all three presets together.
  • Validation: helm template with scrape_configs_file set + a preset enabled fails with the expected error naming the enabled presets.
  • Validation: helm template with mode: deployment + a preset enabled fails with the expected error.
  • Render examples and confirm each preset's scrape job has the expected job_name (node-exporter / kubelet / kubernetes-pods), scheme, metrics_path, relabel rules (including node label injection), and metric_relabel_configs.
  • Deploy to a k3d cluster and verify metrics flow through each receiver.

Adds presets.annotationDiscovery.metrics.prometheus.enabled to the
kube-stack chart. When enabled (requires metrics.enabled: true), the
chart adds a standalone prometheus/annotation_discovery receiver to the
metrics pipeline, configured with Kubernetes pod service discovery and
Prometheus relabel rules to scrape pods on the same node carrying
classic Prometheus annotations (prometheus.io/scrape=true,
prometheus.io/port, prometheus.io/path, prometheus.io/scheme).

A nested option presets.annotationDiscovery.metrics.prometheus.addPrometheusLabels
(default false) gates whether the scrape config emits Prometheus-style
Kubernetes labels (namespace, pod, and all pod labels via
__meta_kubernetes_pod_label_*). When false, only the standard job /
instance labels are produced, leaving the k8sattributes processor as the
sole source of Kubernetes metadata.

Bumps chart version 0.15.0 -> 0.15.1 and enables the new option in the
daemonset-annotation-discovery-metrics example.
@cyrille-leclerc
cyrille-leclerc requested review from a team and jaronoff97 as code owners May 19, 2026 09:59
@cyrille-leclerc cyrille-leclerc changed the title [kube-stack] add prometheus option to annotationDiscovery.metrics preset [kube-stack] add prometheus option to presets/annotationDiscovery/metrics May 19, 2026
…n_discovery

Skip the OpenTelemetry collector's own pods at the Kubernetes service
discovery level by adding `app.kubernetes.io/component!=opentelemetry-collector`
to the kubernetes_sd_configs selector, alongside the existing node field
selector. This is more efficient than a relabel_configs drop rule: the
API server returns only matching pods, so Prometheus never has to fetch
the collector's pods just to discard them.
@cyrille-leclerc

cyrille-leclerc commented May 19, 2026

Copy link
Copy Markdown
Member Author

@dashpole FYI, as I suggested in open-telemetry/opentelemetry-specification#4956 (comment), we could evolve the prometheus/addPrometheusLabels config in this PR so that it not only avoids generating labels such as pod, namespace, etc., but also maps the Prometheus job and instance labels to prometheus.job and prometheus.instance.

This would allow service.name and service.instance.id to remain unset in the PRometheus receiver layer, so that the k8sattributes processor can generate the desired values based on Kubernetes metadata, matching the guidance from SemConv: Specify resource attributes using Kubernetes annotations.
We could temporarily implement this with a transform processor waiting to add the capability in the PRometheus receiver.

@cyrille-leclerc

Copy link
Copy Markdown
Member Author

@jaronoff97 I hope this meets your expectations. The code change is quite small.

@cyrille-leclerc cyrille-leclerc changed the title [kube-stack] add prometheus option to presets/annotationDiscovery/metrics feat(kube-stack) add prometheus option to presets/annotationDiscovery/metrics May 20, 2026
…notations}

Introduces a new top-level `presets.prometheus.*` namespace as a daemonset-local
replacement for the `scrape_configs_file: daemon_scrape_configs.yaml` flow. Each
preset adds a named instance of the prometheus receiver to the metrics pipeline:

  * prometheus/node_exporter   — scrapes node-exporter on ${OTEL_K8S_NODE_IP}:9100
  * prometheus/cadvisor        — scrapes /metrics/cadvisor on the local kubelet
                                 (job_name=kubelet for KPS / legacy parity)
  * prometheus/pod_annotations — scrapes pods on the local node carrying classic
                                 prometheus.io/scrape=true annotations

Renames `presets.annotationDiscovery.metrics.prometheus` → `presets.prometheus.podAnnotations`.
Drops `addPrometheusLabels`; the preset always emits namespace/pod/node and the
full pod-label set, matching daemon_scrape_configs.yaml output.

Chart-enforced constraints (template render fails with a clear error if violated):
  * mutually exclusive with `scrape_configs_file` (the presets are a replacement)
  * require `mode: daemonset`

Time-series compatibility with daemon_scrape_configs.yaml: all three presets are
drop-in compatible — same job/instance/Kubernetes label sets — with additive
extras (a new `node` label on all three, self-scrape filter on podAnnotations).

Bumps chart 0.15.1 → 0.16.0. Adds a new example
examples/daemonset-prometheus-presets demonstrating all three. README gains a
detailed "presets.prometheus.* vs. KPS ServiceMonitors" section under the
existing Kube-Prometheus-Stack compatability heading.
@cyrille-leclerc cyrille-leclerc changed the title feat(kube-stack) add prometheus option to presets/annotationDiscovery/metrics feat(kube-stack): add presets.prometheus.{nodeExporter,cadvisor,podAnnotations} May 26, 2026
@cyrille-leclerc

Copy link
Copy Markdown
Member Author

FYI the build failure looks odd, make check-examples on github actions is unhappy with a few line breaks in generated files when there are no problems on my laptop. I have to investigate more.

@dashpole

Copy link
Copy Markdown
Contributor

The latest iteration of the spec PR just keeps job/instance as-is. Hopefully that simplifies things.

@cyrille-leclerc

Copy link
Copy Markdown
Member Author

@jaronoff97 the build is fixed could you please have a look?

@jaronoff97 jaronoff97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs are incorrect and quite misleading. Please go through them again and correct the errors. The content of the PR's actual helm changes looks good however.

Comment thread charts/opentelemetry-kube-stack/templates/_config.tpl
Comment thread charts/opentelemetry-kube-stack/README.md Outdated
Comment thread charts/opentelemetry-kube-stack/README.md Outdated
Comment thread charts/opentelemetry-kube-stack/README.md Outdated
Comment thread charts/opentelemetry-kube-stack/README.md Outdated
Comment thread charts/opentelemetry-kube-stack/README.md Outdated
@cyrille-leclerc
cyrille-leclerc marked this pull request as draft June 3, 2026 08:27
…eus-prom

# Conflicts:
#	charts/opentelemetry-kube-stack/Chart.yaml
#	charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/bridge.yaml
#	charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/instrumentation.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-logs/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-logs/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-metrics/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-metrics/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/default/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/default/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/isolated-multicollector-deployment/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/isolated-multicollector-deployment/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/no-leader-election-extension/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/no-leader-election-extension/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-api-server/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-controller-manager/service.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-controller-manager/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-dns/service.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-dns/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-etcd/service.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-etcd/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-proxy/service.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-proxy/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-scheduler/service.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-scheduler/servicemonitor.yaml
#	charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/hooks.yaml
#	charts/opentelemetry-kube-stack/examples/secrets-csi-driver/rendered/collector.yaml
#	charts/opentelemetry-kube-stack/examples/secrets-csi-driver/rendered/hooks.yaml
Comment thread charts/opentelemetry-kube-stack/README.md
@cyrille-leclerc
cyrille-leclerc marked this pull request as ready for review June 4, 2026 09:02
@cyrille-leclerc

cyrille-leclerc commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

Thanks for your feedback @jaronoff97. The PR is ready for another review.

I have cleaned up the docs following your recommendations.
I kept the documentation of the differences between metrics captured by daemon_scrape_configs.yaml and by presets.prometheus.* but I stop talking of compatibility.

@jaronoff97 jaronoff97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much better, thank you very much!

@jaronoff97

Copy link
Copy Markdown
Contributor

@dashpole or @TylerHelmuth mind adding your review in here when you get a chance?

Comment thread charts/opentelemetry-kube-stack/README.md Outdated
@dashpole

dashpole commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Looks correct overall to me. I don't have much experience with helm, though

@jaronoff97
jaronoff97 merged commit 8aee4ae into open-telemetry:main Jun 12, 2026
6 checks passed
@cyrille-leclerc
cyrille-leclerc deleted the feat/kube-stack-annotation-discovery-prometheus-prom branch June 15, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants