feat(kube-stack): add presets.prometheus.{nodeExporter,cadvisor,podAnnotations} - #2206
Conversation
Adds presets.annotationDiscovery.metrics.prometheus.enabled to the kube-stack chart. When enabled (requires metrics.enabled: true), the chart adds a standalone prometheus/annotation_discovery receiver to the metrics pipeline, configured with Kubernetes pod service discovery and Prometheus relabel rules to scrape pods on the same node carrying classic Prometheus annotations (prometheus.io/scrape=true, prometheus.io/port, prometheus.io/path, prometheus.io/scheme). A nested option presets.annotationDiscovery.metrics.prometheus.addPrometheusLabels (default false) gates whether the scrape config emits Prometheus-style Kubernetes labels (namespace, pod, and all pod labels via __meta_kubernetes_pod_label_*). When false, only the standard job / instance labels are produced, leaving the k8sattributes processor as the sole source of Kubernetes metadata. Bumps chart version 0.15.0 -> 0.15.1 and enables the new option in the daemonset-annotation-discovery-metrics example.
prometheus option to presets/annotationDiscovery/metrics
…n_discovery Skip the OpenTelemetry collector's own pods at the Kubernetes service discovery level by adding `app.kubernetes.io/component!=opentelemetry-collector` to the kubernetes_sd_configs selector, alongside the existing node field selector. This is more efficient than a relabel_configs drop rule: the API server returns only matching pods, so Prometheus never has to fetch the collector's pods just to discard them.
|
@dashpole FYI, as I suggested in open-telemetry/opentelemetry-specification#4956 (comment), we could evolve the This would allow |
|
@jaronoff97 I hope this meets your expectations. The code change is quite small. |
prometheus option to presets/annotationDiscovery/metricsprometheus option to presets/annotationDiscovery/metrics
…notations}
Introduces a new top-level `presets.prometheus.*` namespace as a daemonset-local
replacement for the `scrape_configs_file: daemon_scrape_configs.yaml` flow. Each
preset adds a named instance of the prometheus receiver to the metrics pipeline:
* prometheus/node_exporter — scrapes node-exporter on ${OTEL_K8S_NODE_IP}:9100
* prometheus/cadvisor — scrapes /metrics/cadvisor on the local kubelet
(job_name=kubelet for KPS / legacy parity)
* prometheus/pod_annotations — scrapes pods on the local node carrying classic
prometheus.io/scrape=true annotations
Renames `presets.annotationDiscovery.metrics.prometheus` → `presets.prometheus.podAnnotations`.
Drops `addPrometheusLabels`; the preset always emits namespace/pod/node and the
full pod-label set, matching daemon_scrape_configs.yaml output.
Chart-enforced constraints (template render fails with a clear error if violated):
* mutually exclusive with `scrape_configs_file` (the presets are a replacement)
* require `mode: daemonset`
Time-series compatibility with daemon_scrape_configs.yaml: all three presets are
drop-in compatible — same job/instance/Kubernetes label sets — with additive
extras (a new `node` label on all three, self-scrape filter on podAnnotations).
Bumps chart 0.15.1 → 0.16.0. Adds a new example
examples/daemonset-prometheus-presets demonstrating all three. README gains a
detailed "presets.prometheus.* vs. KPS ServiceMonitors" section under the
existing Kube-Prometheus-Stack compatability heading.
prometheus option to presets/annotationDiscovery/metrics|
FYI the build failure looks odd, |
|
The latest iteration of the spec PR just keeps job/instance as-is. Hopefully that simplifies things. |
|
@jaronoff97 the build is fixed could you please have a look? |
jaronoff97
left a comment
There was a problem hiding this comment.
The docs are incorrect and quite misleading. Please go through them again and correct the errors. The content of the PR's actual helm changes looks good however.
…eus-prom # Conflicts: # charts/opentelemetry-kube-stack/Chart.yaml # charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/bridge.yaml # charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/cloud-demo/rendered/instrumentation.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-logs/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-logs/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-metrics/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery-metrics/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/daemonset-annotation-discovery/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/default/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/default/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/isolated-multicollector-deployment/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/isolated-multicollector-deployment/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/no-leader-election-extension/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/no-leader-election-extension/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-api-server/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-controller-manager/service.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-controller-manager/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-dns/service.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-dns/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-etcd/service.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-etcd/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-proxy/service.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-proxy/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-scheduler/service.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/exporters/kube-scheduler/servicemonitor.yaml # charts/opentelemetry-kube-stack/examples/prometheus-otel/rendered/hooks.yaml # charts/opentelemetry-kube-stack/examples/secrets-csi-driver/rendered/collector.yaml # charts/opentelemetry-kube-stack/examples/secrets-csi-driver/rendered/hooks.yaml
|
Thanks for your feedback @jaronoff97. The PR is ready for another review. I have cleaned up the docs following your recommendations. |
jaronoff97
left a comment
There was a problem hiding this comment.
much better, thank you very much!
|
@dashpole or @TylerHelmuth mind adding your review in here when you get a chance? |
|
Looks correct overall to me. I don't have much experience with helm, though |
Summary
Adds a new top-level
presets.prometheus.*namespace to the kube-stack chart — a daemonset-local replacement forscrape_configs_file: daemon_scrape_configs.yaml. Three independently-toggleable presets, each adding a dedicated named instance of the prometheus receiver to the daemonset collector's metrics pipeline:prometheus.nodeExporter— scrapes a Prometheus Node Exporter on the local node viastatic_configstargeting${OTEL_K8S_NODE_IP}:9100.prometheus.cadvisor— scrapes cAdvisor from the local kubelet athttps://${OTEL_K8S_NODE_IP}:10250/metrics/cadvisorwith bearer-token auth. Drops the high-cardinality / low-value series by default (same drop list asdaemon_scrape_configs.yaml).prometheus.podAnnotations— scrapes pods on the local node carrying classicprometheus.io/scrape=trueannotations viakubernetes_sd_configswith a node field selector.Sharded by design: each daemonset pod scrapes only its own node, no central Prometheus bottleneck.
UX
Per-preset knobs:
scrapeInterval(default 30s),scrapeTimeout(default 10s), plusportfornodeExporter/cadvisor(podAnnotationsreads port per-pod from theprometheus.io/portannotation).Time-series alignment with the
daemon_scrape_configs.yamljobsAll three presets produce the same identifying labels as
daemon_scrape_configs.yaml—job/instance/ Kubernetes label sets, so existing queries, dashboards, alerts, and recording rules relying on these labels keep working unchanged:daemon_scrape_configs.yamljobnodeExporternode-exporternodelabelcadvisorkubelet(/metrics/cadvisor)job_name: kubeletkept for KPS +daemon_scrape_configs.yamlparitypodAnnotationskubernetes-podsnodelabel + collector self-scrape filterDocumentation
README.md: new subsectionpresets.prometheus.*under the existing Kube-Prometheus-Stack compatability heading. Covers the constraints, discovery model, per-preset label set, alignment withdaemon_scrape_configs.yaml, and remaining duplicate-scrape risks (e.g. the prometheus-node-exporter subchart's own ServiceMonitor).values.yaml: each preset documented inline; constraints called out at the umbrella block.values.schema.json: full schema entries with descriptions;scrape_configs_filedescription notes the mutual exclusivity.Constraints (chart-enforced)
Template-render assertions fail with a clear error if either is violated:
scrape_configs_file— the presets are a replacement, not an addition. The defaultdaemon_scrape_configs.yamlalready has node-exporter, kubelet/cAdvisor, and pod-annotation jobs; running both would cause duplicate scrapes.mode: daemonset— the scrape configs use${OTEL_K8S_NODE_IP}/${OTEL_K8S_NODE_NAME}and node field selectors that only make sense on daemonset collectors.Errors name the specific presets enabled, e.g.
presets.prometheus.{nodeExporter, cadvisor} are a replacement for scrape_configs_file....Why a standalone Prometheus receiver rather than receiver_creator?
The standalone Prometheus receiver reuses the native Prometheus discovery & scrape code paths and config, so users can lift-and-shift existing Prometheus jobs and relabel chains. The receiver_creator path is still available via the existing
presets.annotationDiscovery.metrics.enabled.Test plan
make generate-examples CHARTS=opentelemetry-kube-stackproduces no further diff.helm lint charts/opentelemetry-kube-stackclean.examples/daemonset-prometheus-presets/demonstrates enabling all three presets together.helm templatewithscrape_configs_fileset + a preset enabled fails with the expected error naming the enabled presets.helm templatewithmode: deployment+ a preset enabled fails with the expected error.job_name(node-exporter/kubelet/kubernetes-pods),scheme,metrics_path, relabel rules (includingnodelabel injection), andmetric_relabel_configs.