Prometheus: preserve job/instance when translating Prometheus -> OTLP by dashpole · Pull Request #4956 · open-telemetry/opentelemetry-specification

dashpole · 2026-03-17T19:44:12Z

Part of #4753

Related to open-telemetry/opentelemetry-collector-contrib#45982

Issues with the current state

Prometheus scrape identity (job / instance) and OpenTelemetry semantic identity (service.name / service.instance.id) are similar, but ultimately aren't identical concepts, which manifests in a few ways. Currently, when scraping a target that exposes target_info with service.* labels, receivers are forced to drop either job/instance or service.name/service.instance.id. Additionally, users pushing OTLP to Prometheus expect to query for service.name / service.instance.id like any other resource label, but today must configure server-side flags (keep_identifying_resource_attributes=true) to retain them.

Changes

Separate Storage: Scraped job and instance labels are preserved as distinct OTel Resource Attributes alongside service.name / service.instance.id.
Defaulting & Opt-Out: service.name and service.instance.id MAY default to job and instance when absent on target_info. Implementations MUST provide a configuration option to disable this defaulting behavior.
Aggregated Exporters (OTLP -> Prometheus): Exporters look up stored job and instance resource attributes first, falling back to <service.namespace>/<service.name> and service.instance.id if absent.

Alternatives Considered

Use prometheus.job / prometheus.instance: job and instance are preferred for consistent treatment with other Prometheus labels and for a more intuitive UX in Collector processors / OTTL.
Do not default service.* from job/instance by default: Rejected as the default because it would be a breaking change for existing OTel Collector Prometheus receiver users, and the only benefit is less duplication.

@open-telemetry/prometheus-interoperability @aknuds1 @kln21002 @cyrille-leclerc

ArthurSens · 2026-03-17T21:18:33Z

Hmmmm, to make quicker progress here, would it make sense to focus on the Prometheus -> OTLP transformation and leave OTLP -> Prometheus in a separate PR?

I feel like OTLP -> Prometheus is a lot more complicated, and will require deeper discussions with other parts of OTel, like Infrastructure semantic conventions, OBI, etc 🤔

dashpole · 2026-03-18T00:56:32Z

My main goal is to make sure that everything still round-trips properly. If I make changes to one direction, I usually have to make the inverse change to the other direction to ensure that. I can remove some of the editorial changes to the OTLP -> Prometheus section if that helps. But I would like aggregated exporters to accept prometheus.job and prometheus.instance if we add those to the receiver.

aknuds1

Please see comments :)

github-actions · 2026-04-03T04:04:18Z

This PR was marked stale. It will be closed in 14 days without additional activity.

dashpole · 2026-04-06T19:09:27Z

PoC in the collector: open-telemetry/opentelemetry-collector-contrib@main...dashpole:opentelemetry-collector-contrib:promjob_poc

jmacd

I left a very minor question.

ArthurSens

I did a pass through the PR, and so far, I have just one comment.

I feel like I haven't fully grasped the potential consequences of this change, so I'm taking some extra time to reflect, I hope that's ok :)

ArthurSens

This behavior seems very very similar to the configuration option called keep_identifying_attributes from the Prometheus Server configuration.

I know we're only modifying the data model spec here, not the exporter configuration spec, but I wonder whether we'd like to make adjustments to the exporter spec or the Prometheus configuration so both OTel and Prometheus are aligned.

@aknuds1, I think this is somewhat close to work you've done with @cyrille-leclerc in the past. Do you have anything extra to add?

dashpole · 2026-05-06T18:50:56Z

This behavior seems very very similar to the configuration option called keep_identifying_attributes from the Prometheus Server configuration.

Yeah. If we went with this change, it would make sense to eventually switch that to be true by default (e.g. in the next major version, if we consider it breaking). It would be less likely to be duplicate information, so it would make more sense for users to keep the original ones.

aknuds1 · 2026-05-07T11:10:30Z

I'll have a look.

aknuds1

Please see comments.

aknuds1 · 2026-05-07T17:06:22Z

I think this is somewhat close to work you've done with @cyrille-leclerc in the past. Do you have anything extra to add?

@ArthurSens @dashpole AFAICT, the following implies keep_identifying_resource_attributes=true, without it Prometheus' OTLP endpoint will avoid including service_instance_id and service_name as target_info labels?

The Resource attributes MUST NOT be copied to labels of exported metric families
by default

dashpole · 2026-05-10T02:06:00Z

The Resource attributes MUST NOT be copied to labels of exported metric families
by default

This is meant to say that promote_resource_attributes is empty by default, and promote_all_resource_attributes is false by default (which is the case, currently). LMK if there is wording that can make that clearer.

aknuds1

LGTM sans remaining two suggestions.

jack-berg · 2026-06-09T17:08:20Z

@bogdandrutu, @jmacd, @reyang can you take another look? You're previous approvals were reset.

aknuds1

I have doubts regarding this proposed spec change. Please see comment.

aknuds1 · 2026-06-10T14:29:57Z

+contains all metrics with that `job` and `instance`. `job` and `instance` labels
+MUST be added as resource attributes, and not as metric attributes.


Are you sure it's a good idea to add job and instance labels as resource attributes? OTLP -> Prometheus translation considers (according to spec and implementation) instead service.instance.id and service.name to be the identfiying resource attributes. Also, I think that job and instance resource attributes would be dropped in favour of corresponding labels synthesized by Prometheus when generating target_info.

See this discussion above: #4956 (comment). While our specification stated that service.* attributes were identifying, that was based on a misread of the service attributes specification.

This is now proposing that job and instance are respected if they already exist as resource attributes when ingesting OTLP (since it means that the metric was originally scraped from a prometheus endpoint.

@dashpole What exactly is based on a misread? That Prometheus' OTLP endpoint treats service.instance.id and service.name resource attributes as identifying? If so, how precisely? Changing it would be backwards incompatible, and I would argue outside of this PR's scope.

What about the second problem I raised, i.e. that job and instance resource attributes would be dropped by the Prometheus OTLP endpoint in favour of corresponding labels synthesized from identifying resource attributes when generating the target_info metric?

This is now proposing that job and instance are respected if they already exist as resource attributes when ingesting OTLP

Where is that stated in the PR's proposed text?

since it means that the metric was originally scraped from a prometheus endpoint

@dashpole We can't make that assumption. It's but one possible source of job and instance OTel resource attributes.

We can't make that assumption. It's but one possible source of job and instance OTel resource attributes.

Fair enough. It likely came from a prometheus endpoint, but obviously isn't guaranteed to have. But that isn't really relevant.

What about the second problem I raised, i.e. that job and instance resource attributes would be dropped by the Prometheus OTLP endpoint in favour of corresponding labels synthesized from identifying resource attributes when generating the target_info metric?

I'm not quite sure what you mean. Where in this spec does "labels synthesized from identifying resource attributes when generating the target_info metric" come from? Are you just referring to the fact that we currently derive job/instance from service.name/service.namespace/service.instance.id?

Where in this spec does "labels synthesized from identifying resource attributes when generating the target_info metric" come from?

@dashpole In addition to how it currently works in Prometheus' OTLP endpoint, it's effectively stipulated in the PR's changes:

If job is not present as a resource attribute and service.name is present,
the service.name and service.namespace resource attributes MUST be combined
as <service.namespace>/<service.name>, or <service.name> if namespace is
empty, to form the job metric label; otherwise, job SHOULD be added with an
empty value. If instance is not present as a resource attribute and
service.instance.id is present, service.instance.id MUST be converted to the
instance label; otherwise, instance SHOULD be added with an empty value.

The resulting job and instance labels MUST be added to all OpenTelemetry
metrics that are associated with the Resource, including the target_info metric,
if present.

aknuds1

Please see comment.

aknuds1 · 2026-06-10T17:06:27Z

+If `job` is not present as a resource attribute and `service.name` is present,
+the `service.name` and `service.namespace` resource attributes MUST be combined
+as `<service.namespace>/<service.name>`, or `<service.name>` if namespace is
+empty, to form the `job` metric label; otherwise, `job` SHOULD be added with an
+empty value. If `instance` is not present as a resource attribute and
+`service.instance.id` is present, `service.instance.id` MUST be converted to the
+`instance` label; otherwise, `instance` SHOULD be added with an empty value.


On re-reading the PR, I think this must be where you intend to define that job and instance resource attributes respectively should be identifying iff present, with service.name and service.instance.id as respective fallbacks. That should be clearly spelled out, as the intent is effectively ambiguous as currently written.

There are some issues with such a radical change however:

Users whose resource attributes include job and/or instance (doesn't sound entirely unlikely) will experience that their series identities change, when said attributes differ from (service.namespace/ +)service.name/service.instance.id - or when the latter are absent.

The effect of the keep_identifying_resource_attributes config parameter becomes data-dependent: It can only meaningfully apply to the service.name and service.instance.id resource attributes synthesized into job and instance. When job and/or instance resource attributes take precedence, the former presumably have to be kept on target_info regardless of the parameter.

I thought about this some more in the meantime - I think that projecting the Prometheus meaning of job and instance onto OTel resource attributes is a flawed design. They don't have this intrinsic meaning within the OTel scope, so I do think that the previous proposal within this PR to prefix them with "prometheus." was an objectively better choice. See also my comment above regarding the assumption that job and instance OTel resource attributes would be proof of Prometheus provenance.

We will have to deal with how to make this change in prometheus, but I don't think it will be that difficult. It likely just means it will be opt-in until the next prometheus major release.

I don't quite see how adding a prefix changes whether job/instance have meaning. We copy all other labels from prometheus metrics to otel resource/metric labels without prefixes. It isn't clear to me why these labels in particular need prefixes, but the rest of them don't.

Again, i'm open to having it prefixed, but I don't see why it would be objectively wrong to copy label keys between Prometheus and OpenTelemetry as a general rule.

@dashpole I think there's a misunderstanding around the direction here - AFAICT you are referring to Prometheus -> OTLP conversion, but what I am referring to here is the opposite direction: OTLP -> Prometheus. The specific problem that @krajorama and I are raising is that this proposal would stipulate that the Prometheus OTLP endpoint treat instance and job OTel resource attributes as identifying. We think this is a flawed design, and as a result a problem that will remain even after backwards compatibility itself is no longer required through Prometheus v4.

@krajorama and I would still like have a design doc, so the targeted use cases can be sufficiently clarified. Alternatively we could discuss the proposal during a Prometheus WG call, and produce a doc from that.

as Arve says

krajorama

I agree with @aknuds1 that this is a breaking change for some people.

On Prometheus->OTLP side new labels showing up can in general lead to different results if the query language allows without construct, that is generating series by ignoring some attributes, but allowing everything else.

On OTLP->Prometheus side: user might have knowingly or unknowingly masked job, instance resource attributes by overwriting from service.name, service.id. That would break now as we'd keep job and instance.

Let's discuss at the next WG meeting. See also #4753 (comment)

jmacd · 2026-06-15T18:05:49Z

The (resolved) comment thread #4956 (comment) and the comments above lead me to agree that this will be a breaking change, likely in more than one way. See particularly the comment by @ArthurSens #4956 (comment): we're going to need Collector documentation updates for the users that have become used to the service.* resource attributes.

aknuds1 · 2026-06-22T10:05:02Z

    ([#4605](https://github.com/open-telemetry/opentelemetry-specification/issues/4605))
 - Add optional OpenTelemetry Histogram to Prometheus Native Histogram with Custom Buckets transformation.
  ([#4605](https://github.com/open-telemetry/opentelemetry-specification/issues/4605))
+- Prometheus `job` and `instance` labels are preserved as resource attributes when converting to OTLP.


For my own future reference, so I don't forget; this changelog entry doesn't mention that the PR changes the other way round too: From OTLP to Prometheus.

cyrille-leclerc · 2026-06-24T14:36:21Z

The backward compatibility concerns are real, but I think they're worth it — this PR addresses two pain points that come up a lot in practice:

Correlating Prometheus-scraped metrics with OTel telemetry — right now, the mismatch between job/instance and service.name/service.instance.id makes it genuinely hard to join metrics from the Prometheus receiver with traces, logs, and metrics originating from OTel-instrumented services. Capturing Prometheus job and instance as dedicated resource attributes give us a clean way to carry both without forcing one to win.
Using Prometheus as a backend for OTel infra metrics — many OTel infra metrics lack service.name and service.instance.id. It's great to see the OTel <-> Prometheus spec acknowledging this reality.

Overall a solid step forward — happy to see this getting spec-level attention.

aknuds1 · 2026-06-24T14:40:51Z

Using Prometheus as a backend for OTel infra metrics — many OTel infra metrics lack service.name and service.instance.id. It's great to see the OTel <-> Prometheus spec acknowledging this reality.

How is this solved by respecting job and instance OTel resource attributes as identifying?

cyrille-leclerc · 2026-06-24T18:11:31Z

I want to read the phrasing "If job is not present as a resource attribute and service.name is present..." as a great step forward as it implies that service.name may not be present and this that the current spec of target_info needs to be revisited to clarify what to do of resource attributes when none of job and service.name are present.
Does it make sense?

aknuds1 · 2026-06-24T18:24:45Z

I want to read the phrasing "If job is not present as a resource attribute and service.name is present..." as a great step forward as it implies that service.name may not be present and this that the current spec of target_info needs to be revisited to clarify what to do of resource attributes when none of job and service.name are present. Does it make sense?

The problem is that it doesn't only say that. In practice it says to respect job as an identifying OTel resource attribute. It points to what @krajorama and I have concluded. We need a design doc to hash out in sufficient detail which problems this PR are to solve, and what the solution should look like (and why it should look that way, not least).

cyrille-leclerc · 2026-06-24T20:27:37Z

Point taken.

To keep momentum, I'd suggest prioritizing the Prometheus → OTLP direction and specifically making the derivation of service.name / service.instance.id from job / instance optional. Getting this into the spec matters: without it, real-world OTel + Prometheus deployments could end up diverging from the spec.

jmacd

I think we need a summary; @dashpole will you update the PR description to reflect the nuances discussed in the review thread?

aknuds1

This should be held off on until the design doc reaches consensus.

kangyili · 2026-07-29T15:14:14Z

We scrape KSM with the prometheus receiver and then re-associate each series with the workload it actually describes (the pod, deployment, etc.) using k8s_attributes.

The trouble is that the Prometheus receiver puts service.name = "kube-state-metrics" and service.instance.id = "kube-state-metrics:8080" on the resource. That's the scraper's identity, not the workload the metric is about. And because k8s_attributes only fills in missing values, it can't overwrite them.

Chiming in because this pattern is common enough that a first-class opt-out would save everyone from writing the same workaround to manually drop these attributes

#16285) ## Summary Adds a per-tenant counter `cortex_distributor_otlp_requests_with_job_or_instance_resource_attribute_total{user}` in the distributor. It increments once for each OTLP request whose payload contains at least one `ResourceMetrics` carrying `job` or `instance` as a raw resource attribute key. The detection is folded into the existing ResourceMetrics pass (`inspectOTLPResourceMetrics`, renamed from `observeOTLPFieldsCount`) and short-circuits after the first match, so the overhead is two `pcommon.Map` lookups per resource until a hit — negligible. ⚠️ This is **experimental** and expected to be short-lived. It exists to gather production data for the OpenTelemetry specification PR open-telemetry/opentelemetry-specification#4956, which is deciding how OTel senders should convey `job` and `instance` identity. Once we have enough signal on how prevalent this pattern is across tenants, the metric will most likely be removed. Because it's short-lived and diagnostic in nature, expect it to disappear in a future release; please do not build long-lived dashboards or alerts on it. ## Test plan - [x] `go test ./pkg/distributor/ -run TestInspectOTLPResourceMetrics` — new table-driven test covering no-match, job-only, instance-only, both, and match-on-later-resource cases - [x] `go test ./pkg/distributor/ -run TestHandlerOTLPPush` — existing OTLP handler tests still pass - [x] `go build ./pkg/distributor/... && go vet ./pkg/distributor/...` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

aknuds1 · 2026-08-04T09:42:38Z

@kangyili AFAICT, your problem should be addressed by the "never-derive" option in my proposed design Option C.

opentelemetry-pr-dashboard · 2026-08-11T20:13:56Z

Pull request dashboard status

Waiting on the author · refreshed 2026-08-11 20:13 UTC

Move out of draft to request review.

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

aknuds1 reviewed Mar 18, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

dashpole added this to Prometheus SIG Mar 18, 2026

dashpole moved this to Discussion Needed in Prometheus SIG Mar 18, 2026

github-actions Bot added the Stale label Apr 3, 2026

github-actions Bot removed the Stale label Apr 7, 2026

dashpole changed the title ~~[WIP] Prometheus: update resource translation~~ Prometheus: update resource translation Apr 10, 2026

dashpole marked this pull request as ready for review April 10, 2026 15:41

dashpole requested review from a team as code owners April 10, 2026 15:41

jmacd reviewed Apr 14, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

jmacd approved these changes Apr 14, 2026

View reviewed changes

ArthurSens reviewed Apr 15, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md

ArthurSens moved this from Discussion Needed to In progress in Prometheus SIG Apr 15, 2026

jack-berg assigned dashpole Apr 22, 2026

bogdandrutu reviewed Apr 25, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

bogdandrutu reviewed Apr 25, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

bogdandrutu approved these changes Apr 25, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md

dashpole force-pushed the prom_stabilize_resource branch from 7248acd to 4dda69a Compare May 4, 2026 20:15

ArthurSens reviewed May 6, 2026

View reviewed changes

aknuds1 reviewed May 7, 2026

View reviewed changes

aknuds1 approved these changes May 11, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

Comment thread specification/compatibility/prometheus_and_openmetrics.md Outdated

dashpole commented May 11, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md

dashpole force-pushed the prom_stabilize_resource branch from c5d9afc to 3f74736 Compare June 8, 2026 19:53

cyrille-leclerc reviewed Jun 9, 2026

View reviewed changes

Comment thread specification/compatibility/prometheus_and_openmetrics.md

Comment thread specification/compatibility/prometheus_and_openmetrics.md

Comment thread specification/compatibility/prometheus_and_openmetrics.md

cyrille-leclerc approved these changes Jun 9, 2026

View reviewed changes

ArthurSens mentioned this pull request Jun 9, 2026

OTLP->Prometheus: Clarify what to do when Service Resource Attributes aren't present in the OTLP payload. #5145

Open

dashpole mentioned this pull request Jun 9, 2026

Prometheus -> OTLP: Reconsider server resource attributes from scraped endpoints #5146

Open

Merge branch 'main' into prom_stabilize_resource

f961d80

aknuds1 reviewed Jun 10, 2026

View reviewed changes

ArthurSens mentioned this pull request Jun 10, 2026

exporter/prometheus: Add config option 'resource_constant_labels' open-telemetry/opentelemetry-collector-contrib#48922

Closed

1 task

aknuds1 reviewed Jun 10, 2026

View reviewed changes

krajorama suggested changes Jun 11, 2026

View reviewed changes

aknuds1 reviewed Jun 22, 2026

View reviewed changes

smith approved these changes Jun 26, 2026

View reviewed changes

jmacd reviewed Jul 8, 2026

View reviewed changes

dashpole marked this pull request as draft July 8, 2026 16:46

krajorama mentioned this pull request Jul 20, 2026

[prometheus] Stabilize OTLP->Prometheus: Resource Attributes #4926

Open

aknuds1 suggested changes Jul 24, 2026

View reviewed changes

cyrille-leclerc mentioned this pull request Jul 29, 2026

CAP-3753 add config example for otel kube stack DataDog/opentelemetry-examples#290

Merged

krajorama mentioned this pull request Aug 3, 2026

Distributor: count OTLP requests with job/instance resource attributes grafana/mimir#16285

Merged

3 tasks

opentelemetry-pr-dashboard Bot mentioned this pull request Aug 11, 2026

Pull Request Dashboard #5264

Open

		contains all metrics with that `job` and `instance`. `job` and `instance` labels
		MUST be added as resource attributes, and not as metric attributes.

Uh oh!

Conversation

dashpole commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issues with the current state

Changes

Alternatives Considered

Uh oh!

ArthurSens commented Mar 17, 2026

Uh oh!

dashpole commented Mar 18, 2026

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 3, 2026

Uh oh!

dashpole commented Apr 6, 2026

Uh oh!

Uh oh!

jmacd left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurSens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ArthurSens left a comment

Choose a reason for hiding this comment

Uh oh!

dashpole commented May 6, 2026

Uh oh!

aknuds1 commented May 7, 2026

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aknuds1 commented May 7, 2026

Uh oh!

dashpole commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jack-berg commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aknuds1 Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

dashpole commented Mar 17, 2026 •

edited

Loading

dashpole commented May 10, 2026 •

edited

Loading

jack-berg commented Jun 9, 2026 •

edited

Loading

aknuds1 Jun 11, 2026 •

edited

Loading

aknuds1 Jun 11, 2026 •

edited

Loading

cyrille-leclerc commented Jun 24, 2026 •

edited

Loading

aknuds1 commented Jun 24, 2026 •

edited

Loading

kangyili commented Jul 29, 2026 •

edited

Loading