Discuss adding Gravitino support to the OpenLineage community

[FANNG1]

Background

Gravitino has OpenLineage integration support today, but the plugin is not open sourced in the OpenLineage community. This makes it harder to maintain, review, evolve, and align with OpenLineage's dataset model.

I would like to discuss whether we should first contribute Gravitino support to the OpenLineage community.

Context

OpenLineage dataset naming is datasource-oriented. For example, the Spark Iceberg integration can emit the physical storage dataset as the primary dataset and add a table identifier through the symlinks facet.

For Gravitino, the logical resource model is different. A table is normally identified through:

• metalake
• catalog
• schema
• table

This does not map directly to a single OpenLineage dataset name without choosing a convention.

Proposal

Add Gravitino support in the OpenLineage community first, and keep emitted dataset identifiers consistent with OpenLineage conventions.

On the Gravitino side, the server can translate OpenLineage dataset identifiers into Gravitino's internal resource model. For example, Gravitino could resolve a dataset by using one or more of:

• dataset namespace
• dataset name
• symlinks facet
• catalog dataset facet or custom Gravitino facet
• Gravitino-specific configuration such as the target metalake

This keeps OpenLineage producers aligned with OpenLineage naming, while allowing Gravitino to map lineage events back to metalake.catalog.schema.table.

References:

• OpenLineage naming conventions: https://openlineage.io/docs/spec/naming
• Spark Iceberg handler: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark3/src/main/java/io/openlineage/spark3/agent/lifecycle/plan/catalog/iceberg/IcebergHandler.java
• Spark Iceberg example event with symlinks: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/integrations/container/pysparkV2ReplaceTableAsSelectCompleteEvent.json

[FANNG1]

@jerryshao @yuqi1129 @shaofengshi WDYT?

[FANNG1]

I also looked at the current Polaris OpenLineage discussion and PoC branch:

• Polaris dev thread: https://www.mail-archive.com/dev@polaris.apache.org/msg04430.html
• Follow-up mentioning an earlier/alternative direction: https://www.mail-archive.com/dev@polaris.apache.org/msg04447.html
• PoC branch: https://github.com/iting0321/polaris/tree/data-lineage

Based on the public thread and PoC code, there seem to be two related directions being discussed in Polaris:

1. The current PoC emits OpenLineage events from Polaris-managed Iceberg table operations. It hooks into Polaris persistence events, generates OpenLineage RunEvent JSON for table create/update/drop operations, stores that JSON in the persisted Polaris event under an openlineage property, and adds an HTTP listener that posts the payload to a Marquez/OpenLineage endpoint.
2. The earlier/alternative proposal mentioned in the Polaris thread is to make Polaris an OpenLineage server implementation. In that model, compute engines would send OpenLineage events to Polaris through OpenLineage APIs, and Polaris could either persist lineage internally or forward it to a downstream OpenLineage backend such as Marquez.

The PoC maps Polaris/Iceberg events roughly as follows:

• synthetic job namespace: polaris.<realm>.<catalog>
• synthetic job name: <event_type>:<table_identifier>
• dataset namespace: Iceberg table namespace
• dataset name: Iceberg table name
• datasource facet: derived from the Iceberg table location, with a Polaris URN fallback
• dataset facets: schema, snapshot/version, lifecycle state, and basic output statistics

One important limitation is that the PoC tries to infer CTAS/input lineage at the Polaris server side by tracking AFTER_LOAD_TABLE events and correlating them with later create/update events using spark.app.id or app-id from Iceberg snapshot summaries. That can work for a demo or coarse table-level lineage, but it is heuristic:

• the same Spark app can run multiple independent SQL statements;
• table load order does not always mean data dependency;
• cached data, temporary views, retries, and concurrent operations can make the inferred lineage inaccurate;
• non-Spark engines or engines that do not propagate an app id cannot be correlated;
• Polaris/Gravitino server side does not see the query plan, so it cannot provide operator-level or reliable column-level lineage.

For Gravitino, I think this means we should be careful about relying only on server-side lineage inference. A better separation may be:

• engine-side OpenLineage integrations emit the authoritative query/table lineage, because engines can see the execution plan;
• Gravitino server resolves OpenLineage dataset identifiers back to metalake.catalog.schema.table and enriches the lineage with catalog metadata;
• Gravitino server may still emit supplemental lifecycle/metadata events, such as table create/drop/alter, schema changes, snapshot/version changes, and policy/ownership changes.

This keeps Gravitino aligned with OpenLineage conventions while avoiding inaccurate lineage inference from catalog-server observations alone.

[jerryshao]

Seems reasonable to me. I think you can take a crack on this.