[Storage] Expose Kernel schema in metadata API

[TimothyW553]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/drc-storage-kernel-schema [Files changed]
  • stack/drc-loadtable-storage [Files changed]
    • stack/drc-loadtable-catalog [Files changed]
      • stack/drc-path-credentials [Files changed]
        • stack/drc-createtable-storage [Files changed]
          • stack/drc-createtable-catalog [Files changed]
            • stack/drc-external-createtable [Files changed]

---

Which Delta project/connector is this regarding?

Storage / Kernel

Description

This PR extends the storage metadata abstraction so Delta code can access table schema as a Kernel StructType, not only as schema JSON.

Without this, each caller that needs typed schema has to parse getSchemaString() itself. That makes UC Delta Rest Catalog API integration harder because the typed schema would be converted in multiple places.

This PR adds:

• storage access to Kernel API types.
• A default AbstractMetadata.getSchema() method returning Kernel StructType.
• Implementations for Spark metadata and Kernel metadata adapters.
• A Spark-side test that verifies Metadata.getSchema parses schema JSON through Kernel serde.

AbstractMetadata.getSchemaString() remains for existing callers.

How was this patch tested?

Local verification:

./build/sbt "storage/compile" "storage/testOnly io.delta.storage.commit.uccommitcoordinator.UCTokenBasedRestClientSuite" "spark/testOnly org.apache.spark.sql.delta.ActionSerializerSuite"
./build/sbt scalafmtAll javafmtAll
git diff --check

Downstream stack verification:

./build/sbt "storage/compile" "storage/testOnly io.delta.storage.commit.uccommitcoordinator.UCTokenBasedRestClientSuite"
./build/sbt "spark/testOnly org.apache.spark.sql.delta.catalog.UCDeltaRestCatalogApiSchemaConverterSuite org.apache.spark.sql.delta.catalog.DeltaCatalogClientSuite"
./build/sbt "sparkUnityCatalog/testOnly io.sparkuctest.UCDeltaTableReadTest"

Does this PR introduce any user-facing changes?

No.

[yili-db]

Do you plan to use this io.delta.kernel.unitycatalog.adapters.MetadataAdapter in Delta Storage and Spark?
Can we do that?

[TimothyW553]

no. and im pretty sure we cant do that either.

1. kernel-unitycatalog depends on storage. so if we let storage use this adapter thats a circular dependency
2. spark holds a regular Metadata object - but the constructor for MetadataAdapter takes in kernel metadata - public MetadataAdapter(Metadata kernelMetadata). of course we can convert Spark Metadata to Kernel Metadata but that seems unnecessary

[yili-db]

This would force org.apache.spark.sql.delta.actions.Metadata to import Kernel StructType too. But can it (a non-v2 class) do that?

[TimothyW553]

I changed getSchema() to a default method, so normal AbstractMetadata implementers are not forced to import Kernel StructType. Spark Metadata still overrides it because this PR intentionally exposes Kernel schema at the storage boundary, and Delta-Spark already depends on storage

[yili-db]

I don't know why Kernel API can't be dependency without shading.

[TimothyW553]

yeah i tried just doing storage.dependsOn(kernelApi) but it ran into problems. here's the story:

1. Kernel source code uses normal Jackson names:

com.fasterxml.jackson.databind.ObjectNode

2. When Kernel builds its published jar, Delta renames Jackson packages inside that jar:

com.fasterxml.jackson...

becomes:

io.delta.kernel.shaded.com.fasterxml.jackson...

this way Kernel does not fight with Spark/Hadoop/other libraries that may use different Jackson versions.

3. I first wanted the simple build change:

storage.dependsOn(kernelApi)

4. But in sbt, .dependsOn(kernelApi) does not use Kernel’s final shaded jar. It uses Kernel’s raw compiled class directory.

That raw class directory still references:

com.fasterxml.jackson...

5. Other modules/tests were using the final shaded Kernel jar, which references:

io.delta.kernel.shaded.com.fasterxml.jackson...

6. Then runtime had two different ObjectNode classes:

com.fasterxml.jackson.databind.node.ObjectNode
io.delta.kernel.shaded.com.fasterxml.jackson.databind.node.ObjectNode

Java sees those as totally different classes. So this failed:

  com.fasterxml.jackson.databind.node.ObjectNode cannot be cast to
  io.delta.kernel.shaded.com.fasterxml.jackson.databind.node.ObjectNode

Fix:

Instead of:

storage.dependsOn(kernelApi)

we make storage compile against Kernel’s final shaded jar:

Compile / unmanagedJars += (kernelApi / Compile / packageBin).value

And we still add delta-kernel-api to the published POM, so external users get the normal dependency.