[Storage] Add UC Delta API credential property mapping by TimothyW553 · Pull Request #6677 · delta-io/delta

TimothyW553 · 2026-04-27T23:11:05Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/drc-loadtable-storage [Files changed]
- stack/drc-credential-props [Files changed]
  - stack/drc-loadtable-catalog [Files changed]
    - stack/drc-path-credentials [Files changed]
      - stack/drc-createtable-storage [Files changed]
        
        stack/drc-createtable-catalog [Files changed]
        
        stack/drc-external-createtable [Files changed]

Which Delta project/connector is this regarding?

Spark / Unity Catalog

Description

This PR adds Delta-owned credential property mapping for credentials returned by the UC Delta Rest Catalog API.

UC Delta Rest Catalog API responses contain UC credential models, while Delta file access still needs Hadoop filesystem properties. This PR copies the required UC Spark credential mapping into Delta Spark so later PRs can use the same conversion for both named table credentials and temporary path credentials.

This PR adds:

CredPropsUtil for table-scoped and path-scoped credentials.
AWS, Azure, and GCS credential provider implementations.
Static and renewable credential support.
Optional credential-scoped filesystem overrides.
Delta-owned credential config constants and API client helpers.
Provided Hadoop/GCS dependencies required by the copied providers.
A GCS AccessToken constructor compatibility fix for util-hadoop 3.0.2.

This PR only adds the credential utility layer. Child PRs wire it into UC Delta Rest Catalog API loadTable, path credentials, and create table flows.

How was this patch tested?

Covered by UCDeltaRestCatalogApiCredentialPropsUtilSuite for:

static S3 credential properties
static Azure and GCS credential properties
renewable table credential properties
renewable path credential properties
credential-scoped filesystem overrides
no credential properties for non-UC-vended schemes

Also compiled iceberg/Compile/compile for the GCS compatibility fix.

Does this PR introduce any user-facing changes?

No. This only adds internal Delta Spark credential mapping utilities for the UC Delta Rest Catalog API stack.

TimothyW553 · 2026-04-30T06:20:23Z

+      return self();
+    }
+
+    public T table(String catalog, String schema, String table, String location) {


This mostly follows UC Spark CredPropsUtil; the main extra Delta shape is storing catalog/schema/table/location so renewal can call UC Delta table credential APIs.

TimothyW553 · 2026-04-30T06:20:24Z

+
+package io.delta.storage.unitycatalog.hadoop;
+
+public class UCDeltaRestCatalogApiCredentialConf {


This is a smaller Delta storage copy of UC UCHadoopConf; it keeps only the credential Hadoop keys needed by these helpers.

TimothyW553 · 2026-04-30T06:20:25Z

+    if (conf.get(UCDeltaRestCatalogApiCredentialConf.AZURE_INIT_SAS_TOKEN) != null
+        && conf.get(UCDeltaRestCatalogApiCredentialConf.AZURE_INIT_SAS_TOKEN_EXPIRED_TIME) != null) {
+      String sasToken = conf.get(UCDeltaRestCatalogApiCredentialConf.AZURE_INIT_SAS_TOKEN);
+      Objects.requireNonNull(


This is a straight UC Spark port; the main difference is replacing Guava Preconditions with JDK checks.

TimothyW553 · 2026-04-30T06:20:26Z

+    return new GenericCredential(tempCred);
+  }
+
+  public static GenericCredential fromStorageCredential(StorageCredential storageCredential) {


This adapter is Delta-specific because UC Delta APIs return StorageCredential, while UC Spark renewal returns TemporaryCredentials directly.

TimothyW553 · 2026-04-30T06:20:27Z

+    return credential;
+  }
+
+  protected UCClient ucClient() {


This is the main non-port file: renewal uses Delta UCClient / UCTokenBasedRestClient instead of UC Spark legacy TemporaryCredentialsApi.

TimothyW553 · 2026-04-30T06:20:28Z

+    final long maxSize =
+        Long.getLong(CRED_SCOPED_FS_CACHE_MAX_SIZE, CRED_SCOPED_FS_CACHE_MAX_SIZE_DEFAULT);
+    CACHE =
+        Collections.synchronizedMap(


Same credential-scoped FS logic as UC Spark, but Guava CacheBuilder is replaced with a JDK synchronized LRU map for storage.

openinx · 2026-04-30T17:49:56Z

@@ -0,0 +1,73 @@
+/*


Hi @TimothyW553 , I think one key request is: how could we migrate the tests also from the unitycatalog spark connector to the delta repo also ?

because we don't want any functional regression after we do the migration.

TimothyW553 · 2026-05-01T20:27:09Z

Closing this because PR3 now uses unitycatalog-hadoop directly, so this copied credential mapping PR is no longer needed.

TimothyW553 requested a review from tdas as a code owner April 27, 2026 23:11

TimothyW553 force-pushed the stack/drc-credential-props branch from f4893d8 to 8bc003e Compare April 27, 2026 23:29

TimothyW553 mentioned this pull request Apr 28, 2026

[Spark] Add UC Delta Rest Catalog API path credentials for raw path reads #6682

Open

TimothyW553 force-pushed the stack/drc-credential-props branch 7 times, most recently from e433d7e to eab7835 Compare April 29, 2026 07:57

TimothyW553 changed the title ~~spark: copy UC credential property mapping~~ spark: copy UC Delta Rest Catalog API credential property mapping Apr 29, 2026

TimothyW553 force-pushed the stack/drc-credential-props branch 2 times, most recently from 6dd01e2 to 04445d1 Compare April 29, 2026 23:30

TimothyW553 changed the title ~~spark: copy UC Delta Rest Catalog API credential property mapping~~ spark: copy UC Delta API credential property mapping Apr 30, 2026

TimothyW553 force-pushed the stack/drc-credential-props branch 2 times, most recently from c5085fe to 05f90da Compare April 30, 2026 05:30

TimothyW553 commented Apr 30, 2026

View reviewed changes

TimothyW553 changed the title ~~spark: copy UC Delta API credential property mapping~~ [Spark] Copy UC Delta API credential property mapping Apr 30, 2026

TimothyW553 force-pushed the stack/drc-credential-props branch from 05f90da to 7672549 Compare April 30, 2026 06:28

TimothyW553 requested a review from openinx April 30, 2026 06:35

TimothyW553 requested a review from yili-db April 30, 2026 06:35

TimothyW553 changed the title ~~[Spark] Copy UC Delta API credential property mapping~~ [Storage] Add UC Delta API credential property mapping Apr 30, 2026

TimothyW553 force-pushed the stack/drc-credential-props branch from 7672549 to 3c158ad Compare April 30, 2026 06:57

openinx reviewed Apr 30, 2026

View reviewed changes

TimothyW553 force-pushed the stack/drc-credential-props branch from 3c158ad to 5dacf5d Compare April 30, 2026 18:55

TimothyW553 added 2 commits April 30, 2026 22:11

storage: add UC Delta Rest Catalog API loadTable client APIs

18ea16e

storage: add UC Delta credential property mapping

e0b7139

TimothyW553 force-pushed the stack/drc-credential-props branch from 5dacf5d to e0b7139 Compare April 30, 2026 22:15

TimothyW553 closed this May 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Storage] Add UC Delta API credential property mapping#6677

[Storage] Add UC Delta API credential property mapping#6677
TimothyW553 wants to merge 2 commits into
delta-io:masterfrom
TimothyW553:stack/drc-credential-props

TimothyW553 commented Apr 27, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

TimothyW553 Apr 30, 2026 •

edited

Loading

Uh oh!

openinx Apr 30, 2026

Uh oh!

TimothyW553 commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		package io.delta.storage.unitycatalog.hadoop;

		public class UCDeltaRestCatalogApiCredentialConf {

Conversation

TimothyW553 commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimothyW553 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openinx Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

TimothyW553 commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

TimothyW553 commented Apr 27, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading

TimothyW553 Apr 30, 2026 •

edited

Loading