spark: wire DRC createTable

[TimothyW553]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/drc-loadtable-storage [Files changed]
  • stack/drc-loadtable-catalog [Files changed]
    • stack/drc-path-credentials [Files changed]
      • stack/drc-createtable-storage [Files changed]
        • stack/drc-createtable-catalog [Files changed]
          • stack/drc-external-createtable [Files changed]
            • stack/drc-updatetable-storage [Files changed]
              • stack/drc-updatetable-catalog [Files changed]

---

Which Delta project/connector is this regarding?

Spark / Unity Catalog Delta REST Catalog.

Description

This wires Delta REST Catalog managed CREATE TABLE into the Spark Delta catalog path.

Flow:

1. prepareCreateTable calls the UC DRC staging API for eligible managed creates and gets the UC-managed table location plus required Delta properties.
2. Delta writes the initial transaction log at that staged location.
3. createTable runs after the commit and finalizes the UC table using the committed Snapshot metadata: schema, partition columns, protocol, table properties, and location.

This PR only intercepts managed CREATE shapes without an explicit user location. External create support is added in the next stack PR. Existing paths stay unchanged because the DRC path is gated on Unity Catalog, DRC being enabled and supported, no existing table, a create operation, and prepareCreateTable returning Some.

CreateDeltaTableCommand.createTableFunc now receives (CatalogTable, Snapshot) so the DRC finalization step can use the post-commit Delta metadata instead of pre-commit table input.

How was this patch tested?

Added Spark catalog and schema-converter coverage for the managed create flow, including staging success, unsupported create shapes, disabled/unavailable DRC, staging errors, table-id mismatch, safety-net create failure, createTable request body/namespace assertions, protocol feature ordering, property filtering, and Spark-to-DRC schema conversion.

Ran:

build/sbt 'spark/testOnly org.apache.spark.sql.delta.catalog.DeltaCatalogClientSuite org.apache.spark.sql.delta.catalog.DeltaRestSchemaConverterSuite'

Result: 23 tests passed.

Does this PR introduce any user-facing changes?

No released-version user-facing change. This adds internal plumbing for DRC-backed managed table creation when the experimental catalog option is enabled.