Skip to content

Commit 016025d

Browse files
committed
[Storage] Add tableIdentifier to UCClient getCommits
Adds `TableIdentifier` to `UCClient#getCommits` so UC clients can receive the catalog/schema/table name when fetching commits. The identifier is forwarded from `UCCommitCoordinatorClient` when available in the `TableDescriptor`. Kernel catalog-managed snapshot loading also has an overload that forwards the identifier, and Spark v2 UC snapshot metadata now carries the identifier from `CatalogTable` into that Kernel path. The legacy `UCTokenBasedRestClient` accepts the new argument but keeps sending the existing legacy request fields. `tableIdentifier` is the three-part `catalog.schema.table` name (not the UC UUID `tableId`); callers pass it when they have catalog context and null otherwise, and receivers either require it (rejecting null) or ignore it. Resolves #6784.
1 parent b5e5aec commit 016025d

80 files changed

Lines changed: 45162 additions & 52 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/scheduled_tasks.lock

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"sessionId":"e69e5f03-4812-4feb-bdbf-a2e7922c0e90","pid":202773,"procStart":"1346462","acquiredAt":1778887804850}

.codex

Whitespace-only changes.

AGENTS.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# AGENTS.md
2+
3+
Delta Lake is an open-source Lakehouse storage framework. This monorepo contains Spark, Kernel, Flink, and other connectors.
4+
5+
## Build System
6+
7+
- **Tool**: SBT (`build/sbt`)
8+
- **Scala**: 2.13.17 (Scala 2.12 support dropped)
9+
- **Java**: JDK 17+ required for building
10+
- **Version**: Set in `version.sbt` (`ThisBuild / version := "X.Y.Z-SNAPSHOT"`)
11+
12+
## Project Modules
13+
14+
| Module | Location | Description |
15+
|--------|----------|-------------|
16+
| `spark` | `spark/` | Delta Spark v1 implementation (no tests) |
17+
| `spark-unified` | `spark-unified/` | Published delta-spark JAR (v1+v2 merged) |
18+
| `spark/v2` | `spark/v2/` | Kernel-based DSv2 connector |
19+
| `kernel-api` | `kernel/kernel-api/` | Java APIs for building Delta connectors |
20+
| `kernel-defaults` | `kernel/kernel-defaults/` | Default implementations (Hadoop-based) |
21+
| `storage` | `storage/` | Storage layer (Spark-independent) |
22+
| `flink` | `flink/` | Flink sink connector |
23+
| `iceberg` | `iceberg/` | Iceberg integration |
24+
| `hudi` | `hudi/` | Hudi integration |
25+
26+
**Internal modules** (not published): `sparkV1`, `sparkV1Filtered`, `sparkV2`
27+
28+
## Key Commands
29+
30+
```bash
31+
# Compile and package
32+
build/sbt compile
33+
build/sbt package
34+
35+
# Run all tests
36+
build/sbt test
37+
38+
# Run tests for specific group
39+
python3 run-tests.py --group spark --shard 0
40+
python3 run-tests.py --group kernel --shard 0
41+
42+
# Run a single test suite
43+
build/sbt 'spark/testOnly org.apache.spark.sql.delta.DeltaLogSuite'
44+
45+
# Run a single test method
46+
build/sbt 'spark/testOnly *.DeltaLogSuite -- -z "test name"'
47+
48+
# Kernel tests
49+
build/sbt 'kernelApi/test'
50+
build/sbt 'kernelDefaults/test'
51+
52+
# Flink
53+
build/sbt flink/test
54+
build/sbt flink/assembly
55+
```
56+
57+
## Spark Version Configuration
58+
59+
Defined in `project/CrossSparkVersions.scala`. The `sparkVersion` property controls which Spark version to build against:
60+
61+
```bash
62+
build/sbt -DsparkVersion=4.0 compile # Spark 4.0.x
63+
build/sbt -DsparkVersion=4.1 compile # Spark 4.1.x (default)
64+
build/sbt -DsparkVersion=4.2 compile # Spark 4.2.x (if available)
65+
```
66+
67+
**Current configuration**:
68+
- Default: Spark 4.1.0 (JVM 17)
69+
- Master: None
70+
- Iceberg support: Spark 4.0 only
71+
72+
## Test Sharding
73+
74+
Tests are parallelized using `NUM_SHARDS` and `SHARD_ID` environment variables:
75+
76+
```bash
77+
NUM_SHARDS=8 SHARD_ID=0 build/sbt test # Run shard 0 of 8
78+
TEST_PARALLELISM_COUNT=4 build/sbt test # 4 parallel JVMs
79+
```
80+
81+
Duration data for optimal shard assignment is stored in `project/test-durations.csv`.
82+
83+
## Code Style
84+
85+
```bash
86+
# Scala formatting (scalafmt v3.8.6, config: .scalafmt.conf)
87+
build/sbt scalafmtCheckAll
88+
build/sbt scalafmtAll
89+
90+
# Java formatting (google-java-format)
91+
build/sbt javafmtCheckAll
92+
build/sbt javafmtAll
93+
94+
# Scala style (scalastyle)
95+
build/sbt spark/scalastyle
96+
```
97+
98+
Scala imports are sorted: `java.*`, `scala.*`, `io.delta.*`, `org.apache.spark.sql.delta.*`
99+
100+
## Testing Environment Variables
101+
102+
```bash
103+
DELTA_TESTING=1 # Enables table feature testing
104+
SPARK_LOCAL_IP=127.0.0.1 # For connect tests
105+
```
106+
107+
## Multi-Module Test Configuration
108+
109+
Tests are configured differently per module:
110+
- **spark**: `parallelExecution := false` (can't have multiple Sparks in same JVM)
111+
- **kernel**: Multi-shard parallel execution via `MultiShardMultiJVMTestParallelization`
112+
- **sparkUnityCatalog**: `parallelExecution := false` (UC server per JVM)
113+
114+
## Python Tests
115+
116+
```bash
117+
# Setup (conda environment from python/environment.yml)
118+
conda env create --file python/environment.yml
119+
conda activate delta_python_tests
120+
121+
# Run
122+
python3 python/run-tests.py
123+
```
124+
125+
## Important Notes
126+
127+
1. **Generated code**: `kernel-api` creates `Meta.java` with version at compile time. `sparkV1` generates `python/delta/version.py`.
128+
129+
2. **Shaded kernel JAR**: `kernelApi/Compile/packageBin` runs assembly; other kernel modules depend on it:
130+
```bash
131+
build/sbt 'kernelDefaults/test' # Automatically builds shaded JAR first
132+
```
133+
134+
3. **Iceberg build quirk**: `build/sbt icebergShaded/compile iceberg/compile` may fail on first run—rerun succeeds.
135+
136+
4. **Test resources**: Delta Spark tests look for resources in `spark/src/test/resources/`.
137+
138+
5. **Apache Spark Scala Style Guide** applies: https://spark.apache.org/contributing.html

DO_NOT_MERGE_delta_table_combination_for_streaming.md

Lines changed: 246 additions & 0 deletions
Large diffs are not rendered by default.

Delta-Behind-IRC_High-Level_Design.md

Lines changed: 1033 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)