Partitions with type UUID fail to create transactions

[JosephLenton]

Apache Iceberg Rust version

None

Describe the bug

When making a table with a UUID partition, using the UUID type, transactions to that table crash.

I'm not sure if this is a case that UUIDs just aren't supported. However the type system suggests I should be able to have a UUID type, and for my needs where I work this would end up being extremely common. As a workaround I'm currently working around this by using a String type instead.

If it helps:

• I have this issue against both version 0.10.0 and main.
• I am using the S3 backend.

Btw I am able to work on this issue full time to help get this resolved. I will need some guidance on where, primarily to avoid going off on tangents.

Also thank you guys for this library. It's already been a huge help!

To Reproduce

I don't have a full reproducible example I'm afraid, as the code I am using is split across a large project.

The steps are:

1. Create a table with a UUID field, which uses the type Type::Primitive(PrimitiveType::Uuid)).
2. Add a partition using the UUID field
3. When writing data, use a UUID partition key.
4. Start a transaction.
5. Commit the transaction.

Pseudo code

The pseudo code for this is along the lines of:

// 1. Create a table with a UUID field
let table = TableCreation::builder()
    .name("ExampleTable".to_string())
    .schema(
        IcebergSchema::builder()
            .with_fields([
                // This is a UUID here
                NestedField::required(1, "uuid", Type::Primitive(PrimitiveType::Uuid))
                    .into(),
                NestedField::optional(2, "value", Type::Primitive(PrimitiveType::String))
                    .into(),
            ])
            .build()
            .unwrap(),
    )
    // 2. Add a partition using the UUID field
    .partition_spec(
        UnboundPartitionSpec::builder()
            .with_spec_id(0)
            .add_partition_fields([UnboundPartitionField::builder()
                .source_id(1)
                .field_id(1)
                .name("uuid".to_string())
                .transform(Transform::Identity)
                .build()])?
            .build(),
    )
    .build();

// 3. When writing data, use a UUID partition key.
let uuid_key = Some(Literal::uuid(Uuid::new_v4()));
let data = Struct::from_iter([uuid_key]);
let partition_key = PartitionKey::new(spec, schema, data);

let rolling_builder = todo!(" ... omitted ...");
let data_writer = DataFileWriterBuilder::new(rolling_builder)
    .build(Some(partition_key))
    .await?;

// 4. Start a transaction.
let data_files = todo!(" ... omitted ...");
let catalog = todo!(" ... omitted ...");
let transaction = Transaction::new(&table);
transaction
    .fast_append()
    .add_data_files(data_files)
    .apply(transaction)?
    // 5. Start a transaction.
    .commit(catalog.as_ref())
    .await?; // <-- it is failing here

Stack trace

Here is part of the stack trace. I have excluded the work code stack trace below, and only included the Iceberg parts:

called `Result::unwrap()` on an `Err` value: Iceberg(DataInvalid => Failure in conversion with avro

Source: Could not find matching type in UnionSchema { schemas: [Null, Uuid], variant_index: {Null: 0, Uuid: 1} } for Bytes([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])


Backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1: std::backtrace_rs::backtrace::trace_unsynchronized::<<std::backtrace::Backtrace>::create::{closure#0}>
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2: <std::backtrace::Backtrace>::create
             at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/backtrace.rs:331:13
   3: <iceberg::error::Error>::new::<&str>
             at /iceberg-rust/crates/iceberg/src/error.rs:234:24
   4: <iceberg::error::Error as core::convert::From<apache_avro::error::Error>>::from
             at ~/iceberg-rust/crates/iceberg/src/error.rs:340:17
   5: <core::result::Result<iceberg::spec::manifest_list::manifest_file::ManifestFile, iceberg::error::Error> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible, apache_avro::error::Error>>>::from_residual
             at ~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2189:27
   6: <iceberg::spec::manifest::writer::ManifestWriter>::write_manifest_file::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/spec/manifest/writer.rs:498:21
   7: <iceberg::transaction::snapshot::SnapshotProducer>::write_added_manifest::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:340:38
   8: <iceberg::transaction::snapshot::SnapshotProducer>::produce_manifests::<iceberg::transaction::append::FastAppendOperation, iceberg::transaction::snapshot::DefaultManifestProcess>::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:367:62
   9: <iceberg::transaction::snapshot::SnapshotProducer>::commit::<iceberg::transaction::append::FastAppendOperation, iceberg::transaction::snapshot::DefaultManifestProcess>::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:494:14
  10: <iceberg::transaction::append::FastAppendAction as iceberg::transaction::action::TransactionAction>::commit::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/append.rs:108:14
  11: <core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = core::result::Result<iceberg::transaction::action::ActionCommit, iceberg::error::Error>> + core::marker::Send>> as core::future::future::Future>::poll
             at ~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:133:9
  12: <iceberg::transaction::Transaction>::do_commit::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:233:75
  13: <iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0}::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:195:48
  14: <backon::retry_with_context::RetryWithContext<backon::backoff::exponential::ExponentialBackoff, iceberg::table::Table, iceberg::error::Error, iceberg::transaction::Transaction, <iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0}::{closure#0}, <iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0}, tokio::time::sleep::sleep, <iceberg::transaction::Transaction>::commit::{closure#0}::{closure#1}, for<'a> fn(&'a iceberg::error::Error, core::time::Duration)> as core::future::future::Future>::poll
             at ~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backon-1.6.0/src/retry_with_context.rs:327:58
  15: <iceberg::transaction::Transaction>::commit::{closure#0}
             at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:202:10
  16: my_library::iceberg::iceberg_data_writer::IcebergDataWriter>::flush::{closure#0}
             at ... this and below is our code ...

I would be happy to write a failing test within the Iceberg repo. I'm not 100% sure where the best place would be to recreate this, so some guidance on that would help.

Expected behavior

This is what I would expect:

• It should not error
• I should get a partition on S3 with a hyphen formatted UUID. i.e. s3://.../uuid=4bc971ed-7590-4552-8408-d60928ba76f0/...

Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community

[JosephLenton]

I have gone ahead and implemented a fix on a PR, which seems to get UUIDs working as UUIDs in my own work.

I'm happy to make any changes to it as needed to help get this over the line.