Hi team, I have table.sanitized.sql table schema in Iceberg stored as parquet and when replicating this into lance table using spark and lance storage format 2.2 with blob v2, we noticed storage size is 1.7x larger. Could you help take a look? Please let me know other info you need, thanks!
Additional information:
- Spark: OSS 3.5.6
- Iceberg version: OSS 1.5.2
- Lance-Spark: OSS latest
- Iceberg table size:: ~66TiB, lance table ~115TiB
- blob data size ~90KB average
Lance table creation query
set spark.driver.memory=128g;
set spark.executor.memory=32g;
set spark.executor.memoryOverhead=16g;
set spark.dynamicAllocation.maxExecutors=5000;
set spark.driver.maxResultSize=32g;
set spark.executorEnv.LANCE_INITIAL_UPLOAD_SIZE=5368709120;
SET spark.sql.adaptive.coalescePartitions.enabled=false;
SET spark.sql.shuffle.partitions=50000;
DROP TABLE IF EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2;
CREATE TABLE IF NOT EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2
USING lance
TBLPROPERTIES (
-- 'image_bytes.lance.col1987' = 'blob',
'col1987.lance.encoding' = 'blob',
'file_format_version' = '2.2'
)
AS
SELECT /*+ REPARTITION(50000) */ *
FROM spark_catalog.pyang.lance_multimodal_iceberg_full_iceberg;
Hi team, I have table.sanitized.sql table schema in Iceberg stored as parquet and when replicating this into lance table using spark and lance storage format 2.2 with blob v2, we noticed storage size is 1.7x larger. Could you help take a look? Please let me know other info you need, thanks!
Additional information:
Lance table creation query