[PR] feat: support native Iceberg UUID writes [datafusion-comet]

["sam-1112 (via GitHub)" <gi...@apache.org>]

sam-1112 opened a new pull request, #7198:
URL: https://github.com/apache/datafusion-comet/pull/7198

   ## Which issue does this PR close?
   
   Partially addresses #6959: implements UUID writes. The `unknown` type remains unsupported.
   
   ## Rationale for this change
   
   Spark represents Iceberg UUID columns as strings. The native writer currently declines these columns because Arrow casting does not provide Java-compatible UUID parsing.
   
   This change enables native UUID writes while preserving iceberg-java’s value, metrics and failure-cleanup semantics.
   
   ## What changes are included in this PR?
   
   - Add Java-compatible UUID parsing, including shortened groups, leading plus signs, Unicode digits, group masking and overflow rejection.
   - Convert UUID fields recursively through structs, lists and map keys/values, using Iceberg field IDs to distinguish UUID from `fixed(16)`.
   - Skip invisible values under null parents and outside sliced collection ranges. Preserve the existing cast path for UUID-free subtrees.
   - Allow UUID data columns while retaining fallback for UUID partition/sort sources and unsupported v3 types.
   - Add reproducible Java oracle fixtures, regression coverage, documentation and dedicated UUID benchmark workloads.
   
   Native UUID files use `FIXED_LEN_BYTE_ARRAY(16)` with Iceberg field IDs and big-endian UUID bytes. The pinned iceberg-rust writer omits the Parquet UUID logical annotation. This physical-layout difference is documented and tested: Iceberg readers use the table schema, while standalone Parquet readers relying on the annotation see fixed-length binary.
   
   ## How are these changes tested?
   
   Latest local validation:
   - 9 Rust UUID tests passed, including multi-level null visibility, nonzero map offsets, string representations and `fixed(16)` isolation.
   - 14 Spark 4.1 UUID tests passed, covering native/Java row and metrics parity, footer metadata, writer modes, AQE, eligibility and failure cleanup for flat UUIDs, list elements and map values.
   - Java oracle fixtures matched on JDK 17 and 21: 2,516 UUID inputs plus BMP digit classification.
   - Clippy with warnings denied, compilation and formatting checks passed.
   
   Earlier regression runs passed 174 native Iceberg tests and 211 tests across the complete Spark 4.1 detection/action suites. UUID-focused runs also passed on Spark 3.4/3.5/4.0/4.1 with JDK 17 and Spark 4.0/4.1 with JDK 21. These runs preceded the final test-only coverage additions.
   
   <html><head></head><body><p>Local benchmarks used an Apple M4 (16 GiB), JDK 17, Spark 4.1.3 and Iceberg 1.11.0, with a release native build and local storage. Each workload processed 4,194,304 rows, with at least five measured iterations per arm.</p>
   <p>Writer factor = median time of Comet scan + Java writer / median time of Comet scan + native writer.</p>
   
   UUID workload | Run 1 writer factor | Run 2 writer factor
   -- | -- | --
   Canonical UUIDs | 1.77× | 1.99×
   Canonical UUIDs with approximately 1/8 nulls | 1.53× | 1.08×
   Java-compatible shortened UUIDs | 1.46× | 1.21×
   
   
   <p>All arms passed native-plan and checksum validation. The null workload showed substantial variability: native write times ranged from 206.78 to 715.45 ms in the second run. A non-UUID control was also measured, but no matched pre-change baseline was collected, so it does not establish the absence of a non-UUID performance regression.</p>
   </body></html>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org