[I] Native Iceberg writes: support format-version 3 tables [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6668:
URL: https://github.com/apache/datafusion-comet/issues/6668

   ### What is the problem the feature request solves?
   
   The native Iceberg writer declines every write to a table whose `format-version` is 3 or higher (`requireFormatVersionAtMostTwo` in `CometIcebergNativeWrite`). In the Iceberg 1.11 coverage run of 2026-10-04 (merge-queue run 37239317672), that rule sent 1452 writes to iceberg-java, the third-largest fallback bucket.
   
   The rule is broader than it needs to be. The only thing iceberg-java writes differently for a v3 table is row lineage, and it writes lineage columns only when it rewrites existing rows:
   
   - From Iceberg 1.10, copy-on-write `DELETE`, `UPDATE` and `MERGE` add `_row_id` and `_last_updated_sequence_number` to the write schema (`SparkWriteBuilder`), and `rewrite_data_files` does the same through its rewrite table schema. The values come from each row's metadata.
   - Appends, static and dynamic overwrites, and CTAS/RTAS write the data columns only. The driver gives the new rows their ids at commit time (`first_row_id` inheritance), on the same commit path the native writer already uses.
   - Iceberg 1.8 and 1.9 write no lineage columns at all.
   
   v3 also adds column types the native writer cannot produce (`variant`, `unknown`, `timestamp_ns`, `geometry`, `geography`), which the format-version rule currently keeps out as a side effect.
   
   ### Describe the potential solution
   
   - Admit format-version 3, and keep declining 4, which is still being specified.
   - Decline a write whose write schema includes a metadata column, meaning a field id above `Integer.MAX_VALUE - 200`, the range the spec reserves for them. That covers the lineage-carrying writes on every Iceberg version.
   - Extend the `uuid` column-type rule to the v3-only types.
   
   Writing row lineage natively, so that the copy-on-write and compaction writes can run natively too, is a separate follow-up: #6669.
   
   ### Additional context
   
   Part of #5649. This is one of the restrictions listed in #5643. Related: #3376.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Native Iceberg writes: support format-version 3 tables [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6668: Native Iceberg writes: support format-version 3 tables
URL: https://github.com/apache/datafusion-comet/issues/6668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org