[PR] docs: record the plan for each native Iceberg write fallback [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new pull request, #6961:
URL: https://github.com/apache/datafusion-comet/pull/6961

   ## Which issue does this PR close?
   
   Closes #5643.
   
   ## Rationale for this change
   
   #5643 asks for a keep-or-lift decision on every native Iceberg write eligibility restriction, recorded in the issue and in `iceberg-writes.md`, and #5644 lists it among the things to finish before 1.2.0 ships. Without those decisions there is no way to tell which fallbacks are intended and which are gaps.
   
   ## What changes are included in this PR?
   
   - A "Plans for each fallback" section at the end of "Native Parquet write eligibility" in the user guide's `iceberg-writes.md`. It places every eligibility rule under one of three headings, with the issue that tracks it:
     - **Planned to be lifted**: the object-storage layout (#6956), S3 settings (#6957), merge-on-read on format version 2 (#6240), bloom filters (#5724), format version 3 row lineage (#6669), more storage locations (#6958), rules for settings iceberg-java's writer ignores (#6955), Parquet settings with a parquet-rs equivalent (#6954), `uuid` and `unknown` columns (#6959), and float and double identity partitions (#6851).
     - **Waiting on work in another project**: `hdfs`, format version 3 deletion vectors, table encryption, `variant`, negative zstd levels (apache/arrow-rs#11545), format version 4, and the `timestamp_ns`, `geometry` and `geography` types.
     - **Not planned**: Avro and ORC, user-supplied location providers, `FileIO` classes and encryption managers, S3 features the native client has no equivalent for, unknown `write.parquet.*` keys, settings iceberg-java itself rejects, and out-of-range gzip and brotli levels.
   
     The lists are bullets rather than tables, so a PR that lifts one rule changes one line instead of re-padding a whole table.
   - The contributor guide's eligibility gate section and the `review-comet-iceberg-write-pr` skill point at these lists, so a PR that lifts or adds a rule updates them.
   
   Two outcomes differ from what #5643 anticipated:
   
   - Table encryption waits on upstream rather than being permanent. The pinned iceberg-rust can already write Iceberg-encrypted Parquet (`ParquetWriterBuilder::with_encryption_manager`), but its encryption work (apache/iceberg-rust#2034) is still landing.
   - Deciding the compression-level rule turned up #6960. On Spark 4.1 with Hadoop 3.4 and Iceberg 1.11, iceberg-java fails a gzip write with a numeric `compression-level` and a brotli write without a codec jar, and the native writer currently accepts both.
   
   ## How are these changes tested?
   
   Documentation only; `prettier --check` passes on the changed files. The decisions were checked against the iceberg-java sources at the four pinned versions (1.5.2, 1.8.1, 1.10.0 and 1.11.0) and against iceberg-rust at the pinned revision. The codec behavior behind #6960 and the zstd level decisions was confirmed by running the writes on stock Spark 3.5.8 and 4.1.1 with the Iceberg runtime, with and without a Comet build.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org