You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/23 13:33:09 UTC

[PR] docs: fix stale and missing native Iceberg write details [datafusion-comet]

andygrove opened a new pull request, #6150:
URL: https://github.com/apache/datafusion-comet/pull/6150

   ## Which issue does this PR close?
   
   Closes #6147.
   
   ## Rationale for this change
   
   The user guide pages for Iceberg had several statements about native writes that no longer match the code, and were missing two things a user needs to know before enabling the native writer. Found in an audit of the native Iceberg write path (#5649).
   
   ## What changes are included in this PR?
   
   `docs/source/user-guide/latest/iceberg-writes.md`:
   - The accepted-divergences list said float and double partition directories are rendered with Rust's shortest representation (`f=1` where iceberg-java writes `f=1.0`). Since #5840 they use Java's `Float.toString` / `Double.toString` rules and match iceberg-java, so that divergence is removed. The claim that Iceberg deprecated float and double partitioning in 1.3 is also removed; identity partitioning on float and double is accepted on every Iceberg version Comet tests.
   - The manifest section said the parity suite compares committed manifests byte for byte. `CometIcebergWriteActionSuite` compares the committed `readable_metrics` of tables written by both writers, so the sentence now says that.
   - New paragraph under "Native Parquet write eligibility": the native writer needs a Comet input, so writes fed by a local relation (`INSERT ... VALUES`, `df.writeTo(...).append()` on local data) only use it when `spark.comet.exec.localTableScan.enabled=true`. The configuration example now includes that setting, and its comment for `spark.comet.iceberg.write.enabled` describes the flag as the native writer rather than "eligibility detection".
   - New accepted divergence: high-cardinality columns keep a dictionary page that iceberg-java would not write (#6114).
   
   `docs/source/user-guide/latest/iceberg.md`:
   - The native scan's fallback list included "Iceberg writes (reads are accelerated, writes use Spark)". Writes are not a scan fallback, and there is now a native writer, so the bullet is replaced by a sentence after the list that points to the writes page.
   
   ## How are these changes tested?
   
   Documentation only. `prettier --check` passes on both files. Each corrected statement was checked against the code on `main`: `java_float_string` in `iceberg_partition_path.rs`, the `readable_metrics` comparisons in `CometIcebergWriteActionSuite`, and `CometIcebergNativeWrite.requiresNativeChildren`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] docs: fix stale and missing native Iceberg write details [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6150:
URL: https://github.com/apache/datafusion-comet/pull/6150#issuecomment-5826184291

   Thanks @rich7420 and @sunchao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] docs: fix stale and missing native Iceberg write details [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6150:
URL: https://github.com/apache/datafusion-comet/pull/6150


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org