You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/23 13:18:50 UTC
[I] docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md [datafusion-comet]
andygrove opened a new issue, #6147:
URL: https://github.com/apache/datafusion-comet/issues/6147
### What is the problem the feature request solves?
The Iceberg user guide has errors and gaps about native writes:
- `docs/source/user-guide/latest/iceberg-writes.md:273-276` says float/double partition directories use Rust's shortest representation (`f=1` where iceberg-java writes `f=1.0`). Since #5840 they use Java's `Double.toString` rules and match iceberg-java.
- `iceberg-writes.md:316-319` says the parity suite compares committed manifests byte-for-byte. `CometIcebergWriteActionSuite` compares aggregated `readable_metrics` (min/max, counts), not manifest bytes.
- `iceberg-writes.md` does not say that writes whose input is a local relation, such as `INSERT ... VALUES` and `df.writeTo(...).append()` on a local DataFrame, need `spark.comet.exec.localTableScan.enabled` (default `false`). Without it they silently use iceberg-java even with both write flags on.
- `iceberg-writes.md` does not list #6114 (dictionary page kept for high-cardinality columns) among the accepted divergences.
- `docs/source/user-guide/latest/iceberg.md:184` lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behind `spark.comet.iceberg.write.enabled`; that page should point to `iceberg-writes.md`.
### Describe the potential solution
Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644).
### Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6147: docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md
URL: https://github.com/apache/datafusion-comet/issues/6147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org