You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/17 03:17:54 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-7052-f180d65f18d058ba35528c945153cf9be146d04b created (now 79ab485ff4)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-7052-f180d65f18d058ba35528c945153cf9be146d04b
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 79ab485ff4 docs: list known Parquet and Iceberg issues and add a data source audit log (#7052)
This branch includes the following new commits:
new 79ab485ff4 docs: list known Parquet and Iceberg issues and add a data source audit log (#7052)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: docs: list known Parquet and Iceberg issues and add a data source audit log (#7052)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7052-f180d65f18d058ba35528c945153cf9be146d04b
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 79ab485ff4529cb618618a67934953b8690db741
Author: Andy Grove <ag...@apache.org>
AuthorDate: Sat Oct 17 03:17:32 2026 +0000
docs: list known Parquet and Iceberg issues and add a data source audit log (#7052)
* docs: list known Parquet and Iceberg correctness issues and add a data source audit log
Add a compatibility page that lists the open issues in which a Parquet or
Iceberg read or write gives a different result from Spark without falling
back, with links to the issues, and point to it from the compatibility
overview, the Parquet scan page, the Iceberg guide and the Iceberg writes
page. The datetime rebasing item moves from scans.md to the new page.
Correct the Iceberg writes page's claim that the float/double null-count
difference cannot change IS NULL pruning: iceberg-java's counts are the
wrong ones (apache/iceberg#17561).
Start a Data Source Audits section in the contributor guide, next to the
expression audits, recording the nested-type audits of the native Parquet
and Iceberg scans and writers.
* docs: add Parquet and Iceberg queries that fail where Spark succeeds
Add a section to the known issues page for the open issues in which a
Parquet or Iceberg read or write fails with an error where Spark, or
iceberg-java, succeeds: Parquet and Iceberg reads, object store and
credential settings that the native clients resolve differently from
Hadoop, and memory failures in the native writers.
The page becomes "Known Issues in Parquet and Iceberg Reads and Writes".
The existing sections move under "Results that differ from Spark" and keep
their heading anchors, and the links from the other pages use the new
title.
* docs: drop the Spark 4.1 NullType read limitation
parquet-rs accepts the UNKNOWN logical type on any physical type since
apache/arrow-rs#9855, which the parquet 59 crate includes, so reading a
Spark 4.1 Parquet file with a NullType column no longer fails (#4199).
* docs: address review feedback on the known issues page
Drop #6264 and #6837, which are fixed on main. Scope the datetime rebase
entry to the native Parquet scan and say the Delta scan rebases. Add the
ignored Parquet writer properties from #5304 and note that the row group
failure did not reproduce. Explain how apache/iceberg#17320 covers the
identity partition source case. Reword several entries, break the list of
Spark SQL tests into one bullet each, and note that spark.comet.scan.enabled
is a testing setting.
---
.../data-source-audits/iceberg.md | 108 +++++++++
.../contributor-guide/data-source-audits/index.md | 42 ++++
.../data-source-audits/parquet.md | 108 +++++++++
docs/source/contributor-guide/index.md | 1 +
.../user-guide/latest/compatibility/index.md | 6 +-
.../compatibility/parquet-iceberg-known-issues.md | 266 +++++++++++++++++++++
.../user-guide/latest/compatibility/scans.md | 21 +-
.../latest/compatibility/spark-versions.md | 11 -
docs/source/user-guide/latest/iceberg-writes.md | 13 +-
docs/source/user-guide/latest/iceberg.md | 4 +
docs/source/user-guide/latest/index.rst | 1 +
11 files changed, 553 insertions(+), 28 deletions(-)
diff --git a/docs/source/contributor-guide/data-source-audits/iceberg.md b/docs/source/contributor-guide/data-source-audits/iceberg.md
new file mode 100644
index 0000000000..41f89bd1e0
--- /dev/null
+++ b/docs/source/contributor-guide/data-source-audits/iceberg.md
@@ -0,0 +1,108 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Iceberg Audits
+
+> Audit notes for the native Iceberg scan and the native Iceberg writer. Absence of an entry means
+> the area has not been audited yet. Open issues are listed for users in
+> [Known Issues in Parquet and Iceberg Reads and Writes].
+
+## Native Iceberg scan
+
+### Nested types (audited 2026-10-15)
+
+- **Scope:** structs, arrays and maps. NULLs at every level, field ids, name mappings, schema
+ evolution, deletes and partitions.
+- **Method:** queries compared with Spark with Comet disabled, on Spark 4.1.3 with Iceberg 1.11 at
+ `main` 66c63bd307. A code review of iceberg-rust at the current pin ran alongside, and each
+ finding was reproduced before it was filed.
+- **Filed:** [#7033] (a data file that omits an optional nested field fails the scan), [#7038]
+ (residual predicates on struct fields are not pushed into iceberg-rust) and [#7039] (a file is
+ read in full when the query needs none of its columns). The audit also confirmed the open issues
+ [#6790], [#6794], [#6822], [#6993], [#6782], [#5540] and [#5257]. All are tracked in [#7040].
+- **Matched Spark:** the nested data of the Parquet scan audit written by iceberg-java,
+ merge-on-read position deletes across three-row pages (format v2, and v3 deletion vectors),
+ equality deletes including one with a NULL key, required nested fields under NULL parents, nested
+ UUID, fixed, timestamptz and timestamp_ntz leaves, nested type promotion inside structs, lists
+ and maps, the nested schema evolution fallback for lists and maps, `_partition` with NULL values
+ and spec evolution, and nested partition sources.
+- **Did not reproduce:** reordered struct siblings of the same type swapping values when the column
+ holds a map, in more than 20 probes, CTAS included.
+- **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12
+ reads a NULL struct as a struct of NULLs when the partition source is the struct's first field.
+ [apache/iceberg#17320] fixed it in 1.12.0. Its title names nested initial defaults, but the
+ change also keeps a file column in the read when every projected field of a struct is a constant,
+ and an identity partition value is such a constant. The native scan returns NULL, as 1.12 does.
+- **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11.
+
+## Native Iceberg writer
+
+### Nested types (audited 2026-10-16)
+
+- **Scope:** NULL structs, arrays and maps, NULLs inside them, field ids at every level, manifest
+ metrics, nested partition sources, sort orders, copy-on-write DML, and `NOT NULL` columns.
+- **Method:** each write made natively and with Comet disabled, on Iceberg 1.5.2 (Spark 3.4.3),
+ 1.8.1 (Spark 3.5.9) and 1.11.0 (Spark 4.1.3) at `main` c230bf4e98. The comparison covered the
+ values read back, the manifest metrics of each data file (value, null and NaN counts, lower and
+ upper bounds), the Parquet footer, and the files pruned for nested predicates. A code review of
+ the writer, arrow-rs 59.3 and iceberg-rust ran alongside, and each finding was reproduced before
+ it was filed.
+- **Filed:** [#7049] (writes into `NOT NULL` columns fall back on Spark 3.5 and later when no
+ exchange sits in front of the writer, because Comet has no native `AssertNotNull`) and [#7050]
+ (the user guide's claim about float and double null counts). Both are tracked in [#7051].
+- **Matched iceberg-java:** values, pruning, and field ids for struct fields, list elements and map
+ keys and values, on all three versions. Manifest metrics under the default and `full` metrics
+ modes on all three versions, and on 1.11.0 also under `counts`, `truncate(4)`, per-column
+ overrides on nested fields, `max-inferred-column-defaults`, and tiny pages and row groups. Nested
+ partition sources (identity on `s.a`, `bucket`, `truncate` and `days(ts.t)`), a sort order on a
+ nested field, and copy-on-write `DELETE`, `UPDATE` and `MERGE`. Required nested fields under NULL
+ parents written straight from a native scan, `fixed` fields under NULL structs, and copy-on-write
+ `UPDATE` with `AssertNotNull`, whose check still runs in Spark in front of the columnar shuffle.
+- **Not Comet bugs:** on Iceberg 1.9 and later, the value and null counts of a float or double field
+ under a NULL struct differ from iceberg-java's ([#6659], closed), and iceberg-java's are the wrong
+ ones ([apache/iceberg#17561], with a fix in [apache/iceberg#17560]). With Comet disabled on
+ 1.11.0, `WHERE s.x IS NULL` returned 0 rows instead of 5, while 1.5.2 and 1.8.1 returned 5.
+- **Did not reproduce:** a task failure from validating a required struct under an optional struct,
+ with a required child, when a native scan feeds the writer. A task failure from `fixed(16)` values
+ of the wrong length hidden under NULL structs.
+- **Not measured:** the copy of every column that the writer makes when a float or double sits
+ under a list or map, to count NaNs correctly.
+- **Not covered:** merge-on-read writes (`WriteDelta`), format v3 types, encryption, object stores,
+ and write throughput, which needs a release build.
+
+[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md
+[apache/iceberg#17320]: https://github.com/apache/iceberg/pull/17320
+[apache/iceberg#17560]: https://github.com/apache/iceberg/pull/17560
+[apache/iceberg#17561]: https://github.com/apache/iceberg/issues/17561
+[#5257]: https://github.com/apache/datafusion-comet/issues/5257
+[#5438]: https://github.com/apache/datafusion-comet/issues/5438
+[#5540]: https://github.com/apache/datafusion-comet/issues/5540
+[#6659]: https://github.com/apache/datafusion-comet/issues/6659
+[#6782]: https://github.com/apache/datafusion-comet/issues/6782
+[#6790]: https://github.com/apache/datafusion-comet/issues/6790
+[#6794]: https://github.com/apache/datafusion-comet/issues/6794
+[#6822]: https://github.com/apache/datafusion-comet/issues/6822
+[#6993]: https://github.com/apache/datafusion-comet/issues/6993
+[#7033]: https://github.com/apache/datafusion-comet/issues/7033
+[#7038]: https://github.com/apache/datafusion-comet/issues/7038
+[#7039]: https://github.com/apache/datafusion-comet/issues/7039
+[#7040]: https://github.com/apache/datafusion-comet/issues/7040
+[#7049]: https://github.com/apache/datafusion-comet/issues/7049
+[#7050]: https://github.com/apache/datafusion-comet/issues/7050
+[#7051]: https://github.com/apache/datafusion-comet/issues/7051
diff --git a/docs/source/contributor-guide/data-source-audits/index.md b/docs/source/contributor-guide/data-source-audits/index.md
new file mode 100644
index 0000000000..bc0df6d06a
--- /dev/null
+++ b/docs/source/contributor-guide/data-source-audits/index.md
@@ -0,0 +1,42 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Data Source Audits
+
+This section records audits of Comet's native data source paths, the scans and writers for Parquet
+and Iceberg, in the same way that the [Expression Audits](../expression-audits/index.md) record
+audits of expressions. Each page holds dated entries per component. An entry records:
+
+- what was audited, at which commit, against which Spark and library versions, and how
+- the issues the audit filed, and the open issues it confirmed
+- what was checked and matched, so that it is not re-audited without a reason
+- the candidates that did not reproduce, so that they are not chased again
+- the differences that are bugs in Spark or iceberg-java rather than in Comet
+- what the audit did not cover
+
+A later audit of the same area adds a new dated entry instead of editing an old one, so the history
+stays visible. The open issues that users can hit are listed in the user guide, in
+[Known Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md).
+
+```{toctree}
+:maxdepth: 1
+
+Parquet <parquet>
+Iceberg <iceberg>
+```
diff --git a/docs/source/contributor-guide/data-source-audits/parquet.md b/docs/source/contributor-guide/data-source-audits/parquet.md
new file mode 100644
index 0000000000..ad7fad0efd
--- /dev/null
+++ b/docs/source/contributor-guide/data-source-audits/parquet.md
@@ -0,0 +1,108 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parquet Audits
+
+> Audit notes for the native Parquet scan and the native Parquet writer. Absence of an entry means
+> the area has not been audited yet. Open issues are listed for users in
+> [Known Issues in Parquet and Iceberg Reads and Writes].
+
+## Native Parquet scan
+
+### Nested types (audited 2026-10-15)
+
+- **Scope:** structs, arrays and maps. NULL structs, arrays and maps, NULLs inside them, Parquet
+ field ids, case-insensitive names and read schemas that differ from the file's.
+- **Method:** about 3,000 queries compared with Spark with Comet disabled, on Spark 3.4.3, 3.5.9
+ and 4.1.3 at `main` 66c63bd307. A code review of DataFusion 55.2 and arrow-rs 59.3 ran alongside,
+ and each finding was reproduced before it was filed.
+- **Filed:** [#7029] (INT96 timestamps inside legacy-format lists), [#7030] (a nested column with
+ none of its requested fields in the file), [#7031] (Spark 3.4: a struct matched by field id under
+ a new name, under nested pruning), [#7032] (nested BINARY values that are not valid UTF-8, read as
+ STRING), [#7034] (`TIMESTAMP(NANOS)` read as `timestamp`), [#7035] (a nested encoding that
+ arrow-rs rejects fails the whole file), [#7036] and [#7037] (nested pruning gaps), and
+ [apache/datafusion#26393] upstream for [#7029]. All are tracked in [#7040].
+- **Matched Spark:** NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at
+ every level, in deep (eight levels) and wide (300 fields) structs. Spark-written files in five
+ layouts (default, tiny row groups and pages, the legacy format, v2 pages without dictionaries and
+ v2 tiny pages), read with batch sizes of 1 to 3, with row-filter pushdown on and off and with
+ filter pushdown off. Avro, Thrift, Hive and protobuf list and map encodings, maps with struct and
+ array keys, nested type promotion, and nested timestamp, decimal, boolean and binary leaves
+ (micros, millis, and INT96 in standard layouts). ENUM, JSON, BSON, unsigned and binary-decimal
+ annotations, files written by arrow-rs, partitioned tables, nested schemas that differ across the
+ files of one scan, case-insensitive nested names, nested field ids (renamed, missing and partly
+ present), and both modes of SPARK-53535 on Spark 4.1.
+- **Not Comet bugs:** with `spark.sql.parquet.fieldId.read.enabled=true` and a read schema that
+ swaps two nested field names, Spark's vectorized reader pairs the file's columns with the read
+ fields by the file's names, so it fails or returns no rows, while Comet resolves the fields by
+ id. On Spark 3.5 and later, nested schema pruning drops a root field id, so Spark reads a renamed
+ root as NULL, and Comet matches it there. A map with NULL keys from a non-Spark writer fails in
+ both engines.
+- **Not covered:** Variant ([#5438]), the V2 Parquet source, encrypted files, and the JVM-side
+ conversion paths.
+
+## Native Parquet writer
+
+### Nested types (audited 2026-10-16)
+
+- **Scope:** NULL structs, arrays and maps, NULLs inside them, Parquet field ids, column statistics,
+ and the file's schema and footer.
+- **Method:** each write made natively and by Spark with Comet disabled, on Spark 3.4.3, 3.5.9 and
+ 4.1.3 at `main` c230bf4e98. The comparison covered the values read back by Spark and by Comet,
+ the Parquet schema with its repetition and field ids, the footer key-value metadata and the
+ statistics of every leaf. A code review of the writer and arrow-rs 59.3 ran alongside, and each
+ finding was reproduced before it was filed.
+- **Filed:** [#7047] (the datetime rebase modes are ignored, so Spark rejects or shifts the
+ writer's dates before 1582), [#7048] (`outputTimestampType`, `writeLegacyFormat` and Spark's
+ schema footer key), and a measurement on [#5304]: for nested rows the row-count cap gives one row
+ group per file, 239 MB for 100,000 rows of 100-element arrays, where Spark writes row groups of
+ about 128 MB. All are tracked in [#7051].
+- **Matched Spark:** values, repetition and field ids for NULL structs, arrays and maps, empty
+ arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes
+ input sliced by `LIMIT` and `OFFSET`, filtered input, and nested values computed by expressions
+ before the write. Field ids at every level, with `spark.sql.parquet.fieldId.write.enabled` on and
+ off. Null counts, value counts, minimums and maximums for every leaf. Nested timestamp, decimal,
+ boolean and binary leaves.
+- **Did not reproduce:** a write of void (`NullType`) columns on Spark 3.4 and 3.5, which Spark
+ rejects at analysis before Comet plans the write. A write of ANSI interval columns, which did not
+ reach the native writer on any of the three versions. A task failure from the in-progress row
+ group outgrowing a 256 MB off-heap pool: the 239 MB row group fit.
+- **Not covered:** Variant ([#5433]), partitioned and bucketed writes, which the native writer does
+ not support ([#3193], [#3194]), object-store and HDFS outputs, and write throughput, which needs a
+ release build.
+
+[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md
+[apache/datafusion#26393]: https://github.com/apache/datafusion/issues/26393
+[#3193]: https://github.com/apache/datafusion-comet/issues/3193
+[#3194]: https://github.com/apache/datafusion-comet/issues/3194
+[#5304]: https://github.com/apache/datafusion-comet/issues/5304
+[#5433]: https://github.com/apache/datafusion-comet/issues/5433
+[#5438]: https://github.com/apache/datafusion-comet/issues/5438
+[#7029]: https://github.com/apache/datafusion-comet/issues/7029
+[#7030]: https://github.com/apache/datafusion-comet/issues/7030
+[#7031]: https://github.com/apache/datafusion-comet/issues/7031
+[#7032]: https://github.com/apache/datafusion-comet/issues/7032
+[#7034]: https://github.com/apache/datafusion-comet/issues/7034
+[#7035]: https://github.com/apache/datafusion-comet/issues/7035
+[#7036]: https://github.com/apache/datafusion-comet/issues/7036
+[#7037]: https://github.com/apache/datafusion-comet/issues/7037
+[#7040]: https://github.com/apache/datafusion-comet/issues/7040
+[#7047]: https://github.com/apache/datafusion-comet/issues/7047
+[#7048]: https://github.com/apache/datafusion-comet/issues/7048
+[#7051]: https://github.com/apache/datafusion-comet/issues/7051
diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md
index 1abd8891d2..a773c1b2ed 100644
--- a/docs/source/contributor-guide/index.md
+++ b/docs/source/contributor-guide/index.md
@@ -95,6 +95,7 @@ Tracing <tracing>
:hidden:
Expression Audits <expression-audits/index>
+Data Source Audits <data-source-audits/index>
Supported Spark Configurations <spark_configs_support>
Configuration Conventions <config_conventions>
```
diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md
index a74daf64f7..bc364e14a4 100644
--- a/docs/source/user-guide/latest/compatibility/index.md
+++ b/docs/source/user-guide/latest/compatibility/index.md
@@ -24,6 +24,8 @@ Comet aims to provide consistent results with the version of Apache Spark that i
This guide documents areas where Comet's behavior is known to differ from Spark. Topics are grouped by subsystem:
- **Parquet**: limitations when reading Parquet files.
+- **Parquet and Iceberg known issues**: open issues in which Parquet and Iceberg reads and writes
+ give different results from Spark, or fail where Spark succeeds, without falling back.
- **Floating-point comparison**: NaN and signed-zero handling in comparisons.
- **Regular expressions**: differences between the Rust regexp crate and Java's regex engine.
- **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning.
@@ -119,7 +121,9 @@ divergence:
The following native paths silently return values that differ from Spark for edge-case inputs.
Most also have entries in the per-category expression pages linked above; they are collected here
-so users hunting an unexpected value have a single place to check:
+so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and
+writes have their own list, in
+[Known Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md#results-that-differ-from-spark):
- Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns
diff --git a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md
new file mode 100644
index 0000000000..77a0a2501d
--- /dev/null
+++ b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md
@@ -0,0 +1,266 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Known Issues in Parquet and Iceberg Reads and Writes
+
+This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not
+fall back to Spark. Each entry links to its tracking issue. The entries are in two sections:
+
+- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where
+ Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written
+ files that Spark or other engines read differently from the files Spark writes.
+- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds).
+
+The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in
+the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in
+[Iceberg Writes](../iceberg-writes.md).
+
+Which entries apply depends on which native paths are enabled:
+
+| Native path | Setting | Default |
+| -------------- | ---------------------------------------- | ---------------------- |
+| Parquet scan | `spark.comet.scan.enabled` | `true` |
+| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` |
+| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` |
+| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) |
+
+`spark.comet.scan.enabled` is a testing setting that Comet's own test suites use. It is not meant
+for production, so do not turn it off to avoid an entry on this page.
+
+## Results that differ from Spark
+
+### Parquet reads
+
+- **Dates and timestamps written with the legacy calendar.** Spark before 3.0, and later versions
+ with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`, write these values in the hybrid
+ Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values
+ before October 15, 1582 are wrong in projections and predicates. It ignores
+ `spark.sql.parquet.datetimeRebaseModeInRead` and the file's `org.apache.spark.legacyDateTime`
+ metadata ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). The
+ [Delta scan](../delta.md) rebases.
+- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262
+ read back wrong inside lists written in a legacy layout, as Hive and
+ `spark.sql.parquet.writeLegacyFormat=true` write them
+ ([#7029](https://github.com/apache/datafusion-comet/issues/7029)).
+- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the
+ column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5
+ and 4.0, and to Spark 4.1 and later with
+ `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true`
+ ([#7030](https://github.com/apache/datafusion-comet/issues/7030)).
+- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY
+ column without the UTF8 annotation that is read as `string`, through the read schema or
+ `spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each
+ such nested value. Top-level values match Spark in a release build, because the invalid bytes
+ reach an Arrow string array unchecked. A debug build panics on them
+ ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of
+ [#4764](https://github.com/apache/datafusion-comet/issues/4764)).
+- **Spark 3.4: a struct matched by field id under a different name.** With
+ `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to
+ the file by field id but under another name reads as NULL when Spark prunes its nested fields
+ ([#7031](https://github.com/apache/datafusion-comet/issues/7031)).
+- **Field-id reads where one file column matches a field by id and a sibling by name.** With
+ `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top
+ level. Inside a struct, it returns the values of the field matched by id. Spark's own readers
+ disagree here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)).
+- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and
+ in every column of a scan that has data filters, Comet returns NULL where Spark fails with
+ `ArithmeticException: long overflow`
+ ([#5553](https://github.com/apache/datafusion-comet/issues/5553),
+ [#5739](https://github.com/apache/datafusion-comet/issues/5739)).
+- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for
+ `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz`
+ ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as
+ `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and
+ Variant-annotated fields read as a plain struct
+ ([#5741](https://github.com/apache/datafusion-comet/issues/5741)).
+
+### Iceberg reads
+
+- **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files`
+ procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query
+ also reads a top-level primitive column, the native scan returns NULL for nested fields (struct
+ fields, list elements and map values). Filters on those fields then select the wrong rows
+ ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust
+ upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)).
+- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such
+ a file as NULL, and the native scan returns the values stored in the file
+ ([#6794](https://github.com/apache/datafusion-comet/issues/6794)).
+- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as
+ NULL, and two columns that swapped names return each other's values
+ ([#6822](https://github.com/apache/datafusion-comet/issues/6822)).
+- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or
+ metadata through reflection, it continues with an empty or partial result instead of falling
+ back, which can resolve a column to a different field id
+ ([#5257](https://github.com/apache/datafusion-comet/issues/5257)).
+
+### Parquet writes
+
+The native Parquet writer is experimental and disabled by default. With
+`spark.comet.write.parquet.enabled=true`:
+
+- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is
+ `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark
+ refuses to write, and Spark then fails to read them back under its default read mode. With
+ `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on
+ every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need
+ no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047),
+ [#3427](https://github.com/apache/datafusion-comet/issues/3427)).
+- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are
+ ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default
+ `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's
+ schema footer key is not written either, so a path-based read loses column metadata such as
+ comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ`
+ columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048),
+ [#3425](https://github.com/apache/datafusion-comet/issues/3425)).
+- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** Comet skips the steps Spark runs
+ after a write: file index refresh, cache refresh, partition refresh and statistics update. Later
+ queries can return the table's previous contents until `REFRESH TABLE`
+ ([#6315](https://github.com/apache/datafusion-comet/issues/6315)).
+- **Spark's Parquet writer properties are ignored.** `parquet.block.size`, page size, dictionary,
+ writer version and statistics settings never reach the native writer
+ ([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
+- **An input with no partitions writes no file.** Spark writes one file that holds the schema
+ ([#5303](https://github.com/apache/datafusion-comet/issues/5303)).
+- **Spark SQL tests that still fail with the native writer**, not yet classified:
+ - CTAS schema ([#3418](https://github.com/apache/datafusion-comet/issues/3418))
+ - CTAS with `CHAR_AS_VARCHAR` ([#3419](https://github.com/apache/datafusion-comet/issues/3419))
+ - CTAS with a union ([#3429](https://github.com/apache/datafusion-comet/issues/3429))
+ - `INSERT INTO` with default columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421))
+ - `INSERT INTO` after `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422))
+ - `INSERT INTO` from non-projection queries ([#3423](https://github.com/apache/datafusion-comet/issues/3423))
+ - the `INSERT OVERWRITE` output schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424))
+ - multiple inserts ([#3430](https://github.com/apache/datafusion-comet/issues/3430))
+ - `INSERT OVERWRITE LOCAL DIRECTORY` ([#3420](https://github.com/apache/datafusion-comet/issues/3420))
+ - empty files ([#3417](https://github.com/apache/datafusion-comet/issues/3417))
+ - `onTaskCommit` ([#3428](https://github.com/apache/datafusion-comet/issues/3428))
+
+### Iceberg writes
+
+The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`:
+
+- **Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can
+ read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write
+ instead ([#6960](https://github.com/apache/datafusion-comet/issues/6960)).
+- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, and
+ iceberg-java rejects the value. The native writer also trims or lowercases a few other settings
+ that iceberg-java reads as written
+ ([#7008](https://github.com/apache/datafusion-comet/issues/7008)).
+
+The physical differences that every native data file has, and why they are safe, are listed in
+[Accepted divergences](../iceberg-writes.md#accepted-divergences).
+
+## Queries that fail where Spark succeeds
+
+In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds.
+Comet does not fall back to Spark for them.
+
+### Parquet reads that fail
+
+- **STRING columns that hold invalid UTF-8.** The read fails with
+ `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes
+ ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of
+ [#4764](https://github.com/apache/datafusion-comet/issues/4764)).
+- **Duplicate field names.** Selecting a field by name when several physical siblings match it,
+ including inside structs, arrays and maps, fails with a duplicate-field error
+ ([#5884](https://github.com/apache/datafusion-comet/issues/5884),
+ [#5964](https://github.com/apache/datafusion-comet/issues/5964)).
+ [Parquet Compatibility](scans.md) describes the reads this affects.
+- **A duplicate field id in a column the query does not read.** With a valid read schema and
+ `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same
+ field id fails the scan whenever the table schema has a field with that id, even when the query
+ does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)).
+- **A nested encoding that arrow-rs rejects.** A file fails to read when any column in its schema
+ uses such an encoding, even a column the query does not read
+ ([#7035](https://github.com/apache/datafusion-comet/issues/7035)).
+- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read
+ ([#5188](https://github.com/apache/datafusion-comet/issues/5188)).
+
+### Iceberg reads that fail
+
+- **A data file that omits an optional nested field.** The native scan fails on a data file that
+ lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can
+ produce without any schema change
+ ([#7033](https://github.com/apache/datafusion-comet/issues/7033)).
+- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot**
+ ([#6993](https://github.com/apache/datafusion-comet/issues/6993)).
+- **An equality delete keyed on a nested field.** Every query the native scan runs on the table
+ fails with `field not found`, even one that reads the key
+ ([#6782](https://github.com/apache/datafusion-comet/issues/6782)).
+- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi
+ letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error
+ ([#5540](https://github.com/apache/datafusion-comet/issues/5540)).
+
+### Object store and credential settings
+
+The native Parquet and Iceberg scans resolve storage settings with their own clients rather than
+with Hadoop's. Where the two disagree, a native read can use a different identity or route than
+Spark, and fail where Spark succeeds:
+
+- **S3 settings that Hadoop's S3A client resolves differently**, such as JCEKS-backed secrets and
+ the precedence of per-bucket settings. The native Parquet scan forwards them without checking
+ for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)).
+- **AWS profile credential providers.** The native scans ignore `fs.s3a.auth.profile.name` and
+ `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)),
+ and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use
+ `credential_process` or chains of assumed roles
+ ([#6575](https://github.com/apache/datafusion-comet/issues/6575)).
+- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other
+ configurations resolve differently from Hadoop's ABFS driver
+ ([#6605](https://github.com/apache/datafusion-comet/issues/6605)).
+- **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the
+ credential provider configured with `s3.comet.credential.provider.class`
+ ([#6536](https://github.com/apache/datafusion-comet/issues/6536)).
+
+### Parquet writes that fail
+
+With `spark.comet.write.parquet.enabled=true`:
+
+- **Row groups that outgrow the memory pool.** The native writer cuts row groups by row count
+ only, where Spark uses `parquet.block.size`, and it cannot spill. With wide or nested rows, the
+ open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a
+ failure, because a 239 MB row group fit in a 256 MB pool
+ ([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
+
+### Iceberg writes that fail
+
+With `spark.comet.write.iceberg.enabled=true`:
+
+- **A write that keeps one file open and outgrows the off-heap pool.** An unpartitioned or
+ clustered write fails its task when its open file outgrows the pool, where iceberg-java writes
+ it on the JVM heap. [Failure handling](../iceberg-writes.md#failure-handling) describes the
+ error and the settings that let such a write fit.
+
+## Differences that are not Comet bugs
+
+In these cases Comet's result differs from Spark's, but the bug is in Spark or iceberg-java:
+
+- **iceberg-java 1.9 and later: `IS NULL` on a float or double field under a NULL struct.**
+ iceberg-java leaves the rows where the struct is NULL out of the field's null count, so a filter
+ such as `WHERE s.x IS NULL` can skip data files that hold matching rows, with or without Comet
+ ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). Files written by Comet's
+ native Iceberg writer are not affected.
+- **iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.**
+ iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12
+ does. [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320) is titled for nested
+ initial defaults, but it fixes this case too: it keeps one file column in the read, so the
+ struct's definition level survives when every projected field is a constant, and an identity
+ partition value is such a constant.
+- **Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized
+ reader pairs the file's columns with the read fields by the file's names, so it fails with a type
+ mismatch or returns no rows. Comet resolves the fields by id.
diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md
index 7b7d2c198e..ec6142713f 100644
--- a/docs/source/user-guide/latest/compatibility/scans.md
+++ b/docs/source/user-guide/latest/compatibility/scans.md
@@ -55,18 +55,15 @@ The following features are not supported and cause Comet to fall back to Spark:
`spark.sql.caseSensitive=false`. Spark's analyzer normally rejects such a schema before the scan
is planned.
-The following limitation may produce incorrect results without falling back to Spark:
-
-- No support for datetime rebasing. When reading Parquet files containing dates or timestamps
- written with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` (which is Spark's default for
- data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as
- if they were written using the Proleptic Gregorian calendar. This produces silently-wrong
- values for dates before October 15, 1582 in both projections and predicates. Comet also
- ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level
- `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. Tracked by
- [#5010](https://github.com/apache/datafusion-comet/issues/5010).
-
-The following limitations raise an error at scan time rather than falling back to Spark:
+Some Parquet reads return results that differ from Spark without falling back to Spark, such as
+reads of dates and timestamps written with the legacy hybrid Julian/Gregorian calendar
+([#5010](https://github.com/apache/datafusion-comet/issues/5010)).
+[Known Issues](parquet-iceberg-known-issues.md#parquet-reads) lists them, with the
+issues that track them.
+
+The following limitations raise an error at scan time rather than falling back to Spark. More
+reads that fail where Spark succeeds are listed in
+[Known Issues](parquet-iceberg-known-issues.md#parquet-reads-that-fail):
- Selecting a field by name when multiple physical siblings match, including inside structs,
arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision.
diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md
index 84baf6194f..f178b679f9 100644
--- a/docs/source/user-guide/latest/compatibility/spark-versions.md
+++ b/docs/source/user-guide/latest/compatibility/spark-versions.md
@@ -69,17 +69,6 @@ Spark 4.0.4 is supported with Java 17/21 and Scala 2.13.
Spark 4.1.3 is supported with Java 17/21 and Scala 2.13.
-### Known Limitations
-
-- **`NullType` columns in Parquet files**
- ([#4199](https://github.com/apache/datafusion-comet/issues/4199)): Spark encodes a `NullType`
- column as a Parquet `BOOLEAN` physical type annotated with `LogicalType::Unknown`. The Rust
- `parquet` crate that Comet depends on accepts `Unknown` only when paired with `INT32` and rejects
- any other physical type with `Parquet error: Cannot annotate Unknown from BOOLEAN for field '<name>'`.
- Any attempt to read a Parquet file that contains a `NullType` column fails at decode time before
- Comet's scan runs. Workaround: project the column away, cast it to a concrete type before
- persisting, or read the file with Comet disabled for that query.
-
## Spark 4.2
Spark 4.2.0 is supported with Java 17 and Scala 2.13.
diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md
index bba4ea8ec1..840916411e 100644
--- a/docs/source/user-guide/latest/iceberg-writes.md
+++ b/docs/source/user-guide/latest/iceberg-writes.md
@@ -494,9 +494,14 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed
change pruning decisions.
- On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns
nested under a nullable struct count rows whose parent struct is null (they come from the
- parquet footer), while iceberg-java's writer-tracked counts do not. Both counts inflate by
- the same amount, so the derived null ratios and `IS NULL` / `IS NOT NULL` pruning decisions
- are unaffected.
+ parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the
+ correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null
+ struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a
+ file that contains matching rows
+ ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts
+ never prune such a file. `IS NOT NULL` pruning is the same for both.
All content not listed above — the logical data, statistics values, and manifest metadata —
-must match iceberg-java exactly, or the write falls back.
+must match iceberg-java exactly, or the write falls back. The open bugs in which a native write
+still differs are listed in
+[Known Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-writes).
diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md
index 0febcbb918..0df6319f60 100644
--- a/docs/source/user-guide/latest/iceberg.md
+++ b/docs/source/user-guide/latest/iceberg.md
@@ -201,6 +201,10 @@ The following scenarios will fall back to the JVM Iceberg reader:
name that the scan's other columns don't use (every task of a native scan must give such a
column the same name)
+The cases in which the native reader returns different results from Spark, or fails where Spark
+succeeds, without falling back are listed in
+[Known Issues](compatibility/parquet-iceberg-known-issues.md).
+
Writes are not covered by this list. By default Spark plans Iceberg writes with its own V2 write
operator, and iceberg-java writes the data files. With `spark.comet.write.iceberg.enabled=true`,
Comet plans an Iceberg write with its split-operator plan and writes eligible data files natively;
diff --git a/docs/source/user-guide/latest/index.rst b/docs/source/user-guide/latest/index.rst
index 84b0064647..b4cfe077ca 100644
--- a/docs/source/user-guide/latest/index.rst
+++ b/docs/source/user-guide/latest/index.rst
@@ -60,6 +60,7 @@ the navigation menu to read more.
Overview <compatibility/index>
compatibility/scans
+ compatibility/parquet-iceberg-known-issues
compatibility/floating-point
compatibility/regex
compatibility/operators
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org