You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/09 17:27:55 UTC
[I] Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot [datafusion-comet]
andygrove opened a new issue, #6822:
URL: https://github.com/apache/datafusion-comet/issues/6822
### Describe the bug
A `VERSION AS OF` read of a snapshot taken before a column was renamed returns wrong results from the native Iceberg scan. The renamed column comes back as NULL. When two columns swap names, each one returns the other's values. Spark returns the snapshot's data, and reads of the current snapshot are correct.
`CometIcebergNativeScan.serializePartitions` gives a task with no deletes the current table schema, and a task with deletes `FileScanTask.schema()`, which is also the current schema. It switches to the scan schema only when the scan reads a column the current schema no longer has (`hasHistoricalColumns`). A renamed column keeps its field id, so that check doesn't fire. iceberg-rust then names each output column after the current schema, and Comet matches those columns to the snapshot names Spark expects by name. A renamed column finds no match and reads as NULL. After a swap, `projectFieldIds` also resolves each output name in the task schema first, where the name now belongs to the other column.
### Steps to reproduce
```sql
CREATE TABLE cat.db.t (id INT, a STRING, b STRING) USING iceberg;
INSERT INTO cat.db.t SELECT CAST(id AS INT), concat('A', id), concat('B', id) FROM range(20);
-- <s> is the snapshot id after the insert
ALTER TABLE cat.db.t RENAME COLUMN a TO z;
SELECT id, a FROM cat.db.t VERSION AS OF <s> ORDER BY id;
```
On `main` at 00a4b422f0 and on `branch-1.1` at e9efd9f764 (Spark 4.1, Iceberg 1.11):
| After the snapshot | Query | Spark | Comet |
| --- | --- | --- | --- |
| `a` renamed to `z` | `SELECT id, a ... VERSION AS OF <s>` | `[0,A0] [1,A1] [2,A2] …` | `[0,null] [1,null] [2,null] …` |
| `a` and `b` swapped through a temporary name | `SELECT id, a, b ... VERSION AS OF <s>` | `[0,A0,B0] [1,A1,B1] …` | `[0,B0,A0] [1,B1,A1] …` |
| `a` and `b` swapped | `SELECT id, b ... VERSION AS OF <s>` | `[0,B0] [1,B1] …` | `[0,A0] [1,A1] …` |
The results are the same when the snapshot has position deletes. Spark 3.4 with Iceberg 1.5.2 gives the same results too. I checked that through #6725 with its pruning turned off, which takes the same path as `main` for these queries.
### Expected behavior
The native scan returns the snapshot's data, as Spark does. A `VERSION AS OF` task needs a schema with the snapshot's column names whenever they differ from the current schema's, not only when a column was dropped.
### Additional context
Found while reviewing #6725. That PR reads with Spark's pruned scan schema by default, which carries the snapshot's names, so these queries match Spark with it. The bug stays on `branch-1.1`, with `spark.comet.scan.icebergNative.nestedSchemaPruning.enabled=false`, and for tasks that keep the full schema because the query pruned away a nested partition source or equality-delete key. A swap where the scan also has to append a partition source on one of the swapped columns fails at planning with #6725 (`Invalid schema: multiple fields for name b`), which is discussed on that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #6822:
URL: https://github.com/apache/datafusion-comet/issues/6822#issuecomment-6128964279
Potentially iceberg-rust issue, need to check if there are similar issues raised there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #6822:
URL: https://github.com/apache/datafusion-comet/issues/6822#issuecomment-6129020434
I took this to double check if its iceberg-rust issue and if this already fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot [datafusion-comet]
Posted by "comphead (via GitHub) via github" <gi...@datafusion.apache.org>.
comphead commented on issue #6822:
URL: https://github.com/apache/datafusion-comet/issues/6822#issuecomment-6537196254
setting priority medium as the issue is isolated and only for reading specific snapshots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org