[I] Native Iceberg scan applies equality deletes to the wrong column when column names differ only in case [datafusion-comet]

["dwsmith1983 (via GitHub)" <gi...@apache.org>]

dwsmith1983 opened a new issue, #7129:
URL: https://github.com/apache/datafusion-comet/issues/7129

   ### Describe the bug
   
   The native Iceberg scan applies an equality delete to the wrong column when the table has two columns whose names differ only in case, such as `a` and `A`. Deleted rows come back, or the wrong rows are dropped, and which one happens can change from query to query.
   
   `planner.rs` builds every `FileScanTask` with `.with_case_sensitive(false)` (`native/core/src/execution/planner.rs:2672` on 39380cca8). In the pinned iceberg-rust (1f3bc34), the only reader of that flag outside tests binds the equality-delete predicate: `combined_predicate.bind(task.schema_ref(), task.case_sensitive())` in `crates/iceberg/src/arrow/delete_filter.rs:231`. The predicate refers to columns by name (`caching_delete_file_loader.rs`), and a case-insensitive bind resolves them through `Schema::field_by_name_case_insensitive`. That lookup is a map built from a `HashMap`, so `a` and `A` collapse to one field id, picked by iteration order.
   
   Iceberg's Java reader applies equality deletes by field id, so Spark without Comet returns the right rows.
   
   Found while reproducing #5540.
   
   ### Steps to reproduce
   
   With `spark.sql.caseSensitive=true` and the native Iceberg scan enabled:
   
   1. Create a format-version 2 Iceberg table with columns `a INT, A INT`, and insert `(1, 20), (3, 40), (5, 60)`.
   2. Commit two equality delete files through the Iceberg API: one keyed on `a` with value `1`, and one keyed on `A` with value `40`. (Spark SQL `DELETE` writes position deletes, so this needs the API, as `CometIcebergNativeSuite` does with `commitEqualityDelete`.)
   3. Query the table.
   
   Spark returns only `(5, 60)`. Comet's native scan:
   
   | Query | Spark | Comet |
   |---|---|---|
   | `SELECT A ... ORDER BY A` | `[60]` | `[40], [60]` |
   | `SELECT a ... ORDER BY a` | `[5]` | `[1], [5]` |
   | `SELECT count(*)` | `[1]` | `[2]` |
   
   In the same run, the `A = 40` delete was bound to `a` in one query, and the `a = 1` delete was bound to `A` in another.
   
   ### Expected behavior
   
   The same rows as Spark: every equality delete applies to the column its field id names.
   
   ### Additional context
   
   The bound names come from the task schema: iceberg-rust evolves each delete batch to the task schema by its equality field ids before taking column names. So binding case-sensitively against the same schema matches by field id.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org