You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rich7420 (via GitHub)" <gi...@apache.org> on 2026/10/20 18:23:20 UTC
[I] Native Iceberg scan drops NULL rows for NOT (col <=> value) [datafusion-comet]
rich7420 opened a new issue, #7137:
URL: https://github.com/apache/datafusion-comet/issues/7137
### Describe the bug
Iceberg converts Spark's null-safe `id <=> 1` to `id = 1`. Comet pushes `NOT (id = 1)` into iceberg-rust, which evaluates it as `id != 1` and drops NULL ids. Spark keeps them, because `NOT (NULL <=> 1)` is true. The post-scan filter cannot restore them.
### Steps to reproduce
```sql
CREATE TABLE t (id INT) USING iceberg;
INSERT INTO t VALUES (NULL), (1), (2);
SELECT * FROM t WHERE NOT (id <=> 1);
```
Reproduced at 9e1364914.
### Expected behavior
`NULL` and `2`, as Spark returns. Comet returns only `2`.
### Additional context
Found by @sunchao in #6847, whose partial pushdown also exposed it for mixed predicates. Spark parses `id IS DISTINCT FROM 1` and `b IS NOT TRUE` into the same negated `<=>`. #6847 stops pushing equality under `NOT` for nullable columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org