You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "eneskeles (via GitHub)" <gi...@apache.org> on 2026/10/06 14:18:51 UTC

[I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

eneskeles opened a new issue, #2131:
URL: https://github.com/apache/iceberg-go/issues/2131

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   Came across this one while working on #2111 
   A filter like `NOT(x < 5)` on a float column sometimes matches a NaN row and sometimes does not depending on the other values in the same file.
   
   Scan with `NOT(x < 5)`, one data file per table:
   
   | `x` values in the file | rows returned |
   |---|---|
   | `[10, NaN, 20]` | all three, with the NaN row |
   | `[1, NaN, 2]` | none |
   
   `Delete` with the same filter does the same thing: it removes the NaN row from the first file and keeps it in the second. I see this with both copy-on-write and merge-on-read, on main (0b0828b).
   
   ### Why
   
   The filter is checked in two different forms:
   
   - The row filter runs it as written. `NaN < 5` is false, so `NOT(NaN < 5)` is true and the NaN row matches.
   - The stats checks first call `RewriteNotExpr`, which changes it to `x >= 5`. That is false for NaN. Min/max stats do not include NaN, so the file `[1, NaN, 2]` is skipped before the row filter runs.
   
   ### Repro
   
   ```go
   filter := iceberg.NewNot(iceberg.LessThan(iceberg.Reference("x"), 5.0))
   
   // tblHigh has one file with x = [10, NaN, 20]
   // tblLow  has one file with x = [1, NaN, 2]
   high, _ := tblHigh.Scan(table.WithRowFilter(filter)).ToArrowTable(ctx)
   low, _ := tblLow.Scan(table.WithRowFilter(filter)).ToArrowTable(ctx)
   
   fmt.Println(high.NumRows()) // 3
   fmt.Println(low.NumRows())  // 0
   ```
   
   ### Expected
   
   The NaN row should match or not match the same way in both files.
   
   I think Java rewrites NOT before it evaluates the filter, so `NOT(x < 5)` never matches NaN there. Doing the same here would mean calling `RewriteNotExpr` before the row filter is built too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

Posted by "Revanth14 (via GitHub)" <gi...@apache.org>.
Revanth14 commented on issue #2131:
URL: https://github.com/apache/iceberg-go/issues/2131#issuecomment-6298130509

   I checked Java for this. Its row `Evaluator` doesn't rewrite NOT: it binds the expression directly, and only the metrics evaluators call `rewriteNot`. It also compares doubles with `naturalOrder`, where NaN sorts above every number, so Java's row evaluator says `NaN >= 5` is true while its metrics evaluator prunes `[1, NaN, 2]` on the upper bound. Java has the same split, so it doesn't decide this one.
   
   Two ways to make rows and pruning agree:
   
   - **(a) Rewrite NOT before building the row filter**, as suggested above. Simple, but `NOT(x < 5)` then never matches NaN, unlike IEEE comparison semantics.
   - **(b) NaN-aware NOT rewrite for float/double**: `NOT(x < 5)` → `x >= 5 OR is_nan(x)`. This keeps `NOT(NaN < 5)` true, and pruning stays correct because the `is_nan` branch keeps files with NaN values. It needs the bound type, so it would apply after binding rather than in the exported `RewriteNotExpr`.
   
   I lean towards (b). @zeroshade @laskoviymishka which would you prefer?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #2131:
URL: https://github.com/apache/iceberg-go/issues/2131#issuecomment-6476196401

   I lean towards (b) personally. @laskoviymishka thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

Posted by "zeroshade (via GitHub) via issues" <is...@iceberg.apache.org>.
zeroshade closed issue #2131: NOT(x < 5) on a float column matches NaN rows in some files but not in others
URL: https://github.com/apache/iceberg-go/issues/2131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on issue #2131:
URL: https://github.com/apache/iceberg-go/issues/2131#issuecomment-6493632289

   I'm not entirely sure that we can do option (a) safe anyway, option b looks like obvious choice to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] NOT(x < 5) on a float column matches NaN rows in some files but not in others [iceberg-go]

Posted by "Revanth14 (via GitHub) via issues" <is...@iceberg.apache.org>.
Revanth14 commented on issue #2131:
URL: https://github.com/apache/iceberg-go/issues/2131#issuecomment-6774695701

   @zeroshade This is fixed in #2355 
   we can close this issue as completed.
   @eneskeles 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org