You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JeonDaehong (via GitHub)" <gi...@apache.org> on 2026/10/19 14:06:30 UTC
[PR] [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters [paimon-rust]
JeonDaehong opened a new pull request, #1177:
URL: https://github.com/apache/paimon-rust/pull/1177
### Purpose
Linked issue: close #1174
### Brief change log
The residual filter compares FLOAT/DOUBLE with Arrow kernels, which order floats by their bits. A NaN with the sign bit set, which is what x86 produces for `0.0 / 0.0`, sorts below `-Inf` there and is not equal to `NaN`. Java's `Float.compare` / `Double.compare` treat every NaN as one value above `+Inf`.
`evaluate_column_predicate` now rewrites such NaNs in the column to `NaN` before comparing. The FLOAT/DOUBLE literal is canonicalized the same way. A column without a non-canonical NaN is used as-is.
A probe of 8 values × 8 literals × 6 operators for FLOAT and DOUBLE against paimon-bundle 2.1.0 `PredicateBuilder` went from 112 mismatches out of 768 to 0.
Row evaluation (`eval_row`) and the BTree index are not changed here (see #1174).
### Tests
`float_negative_nan_filter_test` writes one file with `0.0`, a sign-bit NaN, `-2.0` and `NaN`, and checks `<`, `>`, BETWEEN, `= NaN`, `!= NaN`, `= -NaN` and IN on FLOAT and DOUBLE. Without the change, `f < -1.0` returns `[1, 2]`.
`test_bitmap_floating_candidates_preserve_residual_results` (#601) used the residual's old answers as its baseline, for example a sign-bit NaN matching `< 0` and `= NaN` matching only that exact bit pattern. Its expected rows now follow Java. The bitmap index still returns the same rows as the residual in every case.
Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).
### API and Format
None.
### Documentation
None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters [paimon-rust]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on PR #1177:
URL: https://github.com/apache/paimon-rust/pull/1177#issuecomment-6315008390
Added a second commit after checking BTree. BTree keys keep `total_cmp` order, so once the residual follows Java, a table with a BTree index answered differently from a scan without one. `> 0` and `>= -1` dropped sign-bit NaN rows, and `= NaN` / `IN (NaN)` matched only one NaN payload. On main the two agree because both use bit order.
The fix keeps exact BTree answers wherever the key order can't matter. NaN literals are canonicalized. Only a file whose keys hold NaNs, at its first key for ranges or at either end for NaN literals, leaves that predicate to the row filter. Meta pruning and the all-match shortcut skip such files too.
`test_bitmap_floating_candidates_preserve_residual_results` now also runs on BTree and covers `>` / `>=`. `test_evaluate_legacy_{float,double}_btree` and `test_bitmap_floating_bound_fusion_preserves_candidates_and_fallback` pinned bit-order answers for NaN literals and bounds, so they now expect the Java result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters [paimon-rust]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on PR #1177:
URL: https://github.com/apache/paimon-rust/pull/1177#issuecomment-6339734399
@JingsongLi Fair enough, and sorry for the pile of NaN PRs this week. That's on me. I'm closing this one and #1160. The issues can stay as notes in case someone actually runs into it. I'll focus on things that show up in real reads and writes from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters [paimon-rust]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong closed pull request #1177: [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters
URL: https://github.com/apache/paimon-rust/pull/1177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Compare NaN with the sign bit like Java in FLOAT/DOUBLE filters [paimon-rust]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on PR #1177:
URL: https://github.com/apache/paimon-rust/pull/1177#issuecomment-6269156313
@JingsongLi follow-up to your note in #1160, PTAL when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org