You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/30 11:00:02 UTC
[PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
andygrove opened a new pull request, #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447
## Which issue does this PR close?
Closes #6157.
Part of #6385: the "normalize comparisons in the native comparison builder and remove `CometExecRule.normalize`" step.
## Rationale for this change
Spark compares floats with `SQLOrderingUtil.compareDoubles`: `-0.0` equals `0.0`, all NaNs are equal, and NaN sorts above every other value. Arrow's comparison kernels use IEEE 754 total order instead. `CometExecRule.normalize` bridged the two by wrapping comparison operands in `NormalizeNaNAndZero`, but only inside `ProjectExec` and `FilterExec`, so a comparison anywhere else compared raw Arrow values. DataFusion 55 folds `-0.0` in scalar comparisons, which fixed the zero cases, but a NaN with the sign bit set still sorts below every other value, and on x86-64 every NaN that arithmetic produces has that bit set. From #6385, where `-d` turns the NaN in row 3 into a sign-bit NaN on any platform:
| Query | Spark | Comet before |
| --- | --- | --- |
| `SELECT sum(if(-d > 0.0, 1, 0)) FROM t` | `2` | `1` |
| `SELECT count(*) FILTER (WHERE -d >= d) FROM t` | `4` | `3` |
| `SELECT a.id FROM t a JOIN t b ON a.id = b.id AND -a.d >= b.d` | `1, 2, 3, 5` | `1, 2, 5` |
| `SELECT b.id FROM t a JOIN t b ON -a.d > b.d WHERE a.id = 3` | `1, 2, 4, 5` | (no rows) |
| `SELECT id FROM t ORDER BY -d > 0.0, id` | `1, 2, 4, 3, 5` | `1, 2, 3, 4, 5` |
| `SELECT id, x FROM t LATERAL VIEW explode(array(-d > 0.0)) v AS x WHERE id = 3` | `3, true` | `3, false` |
Arrays and structs had the same gap for `<=>`, `<`, `<=`, `>` and `>=` in every operator, Project and Filter included, because the rewrite only wrapped scalar operands (#6157).
## What changes are included in this PR?
- `spark_comparison`, which builds every native comparison, now normalizes float operands for all eight comparison operators: a `FLOAT` or `DOUBLE` operand through `NormalizeNaNAndZero`, and an array or struct with a float leaf through `NormalizeNestedFloats`. Once `-0.0` is folded and NaN canonicalized, Arrow's total order agrees with Spark's. A literal operand is normalized while the plan is built, so it stays a literal. Nested `=` and `<>` keep their `spark_equality` path, which compares without copying.
- The data filters that a native scan pushes into the Parquet reader are built without that normalization. DataFusion's pruning only recognizes a column compared with a literal, so a wrapped column would stop row-group and page pruning for every float predicate. The Filter above the scan still evaluates each of these filters with Spark's semantics. `PhysicalPlanner` becomes `Clone` so that `create_data_filter` can plan them with a copy set to `FloatOperands::Raw`. With row-level pushdown, which is off by default, the reader filters rows by the raw comparison, as it did before this change.
- `CometExecRule.normalize` is removed. It also normalized the divisor of `Divide` and `Remainder`, which is no longer needed: native remainder and ANSI division already treat `-0.0` as a zero divisor, and non-ANSI division checks for a zero divisor with an `=` comparison, which is now normalized natively.
- `NormalizeNaNAndZero` keeps a scalar child a scalar instead of expanding it into a column.
- The floating-point compatibility guide describes the new behavior.
- TPC-DS golden files, regenerated with `dev/regenerate-golden-files.sh` for every Spark version. The plans are unchanged. Queries whose Project or Filter compared or divided doubles (q21, q34, q39a, q39b, q73, q83) count one fewer native expression, the `normalizenanandzero` the rewrite added, and q78 drops `knownfloatingpointnormalized` and `normalizenanandzero` from its dispatched functions.
## How are these changes tested?
- A new `float_comparisons.sql` fixture compares `DOUBLE` and `FLOAT` edge values, including negated NaNs, with every operator, against columns and against literals on either side. It covers Project, Filter, an aggregate argument, a `FILTER` clause, a grouping key, broadcast hash, shuffled hash, sort-merge and nested loop join conditions, a sort key and `explode`. It also compares arrays and structs of floats under all six operators (#6157). On `main` it fails at the first query outside Project and Filter.
- `arithmetic.sql` divides and takes the remainder by `-0.0D` and `double('-0.0')`, the divisors the removed rewrite used to normalize.
- A `CometNativeReaderSuite` test checks that row-group statistics pruning still fires for `d > 500.0D` on a `DOUBLE` column, and a planner unit test checks that `create_data_filter` leaves the column unwrapped where `create_expr` wraps it. The unit test fails if the data filters are built with normalization.
- Unit tests:
- `spark_comparison` on every pair of edge values (both zeros, canonical, sign-bit and payload NaNs, infinities and null) under all eight operators, as two columns and as a column and a literal on either side, for `Float64` and `Float32`, checked against `compare_floats`.
- `<=>`, `<`, `<=`, `>` and `>=` on lists, including lists of different lengths, and on structs, checked against `spark_comparator`.
- Literal folding, wrapping each operand once, `FloatOperands::Raw` leaving operands alone, and non-comparison operators left alone.
- `NormalizeNaNAndZero` keeping scalars.
- The three new `spark_comparison` tests each fail with the normalization turned off.
- Results on macOS aarch64 with the default Spark 4.1 profile:
- Unit tests: 1027 in spark-expr, 566 in core.
- `CometSqlFileTestSuite`: 588. `CometNativeReaderSuite`: 86, plus one test that cancels itself on this Spark version. `CometJoinSuite`: 55.
- `CometExpressionSuite` divide-by-zero tests: 3.
- TPC-DS plan stability v1.4 and v2.7: 129, after regenerating for every Spark version.
Performance: comparisons in Project and Filter cost what they did before, since the rewrite already wrapped their operands in the same `NormalizeNaNAndZero`. Elsewhere a comparison now pays for normalizing its operands. On an M3 Max, a batch of 8192 doubles takes 11.0 µs for a column compared with a column, up from 7.2 µs, and 5.9 µs for a column compared with a literal, up from 4.0 µs. A kernel that compares in Spark's order directly, without copying the operands, could win that back later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#issuecomment-6006823617
@mbutrovich thanks for the review. I updated the description for 449059ca7. I also changed the `spark.comet.parquet.rowFilterPushdown.enabled` doc as you suggested, and the scan tuning page, which made the same claim. Both now say that format-level pruning runs whenever `spark.sql.parquet.filterPushdown` is on, except that a pushed filter comparing `FLOAT` or `DOUBLE` values doesn't prune while the flag is on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4187945415
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this: Parquet pruning recognizes
+ /// a column compared with a literal but not a normalized column. With row-level pushdown the
+ /// reader also evaluates the filters on each row, and a row it drops never reaches Spark's
+ /// Filter above the scan, so any other shape, which pruning cannot use anyway, is normalized.
+ /// A computed operand such as `-d` can hold a NaN with the sign bit set, which a raw
+ /// comparison sorts below every other value.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
Review Comment:
Agreed, and it reaches past that literal. A stored NaN with the sign bit set fails every ordering comparison against the raw column, so with row-level pushdown on, `d > 0.0` dropped it on main as well. Data filters now leave the column raw only when row-level pushdown is off, where the reader prunes with them and the Filter above the scan applies Spark's semantics to each row. With `spark.comet.parquet.rowFilterPushdown.enabled=true` they normalize both sides, so in that mode float comparisons give up statistics pruning instead of dropping rows (2f003c207). `scan_data_filters_normalize_float_columns_with_row_level_pushdown` evaluates `d = -double('NaN')` and `d > 0.0` on a NaN with the sign bit set, a canonical NaN and `1.0`.
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -957,14 +904,12 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
Reproduced, on aarch64 too, since negating the NaN sets the sign bit and the division keeps it. Rather than wait for #6519, I kept the divisor normalization, for `Divide` and `Remainder` only and in Projects and Filters as before (2f003c207). The comment on `normalizeDivisors` says it stays for `percentile_approx` until #6519 is fixed. `nan_divisor.sql` runs your query with ANSI off and on. Without the wrapper Comet returns `[0.0, -0.667]` for the maximum and minimum where Spark returns `[NaN, -0.667]`, and with it they match. The wrapper puts `NormalizeNaNAndZero` back into the TPC-DS plans, so their expression counts are main's again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4178233892
##########
native/spark-expr/src/comet_scalar_funcs.rs:
##########
@@ -315,6 +316,11 @@ pub fn create_comet_physical_fun_with_eval_mode(
// SparkMakeTime already throws on invalid input, so accept the flag here rather
// than falling through to the registry fail-closed path.
"make_time" => Ok(Arc::new(ScalarUDF::new_from_impl(SparkMakeTime::new()))),
+ // Floats, and arrays and structs holding them, need Spark's float ordering. Other types
+ // keep DataFusion's `greatest` and `least` from the registry.
+ "greatest" | "least" if SparkGreatestLeast::handles(&data_type) => Ok(Arc::new(
Review Comment:
[P2] Preserve Spark’s reuse of equivalent extrema expressions before enabling this routing unconditionally. For a Parquet row `(a, b) = (0.0D, -0.0D)`, `SELECT CAST(greatest(a,b) AS STRING), CAST(greatest(b,a) AS STRING) FROM t` returns `('0.0', '0.0')` in Spark with default `spark.sql.subexpressionElimination.enabled=true`. The new native calls retain their respective first arguments, producing `(0.0, -0.0)` and therefore different strings. The previous DataFusion implementation returned positive zero for both calls, so this worsens a previously matching query. This affects observable results and downstream expressions. The compatibility guide documents the divergence, but the default native path remains enabled. Please preserve Spark’s shared representative for equivalent expressions or fall back for affected projections, and test both argument orders together.
Evidence: Fresh Spark 3.5.9 execution over Spark-written Parquet returned `Row(g1='0.0', g2='0.0')` with expression reuse enabled and opposite signs when disabled. An exact-head `PhysicalPlanner::create_expr` probe produced bits `[0x0000000000000000, 0x8000000000000000]` and failed the Spark-result assertion. A before/after kernel probe confirmed the previous DataFusion implementation produced `[0, 0]`. Spark’s supported-version `Greatest.canonicalized` treats argument orders as equivalent. Reproductions: `/tmp/review6447-root-planner-probe.rs` and `/tmp/review6447-root-expression-probe.rs`. Logs: `/tmp/review6447-root-native-planner-probe.log`, `/tmp/review6447-root-native-probe.log`, and `/tmp/review6447-36546-root-spark-cse.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4174326162
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -826,14 +773,12 @@ case class CometExecRule(session: SparkSession)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
[P2] Preserve divisor normalization until native extrema are Spark-compatible. For a Spark-written Parquet `DOUBLE` column `d` containing NaN, `SELECT least(1.0D / (-d), 0.0D), greatest(1.0D / (-d), 0.0D) FROM t` previously matched Spark’s `(0.0, NaN)`. Removing `normalizePlan` passes a negative NaN quotient into the native extrema kernels, which return `(NaN, 0.0)` in both ANSI modes on x86-64. This exposes their existing ordering limitation on a previously correct projection. The hash fix does not protect these consumers. Retain the arithmetic divisor wrapper or correct extrema ordering before removing it, with SQL regression coverage.
Evidence: An exact-head disposable Rust test exercised `create_negate_expr`, the non-ANSI `IfExpr` zero guard or ANSI `checked_div`, and the `least`/`greatest` functions returned by `create_comet_physical_fun`. Restoring the former divisor wrapper produced `(0.0, NaN)` in both modes. Without it, both produced `(NaN, 0.0)` and the regression assertion failed. A fresh Spark 3.5.9 run over Spark-written Parquet returned `(0.0, NaN)` in both modes. Spark’s supported-version extrema use SQL ordering, which places every NaN above finite values. Reproduction: `/tmp/review6447-864c2-session-repro.rs`. Native output: `/tmp/review6447-864c2-session-repro.log`. Spark output: `/tmp/review6447-864c2-session-spark-oracle.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4185761715
##########
native/spark-expr/src/comet_scalar_funcs.rs:
##########
@@ -315,6 +316,11 @@ pub fn create_comet_physical_fun_with_eval_mode(
// SparkMakeTime already throws on invalid input, so accept the flag here rather
// than falling through to the registry fail-closed path.
"make_time" => Ok(Arc::new(ScalarUDF::new_from_impl(SparkMakeTime::new()))),
+ // Floats, and arrays and structs holding them, need Spark's float ordering. Other types
+ // keep DataFusion's `greatest` and `least` from the registry.
+ "greatest" | "least" if SparkGreatestLeast::handles(&data_type) => Ok(Arc::new(
Review Comment:
This one comes from #6457, which merged yesterday, so main already has it. Since the main merge in f2c2fc4e5, `comet_scalar_funcs.rs` isn't part of this diff any more. `floating-point.md` describes the difference at the end of its `min`, `max`, `greatest`, and `least` section.
I don't think there's a single Spark answer to match. On 4.1.3, over `(a, b) = (0.0, -0.0)`, Spark returns `0.0, -0.0, 0.0, -0.0` for `greatest(a, b), greatest(b, a), least(a, b), least(b, a)` once `spark.sql.subexpressionElimination.enabled` is off. That is what Comet returns under either setting. With elimination on, Spark evaluates one expression from each pair and returns `0.0` all four times. A lone `greatest(b, a)` returns `-0.0` from Spark under both settings, and from Comet too. Before #6457 Comet returned `0.0` for it, because DataFusion orders `0.0` above `-0.0`, and that same ordering is the only reason the paired query used to match.
To match the paired case we'd have to replace each expression in a projection with the first semantically equal one before serializing it, which is how `EquivalentExpressions` picks its representative. I can open an issue for that if you think it's worth doing, but I'd rather not hold this PR for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4190049382
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this: Parquet pruning recognizes
+ /// a column compared with a literal but not a normalized column. With row-level pushdown the
+ /// reader also evaluates the filters on each row, and a row it drops never reaches Spark's
+ /// Filter above the scan, so any other shape, which pruning cannot use anyway, is normalized.
+ /// A computed operand such as `-d` can hold a NaN with the sign bit set, which a raw
+ /// comparison sorts below every other value.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
Review Comment:
> With `spark.comet.parquet.rowFilterPushdown.enabled=true` they normalize both sides, so in that mode float comparisons give up statistics pruning instead of dropping rows (2f003c207).
I checked how much pruning that gives up. I planned `d > 500.0D` through `create_data_filter` at 2f003c2 and built a DataFusion `PruningPredicate` from the result. With `pushdown_filters` off, the filter is `d@0 > 500` and the pruning predicate is `d_null_count@1 != row_count@2 AND d_max@0 > 500`. With it on, the filter is `FloatNormalize [child: d@0] > 500` and `PruningPredicate::always_true()` returns `true`. So with the flag on, every `FLOAT` or `DOUBLE` comparison in a data filter loses row-group statistics, page index and bloom filter pruning, not only statistics pruning.
Choosing correct rows over pruning makes sense to me. As I read DataFusion 55's `ParquetSource`, one predicate drives both pruning and the row filter, so keeping pruning in this mode would need a DataFusion change or a separate pruning predicate. Could you open an issue for that and link it from `floating-point.md`, so people who turn the flag on know what they lose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4146865173
##########
spark/src/test/resources/sql-tests/expressions/conditional/float_comparisons.sql:
##########
@@ -0,0 +1,140 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Comparisons follow Spark's SQL ordering for floats (SQLOrderingUtil.compareDoubles): -0.0
+-- equals 0.0, every NaN equals every other NaN, and NaN sorts above every other value. This
+-- must hold in every operator that evaluates a comparison, not only in Project and Filter.
+--
+-- `-d` flips the sign bit, so `-d` of the NaN row is a NaN with the sign bit set on every
+-- platform. That is the NaN that arithmetic produces on x86-64, and Arrow's total order sorts
+-- it below -Infinity.
+
+statement
+CREATE TABLE float_cmp(id INT, d DOUBLE, f FLOAT) USING parquet
+
+statement
+INSERT INTO float_cmp VALUES
+ (1, 0.0D, float('0.0')),
+ (2, double('-0.0'), float('-0.0')),
+ (3, double('NaN'), float('NaN')),
+ (4, 1.0D, float('1.0')),
+ (5, -1.0D, float('-1.0')),
+ (6, double('Infinity'), float('Infinity')),
+ (7, NULL, NULL)
+
+-- Project
+query
+SELECT id, -d = d, -d <=> d, -d < d, -d <= d, -d > d, -d >= d, -d != d FROM float_cmp
+
+query
+SELECT id, -f = f, -f <=> f, -f < f, -f <= f, -f > f, -f >= f, -f != f FROM float_cmp
+
+-- Comparisons against a constant on either side. `-0.0D` and `-0.0F` are literals with the sign
+-- bit set, while `double('NaN')` is a cast, because the suite turns off constant folding.
+query
+SELECT id, -d > 0.0D, -d >= -0.0D, -0.0D = d, -d = double('NaN'), double('NaN') <= -d,
+ -d < double('-0.0'), -f > 0.0F, -0.0F <=> f, -f = float('NaN'), float('-0.0') >= -f
+FROM float_cmp
+
+-- Filter
+query
+SELECT id FROM float_cmp WHERE -d > 0.0D
Review Comment:
Every `WHERE` on a float column here compares a computed operand (`-d` or `-f`), so none has the `column op literal` shape that a native scan pushes down as a data filter. Would it make sense to add a few filters on the stored edge values, for example `WHERE d >= 0.0D`, `WHERE d = double('NaN')` and `WHERE d > double('Infinity')`? I haven't run them, but I'd expect Spark to return the `-0.0` and NaN rows. They would show that the Filter above the scan still applies Spark's ordering after the raw data filters.
##########
native/core/src/execution/planner.rs:
##########
@@ -1007,6 +1017,23 @@ impl PhysicalPlanner {
}
}
+ /// Create a data filter that a scan pushes into the Parquet reader. The filter prunes row
+ /// groups and pages, and rows too when row-level pushdown is enabled. Pruning only recognizes
+ /// a column compared with a literal, so the filter's comparisons leave float operands as they
+ /// are rather than normalizing them. Spark's Filter above the scan evaluates the filter again
+ /// with Spark's semantics.
+ fn create_data_filter(
+ &self,
+ spark_expr: &Expr,
+ input_schema: SchemaRef,
+ ) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
+ let planner = Self {
+ float_operands: FloatOperands::Raw,
Review Comment:
`Raw` applies to every comparison inside a data filter, while the description says pruning only recognizes a column compared with a literal. Would it make sense to leave only that shape raw and normalize the rest? I haven't run this, but with `spark.comet.parquet.rowFilterPushdown.enabled=true` I'd expect `WHERE -d > 0.0D` from `float_comparisons.sql` to lose the NaN row, because the reader would compare the computed sign-bit NaN raw. If `Raw` stays as is, it might be worth saying in the docs that computed operands are affected too.
##########
native/spark-expr/src/float_semantics/normalize.rs:
##########
@@ -240,6 +273,16 @@ pub fn has_float_leaf(dt: &DataType) -> bool {
}
}
+fn is_nested_with_float_leaf(dt: &DataType) -> bool {
Review Comment:
`needs_spark_equality` in `nested_comparison.rs` already tests the same thing (`dt.is_nested() && has_float_leaf(dt)`). Would it be simpler to keep one `pub(crate)` helper and use it from both files?
##########
native/spark-expr/src/float_semantics/normalize.rs:
##########
@@ -87,15 +88,17 @@ impl PhysicalExpr for NormalizeNaNAndZero {
}
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
- let cv = self.child.evaluate(batch)?;
- let array = cv.into_array(batch.num_rows())?;
-
match &self.data_type {
- DataType::Float32 | DataType::Float64 => {
- Ok(ColumnarValue::Array(normalize_floats(&array)))
- }
+ DataType::Float32 | DataType::Float64 => {}
dt => panic!("Unexpected data type {dt:?}"),
}
+ // A scalar stays a scalar, so a constant operand is not expanded into a column.
+ match self.child.evaluate(batch)? {
+ ColumnarValue::Array(array) => Ok(ColumnarValue::Array(normalize_floats(&array))),
+ ColumnarValue::Scalar(value) => {
Review Comment:
Since comparisons outside Project and Filter now pay for this, would it make sense to give `normalize_floats` a read-only fast path? DataFusion's `normalize_float_zero`, which its `apply_cmp` uses, scans first and returns the input `Arc` unless a `-0.0` is present. Here `unary(normalize_float)` always writes a new buffer. I haven't measured it, but I'd expect skipping that allocation in the common case to recover part of the 7.2 to 11.0 µs gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4162029591
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -763,14 +710,12 @@ case class CometExecRule(session: SparkSession)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
[P2] Preserve divisor normalization until NaN hashing is corrected. On x86-64, for a Parquet `DOUBLE` column `d` containing canonical NaN, `SELECT hash(1.0D / (-d)), xxhash64(1.0D / (-d)) FROM t` previously matched Spark. The native path now returns `(-1489914710, 9200374361256412029)` instead of `(-1281358385, -3127944061524951246)` in both ANSI modes. Removing `normalizePlan` also removes the `Divide` divisor wrapper. The new comparison normalization covers the zero-divisor check, but arithmetic still consumes the original negative NaN. This exposes the existing hash limitation on a previously correct projection. Retain the arithmetic wrapper or canonicalize NaNs in both hash paths before removing it.
Evidence: An exact-head disposable Rust test used `create_negate_expr`, the legacy `IfExpr` zero guard or ANSI `checked_div`, and both shipped hash functions. With the former divisor wrapper, quotient bits were `0x7ff8000000000000` and both hashes matched Spark. Without it, bits were `0xfff8000000000000` and both hashes differed. The regression assertion failed in both modes. Spark 3.5.9 executed the SQL over Spark-written Parquet and returned the expected pair in both modes. Spark’s supported-version `hash.scala` implementations canonicalize NaNs through `Double.doubleToLongBits`. Reproduction: `/tmp/review6447-current-1790903311-repro.rs`; native output: `/tmp/review6447-current-1790903311-repro.log`; Spark output: `/tmp/review6447-current-1790903311-spark-oracle.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#issuecomment-5919313999
Notes on the comparison-builder normalization:
- **`spark/src/test/resources/sql-tests/expressions/conditional/float_comparisons.sql:88`** — the fixture covers `=`, `<=>`, `<`, `<=`, `>`, `>=`, `!=`, but there's no `IS DISTINCT FROM` query, which is a separate native operator with its own routing. It's covered in the Rust unit tests, but since this SQL file is the Spark-comparison fixture, please add an `a IS DISTINCT FROM b` case end to end.
- **`float_comparisons.sql`** (end of file) — the nested cases cover one level of `ARRAY<DOUBLE>` and `STRUCT<v: DOUBLE>`. Please add a deeper shape like `ARRAY<STRUCT<v: DOUBLE>>` or `STRUCT<a: ARRAY<DOUBLE>>`. `normalize_nested_floats` recurses, so it should just work, but a deeper column proves it for the ordering operators.
- **`spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala:1538`** — the pruning test writes `range(0,1000)` cast to double, so there's no NaN or `-0.0` in the data. It confirms pruning still fires, but not the other direction: a row group Spark would match because it holds a NaN (Spark treats `NaN > 500.0` as true) must not be pruned away by the raw comparison. Can you confirm a NaN-bearing row group survives here, and consider a small NaN case? This is unchanged by the PR, so a coverage question, not a regression.
- **`native/spark-expr/src/array_funcs/nested_comparison.rs:272`** — for nested ordering operators both operands are deep-copied through `normalize_nested_floats` per batch, unlike nested `=`/`<>` which compare in place. The PR body and module doc already note the planned no-copy comparator; just flagging the cost is real for wide nested columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4147895534
##########
native/spark-expr/src/float_semantics/normalize.rs:
##########
@@ -87,15 +88,17 @@ impl PhysicalExpr for NormalizeNaNAndZero {
}
fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
- let cv = self.child.evaluate(batch)?;
- let array = cv.into_array(batch.num_rows())?;
-
match &self.data_type {
- DataType::Float32 | DataType::Float64 => {
- Ok(ColumnarValue::Array(normalize_floats(&array)))
- }
+ DataType::Float32 | DataType::Float64 => {}
dt => panic!("Unexpected data type {dt:?}"),
}
+ // A scalar stays a scalar, so a constant operand is not expanded into a column.
+ match self.child.evaluate(batch)? {
+ ColumnarValue::Array(array) => Ok(ColumnarValue::Array(normalize_floats(&array))),
+ ColumnarValue::Scalar(value) => {
Review Comment:
I measured it, and the check costs about as much as the copy it saves. Over 8192 doubles with no NaN or `-0.0`, the copy takes 1.71 µs, a branch-free check in blocks of 64 takes 1.66 µs, and the fastest form I found, on the raw bits, 1.28 µs. An `any()` that stops at the first match does not vectorize and took 4.4 µs. So the best case recovers about 0.4 µs per operand, roughly 8% of a column-column comparison, and an array that does need normalizing pays for the check as well as the copy. Most of the gap is the per-value NaN and `-0.0` test itself, so I'd rather close it with a kernel that compares in Spark's order directly, without normalized copies, in a follow-up. Does that work for you?
##########
spark/src/test/resources/sql-tests/expressions/conditional/float_comparisons.sql:
##########
@@ -0,0 +1,140 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Comparisons follow Spark's SQL ordering for floats (SQLOrderingUtil.compareDoubles): -0.0
+-- equals 0.0, every NaN equals every other NaN, and NaN sorts above every other value. This
+-- must hold in every operator that evaluates a comparison, not only in Project and Filter.
+--
+-- `-d` flips the sign bit, so `-d` of the NaN row is a NaN with the sign bit set on every
+-- platform. That is the NaN that arithmetic produces on x86-64, and Arrow's total order sorts
+-- it below -Infinity.
+
+statement
+CREATE TABLE float_cmp(id INT, d DOUBLE, f FLOAT) USING parquet
+
+statement
+INSERT INTO float_cmp VALUES
+ (1, 0.0D, float('0.0')),
+ (2, double('-0.0'), float('-0.0')),
+ (3, double('NaN'), float('NaN')),
+ (4, 1.0D, float('1.0')),
+ (5, -1.0D, float('-1.0')),
+ (6, double('Infinity'), float('Infinity')),
+ (7, NULL, NULL)
+
+-- Project
+query
+SELECT id, -d = d, -d <=> d, -d < d, -d <= d, -d > d, -d >= d, -d != d FROM float_cmp
+
+query
+SELECT id, -f = f, -f <=> f, -f < f, -f <= f, -f > f, -f >= f, -f != f FROM float_cmp
+
+-- Comparisons against a constant on either side. `-0.0D` and `-0.0F` are literals with the sign
+-- bit set, while `double('NaN')` is a cast, because the suite turns off constant folding.
+query
+SELECT id, -d > 0.0D, -d >= -0.0D, -0.0D = d, -d = double('NaN'), double('NaN') <= -d,
+ -d < double('-0.0'), -f > 0.0F, -0.0F <=> f, -f = float('NaN'), float('-0.0') >= -f
+FROM float_cmp
+
+-- Filter
+query
+SELECT id FROM float_cmp WHERE -d > 0.0D
Review Comment:
Added `WHERE d >= 0.0D`, `WHERE f <= -0.0F`, `WHERE d = double('NaN')` and `WHERE d > double('Infinity')`, and the whole file now runs with row-level pushdown both off and on, which reproduces the `-d > 0.0D` case from the `planner.rs` comment. The suite turns off constant folding, so `double('NaN')` stays a cast and only the first two have the `column op literal` shape.
##########
native/core/src/execution/planner.rs:
##########
@@ -1007,6 +1017,23 @@ impl PhysicalPlanner {
}
}
+ /// Create a data filter that a scan pushes into the Parquet reader. The filter prunes row
+ /// groups and pages, and rows too when row-level pushdown is enabled. Pruning only recognizes
+ /// a column compared with a literal, so the filter's comparisons leave float operands as they
+ /// are rather than normalizing them. Spark's Filter above the scan evaluates the filter again
+ /// with Spark's semantics.
+ fn create_data_filter(
+ &self,
+ spark_expr: &Expr,
+ input_schema: SchemaRef,
+ ) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
+ let planner = Self {
+ float_operands: FloatOperands::Raw,
Review Comment:
Confirmed: with `spark.comet.parquet.rowFilterPushdown.enabled=true`, `WHERE -d > 0.0D` returned only row 5, where Spark returns rows 3 and 5. In 12e95e697, `FloatOperands::Raw` leaves only a float column compared with a literal as it is, and still normalizes the literal. Every other comparison in a data filter is normalized, so the NaN row survives the reader, and the pruning test still prunes. The remaining raw shape compares stored values, and Spark writes floats through `doubleToLongBits` and `floatToIntBits`, so a Spark-written file only holds canonical NaNs. The compatibility guide now says that a noncanonical NaN written by another engine can still be filtered differently with row-level pushdown.
##########
native/spark-expr/src/float_semantics/normalize.rs:
##########
@@ -240,6 +273,16 @@ pub fn has_float_leaf(dt: &DataType) -> bool {
}
}
+fn is_nested_with_float_leaf(dt: &DataType) -> bool {
Review Comment:
Done. `needs_spark_equality` is gone, and `nested_comparison.rs` uses `is_nested_with_float_leaf` from `float_semantics`, now `pub(crate)` and written as `dt.is_nested() && has_float_leaf(dt)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4190054555
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this, and only when the Parquet
+ /// reader prunes with them but does not filter rows: Parquet pruning recognizes a column
+ /// compared with a literal but not a normalized column, and Spark's Filter above the scan
+ /// applies Spark's semantics to every row. With row-level pushdown the reader would drop the
+ /// rows such a comparison rejects, including a stored NaN whose bits differ from the
+ /// normalized literal, so the data filters use [`FloatOperands::Normalize`] there instead.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
+ } else if raw && is_literal(&left) && is_float_column(&right, schema) {
+ (normalize_comparison_operand(left, schema)?, right)
Review Comment:
With `FloatOperands::Raw`, the literal is still normalized. Since 2f003c2, `Raw` is used only when the reader prunes and doesn't filter rows. Parquet bloom filter pruning probes the literal's exact bits ([`check_scalar`](https://github.com/apache/datafusion/blob/7d3835c71f30cbd3c3ae4041732267f1f453097a/datafusion/datasource-parquet/src/bloom_filter.rs#L102-L103) hashes the `f64` as is). So `d = -0.0D` now probes for `0.0`.
What happens to a row group whose `d` values are all `-0.0`? I wrote a file with arrow-rs, with a bloom filter on `d` and two `-0.0` rows, and scanned it with the `scan_parquet_file` test helper in `schema_adapter.rs`. With main's data filter (`d@0 = -0`) the scan returns both rows. With the filter `spark_comparison(..., FloatOperands::Raw)` builds at 2f003c2 (`d@0 = 0`) it returns none, with or without column statistics. The Filter above the scan can't bring those rows back, and Spark matches them, since `-0.0 = -0.0`.
Should `Raw` leave a zero literal as it is, as main does, or expand it into both signed zeros the way the `IN` path does for constant lists? Leaving it as it is goes back to main's behavior, where a row group holding only `0.0` is still pruned for `d = -0.0D`. Expanding it covers both cases. Could you also add a test with a bloom filter on a `DOUBLE` column, for example in `CometNativeReaderSuite` with `parquet.bloom.filter.enabled#d`, that checks `WHERE d = -0.0D` returns the `-0.0` rows?
##########
docs/source/user-guide/latest/compatibility/floating-point.md:
##########
@@ -23,20 +23,34 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
-For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
-execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
-also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
-literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
-forms when a list contains zero.
+For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
+normalizes both operands before native execution, including noncanonical NaN literals. This
+applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
+clauses, join conditions, sort keys, and generator arguments.
+
+A native Parquet scan skips row groups with its data filters. So that statistics pruning still
+applies, a `FLOAT` or `DOUBLE` column compared with a constant in a data filter is compared
+without normalizing the column, and the filter above the scan evaluates the comparison again with
+Spark's semantics. Every other comparison in a data filter is normalized. With
+`spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also filters rows with these
+comparisons, so a noncanonical NaN stored in the file, such as one with the sign bit set, can be
+filtered differently from Spark when compared with a constant. Spark's Parquet writer only writes
+canonical NaNs.
Review Comment:
This paragraph describes the data filters as they were before 2f003c2. As I read `data_filter_float_operands`, with `spark.comet.parquet.rowFilterPushdown.enabled=true` both operands are normalized. A noncanonical NaN is no longer filtered differently in that mode, and float comparisons stop pruning instead. What do you think about this wording?
```suggestion
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
still applies, a `FLOAT` or `DOUBLE` column compared with a constant in a data filter is compared
without normalizing the column, and the filter above the scan evaluates the comparison again with
Spark's semantics. Every other comparison in a data filter is normalized. With
`spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data filters
reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE` comparison
in a data filter does not prune row groups or pages.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#issuecomment-5917126017
Added `benches/float_comparison.rs` in 12e95e697, and the description now points to it. It times DataFusion's `BinaryExpr` against `spark_comparison` for a column-column and a column-literal `<`, and `normalize_floats` on its own, over data without special values and with 10% `-0.0`, canonical NaN, or sign-bit NaN. On an M3 Max, per 8192 doubles: column-column 10.4 µs against 6.8 µs, column-literal 5.6 µs against 3.8 µs, and `normalize_floats` 1.75 µs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4190391558
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this, and only when the Parquet
+ /// reader prunes with them but does not filter rows: Parquet pruning recognizes a column
+ /// compared with a literal but not a normalized column, and Spark's Filter above the scan
+ /// applies Spark's semantics to every row. With row-level pushdown the reader would drop the
+ /// rows such a comparison rejects, including a stored NaN whose bits differ from the
+ /// normalized literal, so the data filters use [`FloatOperands::Normalize`] there instead.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
+ } else if raw && is_literal(&left) && is_float_column(&right, schema) {
+ (normalize_comparison_operand(left, schema)?, right)
Review Comment:
Good catch. I went with expanding it (449059ca7). With `FloatOperands::Raw`, `=` against either zero is now `d = -0.0 OR d = 0.0`, which `LiteralGuarantee` turns into one guarantee holding both zeros, so the bloom filter is probed for each. The other operators keep the literal as it is. DataFusion 55's `apply_cmp` folds `-0.0` before it compares, so statistics and page index pruning already treat the two zeros as equal, and only `=` gives the bloom filter anything to probe. I checked that on Spark-written files, whose statistics aren't widened (a row group holding only `-0.0` has `min = max = -0.0`): `=`, `<=>`, `<`, `<=`, `>` and `>=` against a zero returned the same rows as Spark with filter pushdown off, with and without bloom filters.
A NaN literal had the same problem, since no literal has the bits of every NaN a file can hold, and arrow-rs leaves NaNs out of the statistics. So a NaN literal now normalizes the column too, as the `IN` path already does, and gives up pruning.
The new `CometNativeReaderSuite` test writes two files with bloom filters on `d`, one holding `-0.0` and the other `0.0`. It checks that `d = -0.0D` and `d = 0.0D` each return both rows with both bloom filters matching, and that `d = 0.5D` is still pruned by both. It writes out the expected rows instead of comparing with Spark's, because Spark's own reader probes the bloom filter with the literal's bits too and skips the file holding the other zero. There's also a `parquet_exec` test along the lines of your repro, which fails if `=` probes for a single zero.
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this: Parquet pruning recognizes
+ /// a column compared with a literal but not a normalized column. With row-level pushdown the
+ /// reader also evaluates the filters on each row, and a row it drops never reaches Spark's
+ /// Filter above the scan, so any other shape, which pruning cannot use anyway, is normalized.
+ /// A computed operand such as `-d` can hold a NaN with the sign bit set, which a raw
+ /// comparison sorts below every other value.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
Review Comment:
Filed #6702 and linked it from `floating-point.md`. The issue also notes a partial fix that needs no DataFusion change: `=`, `<>`, `<=>` and `IS DISTINCT FROM` against a constant other than NaN could keep the raw column with row-level pushdown too, since DataFusion compares the two zeros as equal and no NaN equals such a constant. That would bring back pruning for equality, but not for `<`, `<=`, `>` and `>=`.
##########
docs/source/user-guide/latest/compatibility/floating-point.md:
##########
@@ -23,20 +23,34 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
-For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
-execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
-also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
-literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
-forms when a list contains zero.
+For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
+normalizes both operands before native execution, including noncanonical NaN literals. This
+applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
+clauses, join conditions, sort keys, and generator arguments.
+
+A native Parquet scan skips row groups with its data filters. So that statistics pruning still
+applies, a `FLOAT` or `DOUBLE` column compared with a constant in a data filter is compared
+without normalizing the column, and the filter above the scan evaluates the comparison again with
+Spark's semantics. Every other comparison in a data filter is normalized. With
+`spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also filters rows with these
+comparisons, so a noncanonical NaN stored in the file, such as one with the sign bit set, can be
+filtered differently from Spark when compared with a constant. Spark's Parquet writer only writes
+canonical NaNs.
Review Comment:
Thanks, I took this wording in 449059ca7, with two additions for what changed there: the column is left raw only against a constant other than NaN, and `=` against either zero checks bloom filters for both zeros. The last sentence links #6702.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4199398258
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
+## Spark's rules
+
+| Rule | `-0.0` and `0.0` | NaN | Used by |
+| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs |
+| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys |
+| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning |
+| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values |
+| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code |
+
+Some functions changed rules in a Spark release, so a native path has to follow the Spark version
+it runs against:
+
+- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and
Review Comment:
You're right, thanks. `mutable.HashSet[Any]` compares boxed values with Scala's `==`, which merges the two zeros and never matches a NaN, the opposite of `Double.equals` on both counts. 6f81c8202 gives Scala `==` its own row in the rules table and moves pre-4.2 `collect_set` under it. I reproduced it on 3.4.3 and 4.1.3: `size(collect_set(d))` is 3 for a group holding `-0.0`, `0.0` and two NaNs, and 4 for three NaNs and two `1.0`s.
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
+## Spark's rules
+
+| Rule | `-0.0` and `0.0` | NaN | Used by |
+| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs |
+| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys |
+| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning |
+| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values |
+| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code |
+
+Some functions changed rules in a Spark release, so a native path has to follow the Spark version
+it runs against:
+
+- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and
Review Comment:
Thanks, confirmed. 3.4.3 has `_data(pos) == k`, and the 3.5 backport of SPARK-45599 is in 3.5.2 but not 3.5.1. 6f81c8202 says that `OpenHashSet` and `OpenHashMap` follow `Double.equals` only from 3.5.2 and 4.0.0, and describes the older matching. That qualifies the `mode` bullet too, because `mode` counts values in an `OpenHashMap`: on 3.4.3, `mode` over three NaNs and two `1.0`s returns `1.0`, where 4.1.3 returns NaN. The new bullet also names the other users, `percentile` and the four array set functions, which handle NaN separately.
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
Review Comment:
It holds. On 3.4.3 and 4.1.3 I wrote four dictionary-encoded files, two whose only zero is `-0.0` and two whose only zero is `0.0`. With either Parquet reader, `d = -0.0D`, `d <=> -0.0D` and `d IN (-0.0D, 5.0D)` returned only the `-0.0` rows, and turning off `parquet.filter.dictionary.enabled` or `spark.sql.parquet.filterPushdown` brought the `0.0` rows back. It isn't limited to equality either. `DictionaryFilter` orders the entries with `Double.compare`, so `d >= 0.0D` skipped a row group whose largest value is `-0.0`, and `d <= -0.0D` one whose smallest is `0.0`. Comet returned every row in each case.
6f81c8202 adds a paragraph with those examples to the compatibility guide, after the one on pruning. It says that Comet's rows are the ones Spark's comparison semantics call for, and that Spark returns them too with `spark.sql.parquet.filterPushdown=false`. The contributor guide's data filter bullet says the same, and that a pruning test writes out its expected rows. The `CometNativeReaderSuite` bloom test writes with dictionary encoding off, so there the bloom filter alone skips the file. I didn't find a Spark or parquet-java issue to link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #6447: fix: normalize float operands in the native comparison builder
URL: https://github.com/apache/datafusion-comet/pull/6447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#issuecomment-6039832572
I appreciate the thorough review @mbutrovich!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4178139408
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -763,14 +710,12 @@ case class CometExecRule(session: SparkSession)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
#6413 has merged, so `hash`, `xxhash64` and the shuffle partitioner canonicalize NaN, and the main merge in b1cc9fc1e brought it in. `nan_divisor.sql` hashes `1.0D / (-d)`, `1.0D % (-d)` and `1.0F % (-f)` with ANSI off and on (864c2b576).
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -826,14 +773,12 @@ case class CometExecRule(session: SparkSession)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
Reproduced, and `max`/`min` of a quotient computed in a projection had the same exposure. #6457 orders floats the Spark way in `min`, `max`, `greatest` and `least`, so rather than keep the divisor wrapper I merged its branch into this one (0aae7f58e), and this PR lands after it. #6457 is in the merge queue; until it merges, this diff also shows its changes, and I'll merge main again once it does.
`nan_divisor.sql` now runs your projection, plus `least(1.0D % (-d), 0.0D)`, `greatest(1.0F % (-f), 0.0F)` and `max`/`min` of a quotient that a `UNION ALL` keeps in a projection below the aggregate, with ANSI off and on (36546d024). The new queries fail on 864c2b576 and pass with #6457. A wider probe over `1.0D / (-d)` and `1.0D % (-d)`, covering the hashes, the extrema, `array_distinct`, `array_union`, `sort_array`, `array_max`, comparisons, and sort and grouping keys, now matches Spark in every case, in both ANSI modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#issuecomment-5981351428
Addressed in a0b1f4db5:
- `float_comparisons.sql` runs `IS DISTINCT FROM` end to end, on scalars, constants, arrays and structs.
- A `float_cmp_deep` table compares `ARRAY<STRUCT<v: DOUBLE>>` and `STRUCT<a: ARRAY<DOUBLE>>` columns under every ordering operator, and negates the leaves so that a NaN two levels down has the sign bit set. Stopping `normalize_nested_floats` after one level fails only those queries.
- The pruning test now puts a NaN in a row group whose other values are all below 500, and checks that `d > 500.0D` still returns it. parquet-mr writes the NaN as the chunk's max statistic, so DataFusion keeps that row group.
- Agreed on the copies for nested ordering. The fix is a comparator that orders nested floats the Spark way in place, as nested `=` and `<>` already compare; that follow-up is #6590.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4185987363
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -957,14 +904,12 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
plan
}
} else {
- val normalizedPlan = normalizePlan(plan)
-
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
- normalizedPlan.transformUp { case p =>
+ plan.transformUp { case p =>
RewriteJoin.rewrite(p)
}
} else {
- normalizedPlan
+ plan
Review Comment:
[P2] [P2] Preserve divisor normalization until percentile ordering handles noncanonical NaNs. For a Spark-written Parquet table `t` containing one NaN in `d`, run `SELECT percentile_approx(q, 1.0D) FROM (SELECT /*+ COALESCE(1) */ q FROM (SELECT 1.0D / (-d) AS q FROM t UNION ALL SELECT 0.0D AS q FROM t))`. Spark returns NaN. On x86-64, removing `normalizePlan` exposes a negative NaN quotient to `QuantileSummaries`, whose `total_cmp` places it below zero, so the native accumulator returns `0.0` in both ANSI modes. The former divisor wrapper made this projection correct. Retain that wrapper or correct percentile NaN ordering before removing it.
Evidence: Fresh exact-head native reproduction exercised `create_negate_expr`, the non-ANSI `IfExpr` division guard or ANSI `checked_div`, and the shipped `ApproxPercentile` accumulator. With divisor normalization, both modes returned NaN. Without it, both returned `Float64(0)`, failing the Spark-result assertion. Fresh Spark 3.5.9 execution of the SQL returned NaN in both modes. Source: `/tmp/review6447-f2c2-session2-consumers.rs`. Logs: `/tmp/review6447-f2c2-session2-consumers.log` and `/tmp/review6447-f2c2-session2-spark.log`.
##########
native/spark-expr/src/array_funcs/nested_comparison.rs:
##########
@@ -219,31 +217,83 @@ impl PhysicalExpr for NestedPredicate {
}
}
-/// Build equality after the planner has reconciled nested operand nullability.
+/// How [`spark_comparison`] treats floating-point operands.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FloatOperands {
+ /// Normalize them, so that the comparison follows Spark's SQL ordering.
+ Normalize,
+ /// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
+ /// other operand. Only a scan's pushed-down data filters use this: Parquet pruning recognizes
+ /// a column compared with a literal but not a normalized column. With row-level pushdown the
+ /// reader also evaluates the filters on each row, and a row it drops never reaches Spark's
+ /// Filter above the scan, so any other shape, which pruning cannot use anyway, is normalized.
+ /// A computed operand such as `-d` can hold a NaN with the sign bit set, which a raw
+ /// comparison sorts below every other value.
+ Raw,
+}
+
+/// Builds a comparison with Spark's SQL ordering for floats, in which `-0.0` equals `0.0`, all
+/// NaNs are equal and NaN sorts above every other value, at any depth of a list or struct.
+///
+/// Arrow compares floats by IEEE 754 total order instead, so each float operand is normalized
+/// first with [`normalize_comparison_operand`], after which the two orders agree. Nested `=` and
+/// `<>` compare with `spark_equality` instead, without building normalized copies of the nested
+/// values. Any other operator, such as `AND`, builds a plain [`BinaryExpr`].
+///
+/// The planner reconciles the nullability of nested operands before calling this.
pub fn spark_comparison(
left: Arc<dyn PhysicalExpr>,
op: Operator,
right: Arc<dyn PhysicalExpr>,
schema: &Schema,
+ float_operands: FloatOperands,
) -> Result<Arc<dyn PhysicalExpr>> {
+ use Operator::*;
+ if !matches!(
+ op,
+ Eq | NotEq | Lt | LtEq | Gt | GtEq | IsDistinctFrom | IsNotDistinctFrom
+ ) {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ }
// An operand whose type does not resolve against this schema falls back to the plain
// comparison, the way `reconcile_nested_comparison_types` already leaves such operands alone.
- let nested = matches!(op, Operator::Eq | Operator::NotEq)
- && match (left.data_type(schema), right.data_type(schema)) {
- (Ok(lt), Ok(_)) => needs_spark_equality(<),
- _ => false,
- };
- if nested {
+ let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
+ return Ok(Arc::new(BinaryExpr::new(left, op, right)));
+ };
+ if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
validate_types(&left, std::slice::from_ref(&right), schema)?;
- Ok(Arc::new(NestedPredicate {
+ return Ok(Arc::new(NestedPredicate {
value: left,
candidates: vec![right],
- negated: op == Operator::NotEq,
+ negated: op == NotEq,
membership: false,
- }))
- } else {
- Ok(Arc::new(BinaryExpr::new(left, op, right)))
+ }));
}
+ let raw = float_operands == FloatOperands::Raw;
+ let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
+ (left, normalize_comparison_operand(right, schema)?)
Review Comment:
[P2] [P2] Do not use one-sided normalization to reject Parquet rows. With `spark.comet.parquet.rowFilterPushdown.enabled=true`, a Parquet column containing negative NaN bits `0xfff8000000000000`, and normal constant folding, `SELECT d FROM t WHERE d = -double('NaN')` previously retained the row. This branch canonicalizes the negative NaN literal while leaving the stored column raw, so Arrow equality now rejects it. Spark considers the NaNs equal. The filter above the scan cannot recover the dropped row. This materially worsens the existing noncanonical-NaN limitation. Keep raw pruning predicates separate from normalized row predicates, or prevent these raw float comparisons from rejecting rows.
Evidence: A fresh planner-level test wrote the negative NaN with ArrowWriter and disabled statistics to isolate row filtering. Through the actual Parquet reader, the base-equivalent predicate retained 1 row, `create_data_filter` at this head retained 0, and the fully normalized predicate retained 1. The regression assertion failed. Fresh Spark 3.5.9 retained the row with Parquet pushdown both off and on, and inspection confirmed the optimized literal bits were `fff8000000000000`. Source: `/tmp/review6447-f2c2-session2-scan-probe.rs`. Logs: `/tmp/review6447-f2c2-session2-scan.log` and `/tmp/review6447-f2c2-session2-spark.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4197855028
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
+## Spark's rules
+
+| Rule | `-0.0` and `0.0` | NaN | Used by |
+| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs |
+| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys |
+| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning |
+| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values |
+| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code |
+
+Some functions changed rules in a Spark release, so a native path has to follow the Spark version
+it runs against:
+
+- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and
Review Comment:
> Please describe Scala collection equality separately and classify pre-4.2 `collect_set` under that rule.
`OpenHashSet`, which line 138 of this page lists under `Double.equals`, also depends on the Spark version. Before [SPARK-45599](https://issues.apache.org/jira/browse/SPARK-45599), `addWithoutResize` matched a key with `_data(pos) == k` ([3.4.3](https://github.com/apache/spark/blob/1eb558c3a6fbdd59e5a305bc3ab12ce748f6511f/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala#L149)), while `DoubleHasher` hashes the `doubleToLongBits` of the value ([3.4.3](https://github.com/apache/spark/blob/1eb558c3a6fbdd59e5a305bc3ab12ce748f6511f/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala#L312-L317)). So two NaNs never match, and `-0.0` and `0.0` start from different slots and match only when probing from one reaches the other. That holds for every 3.4 release, including 3.4.4, and for 3.5.0 and 3.5.1. The [fix](https://github.com/apache/spark/commit/b4118e0dbb505033d6a6a63b4c22c4e4bcf3e8d8) switched to `equals` in 3.5.2 and 4.0.0. When this rule gets r
ewritten, could the page also say that `OpenHashSet` follows `Double.equals` only from 3.5.2 and 4.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4197861645
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
Review Comment:
The new `CometNativeReaderSuite` test says that Spark's own reader probes the bloom filter with the literal's bits and skips the file holding the other zero. As I read parquet-java 1.16.0's `DictionaryFilter`, the same thing happens without bloom filters. It loads the dictionary into a `HashSet` ([L126-L129](https://github.com/apache/parquet-java/blob/402c3810c372d29603e181771acebfecc71bef61/parquet-hadoop/src/main/java/org/apache/parquet/filter2/dictionarylevel/DictionaryFilter.java#L126-L129)) and drops the row group when the set doesn't contain the literal ([L160-L163](https://github.com/apache/parquet-java/blob/402c3810c372d29603e181771acebfecc71bef61/parquet-hadoop/src/main/java/org/apache/parquet/filter2/dictionarylevel/DictionaryFilter.java#L160-L163)), and `Double.equals` keeps the two zeros apart. Parquet files are dictionary-encoded by default. I haven't run this against Spark.
If that holds, `WHERE d = -0.0D` returns rows from Comet that Spark doesn't, whenever a row group holds `0.0` but no `-0.0`. Comet's answer is the one Spark's SQL semantics call for, but someone comparing the two engines will see different results. Should this paragraph say so, and is there a Spark issue we could link for the reader's behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: normalize float operands in the native comparison builder [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6447:
URL: https://github.com/apache/datafusion-comet/pull/6447#discussion_r4196617827
##########
docs/source/contributor-guide/floating_point.md:
##########
@@ -0,0 +1,192 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Floating-Point Semantics
+
+This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
+values. It is aimed at contributors working on an expression or operator that compares, orders,
+hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
+and maps. For user-facing differences from Spark, see
+[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
+to apply these rules systematically is tracked in
+[#6385](https://github.com/apache/datafusion-comet/issues/6385).
+
+The short version: Spark has no single rule for floating-point equality. Each function inherits one
+from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
+which matches none of them. A native implementation must follow the rule of the Spark function it
+replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
+include a NaN with the sign bit set.
+
+## Spark's rules
+
+| Rule | `-0.0` and `0.0` | NaN | Used by |
+| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs |
+| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys |
+| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning |
+| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values |
+| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code |
+
+Some functions changed rules in a Spark release, so a native path has to follow the Spark version
+it runs against:
+
+- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and
Review Comment:
[P2] Correct the pre-4.2 `collect_set` equality rule. Spark uses Scala `mutable.HashSet[Any]`, which merges `-0.0` with `0.0` but retains repeated NaNs. `Double.equals` does the opposite on both cases. The new contributor reference therefore assigns this aggregate the wrong compatibility contract, contradicting both `CometCollectSet.getIncompatibleReasons` and the existing known-gap test. Please describe Scala collection equality separately and classify pre-4.2 `collect_set` under that rule.
Evidence: Fresh Spark 3.5.9 execution over Spark-written Parquet containing ('zero', -0.0), ('zero', 0.0), ('nan', NaN), ('nan', NaN): `SELECT kind, size(collect_set(d)) AS n FROM t GROUP BY kind ORDER BY kind` returned [('nan', 2), ('zero', 1)]. The documented `Double.equals` rule predicts [('nan', 1), ('zero', 2)]. Log: `/tmp/review6447-c5c6-spark-oracle.log`. Spark v3.4.3 and v4.1.3 `expressions/aggregate/collect.scala` confirm `Collect[mutable.HashSet[Any]]` with unchanged floating-point buffer elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org