You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/09/08 18:33:41 UTC
(spark) branch branch-4.3 updated: [SPARK-58897][SQL] Propagate NaN through the infinity norm in vector_norm
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 603f91240811 [SPARK-58897][SQL] Propagate NaN through the infinity norm in vector_norm
603f91240811 is described below
commit 603f9124081142f6e49d59b29dc4625f2f07537b
Author: Sepuri Sai Krishna <sa...@gmail.com>
AuthorDate: Tue Sep 8 18:33:11 2026 +0000
[SPARK-58897][SQL] Propagate NaN through the infinity norm in vector_norm
### What changes were proposed in this pull request?
This PR makes the infinity-norm branch of `vector_norm` propagate `NaN` elements instead of
silently skipping them.
`VectorFunctionImplUtils.vectorInfNorm` computed the maximum absolute value with a hand-rolled
comparison against a running maximum seeded at zero:
```java
float maxAbs = 0.0f;
...
float absVal = Math.abs(vec.getFloat(i));
if (absVal > maxAbs) {
maxAbs = absVal;
}
```
Every comparison involving `NaN` is false under IEEE 754, so a `NaN` element never became the
running maximum and never reached the result. The loop now uses `Math.max`, which returns `NaN`
when either argument is `NaN`. The accumulator is widened to `double` to match `vectorL1Norm`
and `vectorL2Norm` and the method's own "in double precision" contract; the infinity norm of a
float vector cannot overflow the float range, so this part is for consistency rather than a
second fix.
No change is needed in `vectorNormalize`: `vectorNormalizeWithNorm` guards only `norm == 0.0d`,
so a `NaN` norm falls through to the division and yields an all-`NaN` vector, which is what
degree 2.0 already returns.
`VectorNorm` is `RuntimeReplaceable` and lowers to a single `StaticInvoke` on
`VectorFunctionImplUtils`, so the interpreted and codegen paths share this one implementation.
### Why are the changes needed?
`vector_norm` with degree infinity returns a wrong result for any vector containing `NaN`. The
single-element case is the damaging one: a vector of `NaN` is reported as having infinity norm
`0.0`, making it indistinguishable from the zero vector, which in turn makes `vector_normalize`
return `NULL` through its zero-norm path.
```sql
SELECT vector_norm(array(float('nan')), float('inf')); -- 0.0 expected NaN
SELECT vector_norm(array(float('nan'), 5.0F), float('inf')); -- 5.0 expected NaN
SELECT vector_norm(array(float('nan'), float('inf')), float('inf')); -- Infinity expected NaN
SELECT vector_normalize(array(float('nan')), float('inf')); -- NULL expected [NaN]
```
The infinity norm is the only degree that behaves this way; degrees 1.0 and 2.0 both propagate
`NaN`. It is also the only max-like function in Spark that does not treat `NaN` as the largest
value:
```sql
SELECT greatest(float('nan'), 5.0F); -- NaN
SELECT array_max(array(float('nan'), 5.0F)); -- NaN
SELECT max(v) FROM values (float('nan')), (5.0F) AS t(v); -- NaN
SELECT sort_array(array(float('nan'), 5.0F)); -- [5.0, NaN]
```
Since the infinity norm is defined as the maximum absolute value, it should follow the same
convention as `max` and `array_max`.
### Does this PR introduce _any_ user-facing change?
Yes. `vector_norm(v, float('inf'))` now returns `NaN` when `v` contains a `NaN` element, and
`vector_normalize(v, float('inf'))` returns an all-`NaN` vector rather than `NULL`. Both are bug
fixes relative to released 4.2.0.
| Query | Before | After |
| --- | --- | --- |
| `vector_norm(array(float('nan')), float('inf'))` | `0.0` | `NaN` |
| `vector_norm(array(float('nan'), 5.0F), float('inf'))` | `5.0` | `NaN` |
| `vector_norm(array(float('nan'), float('inf')), float('inf'))` | `Infinity` | `NaN` |
| `vector_normalize(array(float('nan')), float('inf'))` | `NULL` | `[NaN]` |
| `vector_normalize(array(float('nan'), 5.0F), float('inf'))` | `[NaN, 1.0]` | `[NaN, NaN]` |
Behaviour for empty vectors (`0.0`), vectors containing `NULL` (`NULL`), and genuinely zero
vectors is unchanged. A `NULL` element continues to take precedence over a `NaN` element.
### How was this patch tested?
Added a `SPARK-58897` section to `sql-tests/inputs/vector-norm.sql` covering the single-element
`NaN` case, `NaN` in either position, `NaN` alongside an infinite element, `NaN` in a longer
16-element vector, the `NULL`-takes-precedence case, the L1 and L2 degrees as regression
guards, and both `vector_normalize` cases. The golden file was regenerated; the resulting diff
is purely additive, with no existing expectation changed.
```
build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z vector"
build/sbt "sql/testOnly org.apache.spark.sql.MiscFunctionsSuite -- -z \"vector functions\""
```
All pass, including the existing `vector-distance.sql` and `vector-agg.sql` golden files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #58149 from SEPURI-SAI-KRISHNA/SPARK-58897-vector-inf-norm-nan.
Authored-by: Sepuri Sai Krishna <sa...@gmail.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit d895f57f367ae39df5e2f94fa6643f66ffe0db9b)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
.../expressions/VectorFunctionImplUtils.java | 14 ++--
.../sql-tests/analyzer-results/vector-norm.sql.out | 73 +++++++++++++++++++
.../resources/sql-tests/inputs/vector-norm.sql | 26 +++++++
.../sql-tests/results/vector-norm.sql.out | 83 ++++++++++++++++++++++
4 files changed, 190 insertions(+), 6 deletions(-)
diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
index c288c5837668..b8681eb6f99b 100644
--- a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
+++ b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
@@ -361,6 +361,8 @@ public class VectorFunctionImplUtils {
/**
* Computes the infinity norm (maximum absolute value) of a float vector, in double precision.
* Returns NULL if the vector contains NULL elements.
+ * Returns NaN if the vector contains NaN elements, following the convention of max and
+ * array_max that NaN compares as larger than any other value.
* Returns 0.0 for empty vectors.
*/
public static Double vectorInfNorm(ArrayData vec) {
@@ -370,18 +372,18 @@ public class VectorFunctionImplUtils {
return 0.0d;
}
- float maxAbs = 0.0f;
+ // Math.max is used rather than a comparison against the running maximum: every comparison
+ // involving NaN is false, so a hand-rolled maximum would skip NaN elements and report the
+ // largest of the remaining ones instead of propagating the NaN.
+ double maxAbs = 0.0d;
for (int i = 0; i < len; i++) {
if (vec.isNullAt(i)) {
return null;
}
- float absVal = Math.abs(vec.getFloat(i));
- if (absVal > maxAbs) {
- maxAbs = absVal;
- }
+ maxAbs = Math.max(maxAbs, Math.abs((double) vec.getFloat(i)));
}
- return (double) maxAbs;
+ return maxAbs;
}
/**
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
index 287c039e6c94..7d453dd514ae 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
@@ -635,3 +635,76 @@ SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F)
-- !query analysis
Project [vector_normalize(array(cast(inf as float), 1.0), 2.0) AS vector_normalize(array(inf, 1.0), 2.0)#x]
+- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan')), float('inf'))
+-- !query analysis
+Project [vector_norm(array(cast(nan as float)), cast(inf as float)) AS vector_norm(array(nan), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), float('inf'))
+-- !query analysis
+Project [vector_norm(array(cast(nan as float), 5.0), cast(inf as float)) AS vector_norm(array(nan, 5.0), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(5.0F, float('nan')), float('inf'))
+-- !query analysis
+Project [vector_norm(array(5.0, cast(nan as float)), cast(inf as float)) AS vector_norm(array(5.0, nan), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), float('inf')), float('inf'))
+-- !query analysis
+Project [vector_norm(array(cast(nan as float), cast(inf as float)), cast(inf as float)) AS vector_norm(array(nan, inf), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), 1.0F)
+-- !query analysis
+Project [vector_norm(array(cast(nan as float), 5.0), 1.0) AS vector_norm(array(nan, 5.0), 1.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), 2.0F)
+-- !query analysis
+Project [vector_norm(array(cast(nan as float), 5.0), 2.0) AS vector_norm(array(nan, 5.0), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(
+ array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, float('nan'), 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
+ float('inf')
+)
+-- !query analysis
+Project [vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, cast(nan as float), 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), cast(inf as float)) AS vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, nan, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), CAST(NULL AS FLOAT)), float('inf'))
+-- !query analysis
+Project [vector_norm(array(cast(nan as float), cast(null as float)), cast(inf as float)) AS vector_norm(array(nan, CAST(NULL AS FLOAT)), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(float('nan')), float('inf'))
+-- !query analysis
+Project [vector_normalize(array(cast(nan as float)), cast(inf as float)) AS vector_normalize(array(nan), inf)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(float('nan'), 5.0F), float('inf'))
+-- !query analysis
+Project [vector_normalize(array(cast(nan as float), 5.0), cast(inf as float)) AS vector_normalize(array(nan, 5.0), inf)#x]
++- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql b/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
index cab7548a628d..932f37792a39 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
@@ -186,3 +186,29 @@ SELECT vector_norm(array(float('inf'), 1.0F), 2.0F);
-- vector_normalize: dividing by an infinite norm yields NaN for the infinite element
SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F);
+
+-- SPARK-58897: NaN elements propagate through the norm, following the convention of max and
+-- array_max that NaN compares as larger than any other value
+
+-- vector_norm: a NaN element makes the infinity norm NaN, not the largest of the other elements
+SELECT vector_norm(array(float('nan')), float('inf'));
+SELECT vector_norm(array(float('nan'), 5.0F), float('inf'));
+SELECT vector_norm(array(5.0F, float('nan')), float('inf'));
+SELECT vector_norm(array(float('nan'), float('inf')), float('inf'));
+
+-- vector_norm: the L1 and L2 norms propagate NaN in the same way
+SELECT vector_norm(array(float('nan'), 5.0F), 1.0F);
+SELECT vector_norm(array(float('nan'), 5.0F), 2.0F);
+
+-- vector_norm: a NaN element anywhere in a longer vector still makes the norm NaN
+SELECT vector_norm(
+ array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, float('nan'), 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
+ float('inf')
+);
+
+-- vector_norm: a NULL element still takes precedence over a NaN element
+SELECT vector_norm(array(float('nan'), CAST(NULL AS FLOAT)), float('inf'));
+
+-- vector_normalize: a NaN norm yields an all-NaN vector, not NULL
+SELECT vector_normalize(array(float('nan')), float('inf'));
+SELECT vector_normalize(array(float('nan'), 5.0F), float('inf'));
diff --git a/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out b/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
index 31a2aa2fb08e..1b2a883df179 100644
--- a/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
@@ -752,3 +752,86 @@ SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F)
struct<vector_normalize(array(inf, 1.0), 2.0):array<float>>
-- !query output
[NaN,0.0]
+
+
+-- !query
+SELECT vector_norm(array(float('nan')), float('inf'))
+-- !query schema
+struct<vector_norm(array(nan), inf):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), float('inf'))
+-- !query schema
+struct<vector_norm(array(nan, 5.0), inf):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(5.0F, float('nan')), float('inf'))
+-- !query schema
+struct<vector_norm(array(5.0, nan), inf):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), float('inf')), float('inf'))
+-- !query schema
+struct<vector_norm(array(nan, inf), inf):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), 1.0F)
+-- !query schema
+struct<vector_norm(array(nan, 5.0), 1.0):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), 5.0F), 2.0F)
+-- !query schema
+struct<vector_norm(array(nan, 5.0), 2.0):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(
+ array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, float('nan'), 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
+ float('inf')
+)
+-- !query schema
+struct<vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, nan, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), inf):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_norm(array(float('nan'), CAST(NULL AS FLOAT)), float('inf'))
+-- !query schema
+struct<vector_norm(array(nan, CAST(NULL AS FLOAT)), inf):float>
+-- !query output
+NULL
+
+
+-- !query
+SELECT vector_normalize(array(float('nan')), float('inf'))
+-- !query schema
+struct<vector_normalize(array(nan), inf):array<float>>
+-- !query output
+[NaN]
+
+
+-- !query
+SELECT vector_normalize(array(float('nan'), 5.0F), float('inf'))
+-- !query schema
+struct<vector_normalize(array(nan, 5.0), inf):array<float>>
+-- !query output
+[NaN,NaN]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org