You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/08/10 05:59:44 UTC
(spark) branch branch-4.3 updated: [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new da54921d8bb8 [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow
da54921d8bb8 is described below
commit da54921d8bb82471df4d8685c23455418a6fac00
Author: Sepuri Sai Krishna <sa...@gmail.com>
AuthorDate: Mon Aug 10 13:58:09 2026 +0800
[SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow
### What changes were proposed in this pull request?
`VectorFunctionImplUtils` accumulates dot products and sums of squares in single precision. Those
intermediate quantities are quadratic in the input values, so they overflow to infinity (or
underflow to zero) long before the final result leaves the float range, and the function returns
`NaN`, `Infinity`, `NULL` or an all-zero vector for inputs whose true result is perfectly
representable as a float.
This PR accumulates the intermediate dot products, sums of squares and absolute sums in `double`
and rounds only the final result back to `float`. `vector_normalize` also keeps the norm in
`double` instead of rounding it to a float before dividing, so a vector whose norm is outside the
float range is still normalized correctly.
Affected functions: `vector_cosine_similarity`, `vector_inner_product`, `vector_l2_distance`,
`vector_norm`, `vector_normalize`.
### Why are the changes needed?
Cosine similarity and normalization are scale invariant, so rescaling the input must not change
the result -- but today it does:
| Query | Before | After (correct) |
| --- | --- | --- |
| `vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F))` | `NaN` | `1.0` |
| `vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F))` | `NULL` | `1.0` |
| `vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))` | `Infinity` | `5.0E19` |
| `vector_norm(array(3.0e19F, 4.0e19F), 2.0F)` | `Infinity` | `5.0E19` |
| `vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)` | `[0.0, 0.0]` | `[0.6, 0.8]` |
| `vector_normalize(array(1.0e-23F, 0.0F), 2.0F)` | `NULL` | `[1.0, 0.0]` |
In every row above the correct result is an ordinary float; only the intermediate sum overflows
(`(3e19)^2 + (4e19)^2 = 2.5e39 > Float.MAX_VALUE`) or underflows (`(1e-23)^2 = 1e-46 < the
smallest positive float`). The wrong values are returned silently, so they propagate into
similarity search results and rankings rather than failing loudly.
Double-precision accumulation is also what other vector similarity implementations do
(for example pgvector), and it removes a rounding error in the common case as well: the existing
`vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F))` result changes from
`0.9746319` to `0.97463185`, which is the correctly rounded float of `32 / sqrt(1078)`
(`0.9746318461970763...`).
These functions were added in 4.2.0 by SPARK-54713 and SPARK-55030, so the wrong results are
present in the 4.2.0 release.
Documented behaviours are unchanged: empty vectors, `NULL` inputs, vectors containing `NULL`
elements, and genuinely zero-magnitude vectors return exactly what they returned before.
### Does this PR introduce _any_ user-facing change?
Yes. The functions listed above now return correct results instead of `NaN`, `Infinity`, `NULL`
or zeros when an intermediate sum overflows or underflows the float range, as shown in the table
above. This is a change relative to 4.2.0, where these functions were introduced.
`vector_cosine_similarity` also changes by one ULP for ordinary inputs, because the result is now
correctly rounded (`0.9746319` -> `0.97463185` for the example above).
### How was this patch tested?
- New golden test cases in `sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql` and
`vector-norm.sql` covering large and small magnitude vectors for all five functions; golden
files regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`.
- Existing suites: `SQLQueryTestSuite` (vector files), `MiscFunctionsSuite`, `ExpressionInfoSuite`
(the `vector_cosine_similarity` example in `ExpressionDescription` is updated to the correctly
rounded value).
On performance: the unrolled loops read through `ArrayData.getFloat()` / `isNullAt()`, and those
accesses dominate the loop, so widening the accumulators is not measurable end to end. Timing the
`vector_cosine_similarity` loop over 1536-dimensional `UnsafeArrayData` inputs (200k calls, JIT
warmed up) gives 5834 ns/call before and 5891 ns/call after in one run, and 5870 vs 5792 ns/call in
the next -- i.e. within run-to-run noise. The arithmetic in isolation (plain `float[]`, no
`ArrayData` indirection) is ~1.3x slower, but it accounts for only ~20% of the loop cost.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57746 from SEPURI-SAI-KRISHNA/SPARK-58544-vector-float-overflow.
Authored-by: Sepuri Sai Krishna <sa...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit d28c23f19f6d03e4cc472a507b0911bfd93829a9)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../expressions/VectorFunctionImplUtils.java | 193 ++++++++++++---------
.../catalyst/expressions/vectorExpressions.scala | 2 +-
.../analyzer-results/vector-distance.sql.out | 63 +++++++
.../sql-tests/analyzer-results/vector-norm.sql.out | 63 +++++++
.../resources/sql-tests/inputs/vector-distance.sql | 35 ++++
.../resources/sql-tests/inputs/vector-norm.sql | 33 ++++
.../sql-tests/results/vector-distance.sql.out | 74 +++++++-
.../sql-tests/results/vector-norm.sql.out | 74 +++++++-
8 files changed, 453 insertions(+), 84 deletions(-)
diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
index 8a3223c588fb..c288c5837668 100644
--- a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
+++ b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java
@@ -31,6 +31,9 @@ public class VectorFunctionImplUtils {
* Returns NULL if either vector contains NULL elements, has zero magnitude, or is empty.
* Throws an exception if vectors have different dimensions.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
+ * The dot product and the squared norms are accumulated in double precision: their magnitudes
+ * are quadratic in the input values, so single precision would overflow to infinity (or
+ * underflow to zero) for vectors whose cosine similarity is perfectly representable as a float.
*/
public static Float vectorCosineSimilarity(ArrayData left, ArrayData right, UTF8String funcName) {
int leftLen = left.numElements();
@@ -45,9 +48,9 @@ public class VectorFunctionImplUtils {
return null;
}
- float dotProduct = 0.0f;
- float norm1Sq = 0.0f;
- float norm2Sq = 0.0f;
+ double dotProduct = 0.0d;
+ double norm1Sq = 0.0d;
+ double norm2Sq = 0.0d;
int i = 0;
int simdLimit = (leftLen / 8) * 8;
@@ -66,15 +69,15 @@ public class VectorFunctionImplUtils {
return null;
}
- float a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
- float a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
- float a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
- float a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
+ double a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
+ double a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
+ double a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
+ double a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
- float b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
- float b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
- float b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
- float b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
+ double b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
+ double b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
+ double b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
+ double b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
dotProduct += a0 * b0 + a1 * b1 + a2 * b2 + a3 * b3 +
a4 * b4 + a5 * b5 + a6 * b6 + a7 * b7;
@@ -90,19 +93,23 @@ public class VectorFunctionImplUtils {
if (left.isNullAt(i) || right.isNullAt(i)) {
return null;
}
- float a = left.getFloat(i);
- float b = right.getFloat(i);
+ double a = left.getFloat(i);
+ double b = right.getFloat(i);
dotProduct += a * b;
norm1Sq += a * a;
norm2Sq += b * b;
i++;
}
- float normProduct = (float) Math.sqrt(norm1Sq * norm2Sq);
- if (normProduct < Float.MIN_NORMAL) {
+ // For vectors of finite elements, `norm1Sq * norm2Sq` cannot overflow in double precision:
+ // both factors are bounded by MAX_ROUNDED_ARRAY_LENGTH * Float.MAX_VALUE^2, so their product
+ // stays well below Double.MAX_VALUE. An element that is already infinite makes the product
+ // infinite and the result NaN, exactly as it did before the accumulators were widened.
+ double normProduct = Math.sqrt(norm1Sq * norm2Sq);
+ if (normProduct == 0.0d) {
return null;
}
- return dotProduct / normProduct;
+ return (float) (dotProduct / normProduct);
}
/**
@@ -111,6 +118,8 @@ public class VectorFunctionImplUtils {
* Returns 0.0 for empty vectors.
* Throws an exception if vectors have different dimensions.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
+ * The dot product is accumulated in double precision so that intermediate terms do not
+ * overflow to infinity when the final result is representable as a float.
*/
public static Float vectorInnerProduct(ArrayData left, ArrayData right, UTF8String funcName) {
int leftLen = left.numElements();
@@ -125,7 +134,7 @@ public class VectorFunctionImplUtils {
return 0.0f;
}
- float dotProduct = 0.0f;
+ double dotProduct = 0.0d;
int i = 0;
int simdLimit = (leftLen / 8) * 8;
@@ -144,15 +153,15 @@ public class VectorFunctionImplUtils {
return null;
}
- float a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
- float a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
- float a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
- float a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
+ double a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
+ double a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
+ double a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
+ double a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
- float b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
- float b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
- float b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
- float b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
+ double b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
+ double b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
+ double b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
+ double b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
dotProduct += a0 * b0 + a1 * b1 + a2 * b2 + a3 * b3 +
a4 * b4 + a5 * b5 + a6 * b6 + a7 * b7;
@@ -164,13 +173,13 @@ public class VectorFunctionImplUtils {
if (left.isNullAt(i) || right.isNullAt(i)) {
return null;
}
- float a = left.getFloat(i);
- float b = right.getFloat(i);
+ double a = left.getFloat(i);
+ double b = right.getFloat(i);
dotProduct += a * b;
i++;
}
- return dotProduct;
+ return (float) dotProduct;
}
/**
@@ -179,6 +188,8 @@ public class VectorFunctionImplUtils {
* Returns 0.0 for empty vectors.
* Throws an exception if vectors have different dimensions.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
+ * The sum of squares is accumulated in double precision: it is quadratic in the input values,
+ * so single precision would overflow to infinity for distances representable as a float.
*/
public static Float vectorL2Distance(ArrayData left, ArrayData right, UTF8String funcName) {
int leftLen = left.numElements();
@@ -193,7 +204,7 @@ public class VectorFunctionImplUtils {
return 0.0f;
}
- float sumSq = 0.0f;
+ double sumSq = 0.0d;
int i = 0;
int simdLimit = (leftLen / 8) * 8;
@@ -212,18 +223,18 @@ public class VectorFunctionImplUtils {
return null;
}
- float a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
- float a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
- float a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
- float a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
+ double a0 = left.getFloat(i), a1 = left.getFloat(i + 1);
+ double a2 = left.getFloat(i + 2), a3 = left.getFloat(i + 3);
+ double a4 = left.getFloat(i + 4), a5 = left.getFloat(i + 5);
+ double a6 = left.getFloat(i + 6), a7 = left.getFloat(i + 7);
- float b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
- float b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
- float b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
- float b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
+ double b0 = right.getFloat(i), b1 = right.getFloat(i + 1);
+ double b2 = right.getFloat(i + 2), b3 = right.getFloat(i + 3);
+ double b4 = right.getFloat(i + 4), b5 = right.getFloat(i + 5);
+ double b6 = right.getFloat(i + 6), b7 = right.getFloat(i + 7);
- float d0 = a0 - b0, d1 = a1 - b1, d2 = a2 - b2, d3 = a3 - b3;
- float d4 = a4 - b4, d5 = a5 - b5, d6 = a6 - b6, d7 = a7 - b7;
+ double d0 = a0 - b0, d1 = a1 - b1, d2 = a2 - b2, d3 = a3 - b3;
+ double d4 = a4 - b4, d5 = a5 - b5, d6 = a6 - b6, d7 = a7 - b7;
sumSq += d0 * d0 + d1 * d1 + d2 * d2 + d3 * d3 +
d4 * d4 + d5 * d5 + d6 * d6 + d7 * d7;
@@ -235,9 +246,9 @@ public class VectorFunctionImplUtils {
if (left.isNullAt(i) || right.isNullAt(i)) {
return null;
}
- float a = left.getFloat(i);
- float b = right.getFloat(i);
- float diff = a - b;
+ double a = left.getFloat(i);
+ double b = right.getFloat(i);
+ double diff = a - b;
sumSq += diff * diff;
i++;
}
@@ -246,19 +257,19 @@ public class VectorFunctionImplUtils {
}
/**
- * Computes the L1 norm (Manhattan norm) of a float vector.
+ * Computes the L1 norm (Manhattan norm) of a float vector, in double precision.
* Returns NULL if the vector contains NULL elements.
* Returns 0.0 for empty vectors.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
*/
- public static Float vectorL1Norm(ArrayData vec) {
+ public static Double vectorL1Norm(ArrayData vec) {
int len = vec.numElements();
if (len == 0) {
- return 0.0f;
+ return 0.0d;
}
- float sum = 0.0f;
+ double sum = 0.0d;
int i = 0;
int simdLimit = (len / 8) * 8;
@@ -273,10 +284,10 @@ public class VectorFunctionImplUtils {
return null;
}
- float a0 = vec.getFloat(i), a1 = vec.getFloat(i + 1);
- float a2 = vec.getFloat(i + 2), a3 = vec.getFloat(i + 3);
- float a4 = vec.getFloat(i + 4), a5 = vec.getFloat(i + 5);
- float a6 = vec.getFloat(i + 6), a7 = vec.getFloat(i + 7);
+ double a0 = vec.getFloat(i), a1 = vec.getFloat(i + 1);
+ double a2 = vec.getFloat(i + 2), a3 = vec.getFloat(i + 3);
+ double a4 = vec.getFloat(i + 4), a5 = vec.getFloat(i + 5);
+ double a6 = vec.getFloat(i + 6), a7 = vec.getFloat(i + 7);
sum += Math.abs(a0) + Math.abs(a1) + Math.abs(a2) + Math.abs(a3) +
Math.abs(a4) + Math.abs(a5) + Math.abs(a6) + Math.abs(a7);
@@ -288,7 +299,7 @@ public class VectorFunctionImplUtils {
if (vec.isNullAt(i)) {
return null;
}
- float a = vec.getFloat(i);
+ double a = vec.getFloat(i);
sum += Math.abs(a);
i++;
}
@@ -297,19 +308,19 @@ public class VectorFunctionImplUtils {
}
/**
- * Computes the L2 norm (Euclidean norm) of a float vector.
+ * Computes the L2 norm (Euclidean norm) of a float vector, in double precision.
* Returns NULL if the vector contains NULL elements.
* Returns 0.0 for empty vectors.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
*/
- public static Float vectorL2Norm(ArrayData vec) {
+ public static Double vectorL2Norm(ArrayData vec) {
int len = vec.numElements();
if (len == 0) {
- return 0.0f;
+ return 0.0d;
}
- float sumSq = 0.0f;
+ double sumSq = 0.0d;
int i = 0;
int simdLimit = (len / 8) * 8;
@@ -324,10 +335,10 @@ public class VectorFunctionImplUtils {
return null;
}
- float a0 = vec.getFloat(i), a1 = vec.getFloat(i + 1);
- float a2 = vec.getFloat(i + 2), a3 = vec.getFloat(i + 3);
- float a4 = vec.getFloat(i + 4), a5 = vec.getFloat(i + 5);
- float a6 = vec.getFloat(i + 6), a7 = vec.getFloat(i + 7);
+ double a0 = vec.getFloat(i), a1 = vec.getFloat(i + 1);
+ double a2 = vec.getFloat(i + 2), a3 = vec.getFloat(i + 3);
+ double a4 = vec.getFloat(i + 4), a5 = vec.getFloat(i + 5);
+ double a6 = vec.getFloat(i + 6), a7 = vec.getFloat(i + 7);
sumSq += a0 * a0 + a1 * a1 + a2 * a2 + a3 * a3 +
a4 * a4 + a5 * a5 + a6 * a6 + a7 * a7;
@@ -339,24 +350,24 @@ public class VectorFunctionImplUtils {
if (vec.isNullAt(i)) {
return null;
}
- float a = vec.getFloat(i);
+ double a = vec.getFloat(i);
sumSq += a * a;
i++;
}
- return (float) Math.sqrt(sumSq);
+ return Math.sqrt(sumSq);
}
/**
- * Computes the infinity norm (maximum absolute value) of a float vector.
+ * Computes the infinity norm (maximum absolute value) of a float vector, in double precision.
* Returns NULL if the vector contains NULL elements.
* Returns 0.0 for empty vectors.
*/
- public static Float vectorInfNorm(ArrayData vec) {
+ public static Double vectorInfNorm(ArrayData vec) {
int len = vec.numElements();
if (len == 0) {
- return 0.0f;
+ return 0.0d;
}
float maxAbs = 0.0f;
@@ -370,7 +381,7 @@ public class VectorFunctionImplUtils {
}
}
- return maxAbs;
+ return (double) maxAbs;
}
/**
@@ -378,15 +389,17 @@ public class VectorFunctionImplUtils {
* Returns NULL if the vector contains NULL elements or if the norm is zero.
* Returns an empty array for empty vectors.
* Uses manual loop unrolling (8 elements at a time) for speculative SIMD optimization.
+ * The norm is taken in double precision so that vectors whose norm is not representable as a
+ * float (or is only representable as a subnormal float) are still normalized correctly.
*/
- public static ArrayData vectorNormalizeWithNorm(ArrayData vec, float norm) {
+ public static ArrayData vectorNormalizeWithNorm(ArrayData vec, double norm) {
int len = vec.numElements();
if (len == 0) {
return vec;
}
- if (norm < Float.MIN_NORMAL) {
+ if (norm == 0.0d) {
return null;
}
@@ -405,14 +418,14 @@ public class VectorFunctionImplUtils {
return null;
}
- result[i] = vec.getFloat(i) / norm;
- result[i + 1] = vec.getFloat(i + 1) / norm;
- result[i + 2] = vec.getFloat(i + 2) / norm;
- result[i + 3] = vec.getFloat(i + 3) / norm;
- result[i + 4] = vec.getFloat(i + 4) / norm;
- result[i + 5] = vec.getFloat(i + 5) / norm;
- result[i + 6] = vec.getFloat(i + 6) / norm;
- result[i + 7] = vec.getFloat(i + 7) / norm;
+ result[i] = (float) (vec.getFloat(i) / norm);
+ result[i + 1] = (float) (vec.getFloat(i + 1) / norm);
+ result[i + 2] = (float) (vec.getFloat(i + 2) / norm);
+ result[i + 3] = (float) (vec.getFloat(i + 3) / norm);
+ result[i + 4] = (float) (vec.getFloat(i + 4) / norm);
+ result[i + 5] = (float) (vec.getFloat(i + 5) / norm);
+ result[i + 6] = (float) (vec.getFloat(i + 6) / norm);
+ result[i + 7] = (float) (vec.getFloat(i + 7) / norm);
i += 8;
}
@@ -421,7 +434,7 @@ public class VectorFunctionImplUtils {
if (vec.isNullAt(i)) {
return null;
}
- result[i] = vec.getFloat(i) / norm;
+ result[i] = (float) (vec.getFloat(i) / norm);
i++;
}
@@ -429,13 +442,13 @@ public class VectorFunctionImplUtils {
}
/**
- * Computes the Lp norm of a float vector using the specified degree.
- * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (L∞).
+ * Computes the Lp norm of a float vector using the specified degree, in double precision.
+ * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (infinity norm).
* Returns NULL if the vector contains NULL elements.
* Returns 0.0 for empty vectors.
* Throws INVALID_VECTOR_NORM_DEGREE if degree is not supported.
*/
- public static Float vectorNorm(ArrayData vec, float degree, UTF8String funcName) {
+ private static Double vectorNormAsDouble(ArrayData vec, float degree, UTF8String funcName) {
// exact floating point comparison for degree since this is direct user input
if (degree == 1.0f) {
return vectorL1Norm(vec);
@@ -448,15 +461,33 @@ public class VectorFunctionImplUtils {
}
}
+ /**
+ * Computes the Lp norm of a float vector using the specified degree.
+ * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (infinity norm).
+ * Returns NULL if the vector contains NULL elements.
+ * Returns 0.0 for empty vectors.
+ * Throws INVALID_VECTOR_NORM_DEGREE if degree is not supported.
+ */
+ public static Float vectorNorm(ArrayData vec, float degree, UTF8String funcName) {
+ Double norm = vectorNormAsDouble(vec, degree, funcName);
+ if (norm == null) {
+ return null;
+ }
+ return (float) norm.doubleValue();
+ }
+
/**
* Normalizes a float vector to unit length using the specified norm degree.
- * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (L∞).
+ * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (infinity norm).
* Returns NULL if the vector contains NULL elements or has zero norm.
* Returns an empty array for empty vectors.
* Throws INVALID_VECTOR_NORM_DEGREE if degree is not supported.
*/
public static ArrayData vectorNormalize(ArrayData vec, float degree, UTF8String funcName) {
- Float norm = vectorNorm(vec, degree, funcName);
+ // The norm is kept in double precision here: rounding it to a float first would turn a norm
+ // that overflows (or underflows) the float range into infinity (or zero) and produce an
+ // all-zero (or NULL) result for a vector that is perfectly normalizable.
+ Double norm = vectorNormAsDouble(vec, degree, funcName);
if (norm == null) {
return null;
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
index d730c8f0be7b..ebda73e42a3b 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
@@ -52,7 +52,7 @@ import org.apache.spark.unsafe.Platform
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
- 0.9746319
+ 0.97463185
""",
since = "4.2.0",
group = "vector_funcs"
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-distance.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-distance.sql.out
index 42239b329730..85b570926cbf 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-distance.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-distance.sql.out
@@ -654,3 +654,66 @@ SELECT vector_l2_distance(
-- !query analysis
Project [vector_l2_distance(array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), array(16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0)) AS vector_l2_distance(array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), array(16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0))#x]
+- OneRowRelation
+
+
+-- !query
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F))
+-- !query analysis
+Project [vector_cosine_similarity(array(3.0E19, 4.0E19), array(3.0E19, 4.0E19)) AS vector_cosine_similarity(array(3.0E19, 4.0E19), array(3.0E19, 4.0E19))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(-3.0e19F, -4.0e19F))
+-- !query analysis
+Project [vector_cosine_similarity(array(3.0E19, 4.0E19), array(-3.0E19, -4.0E19)) AS vector_cosine_similarity(array(3.0E19, 4.0E19), array(-3.0E19, -4.0E19))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_inner_product(array(1.0e20F, 1.0e20F), array(1.0e20F, -1.0e20F))
+-- !query analysis
+Project [vector_inner_product(array(1.0E20, 1.0E20), array(1.0E20, -1.0E20)) AS vector_inner_product(array(1.0E20, 1.0E20), array(1.0E20, -1.0E20))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))
+-- !query analysis
+Project [vector_l2_distance(array(3.0E19, 4.0E19), array(0.0, 0.0)) AS vector_l2_distance(array(3.0E19, 4.0E19), array(0.0, 0.0))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F))
+-- !query analysis
+Project [vector_cosine_similarity(array(1.0E-23, 0.0), array(1.0E-23, 0.0)) AS vector_cosine_similarity(array(1.0E-23, 0.0), array(1.0E-23, 0.0))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_cosine_similarity(array(1.0e-23F, 1.0e-23F), array(1.0e-23F, -1.0e-23F))
+-- !query analysis
+Project [vector_cosine_similarity(array(1.0E-23, 1.0E-23), array(1.0E-23, -1.0E-23)) AS vector_cosine_similarity(array(1.0E-23, 1.0E-23), array(1.0E-23, -1.0E-23))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_cosine_similarity(array(float('inf'), 1.0F), array(1.0F, 1.0F))
+-- !query analysis
+Project [vector_cosine_similarity(array(cast(inf as float), 1.0), array(1.0, 1.0)) AS vector_cosine_similarity(array(inf, 1.0), array(1.0, 1.0))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_inner_product(array(float('inf'), 1.0F), array(1.0F, 1.0F))
+-- !query analysis
+Project [vector_inner_product(array(cast(inf as float), 1.0), array(1.0, 1.0)) AS vector_inner_product(array(inf, 1.0), array(1.0, 1.0))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_l2_distance(array(float('inf'), 1.0F), array(0.0F, 0.0F))
+-- !query analysis
+Project [vector_l2_distance(array(cast(inf as float), 1.0), array(0.0, 0.0)) AS vector_l2_distance(array(inf, 1.0), array(0.0, 0.0))#x]
++- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
index 035ad40cc122..287c039e6c94 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/vector-norm.sql.out
@@ -572,3 +572,66 @@ SELECT vector_norm(
-- !query analysis
Project [vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, cast(null as float), 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), 2.0) AS vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, CAST(NULL AS FLOAT), 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), 2.0)#x]
+- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(3.0e19F, 4.0e19F), 2.0F)
+-- !query analysis
+Project [vector_norm(array(3.0E19, 4.0E19), 2.0) AS vector_norm(array(3.0E19, 4.0E19), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)
+-- !query analysis
+Project [vector_normalize(array(3.0E19, 4.0E19), 2.0) AS vector_normalize(array(3.0E19, 4.0E19), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(3.0e38F, 3.0e38F), 1.0F)
+-- !query analysis
+Project [vector_norm(array(3.0E38, 3.0E38), 1.0) AS vector_norm(array(3.0E38, 3.0E38), 1.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(3.0e38F, 3.0e38F), 1.0F)
+-- !query analysis
+Project [vector_normalize(array(3.0E38, 3.0E38), 1.0) AS vector_normalize(array(3.0E38, 3.0E38), 1.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(1.0e-23F, 0.0F), 2.0F)
+-- !query analysis
+Project [vector_norm(array(1.0E-23, 0.0), 2.0) AS vector_norm(array(1.0E-23, 0.0), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(1.0e-23F, 0.0F), 2.0F)
+-- !query analysis
+Project [vector_normalize(array(1.0E-23, 0.0), 2.0) AS vector_normalize(array(1.0E-23, 0.0), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(1.0e-23F, 1.0e-23F), 2.0F)
+-- !query analysis
+Project [vector_normalize(array(1.0E-23, 1.0E-23), 2.0) AS vector_normalize(array(1.0E-23, 1.0E-23), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_norm(array(float('inf'), 1.0F), 2.0F)
+-- !query analysis
+Project [vector_norm(array(cast(inf as float), 1.0), 2.0) AS vector_norm(array(inf, 1.0), 2.0)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F)
+-- !query analysis
+Project [vector_normalize(array(cast(inf as float), 1.0), 2.0) AS vector_normalize(array(inf, 1.0), 2.0)#x]
++- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql b/sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql
index 24035963260b..ce05649a7d5e 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql
@@ -130,3 +130,38 @@ SELECT vector_l2_distance(
array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, 6.0F, 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
array(16.0F, 15.0F, 14.0F, 13.0F, 12.0F, 11.0F, 10.0F, 9.0F, 8.0F, 7.0F, 6.0F, 5.0F, 4.0F, 3.0F, 2.0F, 1.0F)
);
+
+-- SPARK-58544: large magnitudes, intermediate sums of squares/products must not overflow the
+-- float range
+
+-- vector_cosine_similarity is scale invariant: identical vectors have similarity 1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F));
+
+-- vector_cosine_similarity: opposite vectors have similarity -1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(-3.0e19F, -4.0e19F));
+
+-- vector_inner_product: individual products overflow the float range but cancel out
+SELECT vector_inner_product(array(1.0e20F, 1.0e20F), array(1.0e20F, -1.0e20F));
+
+-- vector_l2_distance: sqrt((3e19)^2 + (4e19)^2) = 5e19
+SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F));
+
+-- SPARK-58544: small magnitudes, intermediate sums of squares/products must not underflow to zero
+
+-- vector_cosine_similarity of identical tiny vectors is 1.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F));
+
+-- vector_cosine_similarity of orthogonal tiny vectors is 0.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 1.0e-23F), array(1.0e-23F, -1.0e-23F));
+
+-- SPARK-58544: elements that are already infinite still propagate to NaN or Infinity; the wider
+-- accumulators do not change that
+
+-- vector_cosine_similarity: the norm is infinite, so the similarity is NaN
+SELECT vector_cosine_similarity(array(float('inf'), 1.0F), array(1.0F, 1.0F));
+
+-- vector_inner_product: the dot product is infinite
+SELECT vector_inner_product(array(float('inf'), 1.0F), array(1.0F, 1.0F));
+
+-- vector_l2_distance: the distance is infinite
+SELECT vector_l2_distance(array(float('inf'), 1.0F), array(0.0F, 0.0F));
diff --git a/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql b/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
index 13eacf854b78..cab7548a628d 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/vector-norm.sql
@@ -153,3 +153,36 @@ SELECT vector_norm(
array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, CAST(NULL AS FLOAT), 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
2.0F
);
+
+-- SPARK-58544: large magnitudes, the intermediate sum of squares must not overflow the float range
+
+-- vector_norm: sqrt((3e19)^2 + (4e19)^2) = 5e19
+SELECT vector_norm(array(3.0e19F, 4.0e19F), 2.0F);
+
+-- vector_normalize: [3e19, 4e19] normalizes to [0.6, 0.8]
+SELECT vector_normalize(array(3.0e19F, 4.0e19F), 2.0F);
+
+-- vector_norm: the L1 norm itself is not representable as a float, so it stays infinite
+SELECT vector_norm(array(3.0e38F, 3.0e38F), 1.0F);
+
+-- vector_normalize: normalization still succeeds when the norm exceeds the float range
+SELECT vector_normalize(array(3.0e38F, 3.0e38F), 1.0F);
+
+-- SPARK-58544: small magnitudes, the intermediate sum of squares must not underflow to zero
+
+-- vector_norm: the L2 norm of a tiny vector is not zero
+SELECT vector_norm(array(1.0e-23F, 0.0F), 2.0F);
+
+-- vector_normalize: normalization is scale invariant, so the result is a unit vector, not NULL
+SELECT vector_normalize(array(1.0e-23F, 0.0F), 2.0F);
+
+SELECT vector_normalize(array(1.0e-23F, 1.0e-23F), 2.0F);
+
+-- SPARK-58544: elements that are already infinite still produce an infinite norm; the wider
+-- accumulators do not change that
+
+-- vector_norm: an infinite element makes the L2 norm infinite
+SELECT vector_norm(array(float('inf'), 1.0F), 2.0F);
+
+-- vector_normalize: dividing by an infinite norm yields NaN for the infinite element
+SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F);
diff --git a/sql/core/src/test/resources/sql-tests/results/vector-distance.sql.out b/sql/core/src/test/resources/sql-tests/results/vector-distance.sql.out
index ccc39c23fb53..c75be8761561 100644
--- a/sql/core/src/test/resources/sql-tests/results/vector-distance.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/vector-distance.sql.out
@@ -4,7 +4,7 @@ SELECT vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F)
-- !query schema
struct<vector_cosine_similarity(array(1.0, 2.0, 3.0), array(4.0, 5.0, 6.0)):float>
-- !query output
-0.9746319
+0.97463185
-- !query
@@ -744,3 +744,75 @@ SELECT vector_l2_distance(
struct<vector_l2_distance(array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), array(16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0)):float>
-- !query output
36.878178
+
+
+-- !query
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F))
+-- !query schema
+struct<vector_cosine_similarity(array(3.0E19, 4.0E19), array(3.0E19, 4.0E19)):float>
+-- !query output
+1.0
+
+
+-- !query
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(-3.0e19F, -4.0e19F))
+-- !query schema
+struct<vector_cosine_similarity(array(3.0E19, 4.0E19), array(-3.0E19, -4.0E19)):float>
+-- !query output
+-1.0
+
+
+-- !query
+SELECT vector_inner_product(array(1.0e20F, 1.0e20F), array(1.0e20F, -1.0e20F))
+-- !query schema
+struct<vector_inner_product(array(1.0E20, 1.0E20), array(1.0E20, -1.0E20)):float>
+-- !query output
+0.0
+
+
+-- !query
+SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))
+-- !query schema
+struct<vector_l2_distance(array(3.0E19, 4.0E19), array(0.0, 0.0)):float>
+-- !query output
+5.0E19
+
+
+-- !query
+SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F))
+-- !query schema
+struct<vector_cosine_similarity(array(1.0E-23, 0.0), array(1.0E-23, 0.0)):float>
+-- !query output
+1.0
+
+
+-- !query
+SELECT vector_cosine_similarity(array(1.0e-23F, 1.0e-23F), array(1.0e-23F, -1.0e-23F))
+-- !query schema
+struct<vector_cosine_similarity(array(1.0E-23, 1.0E-23), array(1.0E-23, -1.0E-23)):float>
+-- !query output
+0.0
+
+
+-- !query
+SELECT vector_cosine_similarity(array(float('inf'), 1.0F), array(1.0F, 1.0F))
+-- !query schema
+struct<vector_cosine_similarity(array(inf, 1.0), array(1.0, 1.0)):float>
+-- !query output
+NaN
+
+
+-- !query
+SELECT vector_inner_product(array(float('inf'), 1.0F), array(1.0F, 1.0F))
+-- !query schema
+struct<vector_inner_product(array(inf, 1.0), array(1.0, 1.0)):float>
+-- !query output
+Infinity
+
+
+-- !query
+SELECT vector_l2_distance(array(float('inf'), 1.0F), array(0.0F, 0.0F))
+-- !query schema
+struct<vector_l2_distance(array(inf, 1.0), array(0.0, 0.0)):float>
+-- !query output
+Infinity
diff --git a/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out b/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
index 8ebd583aa7e4..31a2aa2fb08e 100644
--- a/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/vector-norm.sql.out
@@ -668,7 +668,7 @@ SELECT vector_normalize(
-- !query schema
struct<vector_normalize(array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), 2.0):array<float>>
-- !query output
-[0.025854385,0.05170877,0.07756316,0.10341754,0.12927192,0.15512632,0.1809807,0.20683508,0.23268947,0.25854385,0.28439823,0.31025264,0.33610702,0.3619614,0.38781577,0.41367015]
+[0.025854385,0.05170877,0.07756315,0.10341754,0.12927192,0.1551263,0.1809807,0.20683508,0.23268946,0.25854385,0.28439823,0.3102526,0.336107,0.3619614,0.38781577,0.41367015]
-- !query
@@ -680,3 +680,75 @@ SELECT vector_norm(
struct<vector_norm(array(1.0, 2.0, 3.0, 4.0, 5.0, CAST(NULL AS FLOAT), 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0), 2.0):float>
-- !query output
NULL
+
+
+-- !query
+SELECT vector_norm(array(3.0e19F, 4.0e19F), 2.0F)
+-- !query schema
+struct<vector_norm(array(3.0E19, 4.0E19), 2.0):float>
+-- !query output
+5.0E19
+
+
+-- !query
+SELECT vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)
+-- !query schema
+struct<vector_normalize(array(3.0E19, 4.0E19), 2.0):array<float>>
+-- !query output
+[0.6,0.8]
+
+
+-- !query
+SELECT vector_norm(array(3.0e38F, 3.0e38F), 1.0F)
+-- !query schema
+struct<vector_norm(array(3.0E38, 3.0E38), 1.0):float>
+-- !query output
+Infinity
+
+
+-- !query
+SELECT vector_normalize(array(3.0e38F, 3.0e38F), 1.0F)
+-- !query schema
+struct<vector_normalize(array(3.0E38, 3.0E38), 1.0):array<float>>
+-- !query output
+[0.5,0.5]
+
+
+-- !query
+SELECT vector_norm(array(1.0e-23F, 0.0F), 2.0F)
+-- !query schema
+struct<vector_norm(array(1.0E-23, 0.0), 2.0):float>
+-- !query output
+1.0E-23
+
+
+-- !query
+SELECT vector_normalize(array(1.0e-23F, 0.0F), 2.0F)
+-- !query schema
+struct<vector_normalize(array(1.0E-23, 0.0), 2.0):array<float>>
+-- !query output
+[1.0,0.0]
+
+
+-- !query
+SELECT vector_normalize(array(1.0e-23F, 1.0e-23F), 2.0F)
+-- !query schema
+struct<vector_normalize(array(1.0E-23, 1.0E-23), 2.0):array<float>>
+-- !query output
+[0.70710677,0.70710677]
+
+
+-- !query
+SELECT vector_norm(array(float('inf'), 1.0F), 2.0F)
+-- !query schema
+struct<vector_norm(array(inf, 1.0), 2.0):float>
+-- !query output
+Infinity
+
+
+-- !query
+SELECT vector_normalize(array(float('inf'), 1.0F), 2.0F)
+-- !query schema
+struct<vector_normalize(array(inf, 1.0), 2.0):array<float>>
+-- !query output
+[NaN,0.0]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org