You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/08/04 07:21:05 UTC

[PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

SEPURI-SAI-KRISHNA opened a new pull request, #57746:
URL: https://github.com/apache/spark/pull/57746

   ### What changes were proposed in this pull request?
   
   `VectorFunctionImplUtils` accumulates dot products and sums of squares in single precision. Those
   intermediate quantities are quadratic in the input values, so they overflow to infinity (or
   underflow to zero) long before the final result leaves the float range, and the function returns
   `NaN`, `Infinity`, `NULL` or an all-zero vector for inputs whose true result is perfectly
   representable as a float.
   
   This PR accumulates the intermediate dot products, sums of squares and absolute sums in `double`
   and rounds only the final result back to `float`. `vector_normalize` also keeps the norm in
   `double` instead of rounding it to a float before dividing, so a vector whose norm is outside the
   float range is still normalized correctly.
   
   Affected functions: `vector_cosine_similarity`, `vector_inner_product`, `vector_l2_distance`,
   `vector_norm`, `vector_normalize`.
   
   ### Why are the changes needed?
   
   Cosine similarity and normalization are scale invariant, so rescaling the input must not change
   the result -- but today it does:
   
   | Query | Before | After (correct) |
   | --- | --- | --- |
   | `vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F))` | `NaN` | `1.0` |
   | `vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F))` | `NULL` | `1.0` |
   | `vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))` | `Infinity` | `5.0E19` |
   | `vector_norm(array(3.0e19F, 4.0e19F), 2.0F)` | `Infinity` | `5.0E19` |
   | `vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)` | `[0.0, 0.0]` | `[0.6, 0.8]` |
   | `vector_normalize(array(1.0e-23F, 0.0F), 2.0F)` | `NULL` | `[1.0, 0.0]` |
   
   In every row above the correct result is an ordinary float; only the intermediate sum overflows
   (`(3e19)^2 + (4e19)^2 = 2.5e39 > Float.MAX_VALUE`) or underflows (`(1e-23)^2 = 1e-46 < the
   smallest positive float`). The wrong values are returned silently, so they propagate into
   similarity search results and rankings rather than failing loudly.
   
   Double-precision accumulation is also what other vector similarity implementations do
   (for example pgvector), and it removes a rounding error in the common case as well: the existing
   `vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F))` result changes from
   `0.9746319` to `0.97463185`, which is the correctly rounded float of `32 / sqrt(1078)`
   (`0.9746318461970763...`).
   
   These functions were added in 4.2.0 by SPARK-54713 and SPARK-55030, so the wrong results are
   present in the 4.2.0 release.
   
   Documented behaviours are unchanged: empty vectors, `NULL` inputs, vectors containing `NULL`
   elements, and genuinely zero-magnitude vectors return exactly what they returned before.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. The functions listed above now return correct results instead of `NaN`, `Infinity`, `NULL`
   or zeros when an intermediate sum overflows or underflows the float range, as shown in the table
   above. This is a change relative to 4.2.0, where these functions were introduced.
   
   `vector_cosine_similarity` also changes by one ULP for ordinary inputs, because the result is now
   correctly rounded (`0.9746319` -> `0.97463185` for the example above).
   
   ### How was this patch tested?
   
   - New golden test cases in `sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql` and
     `vector-norm.sql` covering large and small magnitude vectors for all five functions; golden
     files regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`.
   - Existing suites: `SQLQueryTestSuite` (vector files), `MiscFunctionsSuite`, `ExpressionInfoSuite`
     (the `vector_cosine_similarity` example in `@ExpressionDescription` is updated to the correctly
     rounded value).
   
   On performance: the unrolled loops read through `ArrayData.getFloat()` / `isNullAt()`, and those
   accesses dominate the loop, so widening the accumulators is not measurable end to end. Timing the
   `vector_cosine_similarity` loop over 1536-dimensional `UnsafeArrayData` inputs (200k calls, JIT
   warmed up) gives 5834 ns/call before and 5891 ns/call after in one run, and 5870 vs 5792 ns/call in
   the next -- i.e. within run-to-run noise. The arithmetic in isolation (plain `float[]`, no
   `ArrayData` indirection) is ~1.3x slower, but it accounts for only ~20% of the loop cost.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Opus 5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57746:
URL: https://github.com/apache/spark/pull/57746#discussion_r3737581383


##########
sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql:
##########
@@ -130,3 +130,38 @@ SELECT vector_l2_distance(
     array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, 6.0F, 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
     array(16.0F, 15.0F, 14.0F, 13.0F, 12.0F, 11.0F, 10.0F, 9.0F, 8.0F, 7.0F, 6.0F, 5.0F, 4.0F, 3.0F, 2.0F, 1.0F)
 );
+
+-- SPARK-58544: large magnitudes, intermediate sums of squares/products must not overflow the
+-- float range
+
+-- vector_cosine_similarity is scale invariant: identical vectors have similarity 1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F));
+
+-- vector_cosine_similarity: opposite vectors have similarity -1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(-3.0e19F, -4.0e19F));
+
+-- vector_inner_product: individual products overflow the float range but cancel out
+SELECT vector_inner_product(array(1.0e20F, 1.0e20F), array(1.0e20F, -1.0e20F));
+
+-- vector_l2_distance: sqrt((3e19)^2 + (4e19)^2) = 5e19
+SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F));
+
+-- SPARK-58544: small magnitudes, intermediate sums of squares/products must not underflow to zero
+
+-- vector_cosine_similarity of identical tiny vectors is 1.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F));
+
+-- vector_cosine_similarity of orthogonal tiny vectors is 0.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 1.0e-23F), array(1.0e-23F, -1.0e-23F));
+
+-- SPARK-58544: elements that are already infinite still propagate to NaN or Infinity, the wider

Review Comment:
   Please replace this comma splice with a semicolon. The parallel comment in `vector-norm.sql:181` needs the same edit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57746:
URL: https://github.com/apache/spark/pull/57746#issuecomment-5179456200

   Thank you @SEPURI-SAI-KRISHNA! Adding @MaxGekk to also PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #57746:
URL: https://github.com/apache/spark/pull/57746#discussion_r3730126957


##########
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:
##########
@@ -448,6 +460,21 @@ public static Float vectorNorm(ArrayData vec, float degree, UTF8String funcName)
     }
   }
 
+  /**
+   * Computes the Lp norm of a float vector using the specified degree.
+   * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (L∞).

Review Comment:
   Applied. I also changed the identical line in the `vectorNormalize` javadoc (line 480) so the two match; both were pre-existing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #57746:
URL: https://github.com/apache/spark/pull/57746#discussion_r3730137971


##########
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:
##########
@@ -90,19 +93,22 @@ public static Float vectorCosineSimilarity(ArrayData left, ArrayData right, UTF8
       if (left.isNullAt(i) || right.isNullAt(i)) {
         return null;
       }
-      float a = left.getFloat(i);
-      float b = right.getFloat(i);
+      double a = left.getFloat(i);
+      double b = right.getFloat(i);
       dotProduct += a * b;
       norm1Sq += a * a;
       norm2Sq += b * b;
       i++;
     }
 
-    float normProduct = (float) Math.sqrt(norm1Sq * norm2Sq);
-    if (normProduct < Float.MIN_NORMAL) {
+    // `norm1Sq * norm2Sq` cannot overflow in double precision: both factors are bounded by

Review Comment:
   Good catch, the bound as written was unconditional. Reworded to scope it to finite elements and to
   say what happens otherwise:
   
   ```java
   // For vectors of finite elements, `norm1Sq * norm2Sq` cannot overflow in double precision:
   // both factors are bounded by MAX_ROUNDED_ARRAY_LENGTH * Float.MAX_VALUE^2, so their product
   // stays well below Double.MAX_VALUE. An element that is already infinite makes the product
   // infinite and the result NaN, exactly as it did before the accumulators were widened.
   ```
   
   An infinite element cannot produce a finite dot product either -- it is multiplied by either a
   non-zero element (giving +/-Infinity) or by zero (giving NaN) -- so `dotProduct / normProduct`
   is NaN in every such case, both before and after this change.
   
   I also added golden-file coverage for infinite elements so the behaviour is pinned rather than just
   asserted in a comment:
   
   | query | result |
   | --- | --- |
   | `vector_cosine_similarity(array(float('inf'), 1.0F), array(1.0F, 1.0F))` | `NaN` |
   | `vector_inner_product(array(float('inf'), 1.0F), array(1.0F, 1.0F))` | `Infinity` |
   | `vector_l2_distance(array(float('inf'), 1.0F), array(0.0F, 0.0F))` | `Infinity` |
   | `vector_norm(array(float('inf'), 1.0F), 2.0F)` | `Infinity` |
   | `vector_normalize(array(float('inf'), 1.0F), 2.0F)` | `[NaN,0.0]` |
   
   All five are unchanged from before this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57746:
URL: https://github.com/apache/spark/pull/57746#discussion_r3728227504


##########
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:
##########
@@ -448,6 +460,21 @@ public static Float vectorNorm(ArrayData vec, float degree, UTF8String funcName)
     }
   }
 
+  /**
+   * Computes the Lp norm of a float vector using the specified degree.
+   * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (L∞).

Review Comment:
   Spark's repository guidance requires ASCII in code comments.
   
   ```suggestion
      * Supported degrees: 1.0 (L1), 2.0 (L2), Float.POSITIVE_INFINITY (infinity norm).
   ```



##########
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:
##########
@@ -90,19 +93,22 @@ public static Float vectorCosineSimilarity(ArrayData left, ArrayData right, UTF8
       if (left.isNullAt(i) || right.isNullAt(i)) {
         return null;
       }
-      float a = left.getFloat(i);
-      float b = right.getFloat(i);
+      double a = left.getFloat(i);
+      double b = right.getFloat(i);
       dotProduct += a * b;
       norm1Sq += a * a;
       norm2Sq += b * b;
       i++;
     }
 
-    float normProduct = (float) Math.sqrt(norm1Sq * norm2Sq);
-    if (normProduct < Float.MIN_NORMAL) {
+    // `norm1Sq * norm2Sq` cannot overflow in double precision: both factors are bounded by

Review Comment:
   Please narrow this bound to finite float elements. `ARRAY<FLOAT>` also admits infinities, which make an accumulated squared norm infinite, so the factors are not unconditionally bounded.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57746:
URL: https://github.com/apache/spark/pull/57746#issuecomment-5236455174

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/d28c23f19f6d03e4cc472a507b0911bfd93829a9
   - merged into branch-4.x https://github.com/apache/spark/commit/3f6f2cf785ba0aa084137a300a6f4d7d76aa5a85
   - merged into branch-4.3 https://github.com/apache/spark/commit/da54921d8bb82471df4d8685c23455418a6fac00
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57746: [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow
URL: https://github.com/apache/spark/pull/57746


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow [spark]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #57746:
URL: https://github.com/apache/spark/pull/57746#discussion_r3737641505


##########
sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql:
##########
@@ -130,3 +130,38 @@ SELECT vector_l2_distance(
     array(1.0F, 2.0F, 3.0F, 4.0F, 5.0F, 6.0F, 7.0F, 8.0F, 9.0F, 10.0F, 11.0F, 12.0F, 13.0F, 14.0F, 15.0F, 16.0F),
     array(16.0F, 15.0F, 14.0F, 13.0F, 12.0F, 11.0F, 10.0F, 9.0F, 8.0F, 7.0F, 6.0F, 5.0F, 4.0F, 3.0F, 2.0F, 1.0F)
 );
+
+-- SPARK-58544: large magnitudes, intermediate sums of squares/products must not overflow the
+-- float range
+
+-- vector_cosine_similarity is scale invariant: identical vectors have similarity 1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F));
+
+-- vector_cosine_similarity: opposite vectors have similarity -1.0
+SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(-3.0e19F, -4.0e19F));
+
+-- vector_inner_product: individual products overflow the float range but cancel out
+SELECT vector_inner_product(array(1.0e20F, 1.0e20F), array(1.0e20F, -1.0e20F));
+
+-- vector_l2_distance: sqrt((3e19)^2 + (4e19)^2) = 5e19
+SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F));
+
+-- SPARK-58544: small magnitudes, intermediate sums of squares/products must not underflow to zero
+
+-- vector_cosine_similarity of identical tiny vectors is 1.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F));
+
+-- vector_cosine_similarity of orthogonal tiny vectors is 0.0, not NULL
+SELECT vector_cosine_similarity(array(1.0e-23F, 1.0e-23F), array(1.0e-23F, -1.0e-23F));
+
+-- SPARK-58544: elements that are already infinite still propagate to NaN or Infinity, the wider

Review Comment:
   Applied, Thanks for the catch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org