[jira] [Created] (SPARK-59516) Infinity norm of an all-zero SparseVector raises ValueError with NumPy before 2.3

["Akshay Thorat (Jira)" <ji...@apache.org>]

Akshay Thorat created SPARK-59516:
-------------------------------------

             Summary: Infinity norm of an all-zero SparseVector raises ValueError with NumPy before 2.3
                 Key: SPARK-59516
                 URL: https://issues.apache.org/jira/browse/SPARK-59516
             Project: Spark
          Issue Type: Bug
          Components: MLlib, PySpark
    Affects Versions: 5.0.0
            Reporter: Akshay Thorat


h2. Problem

With NumPy versions before 2.3, SparseVector.norm passes an empty values array to numpy.linalg.norm, which raises on the maximum reduction. Dense vectors and sparse vectors storing an explicit zero return 0.0 for the same logical vector. Both pyspark.ml.linalg and pyspark.mllib.linalg are affected.

h2. Reproduction

Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python 3.10.11 and NumPy 2.2.6. No SparkSession is required.

{code:python}
from pyspark.ml.linalg import SparseVector
SparseVector(3, [], []).norm(float("inf"))
# ValueError: zero-size array to reduction operation maximum which has no identity
{code}

h2. Expected behavior

The infinity norm should be 0.0, as it is for DenseVector([0.0, 0.0, 0.0]). Supported NumPy versions should give the same result for positive-dimensional zero vectors.

h2. Proposed fix and verification

Return zero for infinity norm when a positive-dimensional sparse vector has no stored entries, in both linalg APIs. Preserve NumPy behavior for zero-dimensional vectors and invalid norm orders. Regression tests cover both APIs and implicit/explicit zeros.

The regression fails with NumPy 2.2.6 before the fix and passes afterward. NumPy 2.3 independently changed the empty-array norm to return zero; the Spark fix also supports older NumPy releases. See https://numpy.org/doc/2.3/release/2.3.0-notes.html#changes

Pull request: https://github.com/apache/spark/pull/58788



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org