[jira] [Resolved] (SPARK-58897) vector_norm with degree infinity silently ignores NaN elements

["Uroš Bojanić (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uroš Bojanić resolved SPARK-58897.
----------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 58149
[https://github.com/apache/spark/pull/58149]

> vector_norm with degree infinity silently ignores NaN elements
> --------------------------------------------------------------
>
>                 Key: SPARK-58897
>                 URL: https://issues.apache.org/jira/browse/SPARK-58897
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: sepuri sai krishna
>            Assignee: sepuri sai krishna
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> h3. Summary
> The infinity-norm branch of {{vector_norm}} silently drops {{NaN}} elements instead of
> propagating them. Every other degree of {{vector_norm}} propagates {{NaN}}, and every other
> max-like function in Spark treats {{NaN}} as the largest value, so the infinity norm is the
> lone outlier.
> The single-element case is the damaging one: a vector consisting of {{NaN}} is reported as
> having infinity norm {{0.0}}, making it indistinguishable from the zero vector. That in turn
> makes {{vector_normalize}} return {{NULL}} (its zero-norm path) rather than a vector of
> {{NaN}}.
> h3. Reproduction
> {code:sql}
> -- Wrong: NaN is ignored
> SELECT vector_norm(array(float('nan')), float('inf'));                 -- 0.0        expected NaN
> SELECT vector_norm(array(float('nan'), 5.0F), float('inf'));           -- 5.0        expected NaN
> SELECT vector_norm(array(5.0F, float('nan')), float('inf'));           -- 5.0        expected NaN
> SELECT vector_norm(array(float('nan'), float('inf')), float('inf'));   -- Infinity   expected NaN
> -- Correct: every other degree propagates NaN
> SELECT vector_norm(array(float('nan')), 1.0F);                         -- NaN
> SELECT vector_norm(array(float('nan')), 2.0F);                         -- NaN
> SELECT vector_norm(array(float('nan'), 5.0F), 2.0F);                   -- NaN
> -- Knock-on effect in vector_normalize
> SELECT vector_normalize(array(float('nan')), float('inf'));            -- NULL       expected [NaN]
> SELECT vector_normalize(array(float('nan')), 2.0F);                    -- [NaN]
> {code}
> h3. Why this is a bug and not a design choice
> Spark's established convention is that {{NaN}} compares as larger than any other value. All of
> the following agree, and disagree with the infinity norm:
> {code:sql}
> SELECT greatest(float('nan'), 5.0F);                          -- NaN
> SELECT array_max(array(float('nan'), 5.0F));                  -- NaN
> SELECT max(v) FROM values (float('nan')), (5.0F) AS t(v);     -- NaN
> SELECT sort_array(array(float('nan'), 5.0F));                 -- [5.0, NaN]
> {code}
> Since the infinity norm is defined as the maximum absolute value, it should follow the same
> convention that {{max}} and {{array_max}} do.
> h3. Root cause
> {{VectorFunctionImplUtils.vectorInfNorm}}
> (sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java)
> hand-rolls the maximum with a {{>}} comparison seeded at zero:
> {code:java}
> float maxAbs = 0.0f;
> for (int i = 0; i < len; i++) {
>   if (vec.isNullAt(i)) {
>     return null;
>   }
>   float absVal = Math.abs(vec.getFloat(i));
>   if (absVal > maxAbs) {
>     maxAbs = absVal;
>   }
> }
> return (double) maxAbs;
> {code}
> Any comparison involving {{NaN}} is false under IEEE 754, so {{maxAbs}} is never updated to
> {{NaN}} and the {{NaN}} never reaches the result. {{Math.max}}, which returns {{NaN}} when
> either argument is {{NaN}}, gives the expected behaviour.
> h3. Scope
> This is distinct from SPARK-58544, which fixed float overflow in the intermediate sums of the
> same file. That issue explicitly scoped its "correct as-is" cases to empty vectors, NULL
> elements, and genuinely zero magnitude; {{NaN}} input was not covered. The existing behaviour
> for empty vectors ({{0.0}}) and for vectors containing NULL ({{NULL}}) is unchanged by this
> report.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org