[jira] [Commented] (SPARK-58959) Duplicate PIVOT values corrupt the aggregation buffer on the PivotFirst fast path

["Eric Yang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-58959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107192#comment-18107192 ] 

Eric Yang commented on SPARK-58959:
-----------------------------------

I'm working on the fix.

> Duplicate PIVOT values corrupt the aggregation buffer on the PivotFirst fast path
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-58959
>                 URL: https://issues.apache.org/jira/browse/SPARK-58959
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Eric Yang
>            Priority: Major
>
> When a PIVOT ... IN list contains two values that compare equal, the optimized
> PivotFirst fast path writes outside the slots it owns in the aggregation buffer.
> Repro:
>  
>  
> {code:java}
> CREATE OR REPLACE TEMP VIEW t AS SELECT * FROM VALUES (1, 1, 10), (1, 2, 20) AS t(id, k, v);
> SELECT * FROM t PIVOT (sum(v) FOR k IN (1 AS x, 1 AS y));
> java.lang.AssertionError: index (1) should < 1
> at org.apache.spark.sql.catalyst.expressions.UnsafeRow.assertIndexIsValid(UnsafeRow.java:126)
> at org.apache.spark.sql.catalyst.expressions.UnsafeRow.setLong(UnsafeRow.java:224)
> at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.update(PivotFirst.scala)
>  
> {code}
>  
> The standard (non-PivotFirst) path handles the same query correctly, which gives
> the expected answer, one output column per listed pivot value:
>  
> {code:java}
> SELECT * FROM t PIVOT (collect_list(v) FOR k IN (1 AS x, 1 AS y));
> --> [1, [10], [10]]{code}
>  
> Any pair of pivot values that the lookup treats as equal triggers it, e.g.
> IN (1 AS x, 1 AS y), IN ('a' AS x, 'a' AS y), IN (0.0D, -0.0D), and -- for
> collated string columns -- IN ('a', 'A') under UTF8_LCASE.
>  unit tests run with -ea so this surfaces as an AssertionError, but in
> production assertIndexIsValid is a no-op, making this an unchecked out-of-bounds
> write into the Tungsten aggregation buffer page rather than a clean failure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org