You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jia Fan (Jira)" <ji...@apache.org> on 2023/05/07 02:48:00 UTC
[jira] [Resolved] (SPARK-43392) Sequence expression can overflow
[ https://issues.apache.org/jira/browse/SPARK-43392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jia Fan resolved SPARK-43392.
-----------------------------
Resolution: Duplicate
Duplicate with SPARK-43393
> Sequence expression can overflow
> --------------------------------
>
> Key: SPARK-43392
> URL: https://issues.apache.org/jira/browse/SPARK-43392
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Deepayan Patra
> Priority: Major
>
> Spark has a (long-standing) overflow bug in the {{sequence}} expression.
>
> Consider the following operations:
> {{spark.sql("CREATE TABLE foo (l LONG);")}}
> {{spark.sql(s"INSERT INTO foo VALUES (${Long.MaxValue});")}}
> {{spark.sql("SELECT sequence(0, l) FROM foo;").collect()}}
>
> The result of these operations will be:
> {{Array[org.apache.spark.sql.Row] = Array([WrappedArray()])}}
> an unintended consequence of overflow.
>
> The sequence is applied to values {{0}} and {{Long.MaxValue}} with a step size of {{1}} which uses a length computation defined [here|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3451]. In this calculation, with {{{}start = 0{}}}, {{{}stop = Long.MaxValue{}}}, and {{{}step = 1{}}}, the calculated {{len}} overflows to {{{}Long.MinValue{}}}. The computation, in binary looks like:
> {{ 0111111111111111111111111111111111111111111111111111111111111111}}
> {{- 0000000000000000000000000000000000000000000000000000000000000000 }}
> {{------------------------------------------------------------------ 0111111111111111111111111111111111111111111111111111111111111111}}
> {{/ 0000000000000000000000000000000000000000000000000000000000000001}}
> {{------------------------------------------------------------------ 0111111111111111111111111111111111111111111111111111111111111111}}
> {{+ 0000000000000000000000000000000000000000000000000000000000000001}}
> {{------------------------------------------------------------------ 1000000000000000000000000000000000000000000000000000000000000000}}
> {{}}
> The following [check|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3454] passes as the negative {{Long.MinValue}} is still {{{}<= MAX_ROUNDED_ARRAY_LENGTH{}}}. The following cast to {{toInt}} uses this representation and [truncates the upper bits|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3457] resulting in an empty length of 0.{{{}{}}}
> Other overflows are similarly problematic.{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org