(spark) branch branch-4.3 updated: [SPARK-58168][SQL] Do no use Array[Nothing] in Flatten

[ur...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new b037b8cd7c2c [SPARK-58168][SQL] Do no use Array[Nothing] in Flatten
b037b8cd7c2c is described below

commit b037b8cd7c2c517c4f215532a03cbc0029d15fcc
Author: Emil Ejbyfeldt <em...@choreograph.com>
AuthorDate: Sun Aug 23 13:40:00 2026 +0000

    [SPARK-58168][SQL] Do no use Array[Nothing] in Flatten
    
    ### What changes were proposed in this pull request?
    
    ### Why are the changes needed?
    On Scala3 the erasure of `Array[Nothing]` is no longer `[]Object` but `[]scala.runtime.Nothing$` which causes class cast errors like
    ```
    arraycopy: element type mismatch: can not cast one of the elements of java.lang.Object[] to the type of the destination array, scala.runtime.Nothing$
    ```
    when used with a standard library built with Scala3. This is also related https://github.com/scala/scala3/issues/25896 and making Spark usable on Scala 3.8.0 and later.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Existing tests ensure no regressions. That it makes the code more compatible with Scala 3.8 was done by running a local build against the tests in https://github.com/unum-io/tyda (a Scala 3 library that can use Spark as a backend).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #57299 from eejbyfeldt/SPARK-58168.
    
    Authored-by: Emil Ejbyfeldt <em...@choreograph.com>
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
    (cherry picked from commit 923e31286927328497859a909ab965d0920e2622)
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
 .../apache/spark/sql/catalyst/expressions/collectionOperations.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 941772ae157e..492044ff4e38 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -3338,7 +3338,7 @@ case class Flatten(child: Expression) extends UnaryExpression
         throw QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError(
           prettyName, numberOfElements)
       }
-      val flattenedData = new Array(numberOfElements.toInt)
+      val flattenedData = new Array[Any](numberOfElements.toInt)
       var position = 0
       for (ad <- arrayData) {
         val arr = ad.toObjectArray(elementType)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org