[jira] [Commented] (SPARK-29265) Window+collect_list causing single-task operation

["Florentino Sainz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-29265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939423#comment-16939423 ] 

Florentino Sainz commented on SPARK-29265:
------------------------------------------

Ok just realized what's happening, we did have one element with MANY rows inside the same group, when using collect_list we are multiplying the list for each row. (this was not expected tho...)

I changed it to groupBy and now we only have one row with all the values :).

> Window+collect_list causing single-task operation
> -------------------------------------------------
>
>                 Key: SPARK-29265
>                 URL: https://issues.apache.org/jira/browse/SPARK-29265
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.0
>         Environment: Any
>            Reporter: Florentino Sainz
>            Priority: Minor
>
> Hi,
>  
> I had this problem in "real" environments and also made a self-contained test ( [^Test.scala] attached).
> Having this Window definition:
> {code:scala}
> val myWindow = Window.partitionBy($"word").orderBy("word") 
> val filt2 = filtrador.withColumn("avg_Time", collect_list($"number").over(myWindow))
> {code}
>   
> In the test I can see how all elements of my DF are being collected in a single task.
> Unbounded+unordered Window + collect_list seems to be collecting ALL the dataframe in a single executor/task.
> groupBy + collect_list seems to do it as expect (collect_list for each group independently).
>  
> Full Code showing the error (see how the mapPartitions shows 99 rows in one partition) attached in Test.scala, sbt project (src and build.sbt) attached too in TestSpark.zip.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org