You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2017/11/19 05:48:00 UTC
[jira] [Commented] (SPARK-16998) select($"column1",
explode($"column2")) is extremely slow
[ https://issues.apache.org/jira/browse/SPARK-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258340#comment-16258340 ]
Ruslan Dautkhanov commented on SPARK-16998:
-------------------------------------------
Can somebody please help review PR 19683 in SPARK-21657?
As there is still a lot of room for improvement possible in explode code generation for array of structs..
> select($"column1", explode($"column2")) is extremely slow
> ---------------------------------------------------------
>
> Key: SPARK-16998
> URL: https://issues.apache.org/jira/browse/SPARK-16998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: TobiasP
> Assignee: Herman van Hovell
> Fix For: 2.2.0
>
>
> Using a Dataset containing 10.000 rows, each containing null and an array of 5.000 Ints, I observe the following performance (in local mode):
> {noformat}
> scala> time(ds.select(explode($"value")).sample(false, 0.0000001, 1).collect)
> 1.219052 seconds
> res9: Array[org.apache.spark.sql.Row] = Array([3761], [3766], [3196])
> scala> time(ds.select($"dummy", explode($"value")).sample(false, 0.0000001, 1).collect)
> 20.219447 seconds
> res5: Array[org.apache.spark.sql.Row] = Array([null,3761], [null,3766], [null,3196])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org