You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takeshi Yamamuro (JIRA)" <ji...@apache.org> on 2016/11/20 16:04:58 UTC
[jira] [Commented] (SPARK-16998) select($"column1",
explode($"column2")) is extremely slow
[ https://issues.apache.org/jira/browse/SPARK-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15681374#comment-15681374 ]
Takeshi Yamamuro commented on SPARK-16998:
------------------------------------------
[~hvanhovell] Since SPARK-15214 improves this query by ~11x, I think we can also close this ticket;
https://github.com/apache/spark/pull/13065/files#diff-b7bf86a20a79d572f81093300568db6eR152
> select($"column1", explode($"column2")) is extremely slow
> ---------------------------------------------------------
>
> Key: SPARK-16998
> URL: https://issues.apache.org/jira/browse/SPARK-16998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: TobiasP
>
> Using a Dataset containing 10.000 rows, each containing null and an array of 5.000 Ints, I observe the following performance (in local mode):
> {noformat}
> scala> time(ds.select(explode($"value")).sample(false, 0.0000001, 1).collect)
> 1.219052 seconds
> res9: Array[org.apache.spark.sql.Row] = Array([3761], [3766], [3196])
> scala> time(ds.select($"dummy", explode($"value")).sample(false, 0.0000001, 1).collect)
> 20.219447 seconds
> res5: Array[org.apache.spark.sql.Row] = Array([null,3761], [null,3766], [null,3196])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org