You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "TobiasP (JIRA)" <ji...@apache.org> on 2016/08/10 16:26:20 UTC

[jira] [Created] (SPARK-16998) select($"column1", explode($"column2")) is extremely slow

TobiasP created SPARK-16998:
-------------------------------

             Summary: select($"column1", explode($"column2")) is extremely slow
                 Key: SPARK-16998
                 URL: https://issues.apache.org/jira/browse/SPARK-16998
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: TobiasP


Using a Dataset containing 10.000 rows, each containing null and an array of 5.000 Ints, I observe the following performance (in local mode):
{noformat}
scala> time(ds.select(explode($"value")).sample(false, 0.0000001, 1).collect)
1.219052 seconds                                                                
res9: Array[org.apache.spark.sql.Row] = Array([3761], [3766], [3196])

scala> time(ds.select($"dummy", explode($"value")).sample(false, 0.0000001, 1).collect)
20.219447 seconds                                                               
res5: Array[org.apache.spark.sql.Row] = Array([null,3761], [null,3766], [null,3196])
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org