[jira] [Updated] (SPARK-8304) Table with a large number of columns

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-8304:
-----------------------------
    Component/s: SQL

Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

I don't think this is a valid issue report since this doesn't really describe a specific problem other than "it's slow for me".

> Table with a large number of columns
> ------------------------------------
>
>                 Key: SPARK-8304
>                 URL: https://issues.apache.org/jira/browse/SPARK-8304
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.1
>            Reporter: jaeboo jung
>
> SQLContext can't handle any table with a large number of columns. Making dataframe is ok but when a user try to execute query on it, spark doesn't respond. To test, run below code from spark-shell.
> {code:java}
> import org.apache.spark.sql._ 
> import org.apache.spark.sql.types._ 
> val arr = (1 to 500000) 
> val columns = StructType(arr.map(x => StructField("columnNum_"+x , StringType, true))) 
> val data = arr.map(x => arr) 
> val rdd = sc.parallelize(data , 1000).map(Row.fromSeq(_)) 
> val df = sqlContext.createDataFrame(rdd,columns)
> //select few columns among 500,000 columns
> def select1() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select2() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select3() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2","columnNum_3") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select4() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select5() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select6() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5","columnNum_6") 
> println( System.currentTimeMillis - t1 ) 
> } 
> def select7() = { 
> val t1 = System.currentTimeMillis 
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5","columnNum_6","columnNum_7") 
> println( System.currentTimeMillis - t1 ) 
> }  
> {code}
> And the result is,
> {code}
> select1 
> 20552 
> select2 
> 25391 
> select3 
> 29619 
> select4 
> 33695 
> select5 
> 42220 
> select6 
> 44790 
> select7 
> 49101 
> {code}
> Elapsed time for selecting columns is increased about 4000ms after each addition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org