[jira] [Created] (SPARK-14632) randomSplit method fails on dataframes with maps in schema

["Stefano Costantini (JIRA)" <ji...@apache.org>]

Stefano Costantini created SPARK-14632:
------------------------------------------

             Summary: randomSplit method fails on dataframes with maps in schema
                 Key: SPARK-14632
                 URL: https://issues.apache.org/jira/browse/SPARK-14632
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.6.1
            Reporter: Stefano Costantini


Applying the randomSplit method to a dataframe with at least one map in the schema results in an exception
{noformat}
org.apache.spark.sql.AnalysisException: cannot resolve 'features ASC' due to data type mismatch: cannot sort data type map<string,double>;
{noformat}
This bug can be reproduced as follows:
{code}
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

import org.apache.spark.sql._
import org.apache.spark.sql.types._

val arr = Array(("user1", Map("f1" -> 1.0, "f2" -> 1.0)), ("user2", Map("f2" -> 1.0, "f3" -> 1.0)), ("user3",Map("f1" -> 1.0, "f2" -> 1.0)))
val df = sc.parallelize(arr).toDF("user","features")
df.printSchema

val Array(split1, split2) = df.randomSplit(Array(0.7, 0.3), seed = 101L)
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org