[jira] [Created] (SPARK-14330) Spark SQL does not infer correct type for joda DateTime

["Michel Lemay (JIRA)" <ji...@apache.org>]

Michel Lemay created SPARK-14330:
------------------------------------

             Summary: Spark SQL does not infer correct type for joda DateTime
                 Key: SPARK-14330
                 URL: https://issues.apache.org/jira/browse/SPARK-14330
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.6.0
            Reporter: Michel Lemay
            Priority: Minor


Creating a Dataframe on a pojo with joda DateTime:

{code}
val df = sqlContext.createDataFrame(rdd, classOf[MyJavaClassWithJodaDateTime])
df.printSchema
{code}

Digging, I find that JavaTypeInference yields unexpected results:

{code}
val (dataType, _) = org.apache.spark.sql.catalyst.JavaTypeInference.inferDataType(classOf[org.joda.time.DateTime])
dataType.asInstanceOf[org.apache.spark.sql.types.StructType].fields.foreach(println)

StructField(afterNow,BooleanType,false)
StructField(beforeNow,BooleanType,false)
StructField(centuryOfEra,IntegerType,false)
StructField(chronology,StructType(StructField(zone,StructType(StructField(ID,StringType,true), StructField(fixed,BooleanType,false)),true)),t
StructField(dayOfMonth,IntegerType,false)
StructField(dayOfWeek,IntegerType,false)
StructField(dayOfYear,IntegerType,false)
StructField(equalNow,BooleanType,false)
StructField(era,IntegerType,false)
StructField(hourOfDay,IntegerType,false)
StructField(millis,LongType,false)
StructField(millisOfDay,IntegerType,false)
StructField(millisOfSecond,IntegerType,false)
StructField(minuteOfDay,IntegerType,false)
StructField(minuteOfHour,IntegerType,false)
StructField(monthOfYear,IntegerType,false)
StructField(secondOfDay,IntegerType,false)
StructField(secondOfMinute,IntegerType,false)
StructField(weekOfWeekyear,IntegerType,false)
StructField(weekyear,IntegerType,false)
StructField(year,IntegerType,false)
StructField(yearOfCentury,IntegerType,false)
StructField(yearOfEra,IntegerType,false)
StructField(zone,StructType(StructField(ID,StringType,true), StructField(fixed,BooleanType,false)),true)
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org