You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "bing huang (JIRA)" <ji...@apache.org> on 2017/05/25 06:32:04 UTC
[jira] [Updated] (SPARK-20879) Spark SQL will read Date type column
from avro file as Int
[ https://issues.apache.org/jira/browse/SPARK-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
bing huang updated SPARK-20879:
-------------------------------
Attachment: 000000_0.avro
avro file with 5 columns
> Spark SQL will read Date type column from avro file as Int
> ----------------------------------------------------------
>
> Key: SPARK-20879
> URL: https://issues.apache.org/jira/browse/SPARK-20879
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0, 1.6.1, 1.6.2, 1.6.3, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1
> Reporter: bing huang
> Attachments: 000000_0.avro
>
>
> I was using the following code and attached avro file.
> In the avro file we have 5 columns named (date1, date2, date3, start_date,end_date) with type (string, string, date, string, string) respectively.
> You can use hive to verify the type of columns either by "select * from 000000_0.avro" or "describe 000000_0.avro".
> The code of spark sql as below, as you can see the 3rd column's result is int.
> val conf = new SparkConf().setAppName("test").setMaster("local[3]")
> val sc = new SparkContext(conf)
> val sqlContext = new SQLContext(sc)
> val df = sqlContext.read.avro("000000_0.avro")
> df.show(false)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org