You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2016/10/26 17:16:07 UTC
spark infers date to be timestamp type
is there a reason a column with dates in format yyyy-mm-dd in a csv file is
inferred to be TimestampType and not DateType?
thanks! koert
Re: spark infers date to be timestamp type
Posted by Steve Loughran <st...@hortonworks.com>.
CSV type inference isn't really ideal: it does a full scan of a file to determine this; you are doubling the amount of data you need to read. Unless you are just exploring files in your notebook, I'd recommend doing it once, getting the schema from it then using that as the basis for the code snippet where you really define the schema. That's when you can explicitly declare the schema types if the inferred ones aren't great.
(maybe I should write something which prints out the scala/py code for that declaration rather than having to do it by hand...)
On 27 Oct 2016, at 05:55, Hyukjin Kwon <gu...@gmail.com>> wrote:
Hi Koert,
Sorry, I thought you meant this is a regression between 2.0.0 and 2.0.1. I just checked It has not been supporting to infer DateType before[1].
Yes, it only supports to infer such data as timestamps currently.
[1]https://github.com/apache/spark/blob/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala#L85-L92
2016-10-27 9:12 GMT+09:00 Anand Viswanathan <an...@ymail.com>>:
Hi,
you can use the customSchema(for DateType) and specify dateFormat in .option().
or
at spark dataframe side, you can convert the timestamp to date using cast to the column.
Thanks and regards,
Anand Viswanathan
On Oct 26, 2016, at 8:07 PM, Koert Kuipers <ko...@tresata.com>> wrote:
hey,
i create a file called test.csv with contents:
date
2015-01-01
2016-03-05
next i run this code in spark 2.0.1:
spark.read
.format("csv")
.option("header", true)
.option("inferSchema", true)
.load("test.csv")
.printSchema
the result is:
root
|-- date: timestamp (nullable = true)
On Wed, Oct 26, 2016 at 7:35 PM, Hyukjin Kwon <gu...@gmail.com>> wrote:
There are now timestampFormat for TimestampType and dateFormat for DateType.
Do you mind if I ask to share your codes?
On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <ko...@tresata.com>> wrote:
is there a reason a column with dates in format yyyy-mm-dd in a csv file is inferred to be TimestampType and not DateType?
thanks! koert
Re: spark infers date to be timestamp type
Posted by Hyukjin Kwon <gu...@gmail.com>.
Hi Koert,
Sorry, I thought you meant this is a regression between 2.0.0 and 2.0.1. I
just checked It has not been supporting to infer DateType before[1].
Yes, it only supports to infer such data as timestamps currently.
[1]
https://github.com/apache/spark/blob/branch-2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala#L85-L92
2016-10-27 9:12 GMT+09:00 Anand Viswanathan <an...@ymail.com>:
> Hi,
>
> you can use the customSchema(for DateType) and specify dateFormat in
> .option().
> or
> at spark dataframe side, you can convert the timestamp to date using cast
> to the column.
>
> Thanks and regards,
> Anand Viswanathan
>
> On Oct 26, 2016, at 8:07 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
> hey,
> i create a file called test.csv with contents:
> date
> 2015-01-01
> 2016-03-05
>
> next i run this code in spark 2.0.1:
> spark.read
> .format("csv")
> .option("header", true)
> .option("inferSchema", true)
> .load("test.csv")
> .printSchema
>
> the result is:
> root
> |-- date: timestamp (nullable = true)
>
>
> On Wed, Oct 26, 2016 at 7:35 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> There are now timestampFormat for TimestampType and dateFormat for
>> DateType.
>>
>> Do you mind if I ask to share your codes?
>>
>> On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <ko...@tresata.com> wrote:
>>
>>> is there a reason a column with dates in format yyyy-mm-dd in a csv file
>>> is inferred to be TimestampType and not DateType?
>>>
>>> thanks! koert
>>>
>>
>
>
Re: spark infers date to be timestamp type
Posted by Anand Viswanathan <an...@ymail.com.INVALID>.
Hi,
you can use the customSchema(for DateType) and specify dateFormat in .option().
or
at spark dataframe side, you can convert the timestamp to date using cast to the column.
Thanks and regards,
Anand Viswanathan
> On Oct 26, 2016, at 8:07 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
> hey,
> i create a file called test.csv with contents:
> date
> 2015-01-01
> 2016-03-05
>
> next i run this code in spark 2.0.1:
> spark.read
> .format("csv")
> .option("header", true)
> .option("inferSchema", true)
> .load("test.csv")
> .printSchema
>
> the result is:
> root
> |-- date: timestamp (nullable = true)
>
>
> On Wed, Oct 26, 2016 at 7:35 PM, Hyukjin Kwon <gurwls223@gmail.com <ma...@gmail.com>> wrote:
> There are now timestampFormat for TimestampType and dateFormat for DateType.
>
> Do you mind if I ask to share your codes?
>
>
> On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <koert@tresata.com <ma...@tresata.com>> wrote:
> is there a reason a column with dates in format yyyy-mm-dd in a csv file is inferred to be TimestampType and not DateType?
>
> thanks! koert
>
Re: spark infers date to be timestamp type
Posted by Koert Kuipers <ko...@tresata.com>.
hey,
i create a file called test.csv with contents:
date
2015-01-01
2016-03-05
next i run this code in spark 2.0.1:
spark.read
.format("csv")
.option("header", true)
.option("inferSchema", true)
.load("test.csv")
.printSchema
the result is:
root
|-- date: timestamp (nullable = true)
On Wed, Oct 26, 2016 at 7:35 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
> There are now timestampFormat for TimestampType and dateFormat for
> DateType.
>
> Do you mind if I ask to share your codes?
>
> On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <ko...@tresata.com> wrote:
>
>> is there a reason a column with dates in format yyyy-mm-dd in a csv file
>> is inferred to be TimestampType and not DateType?
>>
>> thanks! koert
>>
>
Re: spark infers date to be timestamp type
Posted by Hyukjin Kwon <gu...@gmail.com>.
There are now timestampFormat for TimestampType and dateFormat for DateType.
Do you mind if I ask to share your codes?
On 27 Oct 2016 2:16 a.m., "Koert Kuipers" <ko...@tresata.com> wrote:
> is there a reason a column with dates in format yyyy-mm-dd in a csv file
> is inferred to be TimestampType and not DateType?
>
> thanks! koert
>