You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Bryan Jeffrey <br...@gmail.com> on 2015/11/23 14:40:48 UTC
DateTime Support - Hive Parquet
All,
I am attempting to write objects that include a DateTime properties to a
persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was forced to
convert the DateTime properties to Timestamp properties. I was under the
impression that this issue was fixed in the default Hive supported with
1.5.2 - however, I am still seeing the associated errors.
Is there a bug I can follow to determine when DateTime will be supported
for Parquet?
Regards,
Bryan Jeffrey
Re: DateTime Support - Hive Parquet
Posted by Cheng Lian <li...@gmail.com>.
Oh sorry, you're right. Implicit conversion doesn't affect the schema
inference process.
Just checked that Joda is already a direct dependency of Spark. So I
think it's probably fine to add support for recognizing Joda DateTime as
SQL TimestampType. Would you mind to file a JIRA ticket for it? PR is
also welcomed!
Cheng
On 11/24/15 8:05 PM, Bryan wrote:
>
> Cheng,
>
> I am using Scala. I have an implicit conversion from Joda DateTime to
> timestamp. My tables are defined with Timestamp. However explicit
> conversation appears to be required. Do you have an example of
> implicit conversion for this case? Do you convert on insert or on RDD
> to DF conversion?
>
> Regards,
>
> Bryan Jeffrey
>
> Sent from Outlook Mail
>
>
> *From: *Cheng Lian
> *Sent: *Tuesday, November 24, 2015 6:49 AM
> *To: *Bryan;user
> *Subject: *Re: DateTime Support - Hive Parquet
>
> I see, then this is actually irrelevant to Parquet. I guess can
> support Joda DateTime in Spark SQL reflective schema inference to have
> this, provided that this is a frequent use case and Spark SQL already
> has Joda as a direct dependency.
>
> On the other hand, if you are using Scala, you can write a simple
> implicit conversion method to avoid all the manual conversions.
>
> Cheng
>
> On 11/24/15 7:25 PM, Bryan wrote:
>
> Cheng,
>
> That’s exactly what I was hoping for – native support for writing
> DateTime objects. As it stands Spark 1.5.2 seems to leave no
> option but to do manual conversion (to nanos, Timestamp, etc)
> prior to writing records to hive.
>
> Regards,
>
> Bryan Jeffrey
>
> Sent from Outlook Mail
>
>
> *From: *Cheng Lian
> *Sent: *Tuesday, November 24, 2015 1:42 AM
> *To: *Bryan Jeffrey;user
> *Subject: *Re: DateTime Support - Hive Parquet
>
> Hey Bryan,
>
> What do you mean by "DateTime properties"? Hive and Spark SQL both
>
> support DATE and TIMESTAMP types, but there's no DATETIME type. So I
>
> assume you are referring to Java class DateTime (possibly the one in
>
> joda)? Could you please provide a sample snippet that illustrates
> your
>
> requirement?
>
> Cheng
>
> On 11/23/15 9:40 PM, Bryan Jeffrey wrote:
>
> > All,
>
> >
>
> > I am attempting to write objects that include a DateTime
> properties to
>
> > a persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was
>
> > forced to convert the DateTime properties to Timestamp
> properties. I
>
> > was under the impression that this issue was fixed in the
> default Hive
>
> > supported with 1.5.2 - however, I am still seeing the associated
> errors.
>
> >
>
> > Is there a bug I can follow to determine when DateTime will be
>
> > supported for Parquet?
>
> >
>
> > Regards,
>
> >
>
> > Bryan Jeffrey
>
RE: DateTime Support - Hive Parquet
Posted by Bryan <br...@gmail.com>.
Cheng,
I am using Scala. I have an implicit conversion from Joda DateTime to timestamp. My tables are defined with Timestamp. However explicit conversation appears to be required. Do you have an example of implicit conversion for this case? Do you convert on insert or on RDD to DF conversion?
Regards,
Bryan Jeffrey
Sent from Outlook Mail
From: Cheng Lian
Sent: Tuesday, November 24, 2015 6:49 AM
To: Bryan;user
Subject: Re: DateTime Support - Hive Parquet
I see, then this is actually irrelevant to Parquet. I guess can support Joda DateTime in Spark SQL reflective schema inference to have this, provided that this is a frequent use case and Spark SQL already has Joda as a direct dependency.
On the other hand, if you are using Scala, you can write a simple implicit conversion method to avoid all the manual conversions.
Cheng
On 11/24/15 7:25 PM, Bryan wrote:
Cheng,
That’s exactly what I was hoping for – native support for writing DateTime objects. As it stands Spark 1.5.2 seems to leave no option but to do manual conversion (to nanos, Timestamp, etc) prior to writing records to hive.
Regards,
Bryan Jeffrey
Sent from Outlook Mail
From: Cheng Lian
Sent: Tuesday, November 24, 2015 1:42 AM
To: Bryan Jeffrey;user
Subject: Re: DateTime Support - Hive Parquet
Hey Bryan,
What do you mean by "DateTime properties"? Hive and Spark SQL both
support DATE and TIMESTAMP types, but there's no DATETIME type. So I
assume you are referring to Java class DateTime (possibly the one in
joda)? Could you please provide a sample snippet that illustrates your
requirement?
Cheng
On 11/23/15 9:40 PM, Bryan Jeffrey wrote:
> All,
>
> I am attempting to write objects that include a DateTime properties to
> a persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was
> forced to convert the DateTime properties to Timestamp properties. I
> was under the impression that this issue was fixed in the default Hive
> supported with 1.5.2 - however, I am still seeing the associated errors.
>
> Is there a bug I can follow to determine when DateTime will be
> supported for Parquet?
>
> Regards,
>
> Bryan Jeffrey
Re: DateTime Support - Hive Parquet
Posted by Cheng Lian <li...@gmail.com>.
I see, then this is actually irrelevant to Parquet. I guess can support
Joda DateTime in Spark SQL reflective schema inference to have this,
provided that this is a frequent use case and Spark SQL already has Joda
as a direct dependency.
On the other hand, if you are using Scala, you can write a simple
implicit conversion method to avoid all the manual conversions.
Cheng
On 11/24/15 7:25 PM, Bryan wrote:
>
> Cheng,
>
> That’s exactly what I was hoping for – native support for writing
> DateTime objects. As it stands Spark 1.5.2 seems to leave no option
> but to do manual conversion (to nanos, Timestamp, etc) prior to
> writing records to hive.
>
> Regards,
>
> Bryan Jeffrey
>
> Sent from Outlook Mail
>
>
> *From: *Cheng Lian
> *Sent: *Tuesday, November 24, 2015 1:42 AM
> *To: *Bryan Jeffrey;user
> *Subject: *Re: DateTime Support - Hive Parquet
>
> Hey Bryan,
>
> What do you mean by "DateTime properties"? Hive and Spark SQL both
>
> support DATE and TIMESTAMP types, but there's no DATETIME type. So I
>
> assume you are referring to Java class DateTime (possibly the one in
>
> joda)? Could you please provide a sample snippet that illustrates your
>
> requirement?
>
> Cheng
>
> On 11/23/15 9:40 PM, Bryan Jeffrey wrote:
>
> > All,
>
> >
>
> > I am attempting to write objects that include a DateTime properties to
>
> > a persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was
>
> > forced to convert the DateTime properties to Timestamp properties. I
>
> > was under the impression that this issue was fixed in the default Hive
>
> > supported with 1.5.2 - however, I am still seeing the associated errors.
>
> >
>
> > Is there a bug I can follow to determine when DateTime will be
>
> > supported for Parquet?
>
> >
>
> > Regards,
>
> >
>
> > Bryan Jeffrey
>
RE: DateTime Support - Hive Parquet
Posted by Bryan <br...@gmail.com>.
Cheng,
That’s exactly what I was hoping for – native support for writing DateTime objects. As it stands Spark 1.5.2 seems to leave no option but to do manual conversion (to nanos, Timestamp, etc) prior to writing records to hive.
Regards,
Bryan Jeffrey
Sent from Outlook Mail
From: Cheng Lian
Sent: Tuesday, November 24, 2015 1:42 AM
To: Bryan Jeffrey;user
Subject: Re: DateTime Support - Hive Parquet
Hey Bryan,
What do you mean by "DateTime properties"? Hive and Spark SQL both
support DATE and TIMESTAMP types, but there's no DATETIME type. So I
assume you are referring to Java class DateTime (possibly the one in
joda)? Could you please provide a sample snippet that illustrates your
requirement?
Cheng
On 11/23/15 9:40 PM, Bryan Jeffrey wrote:
> All,
>
> I am attempting to write objects that include a DateTime properties to
> a persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was
> forced to convert the DateTime properties to Timestamp properties. I
> was under the impression that this issue was fixed in the default Hive
> supported with 1.5.2 - however, I am still seeing the associated errors.
>
> Is there a bug I can follow to determine when DateTime will be
> supported for Parquet?
>
> Regards,
>
> Bryan Jeffrey
Re: DateTime Support - Hive Parquet
Posted by Cheng Lian <li...@gmail.com>.
Hey Bryan,
What do you mean by "DateTime properties"? Hive and Spark SQL both
support DATE and TIMESTAMP types, but there's no DATETIME type. So I
assume you are referring to Java class DateTime (possibly the one in
joda)? Could you please provide a sample snippet that illustrates your
requirement?
Cheng
On 11/23/15 9:40 PM, Bryan Jeffrey wrote:
> All,
>
> I am attempting to write objects that include a DateTime properties to
> a persistent table using Spark 1.5.2 / HiveContext. In 1.4.1 I was
> forced to convert the DateTime properties to Timestamp properties. I
> was under the impression that this issue was fixed in the default Hive
> supported with 1.5.2 - however, I am still seeing the associated errors.
>
> Is there a bug I can follow to determine when DateTime will be
> supported for Parquet?
>
> Regards,
>
> Bryan Jeffrey
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org