You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by algermissen1971 <al...@icloud.com> on 2015/07/20 16:37:10 UTC
Joda Time best practice?
Hi,
I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects).
Is there a known best practice to work around these issues?
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Joda Time best practice?
Posted by algermissen1971 <al...@icloud.com>.
On 20 Jul 2015, at 23:20, Harish Butani <rh...@gmail.com> wrote:
> Can you post details on how to reproduce the NPE
Essentially it is like this:
I have a scala case class that contains a Joda DateTime attribute and instances of this class are updated using updateStateByKey. When a certain condition is reached the instances are turned to Json (using spray.json) and are stored in ElasticSearch. just before creating the JSON I call getYear on the date attribute and that fails with NPE.
When insert a getMillis or toString right before the getYear these work just fine.
Jan
>
> On Mon, Jul 20, 2015 at 1:19 PM, algermissen1971 <al...@icloud.com> wrote:
> Hi Harish,
>
> On 20 Jul 2015, at 20:37, Harish Butani <rh...@gmail.com> wrote:
>
> > Hey Jan,
> >
> > Can you provide more details on the serialization and cache issues.
>
> My symptom is that I have a Joda DateTime on which I can call toString and getMillis without problems, but when I call getYear I get a NPE out of the internal AbstractDateTime. Totally strange but seems to align with issues others have.
>
> I am now changing the app to work with millis internally, as that seems to be a performance improvement regarding serialization anyhow.
>
> Thanks,
>
> Jan
>
>
> >
> > If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql.
> >
> > regards,
> > Harish.
> >
> > On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 <al...@icloud.com> wrote:
> > Hi,
> >
> > I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects).
> >
> > Is there a known best practice to work around these issues?
> >
> > Jan
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Joda Time best practice?
Posted by Harish Butani <rh...@gmail.com>.
Can you post details on how to reproduce the NPE
On Mon, Jul 20, 2015 at 1:19 PM, algermissen1971 <algermissen1971@icloud.com
> wrote:
> Hi Harish,
>
> On 20 Jul 2015, at 20:37, Harish Butani <rh...@gmail.com> wrote:
>
> > Hey Jan,
> >
> > Can you provide more details on the serialization and cache issues.
>
> My symptom is that I have a Joda DateTime on which I can call toString and
> getMillis without problems, but when I call getYear I get a NPE out of the
> internal AbstractDateTime. Totally strange but seems to align with issues
> others have.
>
> I am now changing the app to work with millis internally, as that seems to
> be a performance improvement regarding serialization anyhow.
>
> Thanks,
>
> Jan
>
>
> >
> > If you are looking for datetime functionality with spark-sql please
> consider: https://github.com/SparklineData/spark-datetime It provides a
> simple way to combine joda datetime expressions with spark sql.
> >
> > regards,
> > Harish.
> >
> > On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 <
> algermissen1971@icloud.com> wrote:
> > Hi,
> >
> > I am having trouble with Joda Time in a Spark application and saw by now
> that I am not the only one (generally seems to have to do with
> serialization and internal caches of the Joda Time objects).
> >
> > Is there a known best practice to work around these issues?
> >
> > Jan
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
>
>
Re: Joda Time best practice?
Posted by algermissen1971 <al...@icloud.com>.
Hi Harish,
On 20 Jul 2015, at 20:37, Harish Butani <rh...@gmail.com> wrote:
> Hey Jan,
>
> Can you provide more details on the serialization and cache issues.
My symptom is that I have a Joda DateTime on which I can call toString and getMillis without problems, but when I call getYear I get a NPE out of the internal AbstractDateTime. Totally strange but seems to align with issues others have.
I am now changing the app to work with millis internally, as that seems to be a performance improvement regarding serialization anyhow.
Thanks,
Jan
>
> If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql.
>
> regards,
> Harish.
>
> On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 <al...@icloud.com> wrote:
> Hi,
>
> I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects).
>
> Is there a known best practice to work around these issues?
>
> Jan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Joda Time best practice?
Posted by Harish Butani <rh...@gmail.com>.
Hey Jan,
Can you provide more details on the serialization and cache issues.
If you are looking for datetime functionality with spark-sql please
consider: https://github.com/SparklineData/spark-datetime It provides a
simple way to combine joda datetime expressions with spark sql.
regards,
Harish.
On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 <algermissen1971@icloud.com
> wrote:
> Hi,
>
> I am having trouble with Joda Time in a Spark application and saw by now
> that I am not the only one (generally seems to have to do with
> serialization and internal caches of the Joda Time objects).
>
> Is there a known best practice to work around these issues?
>
> Jan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>