You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by David Lauzon <da...@gmail.com> on 2015/11/20 22:43:29 UTC
What is the recommended way to store a Date field in Avro ?
I don't see a Date data type in the spec:
http://avro.apache.org/docs/current/spec.html#schema_primitive
I've seen some old discussions in the ML and issues suggesting to serialize
it using the UNIX epoch in a long value.
However, how do we store events older than 1970 like birthdate ?
RE: What is the recommended way to store a Date field in Avro ?
Posted by "Farkas, Zoltan" <Zo...@pimco.com>.
One option is:
@logicalType("localdate")
record Date {
/** the year */
int year;
/** the month of year */
int month;
/** the day of month */
int day;
}
If you want to , you should be able to use the logicalType facility to map it to an existing type…
--Z
From: David Lauzon [mailto:davidonlaptop@gmail.com]
Sent: Friday, November 20, 2015 4:43 PM
To: user@avro.apache.org
Subject: What is the recommended way to store a Date field in Avro ?
I don't see a Date data type in the spec:
http://avro.apache.org/docs/current/spec.html#schema_primitive
I've seen some old discussions in the ML and issues suggesting to serialize it using the UNIX epoch in a long value.
However, how do we store events older than 1970 like birthdate ?
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction. Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
Re: What is the recommended way to store a Date field in Avro ?
Posted by David Lauzon <da...@gmail.com>.
Can we do the same thing if we generate the Java classes from the avro IDL ?
I am thinking something along the lines of:
record Person {
string name;
@targetClass("java.util.Date")
long dateOfBirth;
@targetClass("java.util.Date")
long registrationDatetime;
}
On Sun, Nov 22, 2015 at 3:51 PM, Evan McClain <ae...@gmail.com> wrote:
> On Sun, Nov 22, 2015 at 03:00:25PM -0500, David Lauzon wrote:
> >
> > 300 million years is more than enough :-)
> > Is there a way to get this long to deserialize automatically to a Java
> > 7 compatible type, like java.util.Date ?
> > (I'm using a framework which currently has issues on JDK 8).
>
> If you are using reflection to encode POJOs as avro, it's easy enough to do
> with DateAsLongEncoding:
>
> @AvroEncode(using=DateAslongEncoding.class)
> Date someDate;
>
> See:
> https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/reflect/DateAsLongEncoding.html
>
> And it's fairly easy to write your own encoder for java 8 java.time.Instant
> when you can upgrade to java 8.
>
> --
> Evan McClain
> https://keybase.io/aeroevan
>
Re: What is the recommended way to store a Date field in Avro ?
Posted by Evan McClain <ae...@gmail.com>.
On Sun, Nov 22, 2015 at 03:00:25PM -0500, David Lauzon wrote:
>
> 300 million years is more than enough :-)
> Is there a way to get this long to deserialize automatically to a Java
> 7 compatible type, like java.util.Date ?
> (I'm using a framework which currently has issues on JDK 8).
If you are using reflection to encode POJOs as avro, it's easy enough to do
with DateAsLongEncoding:
@AvroEncode(using=DateAslongEncoding.class)
Date someDate;
See: https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/reflect/DateAsLongEncoding.html
And it's fairly easy to write your own encoder for java 8 java.time.Instant
when you can upgrade to java 8.
--
Evan McClain
https://keybase.io/aeroevan
Re: What is the recommended way to store a Date field in Avro ?
Posted by David Lauzon <da...@gmail.com>.
300 million years is more than enough :-)
Is there a way to get this long to deserialize automatically to a Java 7
compatible type, like java.util.Date ?
(I'm using a framework which currently has issues on JDK 8).
On Fri, Nov 20, 2015 at 9:10 PM, Evan McClain <ae...@gmail.com> wrote:
> On Fri, Nov 20, 2015 at 04:43:29PM -0500, David Lauzon wrote:
> >
> > I don't see a Date data type in the spec:
> > [1]http://avro.apache.org/docs/current/spec.html#schema_primitive
> > I've seen some old discussions in the ML and issues suggesting to
> > serialize it using the UNIX epoch in a long value.
> > However, how do we store events older than 1970 like birthdate ?
> >
> > References
> >
> > 1. http://avro.apache.org/docs/current/spec.html#schema_primitive
>
> Using ms since UNIX epoch should be fine for dates earlier than 1970 since
> it
> gets stored as a signed long. Dates before 1970 will just have a negative
> value.
>
> If you are dealing with dates *really* far in the past (i.e. almost 300
> million
> years ago) you probably need to do something special.
>
> --
> Evan McClain
> https://keybase.io/aeroevan
>
Re: What is the recommended way to store a Date field in Avro ?
Posted by Evan McClain <ae...@gmail.com>.
On Fri, Nov 20, 2015 at 04:43:29PM -0500, David Lauzon wrote:
>
> I don't see a Date data type in the spec:
> [1]http://avro.apache.org/docs/current/spec.html#schema_primitive
> I've seen some old discussions in the ML and issues suggesting to
> serialize it using the UNIX epoch in a long value.
> However, how do we store events older than 1970 like birthdate ?
>
> References
>
> 1. http://avro.apache.org/docs/current/spec.html#schema_primitive
Using ms since UNIX epoch should be fine for dates earlier than 1970 since it
gets stored as a signed long. Dates before 1970 will just have a negative
value.
If you are dealing with dates *really* far in the past (i.e. almost 300 million
years ago) you probably need to do something special.
--
Evan McClain
https://keybase.io/aeroevan