Adding more timestamp types to on-disk storage formats

[Zoltan Ivanfi <zi...@apache.org>]

Hi,

There is an ongoing effort amongst the SQL engines of the Hadoop stack
to support different timestamp semantics. This development has some
implications for the low-level timestamp types as well. The new
timestamp types added to the different SQL engines will rely on the
decisions of the lower level components about which timestamps
semantics they support and how.

I have created a document to summarize what this means for on-disk
storage formats in general and I am sending it out to multiple dev
mailing lists to let you know about this new requirement and to
initiate a discussion in affected open source components.

The document can be read here:
https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit

Please let me know your thoughts.

Thanks,

Zoltan

Re: Adding more timestamp types to on-disk storage formats

[Zoltan Ivanfi <zi...@cloudera.com.INVALID>]

Hi Adar,

Thanks for reviewing the document. I extended it to address your comment.

Naturally, you are free to choose whichever communication channel you
prefer. Personally, I think that if a comment, question or suggestion is
related to a specific section of the document, it may be better to add it
in Google Docs, since then it can be discussed in context. If it is more
general, then it may be a better candidate for the mailing lists (either
all or just one depending on which components it applies to).

Br,

Zoltan

On Thu, Jan 17, 2019 at 11:11 PM Adar Lieber-Dembo <ad...@cloudera.com>
wrote:

> Hi Zoltan,
>
> I'm not feeling brave enough to continue cc'ing a bunch of other
> Apache projects just yet, but I did leave a comment in your gdoc
> describing what Kudu currently supports and what we could offer in the
> future.
>
> It wasn't clear if you wanted to carry out this discussion over gdoc
> comments or over e-mail; if the latter please let me know and I'll
> respond to your original e-mail (including all the other mailing lists
> this time) with my comment.
>
> On Thu, Jan 17, 2019 at 8:28 AM Zoltan Ivanfi <zi...@apache.org> wrote:
> >
> > Hi,
> >
> > There is an ongoing effort amongst the SQL engines of the Hadoop stack
> > to support different timestamp semantics. This development has some
> > implications for the low-level timestamp types as well. The new
> > timestamp types added to the different SQL engines will rely on the
> > decisions of the lower level components about which timestamps
> > semantics they support and how.
> >
> > I have created a document to summarize what this means for on-disk
> > storage formats in general and I am sending it out to multiple dev
> > mailing lists to let you know about this new requirement and to
> > initiate a discussion in affected open source components.
> >
> > The document can be read here:
> >
> https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit
> >
> > Please let me know your thoughts.
> >
> > Thanks,
> >
> > Zoltan
>

Re: Adding more timestamp types to on-disk storage formats

[Adar Lieber-Dembo <ad...@cloudera.com.INVALID>]

Hi Zoltan,

I'm not feeling brave enough to continue cc'ing a bunch of other
Apache projects just yet, but I did leave a comment in your gdoc
describing what Kudu currently supports and what we could offer in the
future.

It wasn't clear if you wanted to carry out this discussion over gdoc
comments or over e-mail; if the latter please let me know and I'll
respond to your original e-mail (including all the other mailing lists
this time) with my comment.

On Thu, Jan 17, 2019 at 8:28 AM Zoltan Ivanfi <zi...@apache.org> wrote:
>
> Hi,
>
> There is an ongoing effort amongst the SQL engines of the Hadoop stack
> to support different timestamp semantics. This development has some
> implications for the low-level timestamp types as well. The new
> timestamp types added to the different SQL engines will rely on the
> decisions of the lower level components about which timestamps
> semantics they support and how.
>
> I have created a document to summarize what this means for on-disk
> storage formats in general and I am sending it out to multiple dev
> mailing lists to let you know about this new requirement and to
> initiate a discussion in affected open source components.
>
> The document can be read here:
> https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit
>
> Please let me know your thoughts.
>
> Thanks,
>
> Zoltan