Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@apache.org>]

Hello Flink, Calcite, and Beam dev lists!

Linked below is the second document I promised way back in April regarding
a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
for the delay; I thought I was nearly done a while back and then temporal
joins expanded to something much larger than expected).

To repeat what it says in the doc, my hope is that it can serve various
purposes over it's lifetime:

   -
   - A discussion ground for ironing out any remaining features necessary
   for supporting robust streaming semantics in Calcite SQL.

   - A rough, high-level source of truth for tracking efforts underway in
   support of this, currently spanning the Calcite, Flink, and Beam projects.

   - A written specification of the changes that were made, for the sake of
   understanding the delta after the fact.

The first and third points are, IMO, the most important. AFAIK, there are a
few features missing still that need to be defined (e.g., triggers
equivalents via EMIT, robust temporal join support). I'm also proposing a
clear distinction of streams and tables, which I think is important, but
which I believe is not the approach most folks have been taking in this
area. Sorting out these open issues and then having a concise record of the
solutions adopted will be important for providing a solid streaming
experience and teaching folks how to use it.

At any rate, I would much appreciate it if anyone with an interest in this
stuff could please take a look and add comments/suggestions/references to
related work in flight/etc as appropriate. For now please use
comments/suggestions, but if you really want to dive in with edit access,
let me know.

The doc: http://s.apache.org/streaming-sql-spec

-Tyler

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.

-Tyler

On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:

> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>>   -
> >> > > >>>   - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>>   - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>>   support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>>   - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>>   of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.

-Tyler

On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:

> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>>   -
> >> > > >>>   - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>>   - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>>   support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>>   - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>>   of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.

-Tyler

On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:

> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>>   -
> >> > > >>>   - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>>   - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>>   support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>>   - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>>   of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.

I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.

I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.

Julian


On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>>   -
>> > > >>>   - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>>   for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>>   - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>>   - A written specification of the changes that were made, for the
>> > sake
>> > > >>>   of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.

I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.

I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.

Julian


On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>>   -
>> > > >>>   - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>>   for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>>   - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>>   - A written specification of the changes that were made, for the
>> > sake
>> > > >>>   of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.

I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.

I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.

Julian


On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>>   -
>> > > >>>   - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>>   for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>>   - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>>   - A written specification of the changes that were made, for the
>> > sake
>> > > >>>   of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

As promised, I went of the document and made some comments.
I also added a bit of information about the current SQL support in Flink
and its internals.

Thanks, Fabian

2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:

> Hi Tyler,
> Thanks for putting all the efforts into a doc. It is really well written
> and organized.
> I like the most part. The major concern I have is about the "explicit
> trigger". I left a few comments towards this and would like to know what
> the others think about it.
>
> Regards,
> Shaoxuan
>
> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Thanks for the great write up!
> >
> > I think this s very good starting point for a detailed discussion about
> > features, syntax and semantics of streaming SQL.
> > I'll comment on the document in the next days and describe Flink's
> current
> > status, our approaches (or planned approaches) and ask a couple of
> > questions.
> >
> > Thanks, Fabian
> >
> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >
> > > Tyler,
> > >
> > > Thanks for this. I am reading the document thoroughly and will give my
> > > feedback in a day or two.
> > >
> > > Julian
> > >
> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > > wrote:
> > > >
> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > > calcite
> > > > integration and others would be interested.
> > > >
> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > > <ta...@google.com.invalid>
> > > > wrote:
> > > >
> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> > as
> > > >> well. If anyone on the Apex side wants to join in on the fun, your
> > input
> > > >> would be welcomed!
> > > >>
> > > >> -Tyler
> > > >>
> > > >>
> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > > wrote:
> > > >>
> > > >>> Hello Flink, Calcite, and Beam dev lists!
> > > >>>
> > > >>> Linked below is the second document I promised way back in April
> > > >> regarding
> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > > apologies
> > > >>> for the delay; I thought I was nearly done a while back and then
> > > temporal
> > > >>> joins expanded to something much larger than expected).
> > > >>>
> > > >>> To repeat what it says in the doc, my hope is that it can serve
> > various
> > > >>> purposes over it's lifetime:
> > > >>>
> > > >>>   -
> > > >>>   - A discussion ground for ironing out any remaining features
> > > necessary
> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> > > >>>
> > > >>>   - A rough, high-level source of truth for tracking efforts
> underway
> > > in
> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > > >> projects.
> > > >>>
> > > >>>   - A written specification of the changes that were made, for the
> > sake
> > > >>>   of understanding the delta after the fact.
> > > >>>
> > > >>> The first and third points are, IMO, the most important. AFAIK,
> there
> > > are
> > > >>> a few features missing still that need to be defined (e.g.,
> triggers
> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> > > proposing a
> > > >>> clear distinction of streams and tables, which I think is
> important,
> > > but
> > > >>> which I believe is not the approach most folks have been taking in
> > this
> > > >>> area. Sorting out these open issues and then having a concise
> record
> > of
> > > >> the
> > > >>> solutions adopted will be important for providing a solid streaming
> > > >>> experience and teaching folks how to use it.
> > > >>>
> > > >>> At any rate, I would much appreciate it if anyone with an interest
> in
> > > >> this
> > > >>> stuff could please take a look and add comments/suggestions/
> > references
> > > >> to
> > > >>> related work in flight/etc as appropriate. For now please use
> > > >>> comments/suggestions, but if you really want to dive in with edit
> > > access,
> > > >>> let me know.
> > > >>>
> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> > > >>>
> > > >>> -Tyler
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

As promised, I went of the document and made some comments.
I also added a bit of information about the current SQL support in Flink
and its internals.

Thanks, Fabian

2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:

> Hi Tyler,
> Thanks for putting all the efforts into a doc. It is really well written
> and organized.
> I like the most part. The major concern I have is about the "explicit
> trigger". I left a few comments towards this and would like to know what
> the others think about it.
>
> Regards,
> Shaoxuan
>
> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Thanks for the great write up!
> >
> > I think this s very good starting point for a detailed discussion about
> > features, syntax and semantics of streaming SQL.
> > I'll comment on the document in the next days and describe Flink's
> current
> > status, our approaches (or planned approaches) and ask a couple of
> > questions.
> >
> > Thanks, Fabian
> >
> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >
> > > Tyler,
> > >
> > > Thanks for this. I am reading the document thoroughly and will give my
> > > feedback in a day or two.
> > >
> > > Julian
> > >
> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > > wrote:
> > > >
> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > > calcite
> > > > integration and others would be interested.
> > > >
> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > > <ta...@google.com.invalid>
> > > > wrote:
> > > >
> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> > as
> > > >> well. If anyone on the Apex side wants to join in on the fun, your
> > input
> > > >> would be welcomed!
> > > >>
> > > >> -Tyler
> > > >>
> > > >>
> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > > wrote:
> > > >>
> > > >>> Hello Flink, Calcite, and Beam dev lists!
> > > >>>
> > > >>> Linked below is the second document I promised way back in April
> > > >> regarding
> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > > apologies
> > > >>> for the delay; I thought I was nearly done a while back and then
> > > temporal
> > > >>> joins expanded to something much larger than expected).
> > > >>>
> > > >>> To repeat what it says in the doc, my hope is that it can serve
> > various
> > > >>> purposes over it's lifetime:
> > > >>>
> > > >>>   -
> > > >>>   - A discussion ground for ironing out any remaining features
> > > necessary
> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> > > >>>
> > > >>>   - A rough, high-level source of truth for tracking efforts
> underway
> > > in
> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > > >> projects.
> > > >>>
> > > >>>   - A written specification of the changes that were made, for the
> > sake
> > > >>>   of understanding the delta after the fact.
> > > >>>
> > > >>> The first and third points are, IMO, the most important. AFAIK,
> there
> > > are
> > > >>> a few features missing still that need to be defined (e.g.,
> triggers
> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> > > proposing a
> > > >>> clear distinction of streams and tables, which I think is
> important,
> > > but
> > > >>> which I believe is not the approach most folks have been taking in
> > this
> > > >>> area. Sorting out these open issues and then having a concise
> record
> > of
> > > >> the
> > > >>> solutions adopted will be important for providing a solid streaming
> > > >>> experience and teaching folks how to use it.
> > > >>>
> > > >>> At any rate, I would much appreciate it if anyone with an interest
> in
> > > >> this
> > > >>> stuff could please take a look and add comments/suggestions/
> > references
> > > >> to
> > > >>> related work in flight/etc as appropriate. For now please use
> > > >>> comments/suggestions, but if you really want to dive in with edit
> > > access,
> > > >>> let me know.
> > > >>>
> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> > > >>>
> > > >>> -Tyler
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

As promised, I went of the document and made some comments.
I also added a bit of information about the current SQL support in Flink
and its internals.

Thanks, Fabian

2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:

> Hi Tyler,
> Thanks for putting all the efforts into a doc. It is really well written
> and organized.
> I like the most part. The major concern I have is about the "explicit
> trigger". I left a few comments towards this and would like to know what
> the others think about it.
>
> Regards,
> Shaoxuan
>
> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Thanks for the great write up!
> >
> > I think this s very good starting point for a detailed discussion about
> > features, syntax and semantics of streaming SQL.
> > I'll comment on the document in the next days and describe Flink's
> current
> > status, our approaches (or planned approaches) and ask a couple of
> > questions.
> >
> > Thanks, Fabian
> >
> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >
> > > Tyler,
> > >
> > > Thanks for this. I am reading the document thoroughly and will give my
> > > feedback in a day or two.
> > >
> > > Julian
> > >
> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > > wrote:
> > > >
> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > > calcite
> > > > integration and others would be interested.
> > > >
> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > > <ta...@google.com.invalid>
> > > > wrote:
> > > >
> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> > as
> > > >> well. If anyone on the Apex side wants to join in on the fun, your
> > input
> > > >> would be welcomed!
> > > >>
> > > >> -Tyler
> > > >>
> > > >>
> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > > wrote:
> > > >>
> > > >>> Hello Flink, Calcite, and Beam dev lists!
> > > >>>
> > > >>> Linked below is the second document I promised way back in April
> > > >> regarding
> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > > apologies
> > > >>> for the delay; I thought I was nearly done a while back and then
> > > temporal
> > > >>> joins expanded to something much larger than expected).
> > > >>>
> > > >>> To repeat what it says in the doc, my hope is that it can serve
> > various
> > > >>> purposes over it's lifetime:
> > > >>>
> > > >>>   -
> > > >>>   - A discussion ground for ironing out any remaining features
> > > necessary
> > > >>>   for supporting robust streaming semantics in Calcite SQL.
> > > >>>
> > > >>>   - A rough, high-level source of truth for tracking efforts
> underway
> > > in
> > > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > > >> projects.
> > > >>>
> > > >>>   - A written specification of the changes that were made, for the
> > sake
> > > >>>   of understanding the delta after the fact.
> > > >>>
> > > >>> The first and third points are, IMO, the most important. AFAIK,
> there
> > > are
> > > >>> a few features missing still that need to be defined (e.g.,
> triggers
> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> > > proposing a
> > > >>> clear distinction of streams and tables, which I think is
> important,
> > > but
> > > >>> which I believe is not the approach most folks have been taking in
> > this
> > > >>> area. Sorting out these open issues and then having a concise
> record
> > of
> > > >> the
> > > >>> solutions adopted will be important for providing a solid streaming
> > > >>> experience and teaching folks how to use it.
> > > >>>
> > > >>> At any rate, I would much appreciate it if anyone with an interest
> in
> > > >> this
> > > >>> stuff could please take a look and add comments/suggestions/
> > references
> > > >> to
> > > >>> related work in flight/etc as appropriate. For now please use
> > > >>> comments/suggestions, but if you really want to dive in with edit
> > > access,
> > > >>> let me know.
> > > >>>
> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> > > >>>
> > > >>> -Tyler
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Shaoxuan Wang <ws...@gmail.com>]

Hi Tyler,
Thanks for putting all the efforts into a doc. It is really well written
and organized.
I like the most part. The major concern I have is about the "explicit
trigger". I left a few comments towards this and would like to know what
the others think about it.

Regards,
Shaoxuan

On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Thanks for the great write up!
>
> I think this s very good starting point for a detailed discussion about
> features, syntax and semantics of streaming SQL.
> I'll comment on the document in the next days and describe Flink's current
> status, our approaches (or planned approaches) and ask a couple of
> questions.
>
> Thanks, Fabian
>
> 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>
> > Tyler,
> >
> > Thanks for this. I am reading the document thoroughly and will give my
> > feedback in a day or two.
> >
> > Julian
> >
> > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> > wrote:
> > >
> > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > calcite
> > > integration and others would be interested.
> > >
> > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > <ta...@google.com.invalid>
> > > wrote:
> > >
> > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> as
> > >> well. If anyone on the Apex side wants to join in on the fun, your
> input
> > >> would be welcomed!
> > >>
> > >> -Tyler
> > >>
> > >>
> > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > wrote:
> > >>
> > >>> Hello Flink, Calcite, and Beam dev lists!
> > >>>
> > >>> Linked below is the second document I promised way back in April
> > >> regarding
> > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > apologies
> > >>> for the delay; I thought I was nearly done a while back and then
> > temporal
> > >>> joins expanded to something much larger than expected).
> > >>>
> > >>> To repeat what it says in the doc, my hope is that it can serve
> various
> > >>> purposes over it's lifetime:
> > >>>
> > >>>   -
> > >>>   - A discussion ground for ironing out any remaining features
> > necessary
> > >>>   for supporting robust streaming semantics in Calcite SQL.
> > >>>
> > >>>   - A rough, high-level source of truth for tracking efforts underway
> > in
> > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > >> projects.
> > >>>
> > >>>   - A written specification of the changes that were made, for the
> sake
> > >>>   of understanding the delta after the fact.
> > >>>
> > >>> The first and third points are, IMO, the most important. AFAIK, there
> > are
> > >>> a few features missing still that need to be defined (e.g., triggers
> > >>> equivalents via EMIT, robust temporal join support). I'm also
> > proposing a
> > >>> clear distinction of streams and tables, which I think is important,
> > but
> > >>> which I believe is not the approach most folks have been taking in
> this
> > >>> area. Sorting out these open issues and then having a concise record
> of
> > >> the
> > >>> solutions adopted will be important for providing a solid streaming
> > >>> experience and teaching folks how to use it.
> > >>>
> > >>> At any rate, I would much appreciate it if anyone with an interest in
> > >> this
> > >>> stuff could please take a look and add comments/suggestions/
> references
> > >> to
> > >>> related work in flight/etc as appropriate. For now please use
> > >>> comments/suggestions, but if you really want to dive in with edit
> > access,
> > >>> let me know.
> > >>>
> > >>> The doc: http://s.apache.org/streaming-sql-spec
> > >>>
> > >>> -Tyler
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Shaoxuan Wang <ws...@gmail.com>]

Hi Tyler,
Thanks for putting all the efforts into a doc. It is really well written
and organized.
I like the most part. The major concern I have is about the "explicit
trigger". I left a few comments towards this and would like to know what
the others think about it.

Regards,
Shaoxuan

On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Thanks for the great write up!
>
> I think this s very good starting point for a detailed discussion about
> features, syntax and semantics of streaming SQL.
> I'll comment on the document in the next days and describe Flink's current
> status, our approaches (or planned approaches) and ask a couple of
> questions.
>
> Thanks, Fabian
>
> 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>
> > Tyler,
> >
> > Thanks for this. I am reading the document thoroughly and will give my
> > feedback in a day or two.
> >
> > Julian
> >
> > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> > wrote:
> > >
> > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > calcite
> > > integration and others would be interested.
> > >
> > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > <ta...@google.com.invalid>
> > > wrote:
> > >
> > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> as
> > >> well. If anyone on the Apex side wants to join in on the fun, your
> input
> > >> would be welcomed!
> > >>
> > >> -Tyler
> > >>
> > >>
> > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > wrote:
> > >>
> > >>> Hello Flink, Calcite, and Beam dev lists!
> > >>>
> > >>> Linked below is the second document I promised way back in April
> > >> regarding
> > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > apologies
> > >>> for the delay; I thought I was nearly done a while back and then
> > temporal
> > >>> joins expanded to something much larger than expected).
> > >>>
> > >>> To repeat what it says in the doc, my hope is that it can serve
> various
> > >>> purposes over it's lifetime:
> > >>>
> > >>>   -
> > >>>   - A discussion ground for ironing out any remaining features
> > necessary
> > >>>   for supporting robust streaming semantics in Calcite SQL.
> > >>>
> > >>>   - A rough, high-level source of truth for tracking efforts underway
> > in
> > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > >> projects.
> > >>>
> > >>>   - A written specification of the changes that were made, for the
> sake
> > >>>   of understanding the delta after the fact.
> > >>>
> > >>> The first and third points are, IMO, the most important. AFAIK, there
> > are
> > >>> a few features missing still that need to be defined (e.g., triggers
> > >>> equivalents via EMIT, robust temporal join support). I'm also
> > proposing a
> > >>> clear distinction of streams and tables, which I think is important,
> > but
> > >>> which I believe is not the approach most folks have been taking in
> this
> > >>> area. Sorting out these open issues and then having a concise record
> of
> > >> the
> > >>> solutions adopted will be important for providing a solid streaming
> > >>> experience and teaching folks how to use it.
> > >>>
> > >>> At any rate, I would much appreciate it if anyone with an interest in
> > >> this
> > >>> stuff could please take a look and add comments/suggestions/
> references
> > >> to
> > >>> related work in flight/etc as appropriate. For now please use
> > >>> comments/suggestions, but if you really want to dive in with edit
> > access,
> > >>> let me know.
> > >>>
> > >>> The doc: http://s.apache.org/streaming-sql-spec
> > >>>
> > >>> -Tyler
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Shaoxuan Wang <ws...@gmail.com>]

Hi Tyler,
Thanks for putting all the efforts into a doc. It is really well written
and organized.
I like the most part. The major concern I have is about the "explicit
trigger". I left a few comments towards this and would like to know what
the others think about it.

Regards,
Shaoxuan

On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Thanks for the great write up!
>
> I think this s very good starting point for a detailed discussion about
> features, syntax and semantics of streaming SQL.
> I'll comment on the document in the next days and describe Flink's current
> status, our approaches (or planned approaches) and ask a couple of
> questions.
>
> Thanks, Fabian
>
> 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>
> > Tyler,
> >
> > Thanks for this. I am reading the document thoroughly and will give my
> > feedback in a day or two.
> >
> > Julian
> >
> > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> > wrote:
> > >
> > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > calcite
> > > integration and others would be interested.
> > >
> > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > <ta...@google.com.invalid>
> > > wrote:
> > >
> > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> as
> > >> well. If anyone on the Apex side wants to join in on the fun, your
> input
> > >> would be welcomed!
> > >>
> > >> -Tyler
> > >>
> > >>
> > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > wrote:
> > >>
> > >>> Hello Flink, Calcite, and Beam dev lists!
> > >>>
> > >>> Linked below is the second document I promised way back in April
> > >> regarding
> > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > apologies
> > >>> for the delay; I thought I was nearly done a while back and then
> > temporal
> > >>> joins expanded to something much larger than expected).
> > >>>
> > >>> To repeat what it says in the doc, my hope is that it can serve
> various
> > >>> purposes over it's lifetime:
> > >>>
> > >>>   -
> > >>>   - A discussion ground for ironing out any remaining features
> > necessary
> > >>>   for supporting robust streaming semantics in Calcite SQL.
> > >>>
> > >>>   - A rough, high-level source of truth for tracking efforts underway
> > in
> > >>>   support of this, currently spanning the Calcite, Flink, and Beam
> > >> projects.
> > >>>
> > >>>   - A written specification of the changes that were made, for the
> sake
> > >>>   of understanding the delta after the fact.
> > >>>
> > >>> The first and third points are, IMO, the most important. AFAIK, there
> > are
> > >>> a few features missing still that need to be defined (e.g., triggers
> > >>> equivalents via EMIT, robust temporal join support). I'm also
> > proposing a
> > >>> clear distinction of streams and tables, which I think is important,
> > but
> > >>> which I believe is not the approach most folks have been taking in
> this
> > >>> area. Sorting out these open issues and then having a concise record
> of
> > >> the
> > >>> solutions adopted will be important for providing a solid streaming
> > >>> experience and teaching folks how to use it.
> > >>>
> > >>> At any rate, I would much appreciate it if anyone with an interest in
> > >> this
> > >>> stuff could please take a look and add comments/suggestions/
> references
> > >> to
> > >>> related work in flight/etc as appropriate. For now please use
> > >>> comments/suggestions, but if you really want to dive in with edit
> > access,
> > >>> let me know.
> > >>>
> > >>> The doc: http://s.apache.org/streaming-sql-spec
> > >>>
> > >>> -Tyler
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

Thanks for the great write up!

I think this s very good starting point for a detailed discussion about
features, syntax and semantics of streaming SQL.
I'll comment on the document in the next days and describe Flink's current
status, our approaches (or planned approaches) and ask a couple of
questions.

Thanks, Fabian

2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:

> Tyler,
>
> Thanks for this. I am reading the document thoroughly and will give my
> feedback in a day or two.
>
> Julian
>
> > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
> >
> > Thanks for the invitation Tyler. I am sure folks who worked on the
> calcite
> > integration and others would be interested.
> >
> > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> <ta...@google.com.invalid>
> > wrote:
> >
> >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> >> well. If anyone on the Apex side wants to join in on the fun, your input
> >> would be welcomed!
> >>
> >> -Tyler
> >>
> >>
> >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> wrote:
> >>
> >>> Hello Flink, Calcite, and Beam dev lists!
> >>>
> >>> Linked below is the second document I promised way back in April
> >> regarding
> >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> apologies
> >>> for the delay; I thought I was nearly done a while back and then
> temporal
> >>> joins expanded to something much larger than expected).
> >>>
> >>> To repeat what it says in the doc, my hope is that it can serve various
> >>> purposes over it's lifetime:
> >>>
> >>>   -
> >>>   - A discussion ground for ironing out any remaining features
> necessary
> >>>   for supporting robust streaming semantics in Calcite SQL.
> >>>
> >>>   - A rough, high-level source of truth for tracking efforts underway
> in
> >>>   support of this, currently spanning the Calcite, Flink, and Beam
> >> projects.
> >>>
> >>>   - A written specification of the changes that were made, for the sake
> >>>   of understanding the delta after the fact.
> >>>
> >>> The first and third points are, IMO, the most important. AFAIK, there
> are
> >>> a few features missing still that need to be defined (e.g., triggers
> >>> equivalents via EMIT, robust temporal join support). I'm also
> proposing a
> >>> clear distinction of streams and tables, which I think is important,
> but
> >>> which I believe is not the approach most folks have been taking in this
> >>> area. Sorting out these open issues and then having a concise record of
> >> the
> >>> solutions adopted will be important for providing a solid streaming
> >>> experience and teaching folks how to use it.
> >>>
> >>> At any rate, I would much appreciate it if anyone with an interest in
> >> this
> >>> stuff could please take a look and add comments/suggestions/references
> >> to
> >>> related work in flight/etc as appropriate. For now please use
> >>> comments/suggestions, but if you really want to dive in with edit
> access,
> >>> let me know.
> >>>
> >>> The doc: http://s.apache.org/streaming-sql-spec
> >>>
> >>> -Tyler
> >>>
> >>>
> >>>
> >>
>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

Thanks for the great write up!

I think this s very good starting point for a detailed discussion about
features, syntax and semantics of streaming SQL.
I'll comment on the document in the next days and describe Flink's current
status, our approaches (or planned approaches) and ask a couple of
questions.

Thanks, Fabian

2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:

> Tyler,
>
> Thanks for this. I am reading the document thoroughly and will give my
> feedback in a day or two.
>
> Julian
>
> > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
> >
> > Thanks for the invitation Tyler. I am sure folks who worked on the
> calcite
> > integration and others would be interested.
> >
> > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> <ta...@google.com.invalid>
> > wrote:
> >
> >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> >> well. If anyone on the Apex side wants to join in on the fun, your input
> >> would be welcomed!
> >>
> >> -Tyler
> >>
> >>
> >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> wrote:
> >>
> >>> Hello Flink, Calcite, and Beam dev lists!
> >>>
> >>> Linked below is the second document I promised way back in April
> >> regarding
> >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> apologies
> >>> for the delay; I thought I was nearly done a while back and then
> temporal
> >>> joins expanded to something much larger than expected).
> >>>
> >>> To repeat what it says in the doc, my hope is that it can serve various
> >>> purposes over it's lifetime:
> >>>
> >>>   -
> >>>   - A discussion ground for ironing out any remaining features
> necessary
> >>>   for supporting robust streaming semantics in Calcite SQL.
> >>>
> >>>   - A rough, high-level source of truth for tracking efforts underway
> in
> >>>   support of this, currently spanning the Calcite, Flink, and Beam
> >> projects.
> >>>
> >>>   - A written specification of the changes that were made, for the sake
> >>>   of understanding the delta after the fact.
> >>>
> >>> The first and third points are, IMO, the most important. AFAIK, there
> are
> >>> a few features missing still that need to be defined (e.g., triggers
> >>> equivalents via EMIT, robust temporal join support). I'm also
> proposing a
> >>> clear distinction of streams and tables, which I think is important,
> but
> >>> which I believe is not the approach most folks have been taking in this
> >>> area. Sorting out these open issues and then having a concise record of
> >> the
> >>> solutions adopted will be important for providing a solid streaming
> >>> experience and teaching folks how to use it.
> >>>
> >>> At any rate, I would much appreciate it if anyone with an interest in
> >> this
> >>> stuff could please take a look and add comments/suggestions/references
> >> to
> >>> related work in flight/etc as appropriate. For now please use
> >>> comments/suggestions, but if you really want to dive in with edit
> access,
> >>> let me know.
> >>>
> >>> The doc: http://s.apache.org/streaming-sql-spec
> >>>
> >>> -Tyler
> >>>
> >>>
> >>>
> >>
>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Fabian Hueske <fh...@gmail.com>]

Thanks for the great write up!

I think this s very good starting point for a detailed discussion about
features, syntax and semantics of streaming SQL.
I'll comment on the document in the next days and describe Flink's current
status, our approaches (or planned approaches) and ask a couple of
questions.

Thanks, Fabian

2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:

> Tyler,
>
> Thanks for this. I am reading the document thoroughly and will give my
> feedback in a day or two.
>
> Julian
>
> > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
> >
> > Thanks for the invitation Tyler. I am sure folks who worked on the
> calcite
> > integration and others would be interested.
> >
> > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> <ta...@google.com.invalid>
> > wrote:
> >
> >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> >> well. If anyone on the Apex side wants to join in on the fun, your input
> >> would be welcomed!
> >>
> >> -Tyler
> >>
> >>
> >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> wrote:
> >>
> >>> Hello Flink, Calcite, and Beam dev lists!
> >>>
> >>> Linked below is the second document I promised way back in April
> >> regarding
> >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> apologies
> >>> for the delay; I thought I was nearly done a while back and then
> temporal
> >>> joins expanded to something much larger than expected).
> >>>
> >>> To repeat what it says in the doc, my hope is that it can serve various
> >>> purposes over it's lifetime:
> >>>
> >>>   -
> >>>   - A discussion ground for ironing out any remaining features
> necessary
> >>>   for supporting robust streaming semantics in Calcite SQL.
> >>>
> >>>   - A rough, high-level source of truth for tracking efforts underway
> in
> >>>   support of this, currently spanning the Calcite, Flink, and Beam
> >> projects.
> >>>
> >>>   - A written specification of the changes that were made, for the sake
> >>>   of understanding the delta after the fact.
> >>>
> >>> The first and third points are, IMO, the most important. AFAIK, there
> are
> >>> a few features missing still that need to be defined (e.g., triggers
> >>> equivalents via EMIT, robust temporal join support). I'm also
> proposing a
> >>> clear distinction of streams and tables, which I think is important,
> but
> >>> which I believe is not the approach most folks have been taking in this
> >>> area. Sorting out these open issues and then having a concise record of
> >> the
> >>> solutions adopted will be important for providing a solid streaming
> >>> experience and teaching folks how to use it.
> >>>
> >>> At any rate, I would much appreciate it if anyone with an interest in
> >> this
> >>> stuff could please take a look and add comments/suggestions/references
> >> to
> >>> related work in flight/etc as appropriate. For now please use
> >>> comments/suggestions, but if you really want to dive in with edit
> access,
> >>> let me know.
> >>>
> >>> The doc: http://s.apache.org/streaming-sql-spec
> >>>
> >>> -Tyler
> >>>
> >>>
> >>>
> >>
>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

Tyler,

Thanks for this. I am reading the document thoroughly and will give my feedback in a day or two.

Julian

> On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com> wrote:
> 
> Thanks for the invitation Tyler. I am sure folks who worked on the calcite
> integration and others would be interested.
> 
> On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
> wrote:
> 
>> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
>> well. If anyone on the Apex side wants to join in on the fun, your input
>> would be welcomed!
>> 
>> -Tyler
>> 
>> 
>> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>> 
>>> Hello Flink, Calcite, and Beam dev lists!
>>> 
>>> Linked below is the second document I promised way back in April
>> regarding
>>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
>>> for the delay; I thought I was nearly done a while back and then temporal
>>> joins expanded to something much larger than expected).
>>> 
>>> To repeat what it says in the doc, my hope is that it can serve various
>>> purposes over it's lifetime:
>>> 
>>>   -
>>>   - A discussion ground for ironing out any remaining features necessary
>>>   for supporting robust streaming semantics in Calcite SQL.
>>> 
>>>   - A rough, high-level source of truth for tracking efforts underway in
>>>   support of this, currently spanning the Calcite, Flink, and Beam
>> projects.
>>> 
>>>   - A written specification of the changes that were made, for the sake
>>>   of understanding the delta after the fact.
>>> 
>>> The first and third points are, IMO, the most important. AFAIK, there are
>>> a few features missing still that need to be defined (e.g., triggers
>>> equivalents via EMIT, robust temporal join support). I'm also proposing a
>>> clear distinction of streams and tables, which I think is important, but
>>> which I believe is not the approach most folks have been taking in this
>>> area. Sorting out these open issues and then having a concise record of
>> the
>>> solutions adopted will be important for providing a solid streaming
>>> experience and teaching folks how to use it.
>>> 
>>> At any rate, I would much appreciate it if anyone with an interest in
>> this
>>> stuff could please take a look and add comments/suggestions/references
>> to
>>> related work in flight/etc as appropriate. For now please use
>>> comments/suggestions, but if you really want to dive in with edit access,
>>> let me know.
>>> 
>>> The doc: http://s.apache.org/streaming-sql-spec
>>> 
>>> -Tyler
>>> 
>>> 
>>> 
>> 

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

Tyler,

Thanks for this. I am reading the document thoroughly and will give my feedback in a day or two.

Julian

> On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com> wrote:
> 
> Thanks for the invitation Tyler. I am sure folks who worked on the calcite
> integration and others would be interested.
> 
> On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
> wrote:
> 
>> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
>> well. If anyone on the Apex side wants to join in on the fun, your input
>> would be welcomed!
>> 
>> -Tyler
>> 
>> 
>> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>> 
>>> Hello Flink, Calcite, and Beam dev lists!
>>> 
>>> Linked below is the second document I promised way back in April
>> regarding
>>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
>>> for the delay; I thought I was nearly done a while back and then temporal
>>> joins expanded to something much larger than expected).
>>> 
>>> To repeat what it says in the doc, my hope is that it can serve various
>>> purposes over it's lifetime:
>>> 
>>>   -
>>>   - A discussion ground for ironing out any remaining features necessary
>>>   for supporting robust streaming semantics in Calcite SQL.
>>> 
>>>   - A rough, high-level source of truth for tracking efforts underway in
>>>   support of this, currently spanning the Calcite, Flink, and Beam
>> projects.
>>> 
>>>   - A written specification of the changes that were made, for the sake
>>>   of understanding the delta after the fact.
>>> 
>>> The first and third points are, IMO, the most important. AFAIK, there are
>>> a few features missing still that need to be defined (e.g., triggers
>>> equivalents via EMIT, robust temporal join support). I'm also proposing a
>>> clear distinction of streams and tables, which I think is important, but
>>> which I believe is not the approach most folks have been taking in this
>>> area. Sorting out these open issues and then having a concise record of
>> the
>>> solutions adopted will be important for providing a solid streaming
>>> experience and teaching folks how to use it.
>>> 
>>> At any rate, I would much appreciate it if anyone with an interest in
>> this
>>> stuff could please take a look and add comments/suggestions/references
>> to
>>> related work in flight/etc as appropriate. For now please use
>>> comments/suggestions, but if you really want to dive in with edit access,
>>> let me know.
>>> 
>>> The doc: http://s.apache.org/streaming-sql-spec
>>> 
>>> -Tyler
>>> 
>>> 
>>> 
>> 

Re: Towards a spec for robust streaming SQL, Part 2

[Julian Hyde <jh...@apache.org>]

Tyler,

Thanks for this. I am reading the document thoroughly and will give my feedback in a day or two.

Julian

> On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <pr...@datatorrent.com> wrote:
> 
> Thanks for the invitation Tyler. I am sure folks who worked on the calcite
> integration and others would be interested.
> 
> On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
> wrote:
> 
>> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
>> well. If anyone on the Apex side wants to join in on the fun, your input
>> would be welcomed!
>> 
>> -Tyler
>> 
>> 
>> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>> 
>>> Hello Flink, Calcite, and Beam dev lists!
>>> 
>>> Linked below is the second document I promised way back in April
>> regarding
>>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
>>> for the delay; I thought I was nearly done a while back and then temporal
>>> joins expanded to something much larger than expected).
>>> 
>>> To repeat what it says in the doc, my hope is that it can serve various
>>> purposes over it's lifetime:
>>> 
>>>   -
>>>   - A discussion ground for ironing out any remaining features necessary
>>>   for supporting robust streaming semantics in Calcite SQL.
>>> 
>>>   - A rough, high-level source of truth for tracking efforts underway in
>>>   support of this, currently spanning the Calcite, Flink, and Beam
>> projects.
>>> 
>>>   - A written specification of the changes that were made, for the sake
>>>   of understanding the delta after the fact.
>>> 
>>> The first and third points are, IMO, the most important. AFAIK, there are
>>> a few features missing still that need to be defined (e.g., triggers
>>> equivalents via EMIT, robust temporal join support). I'm also proposing a
>>> clear distinction of streams and tables, which I think is important, but
>>> which I believe is not the approach most folks have been taking in this
>>> area. Sorting out these open issues and then having a concise record of
>> the
>>> solutions adopted will be important for providing a solid streaming
>>> experience and teaching folks how to use it.
>>> 
>>> At any rate, I would much appreciate it if anyone with an interest in
>> this
>>> stuff could please take a look and add comments/suggestions/references
>> to
>>> related work in flight/etc as appropriate. For now please use
>>> comments/suggestions, but if you really want to dive in with edit access,
>>> let me know.
>>> 
>>> The doc: http://s.apache.org/streaming-sql-spec
>>> 
>>> -Tyler
>>> 
>>> 
>>> 
>> 

Re: Towards a spec for robust streaming SQL, Part 2

[Pramod Immaneni <pr...@datatorrent.com>]

Thanks for the invitation Tyler. I am sure folks who worked on the calcite
integration and others would be interested.

On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
wrote:

> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> well. If anyone on the Apex side wants to join in on the fun, your input
> would be welcomed!
>
> -Tyler
>
>
> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>
> > Hello Flink, Calcite, and Beam dev lists!
> >
> > Linked below is the second document I promised way back in April
> regarding
> > a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> > for the delay; I thought I was nearly done a while back and then temporal
> > joins expanded to something much larger than expected).
> >
> > To repeat what it says in the doc, my hope is that it can serve various
> > purposes over it's lifetime:
> >
> >    -
> >    - A discussion ground for ironing out any remaining features necessary
> >    for supporting robust streaming semantics in Calcite SQL.
> >
> >    - A rough, high-level source of truth for tracking efforts underway in
> >    support of this, currently spanning the Calcite, Flink, and Beam
> projects.
> >
> >    - A written specification of the changes that were made, for the sake
> >    of understanding the delta after the fact.
> >
> > The first and third points are, IMO, the most important. AFAIK, there are
> > a few features missing still that need to be defined (e.g., triggers
> > equivalents via EMIT, robust temporal join support). I'm also proposing a
> > clear distinction of streams and tables, which I think is important, but
> > which I believe is not the approach most folks have been taking in this
> > area. Sorting out these open issues and then having a concise record of
> the
> > solutions adopted will be important for providing a solid streaming
> > experience and teaching folks how to use it.
> >
> > At any rate, I would much appreciate it if anyone with an interest in
> this
> > stuff could please take a look and add comments/suggestions/references
> to
> > related work in flight/etc as appropriate. For now please use
> > comments/suggestions, but if you really want to dive in with edit access,
> > let me know.
> >
> > The doc: http://s.apache.org/streaming-sql-spec
> >
> > -Tyler
> >
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Pramod Immaneni <pr...@datatorrent.com>]

Thanks for the invitation Tyler. I am sure folks who worked on the calcite
integration and others would be interested.

On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
wrote:

> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> well. If anyone on the Apex side wants to join in on the fun, your input
> would be welcomed!
>
> -Tyler
>
>
> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>
> > Hello Flink, Calcite, and Beam dev lists!
> >
> > Linked below is the second document I promised way back in April
> regarding
> > a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> > for the delay; I thought I was nearly done a while back and then temporal
> > joins expanded to something much larger than expected).
> >
> > To repeat what it says in the doc, my hope is that it can serve various
> > purposes over it's lifetime:
> >
> >    -
> >    - A discussion ground for ironing out any remaining features necessary
> >    for supporting robust streaming semantics in Calcite SQL.
> >
> >    - A rough, high-level source of truth for tracking efforts underway in
> >    support of this, currently spanning the Calcite, Flink, and Beam
> projects.
> >
> >    - A written specification of the changes that were made, for the sake
> >    of understanding the delta after the fact.
> >
> > The first and third points are, IMO, the most important. AFAIK, there are
> > a few features missing still that need to be defined (e.g., triggers
> > equivalents via EMIT, robust temporal join support). I'm also proposing a
> > clear distinction of streams and tables, which I think is important, but
> > which I believe is not the approach most folks have been taking in this
> > area. Sorting out these open issues and then having a concise record of
> the
> > solutions adopted will be important for providing a solid streaming
> > experience and teaching folks how to use it.
> >
> > At any rate, I would much appreciate it if anyone with an interest in
> this
> > stuff could please take a look and add comments/suggestions/references
> to
> > related work in flight/etc as appropriate. For now please use
> > comments/suggestions, but if you really want to dive in with edit access,
> > let me know.
> >
> > The doc: http://s.apache.org/streaming-sql-spec
> >
> > -Tyler
> >
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Pramod Immaneni <pr...@datatorrent.com>]

Thanks for the invitation Tyler. I am sure folks who worked on the calcite
integration and others would be interested.

On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <ta...@google.com.invalid>
wrote:

> +dev@apex.apache.org, since I'm told Apex has a Calcite integration as
> well. If anyone on the Apex side wants to join in on the fun, your input
> would be welcomed!
>
> -Tyler
>
>
> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:
>
> > Hello Flink, Calcite, and Beam dev lists!
> >
> > Linked below is the second document I promised way back in April
> regarding
> > a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> > for the delay; I thought I was nearly done a while back and then temporal
> > joins expanded to something much larger than expected).
> >
> > To repeat what it says in the doc, my hope is that it can serve various
> > purposes over it's lifetime:
> >
> >    -
> >    - A discussion ground for ironing out any remaining features necessary
> >    for supporting robust streaming semantics in Calcite SQL.
> >
> >    - A rough, high-level source of truth for tracking efforts underway in
> >    support of this, currently spanning the Calcite, Flink, and Beam
> projects.
> >
> >    - A written specification of the changes that were made, for the sake
> >    of understanding the delta after the fact.
> >
> > The first and third points are, IMO, the most important. AFAIK, there are
> > a few features missing still that need to be defined (e.g., triggers
> > equivalents via EMIT, robust temporal join support). I'm also proposing a
> > clear distinction of streams and tables, which I think is important, but
> > which I believe is not the approach most folks have been taking in this
> > area. Sorting out these open issues and then having a concise record of
> the
> > solutions adopted will be important for providing a solid streaming
> > experience and teaching folks how to use it.
> >
> > At any rate, I would much appreciate it if anyone with an interest in
> this
> > stuff could please take a look and add comments/suggestions/references
> to
> > related work in flight/etc as appropriate. For now please use
> > comments/suggestions, but if you really want to dive in with edit access,
> > let me know.
> >
> > The doc: http://s.apache.org/streaming-sql-spec
> >
> > -Tyler
> >
> >
> >
>

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

+dev@apex.apache.org, since I'm told Apex has a Calcite integration as
well. If anyone on the Apex side wants to join in on the fun, your input
would be welcomed!

-Tyler


On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:

> Hello Flink, Calcite, and Beam dev lists!
>
> Linked below is the second document I promised way back in April regarding
> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> for the delay; I thought I was nearly done a while back and then temporal
> joins expanded to something much larger than expected).
>
> To repeat what it says in the doc, my hope is that it can serve various
> purposes over it's lifetime:
>
>    -
>    - A discussion ground for ironing out any remaining features necessary
>    for supporting robust streaming semantics in Calcite SQL.
>
>    - A rough, high-level source of truth for tracking efforts underway in
>    support of this, currently spanning the Calcite, Flink, and Beam projects.
>
>    - A written specification of the changes that were made, for the sake
>    of understanding the delta after the fact.
>
> The first and third points are, IMO, the most important. AFAIK, there are
> a few features missing still that need to be defined (e.g., triggers
> equivalents via EMIT, robust temporal join support). I'm also proposing a
> clear distinction of streams and tables, which I think is important, but
> which I believe is not the approach most folks have been taking in this
> area. Sorting out these open issues and then having a concise record of the
> solutions adopted will be important for providing a solid streaming
> experience and teaching folks how to use it.
>
> At any rate, I would much appreciate it if anyone with an interest in this
> stuff could please take a look and add comments/suggestions/references to
> related work in flight/etc as appropriate. For now please use
> comments/suggestions, but if you really want to dive in with edit access,
> let me know.
>
> The doc: http://s.apache.org/streaming-sql-spec
>
> -Tyler
>
>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

+dev@apex.apache.org, since I'm told Apex has a Calcite integration as
well. If anyone on the Apex side wants to join in on the fun, your input
would be welcomed!

-Tyler


On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:

> Hello Flink, Calcite, and Beam dev lists!
>
> Linked below is the second document I promised way back in April regarding
> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> for the delay; I thought I was nearly done a while back and then temporal
> joins expanded to something much larger than expected).
>
> To repeat what it says in the doc, my hope is that it can serve various
> purposes over it's lifetime:
>
>    -
>    - A discussion ground for ironing out any remaining features necessary
>    for supporting robust streaming semantics in Calcite SQL.
>
>    - A rough, high-level source of truth for tracking efforts underway in
>    support of this, currently spanning the Calcite, Flink, and Beam projects.
>
>    - A written specification of the changes that were made, for the sake
>    of understanding the delta after the fact.
>
> The first and third points are, IMO, the most important. AFAIK, there are
> a few features missing still that need to be defined (e.g., triggers
> equivalents via EMIT, robust temporal join support). I'm also proposing a
> clear distinction of streams and tables, which I think is important, but
> which I believe is not the approach most folks have been taking in this
> area. Sorting out these open issues and then having a concise record of the
> solutions adopted will be important for providing a solid streaming
> experience and teaching folks how to use it.
>
> At any rate, I would much appreciate it if anyone with an interest in this
> stuff could please take a look and add comments/suggestions/references to
> related work in flight/etc as appropriate. For now please use
> comments/suggestions, but if you really want to dive in with edit access,
> let me know.
>
> The doc: http://s.apache.org/streaming-sql-spec
>
> -Tyler
>
>
>

Re: Towards a spec for robust streaming SQL, Part 2

[Tyler Akidau <ta...@google.com.INVALID>]

+dev@apex.apache.org, since I'm told Apex has a Calcite integration as
well. If anyone on the Apex side wants to join in on the fun, your input
would be welcomed!

-Tyler


On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org> wrote:

> Hello Flink, Calcite, and Beam dev lists!
>
> Linked below is the second document I promised way back in April regarding
> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
> for the delay; I thought I was nearly done a while back and then temporal
> joins expanded to something much larger than expected).
>
> To repeat what it says in the doc, my hope is that it can serve various
> purposes over it's lifetime:
>
>    -
>    - A discussion ground for ironing out any remaining features necessary
>    for supporting robust streaming semantics in Calcite SQL.
>
>    - A rough, high-level source of truth for tracking efforts underway in
>    support of this, currently spanning the Calcite, Flink, and Beam projects.
>
>    - A written specification of the changes that were made, for the sake
>    of understanding the delta after the fact.
>
> The first and third points are, IMO, the most important. AFAIK, there are
> a few features missing still that need to be defined (e.g., triggers
> equivalents via EMIT, robust temporal join support). I'm also proposing a
> clear distinction of streams and tables, which I think is important, but
> which I believe is not the approach most folks have been taking in this
> area. Sorting out these open issues and then having a concise record of the
> solutions adopted will be important for providing a solid streaming
> experience and teaching folks how to use it.
>
> At any rate, I would much appreciate it if anyone with an interest in this
> stuff could please take a look and add comments/suggestions/references to
> related work in flight/etc as appropriate. For now please use
> comments/suggestions, but if you really want to dive in with edit access,
> let me know.
>
> The doc: http://s.apache.org/streaming-sql-spec
>
> -Tyler
>
>
>