You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Fabian Hueske <fh...@gmail.com> on 2017/08/01 13:40:48 UTC
Re: Towards a spec for robust streaming SQL, Part 2
As promised, I went of the document and made some comments.
I also added a bit of information about the current SQL support in Flink
and its internals.
Thanks, Fabian
2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> Hi Tyler,
> Thanks for putting all the efforts into a doc. It is really well written
> and organized.
> I like the most part. The major concern I have is about the "explicit
> trigger". I left a few comments towards this and would like to know what
> the others think about it.
>
> Regards,
> Shaoxuan
>
> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
> > Thanks for the great write up!
> >
> > I think this s very good starting point for a detailed discussion about
> > features, syntax and semantics of streaming SQL.
> > I'll comment on the document in the next days and describe Flink's
> current
> > status, our approaches (or planned approaches) and ask a couple of
> > questions.
> >
> > Thanks, Fabian
> >
> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >
> > > Tyler,
> > >
> > > Thanks for this. I am reading the document thoroughly and will give my
> > > feedback in a day or two.
> > >
> > > Julian
> > >
> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > > wrote:
> > > >
> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> > > calcite
> > > > integration and others would be interested.
> > > >
> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> > > <ta...@google.com.invalid>
> > > > wrote:
> > > >
> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
> > as
> > > >> well. If anyone on the Apex side wants to join in on the fun, your
> > input
> > > >> would be welcomed!
> > > >>
> > > >> -Tyler
> > > >>
> > > >>
> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
> > > wrote:
> > > >>
> > > >>> Hello Flink, Calcite, and Beam dev lists!
> > > >>>
> > > >>> Linked below is the second document I promised way back in April
> > > >> regarding
> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> > > apologies
> > > >>> for the delay; I thought I was nearly done a while back and then
> > > temporal
> > > >>> joins expanded to something much larger than expected).
> > > >>>
> > > >>> To repeat what it says in the doc, my hope is that it can serve
> > various
> > > >>> purposes over it's lifetime:
> > > >>>
> > > >>> -
> > > >>> - A discussion ground for ironing out any remaining features
> > > necessary
> > > >>> for supporting robust streaming semantics in Calcite SQL.
> > > >>>
> > > >>> - A rough, high-level source of truth for tracking efforts
> underway
> > > in
> > > >>> support of this, currently spanning the Calcite, Flink, and Beam
> > > >> projects.
> > > >>>
> > > >>> - A written specification of the changes that were made, for the
> > sake
> > > >>> of understanding the delta after the fact.
> > > >>>
> > > >>> The first and third points are, IMO, the most important. AFAIK,
> there
> > > are
> > > >>> a few features missing still that need to be defined (e.g.,
> triggers
> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> > > proposing a
> > > >>> clear distinction of streams and tables, which I think is
> important,
> > > but
> > > >>> which I believe is not the approach most folks have been taking in
> > this
> > > >>> area. Sorting out these open issues and then having a concise
> record
> > of
> > > >> the
> > > >>> solutions adopted will be important for providing a solid streaming
> > > >>> experience and teaching folks how to use it.
> > > >>>
> > > >>> At any rate, I would much appreciate it if anyone with an interest
> in
> > > >> this
> > > >>> stuff could please take a look and add comments/suggestions/
> > references
> > > >> to
> > > >>> related work in flight/etc as appropriate. For now please use
> > > >>> comments/suggestions, but if you really want to dive in with edit
> > > access,
> > > >>> let me know.
> > > >>>
> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> > > >>>
> > > >>> -Tyler
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Tyler Akidau <ta...@google.com.INVALID>.
Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.
-Tyler
On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:
> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>> -
> >> > > >>> - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>> for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>> - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>> support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>> - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>> of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Tyler Akidau <ta...@google.com.INVALID>.
Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.
-Tyler
On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:
> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>> -
> >> > > >>> - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>> for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>> - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>> support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>> - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>> of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Tyler Akidau <ta...@google.com.INVALID>.
Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more
I'd like to respond to, but I'm out of time for tonight, so more tomorrow.
-Tyler
On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <jh...@apache.org> wrote:
> I have problems with a couple of the axioms: that a SQL object is
> either a table or a stream, but not both; and that a query is bounded
> if and only if it contains no unbounded streams.
>
> I don't have problems with other axioms, such that a query is either
> bounded or unbounded. And I haven't looked in detail at triggering
> semantics; I don't think there will be major issues, but let's clear
> up the 2 problems above first.
>
> I have added a section "Julian’s thoughts on the fundamentals" to the
> end of the document.
>
> Julian
>
>
> On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> > As promised, I went of the document and made some comments.
> > I also added a bit of information about the current SQL support in Flink
> > and its internals.
> >
> > Thanks, Fabian
> >
> > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
> >
> >> Hi Tyler,
> >> Thanks for putting all the efforts into a doc. It is really well written
> >> and organized.
> >> I like the most part. The major concern I have is about the "explicit
> >> trigger". I left a few comments towards this and would like to know what
> >> the others think about it.
> >>
> >> Regards,
> >> Shaoxuan
> >>
> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com>
> wrote:
> >>
> >> > Thanks for the great write up!
> >> >
> >> > I think this s very good starting point for a detailed discussion
> about
> >> > features, syntax and semantics of streaming SQL.
> >> > I'll comment on the document in the next days and describe Flink's
> >> current
> >> > status, our approaches (or planned approaches) and ask a couple of
> >> > questions.
> >> >
> >> > Thanks, Fabian
> >> >
> >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
> >> >
> >> > > Tyler,
> >> > >
> >> > > Thanks for this. I am reading the document thoroughly and will give
> my
> >> > > feedback in a day or two.
> >> > >
> >> > > Julian
> >> > >
> >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > > wrote:
> >> > > >
> >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
> >> > > calcite
> >> > > > integration and others would be interested.
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
> >> > > <ta...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite
> integration
> >> > as
> >> > > >> well. If anyone on the Apex side wants to join in on the fun,
> your
> >> > input
> >> > > >> would be welcomed!
> >> > > >>
> >> > > >> -Tyler
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <takidau@apache.org
> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Hello Flink, Calcite, and Beam dev lists!
> >> > > >>>
> >> > > >>> Linked below is the second document I promised way back in April
> >> > > >> regarding
> >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
> >> > > apologies
> >> > > >>> for the delay; I thought I was nearly done a while back and then
> >> > > temporal
> >> > > >>> joins expanded to something much larger than expected).
> >> > > >>>
> >> > > >>> To repeat what it says in the doc, my hope is that it can serve
> >> > various
> >> > > >>> purposes over it's lifetime:
> >> > > >>>
> >> > > >>> -
> >> > > >>> - A discussion ground for ironing out any remaining features
> >> > > necessary
> >> > > >>> for supporting robust streaming semantics in Calcite SQL.
> >> > > >>>
> >> > > >>> - A rough, high-level source of truth for tracking efforts
> >> underway
> >> > > in
> >> > > >>> support of this, currently spanning the Calcite, Flink, and
> Beam
> >> > > >> projects.
> >> > > >>>
> >> > > >>> - A written specification of the changes that were made, for
> the
> >> > sake
> >> > > >>> of understanding the delta after the fact.
> >> > > >>>
> >> > > >>> The first and third points are, IMO, the most important. AFAIK,
> >> there
> >> > > are
> >> > > >>> a few features missing still that need to be defined (e.g.,
> >> triggers
> >> > > >>> equivalents via EMIT, robust temporal join support). I'm also
> >> > > proposing a
> >> > > >>> clear distinction of streams and tables, which I think is
> >> important,
> >> > > but
> >> > > >>> which I believe is not the approach most folks have been taking
> in
> >> > this
> >> > > >>> area. Sorting out these open issues and then having a concise
> >> record
> >> > of
> >> > > >> the
> >> > > >>> solutions adopted will be important for providing a solid
> streaming
> >> > > >>> experience and teaching folks how to use it.
> >> > > >>>
> >> > > >>> At any rate, I would much appreciate it if anyone with an
> interest
> >> in
> >> > > >> this
> >> > > >>> stuff could please take a look and add comments/suggestions/
> >> > references
> >> > > >> to
> >> > > >>> related work in flight/etc as appropriate. For now please use
> >> > > >>> comments/suggestions, but if you really want to dive in with
> edit
> >> > > access,
> >> > > >>> let me know.
> >> > > >>>
> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec
> >> > > >>>
> >> > > >>> -Tyler
> >> > > >>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Julian Hyde <jh...@apache.org>.
I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.
I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.
I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.
Julian
On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>> -
>> > > >>> - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>> for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>> - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>> support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>> - A written specification of the changes that were made, for the
>> > sake
>> > > >>> of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Julian Hyde <jh...@apache.org>.
I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.
I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.
I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.
Julian
On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>> -
>> > > >>> - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>> for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>> - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>> support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>> - A written specification of the changes that were made, for the
>> > sake
>> > > >>> of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
Re: Towards a spec for robust streaming SQL, Part 2
Posted by Julian Hyde <jh...@apache.org>.
I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded
if and only if it contains no unbounded streams.
I don't have problems with other axioms, such that a query is either
bounded or unbounded. And I haven't looked in detail at triggering
semantics; I don't think there will be major issues, but let's clear
up the 2 problems above first.
I have added a section "Julian’s thoughts on the fundamentals" to the
end of the document.
Julian
On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <fh...@gmail.com> wrote:
> As promised, I went of the document and made some comments.
> I also added a bit of information about the current SQL support in Flink
> and its internals.
>
> Thanks, Fabian
>
> 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <ws...@gmail.com>:
>
>> Hi Tyler,
>> Thanks for putting all the efforts into a doc. It is really well written
>> and organized.
>> I like the most part. The major concern I have is about the "explicit
>> trigger". I left a few comments towards this and would like to know what
>> the others think about it.
>>
>> Regards,
>> Shaoxuan
>>
>> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>> > Thanks for the great write up!
>> >
>> > I think this s very good starting point for a detailed discussion about
>> > features, syntax and semantics of streaming SQL.
>> > I'll comment on the document in the next days and describe Flink's
>> current
>> > status, our approaches (or planned approaches) and ask a couple of
>> > questions.
>> >
>> > Thanks, Fabian
>> >
>> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <jh...@apache.org>:
>> >
>> > > Tyler,
>> > >
>> > > Thanks for this. I am reading the document thoroughly and will give my
>> > > feedback in a day or two.
>> > >
>> > > Julian
>> > >
>> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > Thanks for the invitation Tyler. I am sure folks who worked on the
>> > > calcite
>> > > > integration and others would be interested.
>> > > >
>> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau
>> > > <ta...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > >> +dev@apex.apache.org, since I'm told Apex has a Calcite integration
>> > as
>> > > >> well. If anyone on the Apex side wants to join in on the fun, your
>> > input
>> > > >> would be welcomed!
>> > > >>
>> > > >> -Tyler
>> > > >>
>> > > >>
>> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <ta...@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> Hello Flink, Calcite, and Beam dev lists!
>> > > >>>
>> > > >>> Linked below is the second document I promised way back in April
>> > > >> regarding
>> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (&
>> > > apologies
>> > > >>> for the delay; I thought I was nearly done a while back and then
>> > > temporal
>> > > >>> joins expanded to something much larger than expected).
>> > > >>>
>> > > >>> To repeat what it says in the doc, my hope is that it can serve
>> > various
>> > > >>> purposes over it's lifetime:
>> > > >>>
>> > > >>> -
>> > > >>> - A discussion ground for ironing out any remaining features
>> > > necessary
>> > > >>> for supporting robust streaming semantics in Calcite SQL.
>> > > >>>
>> > > >>> - A rough, high-level source of truth for tracking efforts
>> underway
>> > > in
>> > > >>> support of this, currently spanning the Calcite, Flink, and Beam
>> > > >> projects.
>> > > >>>
>> > > >>> - A written specification of the changes that were made, for the
>> > sake
>> > > >>> of understanding the delta after the fact.
>> > > >>>
>> > > >>> The first and third points are, IMO, the most important. AFAIK,
>> there
>> > > are
>> > > >>> a few features missing still that need to be defined (e.g.,
>> triggers
>> > > >>> equivalents via EMIT, robust temporal join support). I'm also
>> > > proposing a
>> > > >>> clear distinction of streams and tables, which I think is
>> important,
>> > > but
>> > > >>> which I believe is not the approach most folks have been taking in
>> > this
>> > > >>> area. Sorting out these open issues and then having a concise
>> record
>> > of
>> > > >> the
>> > > >>> solutions adopted will be important for providing a solid streaming
>> > > >>> experience and teaching folks how to use it.
>> > > >>>
>> > > >>> At any rate, I would much appreciate it if anyone with an interest
>> in
>> > > >> this
>> > > >>> stuff could please take a look and add comments/suggestions/
>> > references
>> > > >> to
>> > > >>> related work in flight/etc as appropriate. For now please use
>> > > >>> comments/suggestions, but if you really want to dive in with edit
>> > > access,
>> > > >>> let me know.
>> > > >>>
>> > > >>> The doc: http://s.apache.org/streaming-sql-spec
>> > > >>>
>> > > >>> -Tyler
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>