You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Rui Wang <ru...@google.com.INVALID> on 2019/08/20 01:59:33 UTC
TVF windowing and EMIT syntax support in Calcite
Hi Calcite community,
I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
windowing and EMIT syntax for materialization control.
I tried to search dev@calcite and Calcite JIRA but didn't find those
support above. So is Calcite already support TVF windowing and EMIT syntax
somehow, or it's a future work?
[1]: https://arxiv.org/pdf/1905.12133.pdf
-Rui
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Rui Wang <am...@apache.org>.
Thanks Danny.
In current enumerable implementation, explicit watermark is not needed
because data is already known so we can think watermark already advances to
positive infinity, and every RelNode can close all windows immediately.
Thus [1] make tumbling works for batch workload.
I created [2] to track work related to watermark. After [1] is done, I can
start to think about watermark in calcite.
[1]: https://github.com/apache/calcite/pull/1587
[2]: https://issues.apache.org/jira/browse/CALCITE-3509
-Rui
On Fri, Nov 15, 2019 at 9:40 PM Danny Chan <yu...@gmail.com> wrote:
> No, there is no watermark syntax or implementation in Calcite, watermark
> is needed for the google data stream model, and was implemented well in
> Apache Flink, you can take a reference.(The syntax, the watermark
> strategies translation).
>
>
> Best,
> Danny Chan
> 在 2019年11月16日 +0800 AM1:44，Rui Wang <am...@apache.org>，写道：
> > Is watermark is ever discussed in calcite? In order to make enumerable
> > implementation useful, we will need watermark support. Watermark will be
> > just a value that represents a time and keep advancing to +inf. It
> > generates from sources and propagate to all RelNodes. Emit syntax will
> > require such tool to determine when window will close and then emit data.
> >
> >
> > -Rui
> >
> > On Tue, Aug 20, 2019 at 11:51 PM Danny Chan <yu...@gmail.com>
> wrote:
> >
> > > Sorry, I have no permission to grant you the JIRA contributor
> permission,
> > > you can send another mail asking for this.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年8月21日 +0800 AM11:58，Rui Wang <ru...@google.com.invalid>，写道：
> > > > Thanks Danny. I have filed a sub-task under it: CALCITE-3272.
> > > >
> > > > Can you grant me contributor permission so I can assign it to myself?
> > > >
> > > >
> > > > -Rui
> > > >
> > > > On Tue, Aug 20, 2019 at 8:24 PM Danny Chan <yu...@gmail.com>
> wrote:
> > > >
> > > > > Thanks for bring up this topic, Rui Wang ~
> > > > >
> > > > > I have fired an Umbrella issue [1] about this topic where we can
> create
> > > > > sub-tasks.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/CALCITE-3271
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> > > > > > Hi Calcite community,
> > > > > >
> > > > > > I have seen [1] is published, which proposes TVF style
> > > TUMBLE/HOP/SESSION
> > > > > > windowing and EMIT syntax for materialization control.
> > > > > >
> > > > > > I tried to search dev@calcite and Calcite JIRA but didn't find
> those
> > > > > > support above. So is Calcite already support TVF windowing and
> EMIT
> > > > > syntax
> > > > > > somehow, or it's a future work?
> > > > > >
> > > > > >
> > > > > >
> > > > > > [1]: https://arxiv.org/pdf/1905.12133.pdf
> > > > > >
> > > > > >
> > > > > > -Rui
> > > > >
> > >
>
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Danny Chan <yu...@gmail.com>.
No, there is no watermark syntax or implementation in Calcite, watermark is needed for the google data stream model, and was implemented well in Apache Flink, you can take a reference.(The syntax, the watermark strategies translation).
Best,
Danny Chan
在 2019年11月16日 +0800 AM1:44，Rui Wang <am...@apache.org>，写道：
> Is watermark is ever discussed in calcite? In order to make enumerable
> implementation useful, we will need watermark support. Watermark will be
> just a value that represents a time and keep advancing to +inf. It
> generates from sources and propagate to all RelNodes. Emit syntax will
> require such tool to determine when window will close and then emit data.
>
>
> -Rui
>
> On Tue, Aug 20, 2019 at 11:51 PM Danny Chan <yu...@gmail.com> wrote:
>
> > Sorry, I have no permission to grant you the JIRA contributor permission,
> > you can send another mail asking for this.
> >
> > Best,
> > Danny Chan
> > 在 2019年8月21日 +0800 AM11:58，Rui Wang <ru...@google.com.invalid>，写道：
> > > Thanks Danny. I have filed a sub-task under it: CALCITE-3272.
> > >
> > > Can you grant me contributor permission so I can assign it to myself?
> > >
> > >
> > > -Rui
> > >
> > > On Tue, Aug 20, 2019 at 8:24 PM Danny Chan <yu...@gmail.com> wrote:
> > >
> > > > Thanks for bring up this topic, Rui Wang ~
> > > >
> > > > I have fired an Umbrella issue [1] about this topic where we can create
> > > > sub-tasks.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/CALCITE-3271
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> > > > > Hi Calcite community,
> > > > >
> > > > > I have seen [1] is published, which proposes TVF style
> > TUMBLE/HOP/SESSION
> > > > > windowing and EMIT syntax for materialization control.
> > > > >
> > > > > I tried to search dev@calcite and Calcite JIRA but didn't find those
> > > > > support above. So is Calcite already support TVF windowing and EMIT
> > > > syntax
> > > > > somehow, or it's a future work?
> > > > >
> > > > >
> > > > >
> > > > > [1]: https://arxiv.org/pdf/1905.12133.pdf
> > > > >
> > > > >
> > > > > -Rui
> > > >
> >
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Rui Wang <am...@apache.org>.
Is watermark is ever discussed in calcite? In order to make enumerable
implementation useful, we will need watermark support. Watermark will be
just a value that represents a time and keep advancing to +inf. It
generates from sources and propagate to all RelNodes. Emit syntax will
require such tool to determine when window will close and then emit data.
-Rui
On Tue, Aug 20, 2019 at 11:51 PM Danny Chan <yu...@gmail.com> wrote:
> Sorry, I have no permission to grant you the JIRA contributor permission,
> you can send another mail asking for this.
>
> Best,
> Danny Chan
> 在 2019年8月21日 +0800 AM11:58，Rui Wang <ru...@google.com.invalid>，写道：
> > Thanks Danny. I have filed a sub-task under it: CALCITE-3272.
> >
> > Can you grant me contributor permission so I can assign it to myself?
> >
> >
> > -Rui
> >
> > On Tue, Aug 20, 2019 at 8:24 PM Danny Chan <yu...@gmail.com> wrote:
> >
> > > Thanks for bring up this topic, Rui Wang ~
> > >
> > > I have fired an Umbrella issue [1] about this topic where we can create
> > > sub-tasks.
> > >
> > > [1] https://issues.apache.org/jira/browse/CALCITE-3271
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> > > > Hi Calcite community,
> > > >
> > > > I have seen [1] is published, which proposes TVF style
> TUMBLE/HOP/SESSION
> > > > windowing and EMIT syntax for materialization control.
> > > >
> > > > I tried to search dev@calcite and Calcite JIRA but didn't find those
> > > > support above. So is Calcite already support TVF windowing and EMIT
> > > syntax
> > > > somehow, or it's a future work?
> > > >
> > > >
> > > >
> > > > [1]: https://arxiv.org/pdf/1905.12133.pdf
> > > >
> > > >
> > > > -Rui
> > >
>
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Danny Chan <yu...@gmail.com>.
Sorry, I have no permission to grant you the JIRA contributor permission, you can send another mail asking for this.
Best,
Danny Chan
在 2019年8月21日 +0800 AM11:58，Rui Wang <ru...@google.com.invalid>，写道：
> Thanks Danny. I have filed a sub-task under it: CALCITE-3272.
>
> Can you grant me contributor permission so I can assign it to myself?
>
>
> -Rui
>
> On Tue, Aug 20, 2019 at 8:24 PM Danny Chan <yu...@gmail.com> wrote:
>
> > Thanks for bring up this topic, Rui Wang ~
> >
> > I have fired an Umbrella issue [1] about this topic where we can create
> > sub-tasks.
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-3271
> >
> > Best,
> > Danny Chan
> > 在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> > > Hi Calcite community,
> > >
> > > I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
> > > windowing and EMIT syntax for materialization control.
> > >
> > > I tried to search dev@calcite and Calcite JIRA but didn't find those
> > > support above. So is Calcite already support TVF windowing and EMIT
> > syntax
> > > somehow, or it's a future work?
> > >
> > >
> > >
> > > [1]: https://arxiv.org/pdf/1905.12133.pdf
> > >
> > >
> > > -Rui
> >
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Rui Wang <ru...@google.com.INVALID>.
Thanks Danny. I have filed a sub-task under it: CALCITE-3272.
Can you grant me contributor permission so I can assign it to myself?
-Rui
On Tue, Aug 20, 2019 at 8:24 PM Danny Chan <yu...@gmail.com> wrote:
> Thanks for bring up this topic, Rui Wang ~
>
> I have fired an Umbrella issue [1] about this topic where we can create
> sub-tasks.
>
> [1] https://issues.apache.org/jira/browse/CALCITE-3271
>
> Best,
> Danny Chan
> 在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> > Hi Calcite community,
> >
> > I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
> > windowing and EMIT syntax for materialization control.
> >
> > I tried to search dev@calcite and Calcite JIRA but didn't find those
> > support above. So is Calcite already support TVF windowing and EMIT
> syntax
> > somehow, or it's a future work?
> >
> >
> >
> > [1]: https://arxiv.org/pdf/1905.12133.pdf
> >
> >
> > -Rui
>
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Danny Chan <yu...@gmail.com>.
Thanks for bring up this topic, Rui Wang ~
I have fired an Umbrella issue [1] about this topic where we can create sub-tasks.
[1] https://issues.apache.org/jira/browse/CALCITE-3271
Best,
Danny Chan
在 2019年8月20日 +0800 AM9:59，Rui Wang <ru...@google.com.invalid>，写道：
> Hi Calcite community,
>
> I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
> windowing and EMIT syntax for materialization control.
>
> I tried to search dev@calcite and Calcite JIRA but didn't find those
> support above. So is Calcite already support TVF windowing and EMIT syntax
> somehow, or it's a future work?
>
>
>
> [1]: https://arxiv.org/pdf/1905.12133.pdf
>
>
> -Rui
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Rui Wang <ru...@google.com.INVALID>.
Thanks Julian. If it's nobody who is currently working on it, I would like
to work on it.
I would like to start from adding windowing table functions because I am
trying to support it in Apache Beam SQL. Like you said, how will watermark
work with these table functions would be an interesting area, which I could
explore in BeamSQL.
-Rui
On Mon, Aug 19, 2019 at 8:55 PM Julian Hyde <jh...@apache.org> wrote:
> Calcite has not implemented the syntax in that paper. I would support an
> effort to add it (unsurprising, since I am a co-author of that paper).
>
> EMIT STREAM is equivalent to the current SELECT STREAM syntax.
>
> There is no equivalent in current Calcite of the EMIT AFTER WATERMARK, or
> EMIT STREAM AFTER DELAY.
>
> HOP, TUMBLE and SESSION are supported in Calcite’s SQL parser, but
> following the paper would be replaced with a table function call. We could
> need to add HOP, TUMBLE and SESSION table functions. We would also need to
> make the system aware of how watermarks flow through these table functions
> (an area that the paper does not go into).
>
> Julian
>
> > On Aug 19, 2019, at 6:59 PM, Rui Wang <ru...@google.com.INVALID> wrote:
> >
> > Hi Calcite community,
> >
> > I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
> > windowing and EMIT syntax for materialization control.
> >
> > I tried to search dev@calcite and Calcite JIRA but didn't find those
> > support above. So is Calcite already support TVF windowing and EMIT
> syntax
> > somehow, or it's a future work?
> >
> >
> >
> > [1]: https://arxiv.org/pdf/1905.12133.pdf
> >
> >
> > -Rui
>
>
Re: TVF windowing and EMIT syntax support in Calcite
Posted by Julian Hyde <jh...@apache.org>.
Calcite has not implemented the syntax in that paper. I would support an effort to add it (unsurprising, since I am a co-author of that paper).
EMIT STREAM is equivalent to the current SELECT STREAM syntax.
There is no equivalent in current Calcite of the EMIT AFTER WATERMARK, or EMIT STREAM AFTER DELAY.
HOP, TUMBLE and SESSION are supported in Calcite’s SQL parser, but following the paper would be replaced with a table function call. We could need to add HOP, TUMBLE and SESSION table functions. We would also need to make the system aware of how watermarks flow through these table functions (an area that the paper does not go into).
Julian
> On Aug 19, 2019, at 6:59 PM, Rui Wang <ru...@google.com.INVALID> wrote:
>
> Hi Calcite community,
>
> I have seen [1] is published, which proposes TVF style TUMBLE/HOP/SESSION
> windowing and EMIT syntax for materialization control.
>
> I tried to search dev@calcite and Calcite JIRA but didn't find those
> support above. So is Calcite already support TVF windowing and EMIT syntax
> somehow, or it's a future work?
>
>
>
> [1]: https://arxiv.org/pdf/1905.12133.pdf
>
>
> -Rui