You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Vignesh <vi...@gmail.com> on 2023/01/29 22:20:12 UTC
Declaring monotonic columns in schema for streaming queries.
Hi,
This page <https://calcite.apache.org/docs/stream.html> describes streaming
queries in calcite.
Streaming operators, such as tumbling window require a monotonic column. It
says that we need to declare monotonic columns in the schema.
Is there a way to do it in the create table statement? Flink SQL does it
with a "WATERMARK FOR rowtime_column"
<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table>
syntax in the create table, but that seems to be flink specific.
Thanks,
Vignesh.
Re: Declaring monotonic columns in schema for streaming queries.
Posted by Benchao Li <li...@apache.org>.
Yes, I think you are in the right direction.
Vignesh <vi...@gmail.com> 于2023年2月8日周三 05:31写道:
> Hi,
>
> To do this, I will need to define an implementation of
> CalciteCatalogReader, override the getTable method and return an
> implementation of PreparingTable that returns the right thing for
> getMonotonicity
> <
> https://javadoc.io/static/org.apache.calcite/calcite-core/1.11.0/org/apache/calcite/sql/validate/SqlValidatorTable.html#getMonotonicity(java.lang.String)
> >and
> supportsModality
> <
> https://javadoc.io/static/org.apache.calcite/calcite-core/1.11.0/org/apache/calcite/sql/validate/SqlValidatorTable.html#supportsModality(org.apache.calcite.sql.validate.SqlModality)
> >
> .
> Is above the right approach? Would there be more steps beyond above?
>
> Thanks,
> Vignesh.
>
> On Mon, Jan 30, 2023 at 10:50 PM Vignesh <vi...@gmail.com> wrote:
>
> > Thank you, I will try that.
> >
> > On Sun, Jan 29, 2023, 9:01 PM Benchao Li <li...@apache.org> wrote:
> >
> >> Hi Vignesh,
> >>
> >> AFAIK, there is no such way in Calcite to declare it with DDL. However,
> >> you
> >> can provide it via the statistic for your table, you can take a look at
> >> the
> >> tests[1] in Calcite how we did it.
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/calcite/blob/ae228f64347cc7620d28eff04f5869583c6bf9d5/testkit/src/main/java/org/apache/calcite/test/catalog/MockCatalogReaderSimple.java#L238-L245
> >>
> >> Vignesh <vi...@gmail.com> 于2023年1月30日周一 06:21写道:
> >>
> >> > Hi,
> >> >
> >> > This page <https://calcite.apache.org/docs/stream.html> describes
> >> > streaming
> >> > queries in calcite.
> >> > Streaming operators, such as tumbling window require a monotonic
> >> column. It
> >> > says that we need to declare monotonic columns in the schema.
> >> >
> >> > Is there a way to do it in the create table statement? Flink SQL does
> it
> >> > with a "WATERMARK FOR rowtime_column"
> >> > <
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table
> >> > >
> >> > syntax in the create table, but that seems to be flink specific.
> >> >
> >> >
> >> > Thanks,
> >> > Vignesh.
> >> >
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
> >>
> >
>
--
Best,
Benchao Li
Re: Declaring monotonic columns in schema for streaming queries.
Posted by Vignesh <vi...@gmail.com>.
Hi,
To do this, I will need to define an implementation of
CalciteCatalogReader, override the getTable method and return an
implementation of PreparingTable that returns the right thing for
getMonotonicity
<https://javadoc.io/static/org.apache.calcite/calcite-core/1.11.0/org/apache/calcite/sql/validate/SqlValidatorTable.html#getMonotonicity(java.lang.String)>and
supportsModality
<https://javadoc.io/static/org.apache.calcite/calcite-core/1.11.0/org/apache/calcite/sql/validate/SqlValidatorTable.html#supportsModality(org.apache.calcite.sql.validate.SqlModality)>
.
Is above the right approach? Would there be more steps beyond above?
Thanks,
Vignesh.
On Mon, Jan 30, 2023 at 10:50 PM Vignesh <vi...@gmail.com> wrote:
> Thank you, I will try that.
>
> On Sun, Jan 29, 2023, 9:01 PM Benchao Li <li...@apache.org> wrote:
>
>> Hi Vignesh,
>>
>> AFAIK, there is no such way in Calcite to declare it with DDL. However,
>> you
>> can provide it via the statistic for your table, you can take a look at
>> the
>> tests[1] in Calcite how we did it.
>>
>> [1]
>>
>> https://github.com/apache/calcite/blob/ae228f64347cc7620d28eff04f5869583c6bf9d5/testkit/src/main/java/org/apache/calcite/test/catalog/MockCatalogReaderSimple.java#L238-L245
>>
>> Vignesh <vi...@gmail.com> 于2023年1月30日周一 06:21写道:
>>
>> > Hi,
>> >
>> > This page <https://calcite.apache.org/docs/stream.html> describes
>> > streaming
>> > queries in calcite.
>> > Streaming operators, such as tumbling window require a monotonic
>> column. It
>> > says that we need to declare monotonic columns in the schema.
>> >
>> > Is there a way to do it in the create table statement? Flink SQL does it
>> > with a "WATERMARK FOR rowtime_column"
>> > <
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table
>> > >
>> > syntax in the create table, but that seems to be flink specific.
>> >
>> >
>> > Thanks,
>> > Vignesh.
>> >
>>
>>
>> --
>>
>> Best,
>> Benchao Li
>>
>
Re: Declaring monotonic columns in schema for streaming queries.
Posted by Vignesh <vi...@gmail.com>.
Thank you, I will try that.
On Sun, Jan 29, 2023, 9:01 PM Benchao Li <li...@apache.org> wrote:
> Hi Vignesh,
>
> AFAIK, there is no such way in Calcite to declare it with DDL. However, you
> can provide it via the statistic for your table, you can take a look at the
> tests[1] in Calcite how we did it.
>
> [1]
>
> https://github.com/apache/calcite/blob/ae228f64347cc7620d28eff04f5869583c6bf9d5/testkit/src/main/java/org/apache/calcite/test/catalog/MockCatalogReaderSimple.java#L238-L245
>
> Vignesh <vi...@gmail.com> 于2023年1月30日周一 06:21写道:
>
> > Hi,
> >
> > This page <https://calcite.apache.org/docs/stream.html> describes
> > streaming
> > queries in calcite.
> > Streaming operators, such as tumbling window require a monotonic column.
> It
> > says that we need to declare monotonic columns in the schema.
> >
> > Is there a way to do it in the create table statement? Flink SQL does it
> > with a "WATERMARK FOR rowtime_column"
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table
> > >
> > syntax in the create table, but that seems to be flink specific.
> >
> >
> > Thanks,
> > Vignesh.
> >
>
>
> --
>
> Best,
> Benchao Li
>
Re: Declaring monotonic columns in schema for streaming queries.
Posted by Benchao Li <li...@apache.org>.
Hi Vignesh,
AFAIK, there is no such way in Calcite to declare it with DDL. However, you
can provide it via the statistic for your table, you can take a look at the
tests[1] in Calcite how we did it.
[1]
https://github.com/apache/calcite/blob/ae228f64347cc7620d28eff04f5869583c6bf9d5/testkit/src/main/java/org/apache/calcite/test/catalog/MockCatalogReaderSimple.java#L238-L245
Vignesh <vi...@gmail.com> 于2023年1月30日周一 06:21写道:
> Hi,
>
> This page <https://calcite.apache.org/docs/stream.html> describes
> streaming
> queries in calcite.
> Streaming operators, such as tumbling window require a monotonic column. It
> says that we need to declare monotonic columns in the schema.
>
> Is there a way to do it in the create table statement? Flink SQL does it
> with a "WATERMARK FOR rowtime_column"
> <
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-table
> >
> syntax in the create table, but that seems to be flink specific.
>
>
> Thanks,
> Vignesh.
>
--
Best,
Benchao Li