You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Seth Wiesman <sw...@mediamath.com> on 2017/11/06 13:56:11 UTC

DataStream to Table Api idioms

Hi,

I am experimenting with rewriting some of my datastream projects with the table api and I had some questions on how to express certain idioms. I am using 1.4-SNAPSHOT.


1)       Can I express allowed lateness?

2)       Can I use a custom trigger? More specifically, I have a 24hr window but would like to receive partial results say every hour.

3)       Do window join time intervals have to be constant or can they depend on row attributes. I am running campaigns that have start and end dates and so I would like my join window to be that interval.

Thank you,

Seth Wiesman

Re: DataStream to Table Api idioms

Posted by Seth Wiesman <sw...@mediamath.com>.
Not a problem, thanks for the quick feedback.

https://issues.apache.org/jira/browse/FLINK-7999

Seth Wiesman

From: Fabian Hueske <fh...@gmail.com>
Date: Monday, November 6, 2017 at 9:14 AM
To: Seth Wiesman <sw...@mediamath.com>
Cc: user <us...@flink.apache.org>
Subject: Re: DataStream to Table Api idioms

Hi Seth,

I think the Table API is not there yet to address you use case.

1. Allowed lateness cannot be configured but it is on the list of features that we plan to add in the future.
2. Custom triggers are not supported. We are planning to add an option to support your use case (early firing and updates).
3. The window joins that will be release with 1.4 require constant boundaries (left.time > right.time - X and left.time < right.time + Y).
Variable join window boundaries have not been considered yet and would be quite tricky to implement. Would you mind opening a JIRA issue for this feature?

Best, Fabian

2017-11-06 14:56 GMT+01:00 Seth Wiesman <sw...@mediamath.com>>:
Hi,

I am experimenting with rewriting some of my datastream projects with the table api and I had some questions on how to express certain idioms. I am using 1.4-SNAPSHOT.


1)       Can I express allowed lateness?

2)       Can I use a custom trigger? More specifically, I have a 24hr window but would like to receive partial results say every hour.

3)       Do window join time intervals have to be constant or can they depend on row attributes. I am running campaigns that have start and end dates and so I would like my join window to be that interval.

Thank you,

Seth Wiesman

Re: DataStream to Table Api idioms

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Seth,

I think the Table API is not there yet to address you use case.

1. Allowed lateness cannot be configured but it is on the list of features
that we plan to add in the future.
2. Custom triggers are not supported. We are planning to add an option to
support your use case (early firing and updates).
3. The window joins that will be release with 1.4 require constant
boundaries (left.time > right.time - X and left.time < right.time + Y).
Variable join window boundaries have not been considered yet and would be
quite tricky to implement. Would you mind opening a JIRA issue for this
feature?

Best, Fabian

2017-11-06 14:56 GMT+01:00 Seth Wiesman <sw...@mediamath.com>:

> Hi,
>
>
>
> I am experimenting with rewriting some of my datastream projects with the
> table api and I had some questions on how to express certain idioms. I am
> using 1.4-SNAPSHOT.
>
>
>
> 1)       Can I express allowed lateness?
>
> 2)       Can I use a custom trigger? More specifically, I have a 24hr
> window but would like to receive partial results say every hour.
>
> 3)       Do window join time intervals have to be constant or can they
> depend on row attributes. I am running campaigns that have start and end
> dates and so I would like my join window to be that interval.
>
>
>
> Thank you,
>
>
>
> Seth Wiesman
>
>
>