You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Dyana Rose <dy...@salecycle.com> on 2017/12/29 19:54:25 UTC
Dynamic SessionWindow gaps
I have a use case for non-static Session Window gaps.
For example, given a stream of IoT events, each device type could have a
different gap, and that gap could change while sessions are in flight.
I didn't want to have to run a stream processor for each potential gap
length, not to mention the headache of dealing with changing gaps, so I've
implemented a version of SessionWindows that has one major change; in the
assignWindows method it passes the element to a method to extract the
correct sessionTimeout. (current Flink method for reference:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59
)
Preliminary tests show this working as required and I can't be the only
person with this type of use case for session windows.
Will an issue and PR to add this functionality to the SessionWindow classes
be welcome?
Dyana
Re: Dynamic SessionWindow gaps
Posted by Dyana Rose <dy...@salecycle.com>.
Sounds good, I'm on holiday at the moment. I'll get the ball rolling with a
raised issue when I get in.
Dyana
On 2018-01-02 13:45, Aljoscha Krettek <a....@apache.org> wrote:
> Hi,>
>
> This sounds good! Could you please open a Jira Issue for this and then
open a PR against that Jira issue?>
>
> Btw, I would suggest to implement this as a new type of assigner,
something like DynamicSessionWindows.>
>
> Best,>
> Aljoscha>
>
> > On 29. Dec 2017, at 20:54, Dyana Rose <dy...@salecycle.com> wrote:>
> > >
> > I have a use case for non-static Session Window gaps.>
> > >
> > For example, given a stream of IoT events, each device type could have
a>
> > different gap, and that gap could change while sessions are in flight.>
> > >
> > I didn't want to have to run a stream processor for each potential gap>
> > length, not to mention the headache of dealing with changing gaps, so
I've>
> > implemented a version of SessionWindows that has one major change; in
the>
> > assignWindows method it passes the element to a method to extract the>
> > correct sessionTimeout. (current Flink method for reference:>
> >
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59>
> > )>
> > >
> > Preliminary tests show this working as required and I can't be the
only>
> > person with this type of use case for session windows.>
> > >
> > Will an issue and PR to add this functionality to the SessionWindow
classes>
> > be welcome?>
> > >
> > Dyana>
>
>
Re: Dynamic SessionWindow gaps
Posted by "dyana.rose@salecycle.com" <dy...@salecycle.com>.
I've put up some implementation information and a link to the working branch for comment while I get the documentation written up:
https://issues.apache.org/jira/browse/FLINK-8384
Thanks,
Dyana
On 2018-01-02 13:54, jincheng sun <su...@gmail.com> wrote:
> Hi, Dyana dynamic gap is very useful feature, Agree with the suggestion of
> Aljoscha, looking forward the PR.
>
> Best, Jincheng
>
> 2018-01-02 21:45 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
>
> > Hi,
> >
> > This sounds good! Could you please open a Jira Issue for this and then
> > open a PR against that Jira issue?
> >
> > Btw, I would suggest to implement this as a new type of assigner,
> > something like DynamicSessionWindows.
> >
> > Best,
> > Aljoscha
> >
> > > On 29. Dec 2017, at 20:54, Dyana Rose <dy...@salecycle.com> wrote:
> > >
> > > I have a use case for non-static Session Window gaps.
> > >
> > > For example, given a stream of IoT events, each device type could have a
> > > different gap, and that gap could change while sessions are in flight.
> > >
> > > I didn't want to have to run a stream processor for each potential gap
> > > length, not to mention the headache of dealing with changing gaps, so
> > I've
> > > implemented a version of SessionWindows that has one major change; in the
> > > assignWindows method it passes the element to a method to extract the
> > > correct sessionTimeout. (current Flink method for reference:
> > > https://github.com/apache/flink/blob/master/flink-
> > streaming-java/src/main/java/org/apache/flink/streaming/
> > api/windowing/assigners/EventTimeSessionWindows.java#L59
> > > )
> > >
> > > Preliminary tests show this working as required and I can't be the only
> > > person with this type of use case for session windows.
> > >
> > > Will an issue and PR to add this functionality to the SessionWindow
> > classes
> > > be welcome?
> > >
> > > Dyana
> >
> >
>
Re: Dynamic SessionWindow gaps
Posted by jincheng sun <su...@gmail.com>.
Hi, Dyana dynamic gap is very useful feature, Agree with the suggestion of
Aljoscha, looking forward the PR.
Best, Jincheng
2018-01-02 21:45 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
> Hi,
>
> This sounds good! Could you please open a Jira Issue for this and then
> open a PR against that Jira issue?
>
> Btw, I would suggest to implement this as a new type of assigner,
> something like DynamicSessionWindows.
>
> Best,
> Aljoscha
>
> > On 29. Dec 2017, at 20:54, Dyana Rose <dy...@salecycle.com> wrote:
> >
> > I have a use case for non-static Session Window gaps.
> >
> > For example, given a stream of IoT events, each device type could have a
> > different gap, and that gap could change while sessions are in flight.
> >
> > I didn't want to have to run a stream processor for each potential gap
> > length, not to mention the headache of dealing with changing gaps, so
> I've
> > implemented a version of SessionWindows that has one major change; in the
> > assignWindows method it passes the element to a method to extract the
> > correct sessionTimeout. (current Flink method for reference:
> > https://github.com/apache/flink/blob/master/flink-
> streaming-java/src/main/java/org/apache/flink/streaming/
> api/windowing/assigners/EventTimeSessionWindows.java#L59
> > )
> >
> > Preliminary tests show this working as required and I can't be the only
> > person with this type of use case for session windows.
> >
> > Will an issue and PR to add this functionality to the SessionWindow
> classes
> > be welcome?
> >
> > Dyana
>
>
Re: Dynamic SessionWindow gaps
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
This sounds good! Could you please open a Jira Issue for this and then open a PR against that Jira issue?
Btw, I would suggest to implement this as a new type of assigner, something like DynamicSessionWindows.
Best,
Aljoscha
> On 29. Dec 2017, at 20:54, Dyana Rose <dy...@salecycle.com> wrote:
>
> I have a use case for non-static Session Window gaps.
>
> For example, given a stream of IoT events, each device type could have a
> different gap, and that gap could change while sessions are in flight.
>
> I didn't want to have to run a stream processor for each potential gap
> length, not to mention the headache of dealing with changing gaps, so I've
> implemented a version of SessionWindows that has one major change; in the
> assignWindows method it passes the element to a method to extract the
> correct sessionTimeout. (current Flink method for reference:
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59
> )
>
> Preliminary tests show this working as required and I can't be the only
> person with this type of use case for session windows.
>
> Will an issue and PR to add this functionality to the SessionWindow classes
> be welcome?
>
> Dyana