You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Reza Rokni <re...@google.com> on 2019/05/29 01:27:51 UTC
Timer support in Flink
Hi Flink experts,
I am getting ready to push a PR around a utility class for timeseries join
left.timestamp match to closest right.timestamp where right.timestamp <=
left.timestamp.
It makes very heavy use of Event.Time timers and has to do some manual DoFn
cache work to get around some O(heavy) issues. Wanted to test things
against Flink: In the capability matrix we have "~" for Timer support in
Flink:
https://beam.apache.org/documentation/runners/capability-matrix/
Is that page outdated, if not what are the areas that still need to be
addressed please?
Cheers
Reza
--
This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
Re: Timer support in Flink
Posted by Reza Rokni <re...@google.com>.
I changed the font size and the wording a little, its in this PR:
https://github.com/apache/beam/pull/8773
I started to mess around with moving it around to better position etc.. But
then I quickly remembered why no one lets me near web pages css etc... :-)
Anyone else want to enhance that PR please go for it!
Cheers
Reza
On Tue, 4 Jun 2019 at 15:42, Robert Bradshaw <ro...@google.com> wrote:
> One issue with the fully expanded version is that it's so large it's
> hard to read.
>
> I think it would be useful to make the ~ entries (at least) clickable
> or with hover tool tips. It would be nice to be able to expand columns
> individually as well.
>
> On Tue, Jun 4, 2019 at 7:20 AM Melissa Pashniak <me...@google.com>
> wrote:
> >
> >
> > Yeah, people's eyes likely jump to the big "What is being computed?"
> header first and skip the small font "expand details" (that's what my eyes
> did anyway!) Even just moving the expand/collapse to be AFTER the header of
> the table (or down to the next line) and making the font bigger might help
> a lot. And maybe making the text more explicit: "Click to expand for more
> details".
> >
> > I'm traveling right now so can't take an in-depth look, but this might
> be doable by changing the order of things in [1] and the font size in [2].
> I'll add this info to the JIRA also.
> >
> > [1]
> https://github.com/apache/beam/blame/master/website/src/_includes/capability-matrix.md#L18
> > [2]
> https://github.com/apache/beam/blob/master/website/src/_sass/capability-matrix.scss#L130
> >
> >
> > On Mon, Jun 3, 2019 at 2:15 AM Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> Good point. I think I discovered the detailed view when I made changes
> >> to the source code. Classic tunnel-vision problem :)
> >>
> >> On 30.05.19 12:57, Reza Rokni wrote:
> >> > :-)
> >> >
> >> > https://issues.apache.org/jira/browse/BEAM-7456
> >> >
> >> > On Thu, 30 May 2019 at 18:41, Alex Van Boxel <alex@vanboxel.be
> >> > <ma...@vanboxel.be>> wrote:
> >> >
> >> > Oh... you can expand the matrix. Never saw that, this could indeed
> >> > be better. So it isn't you.
> >> >
> >> > _/
> >> > _/ Alex Van Boxel
> >> >
> >> >
> >> > On Thu, May 30, 2019 at 12:24 PM Reza Rokni <rez@google.com
> >> > <ma...@google.com>> wrote:
> >> >
> >> > PS, until it was just pointed out to me by Max, I had missed
> the
> >> > (expand details) clickable link in the capability matrix.
> >> >
> >> > Probably just me, but do others think it's also easy to miss?
> If
> >> > yes I will raise a Jira for it....
> >> >
> >> > On Wed, 29 May 2019 at 19:52, Reza Rokni <rez@google.com
> >> > <ma...@google.com>> wrote:
> >> >
> >> > Thanx Max!
> >> >
> >> > Reza
> >> >
> >> > On Wed, 29 May 2019, 16:38 Maximilian Michels,
> >> > <mxm@apache.org <ma...@apache.org>> wrote:
> >> >
> >> > Hi Reza,
> >> >
> >> > The detailed view of the capability matrix states:
> "The
> >> > Flink Runner
> >> > supports timers in non-merging windows."
> >> >
> >> > That is still the case. Other than that, timers should
> >> > be working fine.
> >> >
> >> > > It makes very heavy use of Event.Time timers and
> has
> >> > to do some manual DoFn cache work to get around some
> >> > O(heavy) issues.
> >> >
> >> > If you are running on Flink 1.5, timer deletion
> suffers
> >> > from O(n)
> >> > complexity which has been fixed in newer versions.
> >> >
> >> > Cheers,
> >> > Max
> >> >
> >> > On 29.05.19 03:27, Reza Rokni wrote:
> >> > > Hi Flink experts,
> >> > >
> >> > > I am getting ready to push a PR around a utility
> >> > class for timeseries join
> >> > >
> >> > > left.timestamp match to closest right.timestamp
> where
> >> > right.timestamp <=
> >> > > left.timestamp.
> >> > >
> >> > > It makes very heavy use of Event.Time timers and
> has
> >> > to do some manual
> >> > > DoFn cache work to get around some O(heavy) issues.
> >> > Wanted to test
> >> > > things against Flink: In the capability matrix we
> >> > have "~" for Timer
> >> > > support in Flink:
> >> > >
> >> > >
> >> >
> https://beam.apache.org/documentation/runners/capability-matrix/
> >> > >
> >> > > Is that page outdated, if not what are the areas
> that
> >> > still need to be
> >> > > addressed please?
> >> > >
> >> > > Cheers
> >> > >
> >> > > Reza
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > This email may be confidential and privileged. If
> you
> >> > received this
> >> > > communication by mistake, please don't forward it
> to
> >> > anyone else, please
> >> > > erase all copies and attachments, and please let me
> >> > know that it has
> >> > > gone to the wrong person.
> >> > >
> >> > > The above terms reflect a potential business
> >> > arrangement, are provided
> >> > > solely as a basis for further discussion, and are
> not
> >> > intended to be and
> >> > > do not constitute a legally binding obligation. No
> >> > legally binding
> >> > > obligations will be created, implied, or inferred
> >> > until an agreement in
> >> > > final form is executed in writing by all parties
> >> > involved.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > This email may be confidential and privileged. If you received
> >> > this communication by mistake, please don't forward it to
> anyone
> >> > else, please erase all copies and attachments, and please let
> me
> >> > know that it has gone to the wrong person.
> >> >
> >> > The above terms reflect a potential business arrangement, are
> >> > provided solely as a basis for further discussion, and are not
> >> > intended to be and do not constitute a legally binding
> >> > obligation. No legally binding obligations will be created,
> >> > implied, or inferred until an agreement in final form is
> >> > executed in writing by all parties involved.
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > This email may be confidential and privileged. If you received this
> >> > communication by mistake, please don't forward it to anyone else,
> please
> >> > erase all copies and attachments, and please let me know that it has
> >> > gone to the wrong person.
> >> >
> >> > The above terms reflect a potential business arrangement, are provided
> >> > solely as a basis for further discussion, and are not intended to be
> and
> >> > do not constitute a legally binding obligation. No legally binding
> >> > obligations will be created, implied, or inferred until an agreement
> in
> >> > final form is executed in writing by all parties involved.
> >> >
>
--
This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
Re: Timer support in Flink
Posted by Robert Bradshaw <ro...@google.com>.
One issue with the fully expanded version is that it's so large it's
hard to read.
I think it would be useful to make the ~ entries (at least) clickable
or with hover tool tips. It would be nice to be able to expand columns
individually as well.
On Tue, Jun 4, 2019 at 7:20 AM Melissa Pashniak <me...@google.com> wrote:
>
>
> Yeah, people's eyes likely jump to the big "What is being computed?" header first and skip the small font "expand details" (that's what my eyes did anyway!) Even just moving the expand/collapse to be AFTER the header of the table (or down to the next line) and making the font bigger might help a lot. And maybe making the text more explicit: "Click to expand for more details".
>
> I'm traveling right now so can't take an in-depth look, but this might be doable by changing the order of things in [1] and the font size in [2]. I'll add this info to the JIRA also.
>
> [1] https://github.com/apache/beam/blame/master/website/src/_includes/capability-matrix.md#L18
> [2] https://github.com/apache/beam/blob/master/website/src/_sass/capability-matrix.scss#L130
>
>
> On Mon, Jun 3, 2019 at 2:15 AM Maximilian Michels <mx...@apache.org> wrote:
>>
>> Good point. I think I discovered the detailed view when I made changes
>> to the source code. Classic tunnel-vision problem :)
>>
>> On 30.05.19 12:57, Reza Rokni wrote:
>> > :-)
>> >
>> > https://issues.apache.org/jira/browse/BEAM-7456
>> >
>> > On Thu, 30 May 2019 at 18:41, Alex Van Boxel <alex@vanboxel.be
>> > <ma...@vanboxel.be>> wrote:
>> >
>> > Oh... you can expand the matrix. Never saw that, this could indeed
>> > be better. So it isn't you.
>> >
>> > _/
>> > _/ Alex Van Boxel
>> >
>> >
>> > On Thu, May 30, 2019 at 12:24 PM Reza Rokni <rez@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > PS, until it was just pointed out to me by Max, I had missed the
>> > (expand details) clickable link in the capability matrix.
>> >
>> > Probably just me, but do others think it's also easy to miss? If
>> > yes I will raise a Jira for it....
>> >
>> > On Wed, 29 May 2019 at 19:52, Reza Rokni <rez@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > Thanx Max!
>> >
>> > Reza
>> >
>> > On Wed, 29 May 2019, 16:38 Maximilian Michels,
>> > <mxm@apache.org <ma...@apache.org>> wrote:
>> >
>> > Hi Reza,
>> >
>> > The detailed view of the capability matrix states: "The
>> > Flink Runner
>> > supports timers in non-merging windows."
>> >
>> > That is still the case. Other than that, timers should
>> > be working fine.
>> >
>> > > It makes very heavy use of Event.Time timers and has
>> > to do some manual DoFn cache work to get around some
>> > O(heavy) issues.
>> >
>> > If you are running on Flink 1.5, timer deletion suffers
>> > from O(n)
>> > complexity which has been fixed in newer versions.
>> >
>> > Cheers,
>> > Max
>> >
>> > On 29.05.19 03:27, Reza Rokni wrote:
>> > > Hi Flink experts,
>> > >
>> > > I am getting ready to push a PR around a utility
>> > class for timeseries join
>> > >
>> > > left.timestamp match to closest right.timestamp where
>> > right.timestamp <=
>> > > left.timestamp.
>> > >
>> > > It makes very heavy use of Event.Time timers and has
>> > to do some manual
>> > > DoFn cache work to get around some O(heavy) issues.
>> > Wanted to test
>> > > things against Flink: In the capability matrix we
>> > have "~" for Timer
>> > > support in Flink:
>> > >
>> > >
>> > https://beam.apache.org/documentation/runners/capability-matrix/
>> > >
>> > > Is that page outdated, if not what are the areas that
>> > still need to be
>> > > addressed please?
>> > >
>> > > Cheers
>> > >
>> > > Reza
>> > >
>> > >
>> > > --
>> > >
>> > > This email may be confidential and privileged. If you
>> > received this
>> > > communication by mistake, please don't forward it to
>> > anyone else, please
>> > > erase all copies and attachments, and please let me
>> > know that it has
>> > > gone to the wrong person.
>> > >
>> > > The above terms reflect a potential business
>> > arrangement, are provided
>> > > solely as a basis for further discussion, and are not
>> > intended to be and
>> > > do not constitute a legally binding obligation. No
>> > legally binding
>> > > obligations will be created, implied, or inferred
>> > until an agreement in
>> > > final form is executed in writing by all parties
>> > involved.
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > This email may be confidential and privileged. If you received
>> > this communication by mistake, please don't forward it to anyone
>> > else, please erase all copies and attachments, and please let me
>> > know that it has gone to the wrong person.
>> >
>> > The above terms reflect a potential business arrangement, are
>> > provided solely as a basis for further discussion, and are not
>> > intended to be and do not constitute a legally binding
>> > obligation. No legally binding obligations will be created,
>> > implied, or inferred until an agreement in final form is
>> > executed in writing by all parties involved.
>> >
>> >
>> >
>> > --
>> >
>> > This email may be confidential and privileged. If you received this
>> > communication by mistake, please don't forward it to anyone else, please
>> > erase all copies and attachments, and please let me know that it has
>> > gone to the wrong person.
>> >
>> > The above terms reflect a potential business arrangement, are provided
>> > solely as a basis for further discussion, and are not intended to be and
>> > do not constitute a legally binding obligation. No legally binding
>> > obligations will be created, implied, or inferred until an agreement in
>> > final form is executed in writing by all parties involved.
>> >
Re: Timer support in Flink
Posted by Melissa Pashniak <me...@google.com>.
Yeah, people's eyes likely jump to the big "What is being computed?" header
first and skip the small font "expand details" (that's what my eyes did
anyway!) Even just moving the expand/collapse to be AFTER the header of the
table (or down to the next line) and making the font bigger might help a
lot. And maybe making the text more explicit: "Click to expand for more
details".
I'm traveling right now so can't take an in-depth look, but this might be
doable by changing the order of things in [1] and the font size in [2].
I'll add this info to the JIRA also.
[1]
https://github.com/apache/beam/blame/master/website/src/_includes/capability-matrix.md#L18
[2]
https://github.com/apache/beam/blob/master/website/src/_sass/capability-matrix.scss#L130
On Mon, Jun 3, 2019 at 2:15 AM Maximilian Michels <mx...@apache.org> wrote:
> Good point. I think I discovered the detailed view when I made changes
> to the source code. Classic tunnel-vision problem :)
>
> On 30.05.19 12:57, Reza Rokni wrote:
> > :-)
> >
> > https://issues.apache.org/jira/browse/BEAM-7456
> >
> > On Thu, 30 May 2019 at 18:41, Alex Van Boxel <alex@vanboxel.be
> > <ma...@vanboxel.be>> wrote:
> >
> > Oh... you can expand the matrix. Never saw that, this could indeed
> > be better. So it isn't you.
> >
> > _/
> > _/ Alex Van Boxel
> >
> >
> > On Thu, May 30, 2019 at 12:24 PM Reza Rokni <rez@google.com
> > <ma...@google.com>> wrote:
> >
> > PS, until it was just pointed out to me by Max, I had missed the
> > (expand details) clickable link in the capability matrix.
> >
> > Probably just me, but do others think it's also easy to miss? If
> > yes I will raise a Jira for it....
> >
> > On Wed, 29 May 2019 at 19:52, Reza Rokni <rez@google.com
> > <ma...@google.com>> wrote:
> >
> > Thanx Max!
> >
> > Reza
> >
> > On Wed, 29 May 2019, 16:38 Maximilian Michels,
> > <mxm@apache.org <ma...@apache.org>> wrote:
> >
> > Hi Reza,
> >
> > The detailed view of the capability matrix states: "The
> > Flink Runner
> > supports timers in non-merging windows."
> >
> > That is still the case. Other than that, timers should
> > be working fine.
> >
> > > It makes very heavy use of Event.Time timers and has
> > to do some manual DoFn cache work to get around some
> > O(heavy) issues.
> >
> > If you are running on Flink 1.5, timer deletion suffers
> > from O(n)
> > complexity which has been fixed in newer versions.
> >
> > Cheers,
> > Max
> >
> > On 29.05.19 03:27, Reza Rokni wrote:
> > > Hi Flink experts,
> > >
> > > I am getting ready to push a PR around a utility
> > class for timeseries join
> > >
> > > left.timestamp match to closest right.timestamp where
> > right.timestamp <=
> > > left.timestamp.
> > >
> > > It makes very heavy use of Event.Time timers and has
> > to do some manual
> > > DoFn cache work to get around some O(heavy) issues.
> > Wanted to test
> > > things against Flink: In the capability matrix we
> > have "~" for Timer
> > > support in Flink:
> > >
> > >
> >
> https://beam.apache.org/documentation/runners/capability-matrix/
> > >
> > > Is that page outdated, if not what are the areas that
> > still need to be
> > > addressed please?
> > >
> > > Cheers
> > >
> > > Reza
> > >
> > >
> > > --
> > >
> > > This email may be confidential and privileged. If you
> > received this
> > > communication by mistake, please don't forward it to
> > anyone else, please
> > > erase all copies and attachments, and please let me
> > know that it has
> > > gone to the wrong person.
> > >
> > > The above terms reflect a potential business
> > arrangement, are provided
> > > solely as a basis for further discussion, and are not
> > intended to be and
> > > do not constitute a legally binding obligation. No
> > legally binding
> > > obligations will be created, implied, or inferred
> > until an agreement in
> > > final form is executed in writing by all parties
> > involved.
> > >
> >
> >
> >
> > --
> >
> > This email may be confidential and privileged. If you received
> > this communication by mistake, please don't forward it to anyone
> > else, please erase all copies and attachments, and please let me
> > know that it has gone to the wrong person.
> >
> > The above terms reflect a potential business arrangement, are
> > provided solely as a basis for further discussion, and are not
> > intended to be and do not constitute a legally binding
> > obligation. No legally binding obligations will be created,
> > implied, or inferred until an agreement in final form is
> > executed in writing by all parties involved.
> >
> >
> >
> > --
> >
> > This email may be confidential and privileged. If you received this
> > communication by mistake, please don't forward it to anyone else, please
> > erase all copies and attachments, and please let me know that it has
> > gone to the wrong person.
> >
> > The above terms reflect a potential business arrangement, are provided
> > solely as a basis for further discussion, and are not intended to be and
> > do not constitute a legally binding obligation. No legally binding
> > obligations will be created, implied, or inferred until an agreement in
> > final form is executed in writing by all parties involved.
> >
>
Re: Timer support in Flink
Posted by Maximilian Michels <mx...@apache.org>.
Good point. I think I discovered the detailed view when I made changes
to the source code. Classic tunnel-vision problem :)
On 30.05.19 12:57, Reza Rokni wrote:
> :-)
>
> https://issues.apache.org/jira/browse/BEAM-7456
>
> On Thu, 30 May 2019 at 18:41, Alex Van Boxel <alex@vanboxel.be
> <ma...@vanboxel.be>> wrote:
>
> Oh... you can expand the matrix. Never saw that, this could indeed
> be better. So it isn't you.
>
> _/
> _/ Alex Van Boxel
>
>
> On Thu, May 30, 2019 at 12:24 PM Reza Rokni <rez@google.com
> <ma...@google.com>> wrote:
>
> PS, until it was just pointed out to me by Max, I had missed the
> (expand details) clickable link in the capability matrix.
>
> Probably just me, but do others think it's also easy to miss? If
> yes I will raise a Jira for it....
>
> On Wed, 29 May 2019 at 19:52, Reza Rokni <rez@google.com
> <ma...@google.com>> wrote:
>
> Thanx Max!
>
> Reza
>
> On Wed, 29 May 2019, 16:38 Maximilian Michels,
> <mxm@apache.org <ma...@apache.org>> wrote:
>
> Hi Reza,
>
> The detailed view of the capability matrix states: "The
> Flink Runner
> supports timers in non-merging windows."
>
> That is still the case. Other than that, timers should
> be working fine.
>
> > It makes very heavy use of Event.Time timers and has
> to do some manual DoFn cache work to get around some
> O(heavy) issues.
>
> If you are running on Flink 1.5, timer deletion suffers
> from O(n)
> complexity which has been fixed in newer versions.
>
> Cheers,
> Max
>
> On 29.05.19 03:27, Reza Rokni wrote:
> > Hi Flink experts,
> >
> > I am getting ready to push a PR around a utility
> class for timeseries join
> >
> > left.timestamp match to closest right.timestamp where
> right.timestamp <=
> > left.timestamp.
> >
> > It makes very heavy use of Event.Time timers and has
> to do some manual
> > DoFn cache work to get around some O(heavy) issues.
> Wanted to test
> > things against Flink: In the capability matrix we
> have "~" for Timer
> > support in Flink:
> >
> >
> https://beam.apache.org/documentation/runners/capability-matrix/
> >
> > Is that page outdated, if not what are the areas that
> still need to be
> > addressed please?
> >
> > Cheers
> >
> > Reza
> >
> >
> > --
> >
> > This email may be confidential and privileged. If you
> received this
> > communication by mistake, please don't forward it to
> anyone else, please
> > erase all copies and attachments, and please let me
> know that it has
> > gone to the wrong person.
> >
> > The above terms reflect a potential business
> arrangement, are provided
> > solely as a basis for further discussion, and are not
> intended to be and
> > do not constitute a legally binding obligation. No
> legally binding
> > obligations will be created, implied, or inferred
> until an agreement in
> > final form is executed in writing by all parties
> involved.
> >
>
>
>
> --
>
> This email may be confidential and privileged. If you received
> this communication by mistake, please don't forward it to anyone
> else, please erase all copies and attachments, and please let me
> know that it has gone to the wrong person.
>
> The above terms reflect a potential business arrangement, are
> provided solely as a basis for further discussion, and are not
> intended to be and do not constitute a legally binding
> obligation. No legally binding obligations will be created,
> implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>
>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has
> gone to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and
> do not constitute a legally binding obligation. No legally binding
> obligations will be created, implied, or inferred until an agreement in
> final form is executed in writing by all parties involved.
>
Re: Timer support in Flink
Posted by Reza Rokni <re...@google.com>.
:-)
https://issues.apache.org/jira/browse/BEAM-7456
On Thu, 30 May 2019 at 18:41, Alex Van Boxel <al...@vanboxel.be> wrote:
> Oh... you can expand the matrix. Never saw that, this could indeed be
> better. So it isn't you.
>
> _/
> _/ Alex Van Boxel
>
>
> On Thu, May 30, 2019 at 12:24 PM Reza Rokni <re...@google.com> wrote:
>
>> PS, until it was just pointed out to me by Max, I had missed the (expand
>> details) clickable link in the capability matrix.
>>
>> Probably just me, but do others think it's also easy to miss? If yes I
>> will raise a Jira for it....
>>
>> On Wed, 29 May 2019 at 19:52, Reza Rokni <re...@google.com> wrote:
>>
>>> Thanx Max!
>>>
>>> Reza
>>>
>>> On Wed, 29 May 2019, 16:38 Maximilian Michels, <mx...@apache.org> wrote:
>>>
>>>> Hi Reza,
>>>>
>>>> The detailed view of the capability matrix states: "The Flink Runner
>>>> supports timers in non-merging windows."
>>>>
>>>> That is still the case. Other than that, timers should be working fine.
>>>>
>>>> > It makes very heavy use of Event.Time timers and has to do some
>>>> manual DoFn cache work to get around some O(heavy) issues.
>>>>
>>>> If you are running on Flink 1.5, timer deletion suffers from O(n)
>>>> complexity which has been fixed in newer versions.
>>>>
>>>> Cheers,
>>>> Max
>>>>
>>>> On 29.05.19 03:27, Reza Rokni wrote:
>>>> > Hi Flink experts,
>>>> >
>>>> > I am getting ready to push a PR around a utility class for
>>>> timeseries join
>>>> >
>>>> > left.timestamp match to closest right.timestamp where right.timestamp
>>>> <=
>>>> > left.timestamp.
>>>> >
>>>> > It makes very heavy use of Event.Time timers and has to do some
>>>> manual
>>>> > DoFn cache work to get around some O(heavy) issues. Wanted to test
>>>> > things against Flink: In the capability matrix we have "~" for Timer
>>>> > support in Flink:
>>>> >
>>>> > https://beam.apache.org/documentation/runners/capability-matrix/
>>>> >
>>>> > Is that page outdated, if not what are the areas that still need to
>>>> be
>>>> > addressed please?
>>>> >
>>>> > Cheers
>>>> >
>>>> > Reza
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > This email may be confidential and privileged. If you received this
>>>> > communication by mistake, please don't forward it to anyone else,
>>>> please
>>>> > erase all copies and attachments, and please let me know that it has
>>>> > gone to the wrong person.
>>>> >
>>>> > The above terms reflect a potential business arrangement, are
>>>> provided
>>>> > solely as a basis for further discussion, and are not intended to be
>>>> and
>>>> > do not constitute a legally binding obligation. No legally binding
>>>> > obligations will be created, implied, or inferred until an agreement
>>>> in
>>>> > final form is executed in writing by all parties involved.
>>>> >
>>>>
>>>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>>
>> The above terms reflect a potential business arrangement, are provided
>> solely as a basis for further discussion, and are not intended to be and do
>> not constitute a legally binding obligation. No legally binding obligations
>> will be created, implied, or inferred until an agreement in final form is
>> executed in writing by all parties involved.
>>
>
--
This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
Re: Timer support in Flink
Posted by Alex Van Boxel <al...@vanboxel.be>.
Oh... you can expand the matrix. Never saw that, this could indeed be
better. So it isn't you.
_/
_/ Alex Van Boxel
On Thu, May 30, 2019 at 12:24 PM Reza Rokni <re...@google.com> wrote:
> PS, until it was just pointed out to me by Max, I had missed the (expand
> details) clickable link in the capability matrix.
>
> Probably just me, but do others think it's also easy to miss? If yes I
> will raise a Jira for it....
>
> On Wed, 29 May 2019 at 19:52, Reza Rokni <re...@google.com> wrote:
>
>> Thanx Max!
>>
>> Reza
>>
>> On Wed, 29 May 2019, 16:38 Maximilian Michels, <mx...@apache.org> wrote:
>>
>>> Hi Reza,
>>>
>>> The detailed view of the capability matrix states: "The Flink Runner
>>> supports timers in non-merging windows."
>>>
>>> That is still the case. Other than that, timers should be working fine.
>>>
>>> > It makes very heavy use of Event.Time timers and has to do some manual
>>> DoFn cache work to get around some O(heavy) issues.
>>>
>>> If you are running on Flink 1.5, timer deletion suffers from O(n)
>>> complexity which has been fixed in newer versions.
>>>
>>> Cheers,
>>> Max
>>>
>>> On 29.05.19 03:27, Reza Rokni wrote:
>>> > Hi Flink experts,
>>> >
>>> > I am getting ready to push a PR around a utility class for
>>> timeseries join
>>> >
>>> > left.timestamp match to closest right.timestamp where right.timestamp
>>> <=
>>> > left.timestamp.
>>> >
>>> > It makes very heavy use of Event.Time timers and has to do some manual
>>> > DoFn cache work to get around some O(heavy) issues. Wanted to test
>>> > things against Flink: In the capability matrix we have "~" for Timer
>>> > support in Flink:
>>> >
>>> > https://beam.apache.org/documentation/runners/capability-matrix/
>>> >
>>> > Is that page outdated, if not what are the areas that still need to be
>>> > addressed please?
>>> >
>>> > Cheers
>>> >
>>> > Reza
>>> >
>>> >
>>> > --
>>> >
>>> > This email may be confidential and privileged. If you received this
>>> > communication by mistake, please don't forward it to anyone else,
>>> please
>>> > erase all copies and attachments, and please let me know that it has
>>> > gone to the wrong person.
>>> >
>>> > The above terms reflect a potential business arrangement, are provided
>>> > solely as a basis for further discussion, and are not intended to be
>>> and
>>> > do not constitute a legally binding obligation. No legally binding
>>> > obligations will be created, implied, or inferred until an agreement
>>> in
>>> > final form is executed in writing by all parties involved.
>>> >
>>>
>>
>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
Re: Timer support in Flink
Posted by Reza Rokni <re...@google.com>.
PS, until it was just pointed out to me by Max, I had missed the (expand
details) clickable link in the capability matrix.
Probably just me, but do others think it's also easy to miss? If yes I will
raise a Jira for it....
On Wed, 29 May 2019 at 19:52, Reza Rokni <re...@google.com> wrote:
> Thanx Max!
>
> Reza
>
> On Wed, 29 May 2019, 16:38 Maximilian Michels, <mx...@apache.org> wrote:
>
>> Hi Reza,
>>
>> The detailed view of the capability matrix states: "The Flink Runner
>> supports timers in non-merging windows."
>>
>> That is still the case. Other than that, timers should be working fine.
>>
>> > It makes very heavy use of Event.Time timers and has to do some manual
>> DoFn cache work to get around some O(heavy) issues.
>>
>> If you are running on Flink 1.5, timer deletion suffers from O(n)
>> complexity which has been fixed in newer versions.
>>
>> Cheers,
>> Max
>>
>> On 29.05.19 03:27, Reza Rokni wrote:
>> > Hi Flink experts,
>> >
>> > I am getting ready to push a PR around a utility class for
>> timeseries join
>> >
>> > left.timestamp match to closest right.timestamp where right.timestamp
>> <=
>> > left.timestamp.
>> >
>> > It makes very heavy use of Event.Time timers and has to do some manual
>> > DoFn cache work to get around some O(heavy) issues. Wanted to test
>> > things against Flink: In the capability matrix we have "~" for Timer
>> > support in Flink:
>> >
>> > https://beam.apache.org/documentation/runners/capability-matrix/
>> >
>> > Is that page outdated, if not what are the areas that still need to be
>> > addressed please?
>> >
>> > Cheers
>> >
>> > Reza
>> >
>> >
>> > --
>> >
>> > This email may be confidential and privileged. If you received this
>> > communication by mistake, please don't forward it to anyone else,
>> please
>> > erase all copies and attachments, and please let me know that it has
>> > gone to the wrong person.
>> >
>> > The above terms reflect a potential business arrangement, are provided
>> > solely as a basis for further discussion, and are not intended to be
>> and
>> > do not constitute a legally binding obligation. No legally binding
>> > obligations will be created, implied, or inferred until an agreement in
>> > final form is executed in writing by all parties involved.
>> >
>>
>
--
This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
Re: Timer support in Flink
Posted by Reza Rokni <re...@google.com>.
Thanx Max!
Reza
On Wed, 29 May 2019, 16:38 Maximilian Michels, <mx...@apache.org> wrote:
> Hi Reza,
>
> The detailed view of the capability matrix states: "The Flink Runner
> supports timers in non-merging windows."
>
> That is still the case. Other than that, timers should be working fine.
>
> > It makes very heavy use of Event.Time timers and has to do some manual
> DoFn cache work to get around some O(heavy) issues.
>
> If you are running on Flink 1.5, timer deletion suffers from O(n)
> complexity which has been fixed in newer versions.
>
> Cheers,
> Max
>
> On 29.05.19 03:27, Reza Rokni wrote:
> > Hi Flink experts,
> >
> > I am getting ready to push a PR around a utility class for
> timeseries join
> >
> > left.timestamp match to closest right.timestamp where right.timestamp <=
> > left.timestamp.
> >
> > It makes very heavy use of Event.Time timers and has to do some manual
> > DoFn cache work to get around some O(heavy) issues. Wanted to test
> > things against Flink: In the capability matrix we have "~" for Timer
> > support in Flink:
> >
> > https://beam.apache.org/documentation/runners/capability-matrix/
> >
> > Is that page outdated, if not what are the areas that still need to be
> > addressed please?
> >
> > Cheers
> >
> > Reza
> >
> >
> > --
> >
> > This email may be confidential and privileged. If you received this
> > communication by mistake, please don't forward it to anyone else, please
> > erase all copies and attachments, and please let me know that it has
> > gone to the wrong person.
> >
> > The above terms reflect a potential business arrangement, are provided
> > solely as a basis for further discussion, and are not intended to be and
> > do not constitute a legally binding obligation. No legally binding
> > obligations will be created, implied, or inferred until an agreement in
> > final form is executed in writing by all parties involved.
> >
>
Re: Timer support in Flink
Posted by Maximilian Michels <mx...@apache.org>.
Hi Reza,
The detailed view of the capability matrix states: "The Flink Runner
supports timers in non-merging windows."
That is still the case. Other than that, timers should be working fine.
> It makes very heavy use of Event.Time timers and has to do some manual DoFn cache work to get around some O(heavy) issues.
If you are running on Flink 1.5, timer deletion suffers from O(n)
complexity which has been fixed in newer versions.
Cheers,
Max
On 29.05.19 03:27, Reza Rokni wrote:
> Hi Flink experts,
>
> I am getting ready to push a PR around a utility class for timeseries join
>
> left.timestamp match to closest right.timestamp where right.timestamp <=
> left.timestamp.
>
> It makes very heavy use of Event.Time timers and has to do some manual
> DoFn cache work to get around some O(heavy) issues. Wanted to test
> things against Flink: In the capability matrix we have "~" for Timer
> support in Flink:
>
> https://beam.apache.org/documentation/runners/capability-matrix/
>
> Is that page outdated, if not what are the areas that still need to be
> addressed please?
>
> Cheers
>
> Reza
>
>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has
> gone to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and
> do not constitute a legally binding obligation. No legally binding
> obligations will be created, implied, or inferred until an agreement in
> final form is executed in writing by all parties involved.
>