You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2016/11/01 15:39:36 UTC
Re: [DISCUSSION] Timely function interface and timer params
Hi,
yes, I think exposing a simple form of namespaces as String would be good.
By the way, do we also need access to state with namespaces then?
@Gyula: Please go ahead and open the Jira issue.
Cheers,
Aljoscha
On Sat, 29 Oct 2016 at 17:28 Gyula Fóra <gy...@gmail.com> wrote:
> Thanks for the feedback guys,
>
> I think exposing the namespace in a simplified form in the user facing API
> is I think a very good idea, that already let's the users implement
> practically anything they want. Maybe doing it as a simple string as Jamie
> suggested would be a nice way to do it and that would serve as a label or
> timer-id for the user. Thinking of it as a label/id is probably a much
> simpler concept than the "namespace".
>
> Should we open a JIRA for this? Judging from the internal timer service
> this should be a fairly straight forward extension as Aljoscha pointed out.
>
> Gyula
>
> Jamie Grier <ja...@data-artisans.com> ezt írta (időpont: 2016. okt. 29.,
> Szo, 15:37):
>
> > Hi guys,
> >
> > Good points, Gyula. I think it would be much easier on a user if there
> > could be multiple timers in flight per key. I prefer the second
> approach,
> > though, where a user associates some bit of metadata with the timer and
> we
> > pass it back to them in the onTimer() callback, otherwise they are forced
> > to maintain this state themselves.
> >
> > It looks to me like somehow exposing the namespaces, even if it's simpler
> > and just a string, is the way to go.
> >
> > I'm really excited by this guys! I think the TimelyFlatMap and
> > TimelyCoFlatMap are going to get a LOT of use. This is gonna make a lot
> of
> > people happy.
> >
> > -Jamie
> >
> >
> > On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > Hi Gyula,
> > > if you look at the internal API you'll notice that it is pretty much
> like
> > > your second proposal. Just for reference, the interface is roughly
> this:
> > >
> > > public interface InternalTimerService<N> {
> > > long currentProcessingTime();
> > > long currentWatermark();
> > > void registerProcessingTimeTimer(N namespace, long time);
> > > void deleteProcessingTimeTimer(N namespace, long time);
> > > void registerEventTimeTimer(N namespace, long time);
> > > void deleteEventTimeTimer(N namespace, long time);
> > > }
> > >
> > > that namespace bit can be anything for which you can provide a
> > > TypeSerializer.
> > >
> > > IMHO, this goes back a bit to the discussion about adding a low level
> > > operator-like interface that allows pretty much anything a custom
> > operator
> > > can do but with a defined, stable interface. The internal operator
> > > interface is somewhat in flux, still, so I wouldn't recommend people to
> > use
> > > it directly.
> > >
> > > The only thing missing, really, from TimelyFlatMap is access to
> > namespaces
> > > for state and timers. With that, you could implement even the
> > > WindowOperator as a TimelyFlatMap since I worked on abstracting
> > everything
> > > that it uses away behind interfaces that any operator can use. The last
> > > pice, the generic timer API went in last, of course. :-)
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <gy...@apache.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > I was thinking about the methods provided by the timely functions and
> > the
> > > > timerservice and I am wondering if it makes sense to change them a
> > little
> > > > so they can cover a wider set of use case. Maybe I missed something
> > > > completely obvious so please shoot me down in that case :)
> > > >
> > > > Currently the user gets a TimerService to register timers that will
> in
> > > the
> > > > future call the onTimer method. It is not completely obvious to me
> how
> > > > would I implement a function that needs to trigger two types of
> > callbacks
> > > > in the future. If I get only one onTimer method I should be able to
> > pass
> > > in
> > > > some sort of parameter or flag so I can branch in my onTimer
> > > > implementation.
> > > >
> > > > As parameters are not supported I am left with states that are scoped
> > to
> > > > the keys which is also pretty useless if I want to trigger different
> > > timed
> > > > actions for the same keys.
> > > >
> > > > I know this is quite tricky but I see some alternative options:
> > > > - The register timer method returns a unique (per key) timer id, so
> we
> > > can
> > > > associate state with this id to fetch info about the timer
> registered.
> > > (We
> > > > could also remove timers with this id and maybe add methods to remove
> > all
> > > > for the current key)
> > > > - Allow the users to pass a custom parameter when registering the
> > > > callback, and the parameter would be passed to the onTimer method
> > > > - Allow users to pass custom callback functions when registering the
> > > > timers, but this would mean we have to support some sort of context
> for
> > > > accessing the state (like the window context before)
> > > > - We could go for an annotation based API like in beam but thats
> > > probably
> > > > not good to mix in the current ones
> > > >
> > > > I personally prefer the first one.
> > > >
> > > > What do you think?
> > > >
> > > > Regards,
> > > > Gyula
> > > >
> > >
> >
> >
> >
> > --
> >
> > Jamie Grier
> > data Artisans, Director of Applications Engineering
> > @jamiegrier <https://twitter.com/jamiegrier>
> > jamie@data-artisans.com
> >
>
Re: [DISCUSSION] Timely function interface and timer params
Posted by Gyula Fóra <gy...@gmail.com>.
Hi,
I opened this:
https://issues.apache.org/jira/browse/FLINK-4992
With this users can implement practically anything depending how they use
the parameter so changing the state access logic is not necessary.
Cheers,
Gyula
Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2016. nov. 1., K,
16:39):
> Hi,
> yes, I think exposing a simple form of namespaces as String would be good.
> By the way, do we also need access to state with namespaces then?
>
> @Gyula: Please go ahead and open the Jira issue.
>
> Cheers,
> Aljoscha
>
> On Sat, 29 Oct 2016 at 17:28 Gyula Fóra <gy...@gmail.com> wrote:
>
> > Thanks for the feedback guys,
> >
> > I think exposing the namespace in a simplified form in the user facing
> API
> > is I think a very good idea, that already let's the users implement
> > practically anything they want. Maybe doing it as a simple string as
> Jamie
> > suggested would be a nice way to do it and that would serve as a label or
> > timer-id for the user. Thinking of it as a label/id is probably a much
> > simpler concept than the "namespace".
> >
> > Should we open a JIRA for this? Judging from the internal timer service
> > this should be a fairly straight forward extension as Aljoscha pointed
> out.
> >
> > Gyula
> >
> > Jamie Grier <ja...@data-artisans.com> ezt írta (időpont: 2016. okt. 29.,
> > Szo, 15:37):
> >
> > > Hi guys,
> > >
> > > Good points, Gyula. I think it would be much easier on a user if there
> > > could be multiple timers in flight per key. I prefer the second
> > approach,
> > > though, where a user associates some bit of metadata with the timer and
> > we
> > > pass it back to them in the onTimer() callback, otherwise they are
> forced
> > > to maintain this state themselves.
> > >
> > > It looks to me like somehow exposing the namespaces, even if it's
> simpler
> > > and just a string, is the way to go.
> > >
> > > I'm really excited by this guys! I think the TimelyFlatMap and
> > > TimelyCoFlatMap are going to get a LOT of use. This is gonna make a
> lot
> > of
> > > people happy.
> > >
> > > -Jamie
> > >
> > >
> > > On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > >
> > > > Hi Gyula,
> > > > if you look at the internal API you'll notice that it is pretty much
> > like
> > > > your second proposal. Just for reference, the interface is roughly
> > this:
> > > >
> > > > public interface InternalTimerService<N> {
> > > > long currentProcessingTime();
> > > > long currentWatermark();
> > > > void registerProcessingTimeTimer(N namespace, long time);
> > > > void deleteProcessingTimeTimer(N namespace, long time);
> > > > void registerEventTimeTimer(N namespace, long time);
> > > > void deleteEventTimeTimer(N namespace, long time);
> > > > }
> > > >
> > > > that namespace bit can be anything for which you can provide a
> > > > TypeSerializer.
> > > >
> > > > IMHO, this goes back a bit to the discussion about adding a low level
> > > > operator-like interface that allows pretty much anything a custom
> > > operator
> > > > can do but with a defined, stable interface. The internal operator
> > > > interface is somewhat in flux, still, so I wouldn't recommend people
> to
> > > use
> > > > it directly.
> > > >
> > > > The only thing missing, really, from TimelyFlatMap is access to
> > > namespaces
> > > > for state and timers. With that, you could implement even the
> > > > WindowOperator as a TimelyFlatMap since I worked on abstracting
> > > everything
> > > > that it uses away behind interfaces that any operator can use. The
> last
> > > > pice, the generic timer API went in last, of course. :-)
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <gy...@apache.org> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I was thinking about the methods provided by the timely functions
> and
> > > the
> > > > > timerservice and I am wondering if it makes sense to change them a
> > > little
> > > > > so they can cover a wider set of use case. Maybe I missed something
> > > > > completely obvious so please shoot me down in that case :)
> > > > >
> > > > > Currently the user gets a TimerService to register timers that will
> > in
> > > > the
> > > > > future call the onTimer method. It is not completely obvious to me
> > how
> > > > > would I implement a function that needs to trigger two types of
> > > callbacks
> > > > > in the future. If I get only one onTimer method I should be able to
> > > pass
> > > > in
> > > > > some sort of parameter or flag so I can branch in my onTimer
> > > > > implementation.
> > > > >
> > > > > As parameters are not supported I am left with states that are
> scoped
> > > to
> > > > > the keys which is also pretty useless if I want to trigger
> different
> > > > timed
> > > > > actions for the same keys.
> > > > >
> > > > > I know this is quite tricky but I see some alternative options:
> > > > > - The register timer method returns a unique (per key) timer id,
> so
> > we
> > > > can
> > > > > associate state with this id to fetch info about the timer
> > registered.
> > > > (We
> > > > > could also remove timers with this id and maybe add methods to
> remove
> > > all
> > > > > for the current key)
> > > > > - Allow the users to pass a custom parameter when registering the
> > > > > callback, and the parameter would be passed to the onTimer method
> > > > > - Allow users to pass custom callback functions when registering
> the
> > > > > timers, but this would mean we have to support some sort of context
> > for
> > > > > accessing the state (like the window context before)
> > > > > - We could go for an annotation based API like in beam but thats
> > > > probably
> > > > > not good to mix in the current ones
> > > > >
> > > > > I personally prefer the first one.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Regards,
> > > > > Gyula
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Jamie Grier
> > > data Artisans, Director of Applications Engineering
> > > @jamiegrier <https://twitter.com/jamiegrier>
> > > jamie@data-artisans.com
> > >
> >
>