Using event timestamps

[Gyula Fóra <gy...@apache.org>]

Hey All!

Is there a proper way of using a Flink Streaming source with event
timestamps and watermarks? What I mean here is instead of implementing a
custom SourceFunction, use an existing one and provide some Timestamp
extractor (like the one currently used for Time windows), which will also
automatically generate the watermarks based on event time (assuming that
timestamps are partially ordered).

Maybe I missed something, or there might be somebody working on this
already :)

Cheers,
Gyula

Re: Using event timestamps

[Gyula Fóra <gy...@gmail.com>]

Then as a first step, I will open a Jira for it :)

Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. szept. 14.,
H, 10:18):

> Hi,
> there is nothing like that in there yet but it would be very nice to have
> it. :D Also, as far as I know no-one is working on this right now.
>
> Cheers,
> Aljoscha
>
> On Mon, 14 Sep 2015 at 06:27 Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hey Gyula,
> >
> > I have been recently looking at the streaming UdfOperators and can not
> > recall a utility for the sources that you are looking for, but maybe I am
> > also missing it. :)
> > It would be a convenient addition though.
> >
> > Best,
> >
> > Marton
> >
> > On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <gy...@apache.org> wrote:
> >
> > > Hey All!
> > >
> > > Is there a proper way of using a Flink Streaming source with event
> > > timestamps and watermarks? What I mean here is instead of implementing
> a
> > > custom SourceFunction, use an existing one and provide some Timestamp
> > > extractor (like the one currently used for Time windows), which will
> also
> > > automatically generate the watermarks based on event time (assuming
> that
> > > timestamps are partially ordered).
> > >
> > > Maybe I missed something, or there might be somebody working on this
> > > already :)
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>

Re: Using event timestamps

[Aljoscha Krettek <al...@apache.org>]

Hi,
there is nothing like that in there yet but it would be very nice to have
it. :D Also, as far as I know no-one is working on this right now.

Cheers,
Aljoscha

On Mon, 14 Sep 2015 at 06:27 Márton Balassi <ba...@gmail.com>
wrote:

> Hey Gyula,
>
> I have been recently looking at the streaming UdfOperators and can not
> recall a utility for the sources that you are looking for, but maybe I am
> also missing it. :)
> It would be a convenient addition though.
>
> Best,
>
> Marton
>
> On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <gy...@apache.org> wrote:
>
> > Hey All!
> >
> > Is there a proper way of using a Flink Streaming source with event
> > timestamps and watermarks? What I mean here is instead of implementing a
> > custom SourceFunction, use an existing one and provide some Timestamp
> > extractor (like the one currently used for Time windows), which will also
> > automatically generate the watermarks based on event time (assuming that
> > timestamps are partially ordered).
> >
> > Maybe I missed something, or there might be somebody working on this
> > already :)
> >
> > Cheers,
> > Gyula
> >
>

Re: Using event timestamps

[Márton Balassi <ba...@gmail.com>]

Hey Gyula,

I have been recently looking at the streaming UdfOperators and can not
recall a utility for the sources that you are looking for, but maybe I am
also missing it. :)
It would be a convenient addition though.

Best,

Marton

On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <gy...@apache.org> wrote:

> Hey All!
>
> Is there a proper way of using a Flink Streaming source with event
> timestamps and watermarks? What I mean here is instead of implementing a
> custom SourceFunction, use an existing one and provide some Timestamp
> extractor (like the one currently used for Time windows), which will also
> automatically generate the watermarks based on event time (assuming that
> timestamps are partially ordered).
>
> Maybe I missed something, or there might be somebody working on this
> already :)
>
> Cheers,
> Gyula
>