You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2016/10/12 09:55:31 UTC

Re: currentLowWatermark metric not reported for all tasks?

I added a JIRA for this feature request:
https://issues.apache.org/jira/browse/FLINK-4812

On Fri, Sep 30, 2016 at 6:13 PM, dan bress <da...@gmail.com> wrote:

> Awesome!  It would definitely help me troubleshoot lagging watermarks if i
> can see what watermark all my sources have seen.  Thanks for looking into
> this!
>
> Dan
>
> On Fri, Sep 30, 2016 at 5:48 AM Stephan Ewen <se...@apache.org> wrote:
>
> > I think what you describe, Chesnay, is exactly what we should do...
> >
> > On Fri, Sep 30, 2016 at 1:15 PM, Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > Hello Dan,
> > >
> > > the technical reason is that this metric is only collected in the
> > > *InputProcessor classes, which aren't used for source tasks.
> > >
> > > I do recall that there were discussions about source watermarks, but
> > > frankly i don't remember why we didn't add them.
> > >
> > > In order to add them one would only have to modify the SourceContext
> > > classes to a) store the last emitted watermark and b) expose it
> through a
> > > metric.
> > >
> > > Regards,
> > > Chesnay
> > >
> > >
> > > On 29.09.2016 23:03, dan bress wrote:
> > >
> > >> I am trying to measure the currentLowWatermark throughout my dataflow,
> > but
> > >> I am not seeing it for tasks with sources.  For those tasks I see
> these
> > >> metrics:
> > >>
> > >> lastCheckpointSize
> > >> numBytesInLocal
> > >> numBytesInRemote
> > >> numBytesOut
> > >>
> > >> why am I not seeing currentLowWatermark on these tasks?
> > >>
> > >> Thanks!
> > >>
> > >> Dan
> > >>
> > >>
> > >
> >
>

Re: currentLowWatermark metric not reported for all tasks?

Posted by dan bress <da...@gmail.com>.
Thank you Robert!

On Wed, Oct 12, 2016 at 2:55 AM Robert Metzger <rm...@apache.org> wrote:

> I added a JIRA for this feature request:
> https://issues.apache.org/jira/browse/FLINK-4812
>
> On Fri, Sep 30, 2016 at 6:13 PM, dan bress <da...@gmail.com> wrote:
>
> > Awesome!  It would definitely help me troubleshoot lagging watermarks if
> i
> > can see what watermark all my sources have seen.  Thanks for looking into
> > this!
> >
> > Dan
> >
> > On Fri, Sep 30, 2016 at 5:48 AM Stephan Ewen <se...@apache.org> wrote:
> >
> > > I think what you describe, Chesnay, is exactly what we should do...
> > >
> > > On Fri, Sep 30, 2016 at 1:15 PM, Chesnay Schepler <ch...@apache.org>
> > > wrote:
> > >
> > > > Hello Dan,
> > > >
> > > > the technical reason is that this metric is only collected in the
> > > > *InputProcessor classes, which aren't used for source tasks.
> > > >
> > > > I do recall that there were discussions about source watermarks, but
> > > > frankly i don't remember why we didn't add them.
> > > >
> > > > In order to add them one would only have to modify the SourceContext
> > > > classes to a) store the last emitted watermark and b) expose it
> > through a
> > > > metric.
> > > >
> > > > Regards,
> > > > Chesnay
> > > >
> > > >
> > > > On 29.09.2016 23:03, dan bress wrote:
> > > >
> > > >> I am trying to measure the currentLowWatermark throughout my
> dataflow,
> > > but
> > > >> I am not seeing it for tasks with sources.  For those tasks I see
> > these
> > > >> metrics:
> > > >>
> > > >> lastCheckpointSize
> > > >> numBytesInLocal
> > > >> numBytesInRemote
> > > >> numBytesOut
> > > >>
> > > >> why am I not seeing currentLowWatermark on these tasks?
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Dan
> > > >>
> > > >>
> > > >
> > >
> >
>