You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Kirschnick, Johannes" <jo...@tu-berlin.de> on 2015/02/14 16:31:51 UTC
Measuring Iteration Timings
Hi,
I have a simple question on to how one could measure the individual execution times for an iteration step, without parsing the logs :)
Is there a best practise for that?
I'm interested in producing something like this
http://data-artisans.com/img/blog/bulk_runtimes.png
>From the blog
http://data-artisans.com/data-analysis-with-flink.html
Johannes
Re: Measuring Iteration Timings
Posted by Stephan Ewen <se...@apache.org>.
Johannes!
You can also customize your messages by printing a message to standard out
in the open() and close() methods of your functions. You have to extend the
RichFunction in that case.
Greetings,
Stephan
On Sat, Feb 14, 2015 at 4:47 PM, Kirschnick, Johannes <
johannes.kirschnick@tu-berlin.de> wrote:
> Hi,
>
> thanks for the quick answer - this tool might just do the job for the
> moment.
>
> Johannes
> ________________________________________
> Von: Robert Metzger <rm...@apache.org>
> Gesendet: Samstag, 14. Februar 2015 16:38
> An: dev@flink.apache.org
> Betreff: Re: Measuring Iteration Timings
>
> Hi,
>
> Right now, you have to parse the logs (of the TaskManager running the Sync
> Task).
> We've created the linked images using this (very hacky) tool:
>
> https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
> .
>
> I'm currently working on improving the monitoring, but realistically it
> will take some time until we expose iteration details in the web frontend.
> (Contributions are welcome as always ;) )
>
> Best,
> Robert
>
>
>
> On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
> johannes.kirschnick@tu-berlin.de> wrote:
>
> > Hi,
> >
> >
> > I have a simple question on to how one could measure the individual
> > execution times for an iteration step, without parsing the logs :)
> >
> >
> > Is there a best practise for that?
> >
> >
> > I'm interested in producing something like this
> >
> > http://data-artisans.com/img/blog/bulk_runtimes.png
> >
> >
> > From the blog
> >
> > http://data-artisans.com/data-analysis-with-flink.html
> >
> >
> >
> > Johannes
> >
>
Re: Measuring Iteration Timings
Posted by "Kirschnick, Johannes" <jo...@tu-berlin.de>.
Hi,
thanks for the quick answer - this tool might just do the job for the moment.
Johannes
________________________________________
Von: Robert Metzger <rm...@apache.org>
Gesendet: Samstag, 14. Februar 2015 16:38
An: dev@flink.apache.org
Betreff: Re: Measuring Iteration Timings
Hi,
Right now, you have to parse the logs (of the TaskManager running the Sync
Task).
We've created the linked images using this (very hacky) tool:
https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
.
I'm currently working on improving the monitoring, but realistically it
will take some time until we expose iteration details in the web frontend.
(Contributions are welcome as always ;) )
Best,
Robert
On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
johannes.kirschnick@tu-berlin.de> wrote:
> Hi,
>
>
> I have a simple question on to how one could measure the individual
> execution times for an iteration step, without parsing the logs :)
>
>
> Is there a best practise for that?
>
>
> I'm interested in producing something like this
>
> http://data-artisans.com/img/blog/bulk_runtimes.png
>
>
> From the blog
>
> http://data-artisans.com/data-analysis-with-flink.html
>
>
>
> Johannes
>
Re: Measuring Iteration Timings
Posted by Robert Metzger <rm...@apache.org>.
Hi,
Right now, you have to parse the logs (of the TaskManager running the Sync
Task).
We've created the linked images using this (very hacky) tool:
https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
.
I'm currently working on improving the monitoring, but realistically it
will take some time until we expose iteration details in the web frontend.
(Contributions are welcome as always ;) )
Best,
Robert
On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
johannes.kirschnick@tu-berlin.de> wrote:
> Hi,
>
>
> I have a simple question on to how one could measure the individual
> execution times for an iteration step, without parsing the logs :)
>
>
> Is there a best practise for that?
>
>
> I'm interested in producing something like this
>
> http://data-artisans.com/img/blog/bulk_runtimes.png
>
>
> From the blog
>
> http://data-artisans.com/data-analysis-with-flink.html
>
>
>
> Johannes
>