You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by otherwise777 <wo...@onzichtbaar.net> on 2016/12/09 13:12:14 UTC
How to analyze space usage of Flink algorithms
Currently i'm doing some analysis for some algorithms that i use in Flink,
I'm interested in the Space and time it takes to execute them. For the Time
i used getNetRuntime() in the executionenvironment, but I have no idea how
to analyse the amount of space an algorithm uses.
Space can mean different things here, like Heap space, disk space, overal
memory or allocated memory. I would like to analyze some of these.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to analyze space usage of Flink algorithms
Posted by Fabian Hueske <fh...@gmail.com>.
Your functions do not need to implement RichFunction (although, each
function can be a RichFunction and it should not be a problem to adapt the
job).
The system metrics are automatically collected. Metrics are exposed via a
Reporter [1].
So you do not need to take care of the collection but rather specify where
the collected metrics should be reported to.
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html#reporter
2016-12-19 9:59 GMT+01:00 otherwise777 <wo...@onzichtbaar.net>:
> Thank you for your reply,
> I'm afraid i still don't understand it, the part i don't understand is how
> to actually analyze it. It's ok if i can just analyze the system instead of
> the actual job, but how would i actually do that?
> I don't have any function in my program that extends the richfunction
> afaik,
> so how would i call the getRuntimeContext() to print or store it?
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-
> space-usage-of-Flink-algorithms-tp10555p10686.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: How to analyze space usage of Flink algorithms
Posted by otherwise777 <wo...@onzichtbaar.net>.
Thank you for your reply,
I'm afraid i still don't understand it, the part i don't understand is how
to actually analyze it. It's ok if i can just analyze the system instead of
the actual job, but how would i actually do that?
I don't have any function in my program that extends the richfunction afaik,
so how would i call the getRuntimeContext() to print or store it?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555p10686.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to analyze space usage of Flink algorithms
Posted by Fabian Hueske <fh...@gmail.com>.
The system metrics [1] are only available on a system level, i.e. not for
an individual job.
The reason is that multiple job might run concurrently on the same task
manager JVM process. So it would not be possible to separate their heap
usage.
The same would be true for the approach that monitors the task manager tmp
directory.
You would need to correlate your measurements with the time range in which
a job is executed.
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html#system-metrics
2016-12-16 9:08 GMT+01:00 otherwise777 <wo...@onzichtbaar.net>:
> Hey Fabian,
>
> Thanks for the quick reply,
> I was looking through the flink metrics [1] but i couldn't find anything in
> there how to analyze the environment from start to finish, only for
> functions that extend the richmapfunction
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.1/apis/metrics.html#list-of-all-variables
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-
> space-usage-of-Flink-algorithms-tp10555p10661.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: How to analyze space usage of Flink algorithms
Posted by otherwise777 <wo...@onzichtbaar.net>.
Hey Fabian,
Thanks for the quick reply,
I was looking through the flink metrics [1] but i couldn't find anything in
there how to analyze the environment from start to finish, only for
functions that extend the richmapfunction
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html#list-of-all-variables
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555p10661.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to analyze space usage of Flink algorithms
Posted by Greg Hogan <co...@greghogan.com>.
This does sound like a nice feature, both per-job and per-taskmanager bytes
written to and read from disk.
On Fri, Dec 9, 2016 at 8:51 AM, Chesnay Schepler <ch...@apache.org> wrote:
> We do not measure how much data we are spilling to disk.
>
>
> On 09.12.2016 14:43, Fabian Hueske wrote:
>
> Hi,
>
> the heap mem usage should be available via Flink's metrics system.
> Not sure if that also captures spilled data. Chesnay (in CC) should know
> that.
>
> If the spilled data is not available as a metric, you can try to write a
> small script that monitors the directories to which Flink spills (Config
> parameter: taskmanager.tmp.dirs [1]).
> The script would repeatedly list all files and keep for each file the max
> size (files are deleted once the are not used anymore). This is not super
> precise but might be good enough.
>
> Hope this helps,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.1/setup/config.html#jobmanager-amp-taskmanager
>
> 2016-12-09 14:12 GMT+01:00 otherwise777 <wo...@onzichtbaar.net>:
>
>> Currently i'm doing some analysis for some algorithms that i use in Flink,
>> I'm interested in the Space and time it takes to execute them. For the
>> Time
>> i used getNetRuntime() in the executionenvironment, but I have no idea how
>> to analyse the amount of space an algorithm uses.
>> Space can mean different things here, like Heap space, disk space, overal
>> memory or allocated memory. I would like to analyze some of these.
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/How-to-analyze-spac
>> e-usage-of-Flink-algorithms-tp10555.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
>
Re: How to analyze space usage of Flink algorithms
Posted by Chesnay Schepler <ch...@apache.org>.
We do not measure how much data we are spilling to disk.
On 09.12.2016 14:43, Fabian Hueske wrote:
> Hi,
>
> the heap mem usage should be available via Flink's metrics system.
> Not sure if that also captures spilled data. Chesnay (in CC) should
> know that.
>
> If the spilled data is not available as a metric, you can try to write
> a small script that monitors the directories to which Flink spills
> (Config parameter: |taskmanager.tmp.dirs| [1]).
> The script would repeatedly list all files and keep for each file the
> max size (files are deleted once the are not used anymore). This is
> not super precise but might be good enough.
>
> Hope this helps,
> Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/config.html#jobmanager-amp-taskmanager
>
> 2016-12-09 14:12 GMT+01:00 otherwise777 <wouter@onzichtbaar.net
> <ma...@onzichtbaar.net>>:
>
> Currently i'm doing some analysis for some algorithms that i use
> in Flink,
> I'm interested in the Space and time it takes to execute them. For
> the Time
> i used getNetRuntime() in the executionenvironment, but I have no
> idea how
> to analyse the amount of space an algorithm uses.
> Space can mean different things here, like Heap space, disk space,
> overal
> memory or allocated memory. I would like to analyze some of these.
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html>
> Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>
Re: How to analyze space usage of Flink algorithms
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
the heap mem usage should be available via Flink's metrics system.
Not sure if that also captures spilled data. Chesnay (in CC) should know
that.
If the spilled data is not available as a metric, you can try to write a
small script that monitors the directories to which Flink spills (Config
parameter: taskmanager.tmp.dirs [1]).
The script would repeatedly list all files and keep for each file the max
size (files are deleted once the are not used anymore). This is not super
precise but might be good enough.
Hope this helps,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/config.html#jobmanager-amp-taskmanager
2016-12-09 14:12 GMT+01:00 otherwise777 <wo...@onzichtbaar.net>:
> Currently i'm doing some analysis for some algorithms that i use in Flink,
> I'm interested in the Space and time it takes to execute them. For the Time
> i used getNetRuntime() in the executionenvironment, but I have no idea how
> to analyse the amount of space an algorithm uses.
> Space can mean different things here, like Heap space, disk space, overal
> memory or allocated memory. I would like to analyze some of these.
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-
> space-usage-of-Flink-algorithms-tp10555.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>