You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stefanos Antaris <an...@gmail.com> on 2015/08/07 17:07:05 UTC
extract workers' resources stats
Hi to all.
I am working an a research project using flink and i would like to extract the CPU + RAM resources consumed on each worker in order to include the stats in my paper. Can anyone advice me on how could i extract them?
Thanks in advance.
Stefanos
Re: extract workers' resources stats
Posted by Stefanos Antaris <an...@gmail.com>.
Hi to all.
Thank you for your quick response. I will try both of your solutions in
order to find which one I the best.
Thanks,
Stefanos
On 7 Aug 2015 20:40, "Robert Metzger" <rm...@apache.org> wrote:
> Hi Stefanos,
>
> you can also write yourself a little script/tool which is periodically
> requesting the following JSON from the JobManager:
>
> http://localhost:8081/setupInfo?get=taskmanagers&_=1438972693441
>
> It returns a JSON string like this:
>
>
> {"taskmanagers":[{"path":"akka:\/\/flink\/user\/taskmanager","dataPort":51695,"timeSinceLastHeartbeat":3,"slotsNumber":1,"freeSlots":1,"cpuCores":8,"physicalMemory":15941,"freeMemory":736,"managedMemory":461,"instanceID":"187cdbc1de0acdb02b5bdc796cefceaf","metrics":{"version":"3.0.0","gauges":{"cpuLoad":{"value":0.09425878320479862},"gc.PS-MarkSweep.count":{"value":6},"gc.PS-MarkSweep.time":{"value":764},"gc.PS-Scavenge.count":{"value":3},"gc.PS-Scavenge.time":{"value":121},"load":{"value":3.04},"memory.heap.committed":{"value":771751936},"memory.heap.init":{"value":805306368},"memory.heap.max":{"value":771751936},"memory.heap.usage":{"value":0.741875586302384},"memory.heap.used":{"value":572543920},"memory.non-heap.committed":{"value":50790400},"memory.non-heap.init":{"value":24576000},"memory.non-heap.max":{"value":318767104},"memory.non-heap.usage":{"value":0.09098903756392629},"memory.non-heap.used":{"value":29004312},"memory.pools.Code-Cache.usage":{"value":0.020928700764973957},"memory.pools.PS-Eden-Space.usage":{"value":0.1786344051361084},"memory.pools.PS-Old-Gen.usage":{"value":0.9994582533836365},"memory.pools.PS-Perm-Gen.usage":{"value":0.10412535071372986},"memory.pools.PS-Survivor-Space.usage":{"value":0},"memory.total.committed":{"value":822542336},"memory.total.init":{"value":829882368},"memory.total.max":{"value":1090519040},"memory.total.used":{"value":601962568}},"counters":{},"histograms":{},"meters":{},"timers":{}}}]}
>
>
>
> On Fri, Aug 7, 2015 at 5:43 PM, Andra Lungu <lu...@gmail.com> wrote:
>
>> Hi,
>>
>> You can use something like iostat to extract the CPU usage.
>> For instance, I call this script on the JM node:
>> #!/usr/bin/env bash
>>
>> lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s`
>> echo "${lines}"
>> IFS="," read -ra hostnames <<<"${lines}"
>> for line in "${hostnames[@]}"; do
>> ssh ${line} `iostat -xtc 5 3 > /home/andra.lungu/cpu_stats/${line}` &
>> done
>>
>> And then I parse the individual output in the cpu_stats folder with
>> python to form charts.
>> To my knowledge. Flink does not do something like that for you, at least
>> not yet :)
>>
>> Bottom line is that you can use linux commands: iostat, dstat, htop, etc.
>> Hope this helps!
>> Andra
>>
>> On Fri, Aug 7, 2015 at 5:07 PM, Stefanos Antaris <
>> antaris.stefanos@gmail.com> wrote:
>>
>>> Hi to all.
>>>
>>> I am working an a research project using flink and i would like to
>>> extract the CPU + RAM resources consumed on each worker in order to include
>>> the stats in my paper. Can anyone advice me on how could i extract them?
>>>
>>> Thanks in advance.
>>> Stefanos
>>
>>
>>
>
Re: extract workers' resources stats
Posted by Robert Metzger <rm...@apache.org>.
Hi Stefanos,
you can also write yourself a little script/tool which is periodically
requesting the following JSON from the JobManager:
http://localhost:8081/setupInfo?get=taskmanagers&_=1438972693441
It returns a JSON string like this:
{"taskmanagers":[{"path":"akka:\/\/flink\/user\/taskmanager","dataPort":51695,"timeSinceLastHeartbeat":3,"slotsNumber":1,"freeSlots":1,"cpuCores":8,"physicalMemory":15941,"freeMemory":736,"managedMemory":461,"instanceID":"187cdbc1de0acdb02b5bdc796cefceaf","metrics":{"version":"3.0.0","gauges":{"cpuLoad":{"value":0.09425878320479862},"gc.PS-MarkSweep.count":{"value":6},"gc.PS-MarkSweep.time":{"value":764},"gc.PS-Scavenge.count":{"value":3},"gc.PS-Scavenge.time":{"value":121},"load":{"value":3.04},"memory.heap.committed":{"value":771751936},"memory.heap.init":{"value":805306368},"memory.heap.max":{"value":771751936},"memory.heap.usage":{"value":0.741875586302384},"memory.heap.used":{"value":572543920},"memory.non-heap.committed":{"value":50790400},"memory.non-heap.init":{"value":24576000},"memory.non-heap.max":{"value":318767104},"memory.non-heap.usage":{"value":0.09098903756392629},"memory.non-heap.used":{"value":29004312},"memory.pools.Code-Cache.usage":{"value":0.020928700764973957},"memory.pools.PS-Eden-Space.usage":{"value":0.1786344051361084},"memory.pools.PS-Old-Gen.usage":{"value":0.9994582533836365},"memory.pools.PS-Perm-Gen.usage":{"value":0.10412535071372986},"memory.pools.PS-Survivor-Space.usage":{"value":0},"memory.total.committed":{"value":822542336},"memory.total.init":{"value":829882368},"memory.total.max":{"value":1090519040},"memory.total.used":{"value":601962568}},"counters":{},"histograms":{},"meters":{},"timers":{}}}]}
On Fri, Aug 7, 2015 at 5:43 PM, Andra Lungu <lu...@gmail.com> wrote:
> Hi,
>
> You can use something like iostat to extract the CPU usage.
> For instance, I call this script on the JM node:
> #!/usr/bin/env bash
>
> lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s`
> echo "${lines}"
> IFS="," read -ra hostnames <<<"${lines}"
> for line in "${hostnames[@]}"; do
> ssh ${line} `iostat -xtc 5 3 > /home/andra.lungu/cpu_stats/${line}` &
> done
>
> And then I parse the individual output in the cpu_stats folder with python
> to form charts.
> To my knowledge. Flink does not do something like that for you, at least
> not yet :)
>
> Bottom line is that you can use linux commands: iostat, dstat, htop, etc.
> Hope this helps!
> Andra
>
> On Fri, Aug 7, 2015 at 5:07 PM, Stefanos Antaris <
> antaris.stefanos@gmail.com> wrote:
>
>> Hi to all.
>>
>> I am working an a research project using flink and i would like to
>> extract the CPU + RAM resources consumed on each worker in order to include
>> the stats in my paper. Can anyone advice me on how could i extract them?
>>
>> Thanks in advance.
>> Stefanos
>
>
>
Re: extract workers' resources stats
Posted by Andra Lungu <lu...@gmail.com>.
Hi,
You can use something like iostat to extract the CPU usage.
For instance, I call this script on the JM node:
#!/usr/bin/env bash
lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s`
echo "${lines}"
IFS="," read -ra hostnames <<<"${lines}"
for line in "${hostnames[@]}"; do
ssh ${line} `iostat -xtc 5 3 > /home/andra.lungu/cpu_stats/${line}` &
done
And then I parse the individual output in the cpu_stats folder with python
to form charts.
To my knowledge. Flink does not do something like that for you, at least
not yet :)
Bottom line is that you can use linux commands: iostat, dstat, htop, etc.
Hope this helps!
Andra
On Fri, Aug 7, 2015 at 5:07 PM, Stefanos Antaris <antaris.stefanos@gmail.com
> wrote:
> Hi to all.
>
> I am working an a research project using flink and i would like to extract
> the CPU + RAM resources consumed on each worker in order to include the
> stats in my paper. Can anyone advice me on how could i extract them?
>
> Thanks in advance.
> Stefanos