You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by VJ Shalish <vj...@gmail.com> on 2014/09/17 05:14:52 UTC
CPU RAM
Hi
I need to get the CPU utilisation, RAM usage, Network IO and other metrics
using Java program. Can anyone help me on this?
Thanks
Shalish.
Re: CPU RAM
Posted by VJ Shalish <vj...@gmail.com>.
Thank u for the response Amit.
So is it that, we cannot measure the CPU consumption, RAM usage of a spark
job through a Java program?
On Tue, Sep 16, 2014 at 11:23 PM, Amit <ku...@gmail.com> wrote:
> Not particularly related to Spark, but you can check out SIGAR API. It
> let's you get CPU, Memory, Network, Filesystem and process based metrics.
>
> Amit
> On Sep 16, 2014, at 20:14, VJ Shalish <vj...@gmail.com> wrote:
>
> > Hi
> >
> > I need to get the CPU utilisation, RAM usage, Network IO and other
> metrics using Java program. Can anyone help me on this?
> >
> > Thanks
> > Shalish.
>
Re: CPU RAM
Posted by VJ Shalish <vj...@gmail.com>.
Hi
I need the same through Java.
Doesn't the SPark API support this?
On Wed, Sep 17, 2014 at 2:48 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Ganglia does give you a cluster wide and per machine utilization of
> resources, but i don't think it gives your per Spark Job. If you want to
> build something from scratch then you can follow up like :
>
> 1. Login to the machine
> 2. Get the PIDs
> 3. For network IO per process, you can have a look at
> http://nethogs.sourceforge.net/
> 4. You can make use of the information in /proc/[pid]/stat and /proc/stat
> to estimate CPU usage and all
>
>
> Similarly you can get any metric of process once you have the PID.
>
>
> Thanks
> Best Regards
>
> On Wed, Sep 17, 2014 at 8:59 AM, VJ Shalish <vj...@gmail.com> wrote:
>
>> Sorry for the confusion Team.
>> My requirement is to measure the CPU utilisation, RAM usage, Network IO
>> and other metrics of a SPARK JOB using Java program.
>> Please help on the same.
>>
>> On Tue, Sep 16, 2014 at 11:23 PM, Amit <ku...@gmail.com> wrote:
>>
>>> Not particularly related to Spark, but you can check out SIGAR API. It
>>> let's you get CPU, Memory, Network, Filesystem and process based metrics.
>>>
>>> Amit
>>> On Sep 16, 2014, at 20:14, VJ Shalish <vj...@gmail.com> wrote:
>>>
>>> > Hi
>>> >
>>> > I need to get the CPU utilisation, RAM usage, Network IO and other
>>> metrics using Java program. Can anyone help me on this?
>>> >
>>> > Thanks
>>> > Shalish.
>>>
>>
>>
>
Re: CPU RAM
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Ganglia does give you a cluster wide and per machine utilization of
resources, but i don't think it gives your per Spark Job. If you want to
build something from scratch then you can follow up like :
1. Login to the machine
2. Get the PIDs
3. For network IO per process, you can have a look at
http://nethogs.sourceforge.net/
4. You can make use of the information in /proc/[pid]/stat and /proc/stat
to estimate CPU usage and all
Similarly you can get any metric of process once you have the PID.
Thanks
Best Regards
On Wed, Sep 17, 2014 at 8:59 AM, VJ Shalish <vj...@gmail.com> wrote:
> Sorry for the confusion Team.
> My requirement is to measure the CPU utilisation, RAM usage, Network IO
> and other metrics of a SPARK JOB using Java program.
> Please help on the same.
>
> On Tue, Sep 16, 2014 at 11:23 PM, Amit <ku...@gmail.com> wrote:
>
>> Not particularly related to Spark, but you can check out SIGAR API. It
>> let's you get CPU, Memory, Network, Filesystem and process based metrics.
>>
>> Amit
>> On Sep 16, 2014, at 20:14, VJ Shalish <vj...@gmail.com> wrote:
>>
>> > Hi
>> >
>> > I need to get the CPU utilisation, RAM usage, Network IO and other
>> metrics using Java program. Can anyone help me on this?
>> >
>> > Thanks
>> > Shalish.
>>
>
>
Re: CPU RAM
Posted by VJ Shalish <vj...@gmail.com>.
Sorry for the confusion Team.
My requirement is to measure the CPU utilisation, RAM usage, Network IO and
other metrics of a SPARK JOB using Java program.
Please help on the same.
On Tue, Sep 16, 2014 at 11:23 PM, Amit <ku...@gmail.com> wrote:
> Not particularly related to Spark, but you can check out SIGAR API. It
> let's you get CPU, Memory, Network, Filesystem and process based metrics.
>
> Amit
> On Sep 16, 2014, at 20:14, VJ Shalish <vj...@gmail.com> wrote:
>
> > Hi
> >
> > I need to get the CPU utilisation, RAM usage, Network IO and other
> metrics using Java program. Can anyone help me on this?
> >
> > Thanks
> > Shalish.
>
Re: CPU RAM
Posted by Amit <ku...@gmail.com>.
Not particularly related to Spark, but you can check out SIGAR API. It let's you get CPU, Memory, Network, Filesystem and process based metrics.
Amit
On Sep 16, 2014, at 20:14, VJ Shalish <vj...@gmail.com> wrote:
> Hi
>
> I need to get the CPU utilisation, RAM usage, Network IO and other metrics using Java program. Can anyone help me on this?
>
> Thanks
> Shalish.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org