You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Sam Taha <ta...@gmail.com> on 2013/10/17 13:55:49 UTC
Resource utilization stats
Is there currently a way for a Framework to get back CPU/memory usage
statistics as a task is running or after the task has finished? From Java
API specifically?
Thanks,
Sam Taha
http://www.grandlogic.com
Re: Resource utilization stats
Posted by Sam Taha <ta...@gmail.com>.
I am looking to capture how much cpu/memory a specific task consumed during
its lifetime and report on this per job/task run.
For example, my JobTracker tracks each job run and let's the user see
details of current and historical job runs on a per job run basis. Then I
also have a separate Stats Rreporting UI that shows trends of job over
time. But all these stats are per job/task run.
For overall slave level stats (historical and current), I can probably just
point the user to the Mesos web UIs, since this more cluster level details
that the Mesos UI is probably better suited for.
Thanks,
Sam Taha
http://www.grandlogic.com
On Fri, Oct 18, 2013 at 2:40 PM, Benjamin Mahler
<be...@gmail.com>wrote:
> The stats on that endpoint are instantaneous values, we keep historical
> stats but do not yet expose them.
>
> What kind of "basic usage stats" would you benefit from receiving upon
> TASK_FINISHED?
> I would imagine the resource consumption statistics are not terribly
> useful without also receiving historical data or being able to retrieve
> them periodically.
>
> If it's for the purposes of your UI, maybe a flexible endpoint on the
> slave would be more beneficial so you can retrieve timeseries style
> information?
>
>
> On Fri, Oct 18, 2013 at 11:11 AM, Sam Taha <ta...@gmail.com> wrote:
>
>> Thanks Ben. I will see if I can incorporate this in my reporting GUI. Do
>> you know how long lived these are stats are?
>>
>> Ideally, I am thinking it would be nice to return basic usage stats maybe
>> via the TaskStatus so that I can capture them when they are returned when
>> the TASK_FINISHED event is fired. I capture stats in my local app database
>> as I track the events per job.
>>
>> Thanks,
>> Sam Taha
>>
>> http://www.grandlogic.com
>>
>>
>> On Fri, Oct 18, 2013 at 1:34 PM, Benjamin Mahler <
>> benjamin.mahler@gmail.com> wrote:
>>
>>> Hey Sam,
>>>
>>> You can use the json monitoring endpoint on the slave to retrieve cpu
>>> usage information.
>>>
>>> $ curl localhost:5051/monitor/statistics.json
>>>
>>> Take a look at the data, this is how the webui displays cpu usage
>>> information.
>>>
>>>
>>> On Thu, Oct 17, 2013 at 12:10 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>
>>>> MESOS-581 is tracking cpu/mem usage of master/slave process themselves.
>>>>
>>>> I think what you are looking for is close to
>>>> https://issues.apache.org/jira/browse/MESOS-62
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha <ta...@gmail.com> wrote:
>>>>
>>>>> https://issues.apache.org/jira/browse/MESOS-581
>>>>>
>>>>>
>>>>> On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
>>>>>
>>>>>> Looks like it is not implemented
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>>>>>>
>>>>>>> Is there currently a way for a Framework to get back CPU/memory
>>>>>>> usage statistics as a task is running or after the task has finished? From
>>>>>>> Java API specifically?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sam Taha
>>>>>>>
>>>>>>> http://www.grandlogic.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Resource utilization stats
Posted by Benjamin Mahler <be...@gmail.com>.
The stats on that endpoint are instantaneous values, we keep historical
stats but do not yet expose them.
What kind of "basic usage stats" would you benefit from receiving upon
TASK_FINISHED?
I would imagine the resource consumption statistics are not terribly useful
without also receiving historical data or being able to retrieve them
periodically.
If it's for the purposes of your UI, maybe a flexible endpoint on the slave
would be more beneficial so you can retrieve timeseries style information?
On Fri, Oct 18, 2013 at 11:11 AM, Sam Taha <ta...@gmail.com> wrote:
> Thanks Ben. I will see if I can incorporate this in my reporting GUI. Do
> you know how long lived these are stats are?
>
> Ideally, I am thinking it would be nice to return basic usage stats maybe
> via the TaskStatus so that I can capture them when they are returned when
> the TASK_FINISHED event is fired. I capture stats in my local app database
> as I track the events per job.
>
> Thanks,
> Sam Taha
>
> http://www.grandlogic.com
>
>
> On Fri, Oct 18, 2013 at 1:34 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Hey Sam,
>>
>> You can use the json monitoring endpoint on the slave to retrieve cpu
>> usage information.
>>
>> $ curl localhost:5051/monitor/statistics.json
>>
>> Take a look at the data, this is how the webui displays cpu usage
>> information.
>>
>>
>> On Thu, Oct 17, 2013 at 12:10 PM, Vinod Kone <vi...@gmail.com> wrote:
>>
>>> MESOS-581 is tracking cpu/mem usage of master/slave process themselves.
>>>
>>> I think what you are looking for is close to
>>> https://issues.apache.org/jira/browse/MESOS-62
>>>
>>>
>>> On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha <ta...@gmail.com> wrote:
>>>
>>>> https://issues.apache.org/jira/browse/MESOS-581
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
>>>>
>>>>> Looks like it is not implemented
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>>>>>
>>>>>> Is there currently a way for a Framework to get back CPU/memory usage
>>>>>> statistics as a task is running or after the task has finished? From Java
>>>>>> API specifically?
>>>>>>
>>>>>> Thanks,
>>>>>> Sam Taha
>>>>>>
>>>>>> http://www.grandlogic.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Resource utilization stats
Posted by Sam Taha <ta...@gmail.com>.
Thanks Ben. I will see if I can incorporate this in my reporting GUI. Do
you know how long lived these are stats are?
Ideally, I am thinking it would be nice to return basic usage stats maybe
via the TaskStatus so that I can capture them when they are returned when
the TASK_FINISHED event is fired. I capture stats in my local app database
as I track the events per job.
Thanks,
Sam Taha
http://www.grandlogic.com
On Fri, Oct 18, 2013 at 1:34 PM, Benjamin Mahler
<be...@gmail.com>wrote:
> Hey Sam,
>
> You can use the json monitoring endpoint on the slave to retrieve cpu
> usage information.
>
> $ curl localhost:5051/monitor/statistics.json
>
> Take a look at the data, this is how the webui displays cpu usage
> information.
>
>
> On Thu, Oct 17, 2013 at 12:10 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> MESOS-581 is tracking cpu/mem usage of master/slave process themselves.
>>
>> I think what you are looking for is close to
>> https://issues.apache.org/jira/browse/MESOS-62
>>
>>
>> On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha <ta...@gmail.com> wrote:
>>
>>> https://issues.apache.org/jira/browse/MESOS-581
>>>
>>>
>>> On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
>>>
>>>> Looks like it is not implemented
>>>>
>>>>
>>>>
>>>> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>>>>
>>>>> Is there currently a way for a Framework to get back CPU/memory usage
>>>>> statistics as a task is running or after the task has finished? From Java
>>>>> API specifically?
>>>>>
>>>>> Thanks,
>>>>> Sam Taha
>>>>>
>>>>> http://www.grandlogic.com
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Resource utilization stats
Posted by Benjamin Mahler <be...@gmail.com>.
Hey Sam,
You can use the json monitoring endpoint on the slave to retrieve cpu usage
information.
$ curl localhost:5051/monitor/statistics.json
Take a look at the data, this is how the webui displays cpu usage
information.
On Thu, Oct 17, 2013 at 12:10 PM, Vinod Kone <vi...@gmail.com> wrote:
> MESOS-581 is tracking cpu/mem usage of master/slave process themselves.
>
> I think what you are looking for is close to
> https://issues.apache.org/jira/browse/MESOS-62
>
>
> On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha <ta...@gmail.com> wrote:
>
>> https://issues.apache.org/jira/browse/MESOS-581
>>
>>
>> On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
>>
>>> Looks like it is not implemented
>>>
>>>
>>>
>>> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>>>
>>>> Is there currently a way for a Framework to get back CPU/memory usage
>>>> statistics as a task is running or after the task has finished? From Java
>>>> API specifically?
>>>>
>>>> Thanks,
>>>> Sam Taha
>>>>
>>>> http://www.grandlogic.com
>>>>
>>>
>>>
>>
>
Re: Resource utilization stats
Posted by Vinod Kone <vi...@gmail.com>.
MESOS-581 is tracking cpu/mem usage of master/slave process themselves.
I think what you are looking for is close to
https://issues.apache.org/jira/browse/MESOS-62
On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha <ta...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/MESOS-581
>
>
> On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
>
>> Looks like it is not implemented
>>
>>
>>
>> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>>
>>> Is there currently a way for a Framework to get back CPU/memory usage
>>> statistics as a task is running or after the task has finished? From Java
>>> API specifically?
>>>
>>> Thanks,
>>> Sam Taha
>>>
>>> http://www.grandlogic.com
>>>
>>
>>
>
Re: Resource utilization stats
Posted by Sam Taha <ta...@gmail.com>.
https://issues.apache.org/jira/browse/MESOS-581
On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha <ta...@gmail.com> wrote:
> Looks like it is not implemented
>
>
>
> On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
>
>> Is there currently a way for a Framework to get back CPU/memory usage
>> statistics as a task is running or after the task has finished? From Java
>> API specifically?
>>
>> Thanks,
>> Sam Taha
>>
>> http://www.grandlogic.com
>>
>
>
Re: Resource utilization stats
Posted by Sam Taha <ta...@gmail.com>.
Looks like it is not implemented
On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha <ta...@gmail.com> wrote:
> Is there currently a way for a Framework to get back CPU/memory usage
> statistics as a task is running or after the task has finished? From Java
> API specifically?
>
> Thanks,
> Sam Taha
>
> http://www.grandlogic.com
>