You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Luis Fernando De Pombo <lf...@gmail.com> on 2015/06/26 04:47:12 UTC
Re: what does the utilization metric in SamzaContainerMetrics show?
Hi Shadi,
Thanks for asking. This metric tracks the utilization of the event loop
<https://samza.apache.org/learn/documentation/0.8/container/event-loop.html>
within a samza container, which uses a single thread, that is in charge of
reading and writing messages, flushing metrics, checkpointing, and
windowing. It is important to track the utilization (aka "duty cycle") of
any event loop, which is the sum of all the timings (activeMs) divided by
the window length (totalMs).
You are right in that most of the time this value will be close to 1, which
represents complete utilization. However when the event loop starts to have
idle time, this metric will give you an idea of how much headroom you have
before the job will start to seriously fall behind.
I hope that answers your question!
ᐧ
On Thu, Jun 25, 2015 at 6:39 PM, Shadi Noghabi <
snoghabi@linkedin.com.invalid> wrote:
> Hi,
>
> I was wondering what does this utilization metric in the
> SamzaContainerMetrics show? I am asking this sine in the code it is
> calculated as below:
>
> while (!shutdownNow) {
> val loopStartTime = clock();
> process
> window
> commit
> val totalMs = clock() - loopStartTime
> metrics.utilization.set(activeMs.toFloat/totalMs)
> activeMs = 0L
> }
>
> Where the totalMs is the time it takes to start calling process until
> commit is done which is equal to the time it takes to run process, window,
> and commit. And they way activeMs is calculated is by summing up the time
> it takes to call process, window and commit, which means these two values
> are going to be almost the same and the utilization is always going to be
> almost 1.
>
> I was just wondering what is the point of doing this?
>
>
>
>
Re: what does the utilization metric in SamzaContainerMetrics show?
Posted by Luis Fernando De Pombo <lf...@gmail.com>.
Shadi, you are absolutely right. I created a diff (
https://reviews.apache.org/r/36274/) to properly calculate activeMs. Feel
free to leave any comments there. Thanks for pointing this out :)
ᐧ
On Fri, Jun 26, 2015 at 9:40 AM, Shadi Noghabi <
snoghabi@linkedin.com.invalid> wrote:
>
> Thanks for your prompt reply. I just don’t see when will there be an idle
> time that is not added to the activeMs but is counted in the totalMs.
> Because as far as I can tell, if one of the process,window, or commit
> takes more time it will be reflected in the activeMs value as well.
>
>
>
> On 6/25/15, 7:47 PM, "Luis Fernando De Pombo" <lf...@gmail.com> wrote:
>
> >Hi Shadi,
> >Thanks for asking. This metric tracks the utilization of the event loop
> ><
> https://samza.apache.org/learn/documentation/0.8/container/event-loop.htm
> >l>
> >within a samza container, which uses a single thread, that is in charge of
> >reading and writing messages, flushing metrics, checkpointing, and
> >windowing. It is important to track the utilization (aka "duty cycle") of
> >any event loop, which is the sum of all the timings (activeMs) divided by
> >the window length (totalMs).
> >
> >You are right in that most of the time this value will be close to 1,
> >which
> >represents complete utilization. However when the event loop starts to
> >have
> >idle time, this metric will give you an idea of how much headroom you have
> >before the job will start to seriously fall behind.
> >
> >I hope that answers your question!
> >
> >ᐧ
> >
> >On Thu, Jun 25, 2015 at 6:39 PM, Shadi Noghabi <
> >snoghabi@linkedin.com.invalid> wrote:
> >
> >> Hi,
> >>
> >> I was wondering what does this utilization metric in the
> >> SamzaContainerMetrics show? I am asking this sine in the code it is
> >> calculated as below:
> >>
> >> while (!shutdownNow) {
> >> val loopStartTime = clock();
> >> process
> >> window
> >> commit
> >> val totalMs = clock() - loopStartTime
> >> metrics.utilization.set(activeMs.toFloat/totalMs)
> >> activeMs = 0L
> >> }
> >>
> >> Where the totalMs is the time it takes to start calling process until
> >> commit is done which is equal to the time it takes to run process,
> >>window,
> >> and commit. And they way activeMs is calculated is by summing up the
> >>time
> >> it takes to call process, window and commit, which means these two
> >>values
> >> are going to be almost the same and the utilization is always going to
> >>be
> >> almost 1.
> >>
> >> I was just wondering what is the point of doing this?
> >>
> >>
> >>
> >>
>
>
Re: what does the utilization metric in SamzaContainerMetrics show?
Posted by Shadi Noghabi <sn...@linkedin.com.INVALID>.
Thanks for your prompt reply. I just don’t see when will there be an idle
time that is not added to the activeMs but is counted in the totalMs.
Because as far as I can tell, if one of the process,window, or commit
takes more time it will be reflected in the activeMs value as well.
On 6/25/15, 7:47 PM, "Luis Fernando De Pombo" <lf...@gmail.com> wrote:
>Hi Shadi,
>Thanks for asking. This metric tracks the utilization of the event loop
><https://samza.apache.org/learn/documentation/0.8/container/event-loop.htm
>l>
>within a samza container, which uses a single thread, that is in charge of
>reading and writing messages, flushing metrics, checkpointing, and
>windowing. It is important to track the utilization (aka "duty cycle") of
>any event loop, which is the sum of all the timings (activeMs) divided by
>the window length (totalMs).
>
>You are right in that most of the time this value will be close to 1,
>which
>represents complete utilization. However when the event loop starts to
>have
>idle time, this metric will give you an idea of how much headroom you have
>before the job will start to seriously fall behind.
>
>I hope that answers your question!
>
>ᐧ
>
>On Thu, Jun 25, 2015 at 6:39 PM, Shadi Noghabi <
>snoghabi@linkedin.com.invalid> wrote:
>
>> Hi,
>>
>> I was wondering what does this utilization metric in the
>> SamzaContainerMetrics show? I am asking this sine in the code it is
>> calculated as below:
>>
>> while (!shutdownNow) {
>> val loopStartTime = clock();
>> process
>> window
>> commit
>> val totalMs = clock() - loopStartTime
>> metrics.utilization.set(activeMs.toFloat/totalMs)
>> activeMs = 0L
>> }
>>
>> Where the totalMs is the time it takes to start calling process until
>> commit is done which is equal to the time it takes to run process,
>>window,
>> and commit. And they way activeMs is calculated is by summing up the
>>time
>> it takes to call process, window and commit, which means these two
>>values
>> are going to be almost the same and the utilization is always going to
>>be
>> almost 1.
>>
>> I was just wondering what is the point of doing this?
>>
>>
>>
>>