You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Abhishek Agarwal <ab...@gmail.com> on 2015/08/05 13:04:53 UTC
Storm metrics are not being logged
Hi,
I have created few CountMetrics in my bolt which I initialize and register
in the prepare method of my bolt. I run my topology on intellij in debug
mode and I have verified that metrics increases. Yet, the reporting seems
to be inconsistent. Sometimes the metrics get logged and other times it is
not the case. What could be the reason which could result in something like
this?
--
Regards,
Abhishek Agarwal
Re: Storm metrics are not being logged
Posted by Abhishek Agarwal <ab...@gmail.com>.
Bolt execution slowness was indeed the reason. After I increased the
parallelism of bolt tasks, metrics started to get logged.
On Thu, Aug 6, 2015 at 2:04 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> I am just guessing here. The rate at which bolt executes the tuple, would
> affect the rate at which metric is being logged. A metric tick is sent on a
> different stream after every 60 seconds. Metrics are logged by the executor
> when that tick is consumed. But tick itself won't be consumed unless all
> the input tuples sent before that are consumed. Isn't it?
>
> On Thu, Aug 6, 2015 at 1:41 AM, Abhishek Agarwal <ab...@gmail.com>
> wrote:
>
>> That setting was set to 60 seconds. Metrics do get logged but the timings
>> are totally inconsistent. I am noting the same behavior for builtin metrics
>> too. Observed time interval between metric logs is not constant and differs
>> from what is specified in configuration.
>>
>> On Wed, Aug 5, 2015 at 8:12 PM, Hong Jeon <hj...@cornell.edu> wrote:
>>
>>> What is the setting of your "topology.builtin.metrics.bucket.secs" in
>>> your storm.yaml?
>>> You can also set this value programatically through the Config object.
>>> Basically, all of the registered metrics in your topology will be
>>> reported to your metrics consumer every bucket seconds you specify.
>>>
>>> Maybe you are only seeing the CountMetrics get written to the log every
>>> so often, while your debug happens more frequently?
>>>
>>> On Wed, Aug 5, 2015 at 7:04 AM, Abhishek Agarwal <ab...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I have created few CountMetrics in my bolt which I initialize and
>>>> register in the prepare method of my bolt. I run my topology on intellij in
>>>> debug mode and I have verified that metrics increases. Yet, the reporting
>>>> seems to be inconsistent. Sometimes the metrics get logged and other times
>>>> it is not the case. What could be the reason which could result in
>>>> something like this?
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Abhishek Agarwal
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Abhishek Agarwal
>>
>>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>
--
Regards,
Abhishek Agarwal
Re: Storm metrics are not being logged
Posted by Abhishek Agarwal <ab...@gmail.com>.
I am just guessing here. The rate at which bolt executes the tuple, would
affect the rate at which metric is being logged. A metric tick is sent on a
different stream after every 60 seconds. Metrics are logged by the executor
when that tick is consumed. But tick itself won't be consumed unless all
the input tuples sent before that are consumed. Isn't it?
On Thu, Aug 6, 2015 at 1:41 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> That setting was set to 60 seconds. Metrics do get logged but the timings
> are totally inconsistent. I am noting the same behavior for builtin metrics
> too. Observed time interval between metric logs is not constant and differs
> from what is specified in configuration.
>
> On Wed, Aug 5, 2015 at 8:12 PM, Hong Jeon <hj...@cornell.edu> wrote:
>
>> What is the setting of your "topology.builtin.metrics.bucket.secs" in
>> your storm.yaml?
>> You can also set this value programatically through the Config object.
>> Basically, all of the registered metrics in your topology will be
>> reported to your metrics consumer every bucket seconds you specify.
>>
>> Maybe you are only seeing the CountMetrics get written to the log every
>> so often, while your debug happens more frequently?
>>
>> On Wed, Aug 5, 2015 at 7:04 AM, Abhishek Agarwal <ab...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I have created few CountMetrics in my bolt which I initialize and
>>> register in the prepare method of my bolt. I run my topology on intellij in
>>> debug mode and I have verified that metrics increases. Yet, the reporting
>>> seems to be inconsistent. Sometimes the metrics get logged and other times
>>> it is not the case. What could be the reason which could result in
>>> something like this?
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Abhishek Agarwal
>>>
>>>
>>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>
--
Regards,
Abhishek Agarwal
Re: Storm metrics are not being logged
Posted by Abhishek Agarwal <ab...@gmail.com>.
That setting was set to 60 seconds. Metrics do get logged but the timings
are totally inconsistent. I am noting the same behavior for builtin metrics
too. Observed time interval between metric logs is not constant and differs
from what is specified in configuration.
On Wed, Aug 5, 2015 at 8:12 PM, Hong Jeon <hj...@cornell.edu> wrote:
> What is the setting of your "topology.builtin.metrics.bucket.secs" in your
> storm.yaml?
> You can also set this value programatically through the Config object.
> Basically, all of the registered metrics in your topology will be reported
> to your metrics consumer every bucket seconds you specify.
>
> Maybe you are only seeing the CountMetrics get written to the log every so
> often, while your debug happens more frequently?
>
> On Wed, Aug 5, 2015 at 7:04 AM, Abhishek Agarwal <ab...@gmail.com>
> wrote:
>
>> Hi,
>> I have created few CountMetrics in my bolt which I initialize and
>> register in the prepare method of my bolt. I run my topology on intellij in
>> debug mode and I have verified that metrics increases. Yet, the reporting
>> seems to be inconsistent. Sometimes the metrics get logged and other times
>> it is not the case. What could be the reason which could result in
>> something like this?
>>
>>
>>
>> --
>> Regards,
>> Abhishek Agarwal
>>
>>
>
--
Regards,
Abhishek Agarwal
Re: Storm metrics are not being logged
Posted by Hong Jeon <hj...@cornell.edu>.
What is the setting of your "topology.builtin.metrics.bucket.secs" in your
storm.yaml?
You can also set this value programatically through the Config object.
Basically, all of the registered metrics in your topology will be reported
to your metrics consumer every bucket seconds you specify.
Maybe you are only seeing the CountMetrics get written to the log every so
often, while your debug happens more frequently?
On Wed, Aug 5, 2015 at 7:04 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> Hi,
> I have created few CountMetrics in my bolt which I initialize and register
> in the prepare method of my bolt. I run my topology on intellij in debug
> mode and I have verified that metrics increases. Yet, the reporting seems
> to be inconsistent. Sometimes the metrics get logged and other times it is
> not the case. What could be the reason which could result in something like
> this?
>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>