You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by P Ghosh <ja...@gmail.com> on 2014/04/23 00:19:56 UTC
Understanding metrics.log
I added metrics to my storm implementation by implementing IMetric. It is
working and can see the metrics log populated with all stats. I've a 3 node
(3 worker) and 1 nimbus/zookeeper in Development.
On WORKER1's Metrics Log I can see , some metrics with reference to
WORKER2 and WORKER3. For example,
2014-04-22 16:42:14,829 23002 1398184934 *worker3*:9953
17:normalizeBolt null {bolt.timeconsumed={8,
1657, 5135, 41}}
2014-04-22 16:42:14,830 23003 1398184934 *worker3*:9953
38:normalizeBolt null {db.searchip={3, 77, 221,
7}, fetch-doc={3, 32, 101, 8}}
2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
31:normalizeBolt null {bolt.timeconsumed={7,
1638, 5036, 36}}
2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
10:normalizeBolt null {db.searchip={3, 86, 165,
6}, fetch-doc={3, 17, 53, 7}}
2014-04-22 16:42:14,865 23038 1398184934 *worker1*:9953
44:normalizeSpout null {spout.timeconsumed={656,
8454, 56490, 10}}
*My questions, *
Why I'm seeing the stats of WORKER3 in WORKER1's log file ? My
expectation was each worker will just log it's own metrics and it's upto
the log collector (like splunk or logstash) to merge and interpret.
Thanks,
Prasun
Re: Understanding metrics.log
Posted by P Ghosh <ja...@gmail.com>.
Thanks Danijel, for that information. So, I assume to get a complete
picture I should be merging metrics logs from each of the workers and
then conclude the count. Right ?
Prasun
On Wed, Apr 23, 2014 at 1:39 AM, Danijel Schiavuzzi
<da...@schiavuzzi.com>wrote:
> That's because the metrics consumer you implemented runs on the cluster as
> an actual Storm bolt itself, consuming the metrics from the metrics stream.
> If you enable the "Show System Stats" option in Storm UI, you'll see your
> metrics bolt running on the cluster.
>
> Danijel Schiavuzzi
> www.schiavuzzi.com
>
>
> On Wed, Apr 23, 2014 at 12:19 AM, P Ghosh <ja...@gmail.com> wrote:
>
>> I added metrics to my storm implementation by implementing IMetric. It is
>> working and can see the metrics log populated with all stats. I've a 3 node
>> (3 worker) and 1 nimbus/zookeeper in Development.
>>
>> On WORKER1's Metrics Log I can see , some metrics with reference to
>> WORKER2 and WORKER3. For example,
>>
>> 2014-04-22 16:42:14,829 23002 1398184934 *worker3*:9953
>> 17:normalizeBolt null {bolt.timeconsumed={8,
>> 1657, 5135, 41}}
>> 2014-04-22 16:42:14,830 23003 1398184934 *worker3*:9953
>> 38:normalizeBolt null {db.searchip={3, 77, 221,
>> 7}, fetch-doc={3, 32, 101, 8}}
>> 2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
>> 31:normalizeBolt null {bolt.timeconsumed={7,
>> 1638, 5036, 36}}
>> 2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
>> 10:normalizeBolt null {db.searchip={3, 86, 165,
>> 6}, fetch-doc={3, 17, 53, 7}}
>> 2014-04-22 16:42:14,865 23038 1398184934 *worker1*:9953
>> 44:normalizeSpout null {spout.timeconsumed={656,
>> 8454, 56490, 10}}
>>
>>
>> *My questions, *
>> Why I'm seeing the stats of WORKER3 in WORKER1's log file ? My
>> expectation was each worker will just log it's own metrics and it's upto
>> the log collector (like splunk or logstash) to merge and interpret.
>>
>> Thanks,
>> Prasun
>>
>>
>
>
> --
> Danijel Schiavuzzi
>
> E: danijel@schiavuzzi.com
> W: www.schiavuzzi.com
> T: +385989035562
> Skype: danijels7
>
Re: Understanding metrics.log
Posted by Danijel Schiavuzzi <da...@schiavuzzi.com>.
That's because the metrics consumer you implemented runs on the cluster as
an actual Storm bolt itself, consuming the metrics from the metrics stream.
If you enable the "Show System Stats" option in Storm UI, you'll see your
metrics bolt running on the cluster.
Danijel Schiavuzzi
www.schiavuzzi.com
On Wed, Apr 23, 2014 at 12:19 AM, P Ghosh <ja...@gmail.com> wrote:
> I added metrics to my storm implementation by implementing IMetric. It is
> working and can see the metrics log populated with all stats. I've a 3 node
> (3 worker) and 1 nimbus/zookeeper in Development.
>
> On WORKER1's Metrics Log I can see , some metrics with reference to
> WORKER2 and WORKER3. For example,
>
> 2014-04-22 16:42:14,829 23002 1398184934 *worker3*:9953
> 17:normalizeBolt null {bolt.timeconsumed={8,
> 1657, 5135, 41}}
> 2014-04-22 16:42:14,830 23003 1398184934 *worker3*:9953
> 38:normalizeBolt null {db.searchip={3, 77, 221,
> 7}, fetch-doc={3, 32, 101, 8}}
> 2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
> 31:normalizeBolt null {bolt.timeconsumed={7,
> 1638, 5036, 36}}
> 2014-04-22 16:42:14,833 23006 1398184934 *worker3*:9953
> 10:normalizeBolt null {db.searchip={3, 86, 165,
> 6}, fetch-doc={3, 17, 53, 7}}
> 2014-04-22 16:42:14,865 23038 1398184934 *worker1*:9953
> 44:normalizeSpout null {spout.timeconsumed={656,
> 8454, 56490, 10}}
>
>
> *My questions, *
> Why I'm seeing the stats of WORKER3 in WORKER1's log file ? My
> expectation was each worker will just log it's own metrics and it's upto
> the log collector (like splunk or logstash) to merge and interpret.
>
> Thanks,
> Prasun
>
>
--
Danijel Schiavuzzi
E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7