Storm UI metrics data is cleared sometimes

[Chen Junfeng <k-...@hotmail.com>]

Hi buddies,

I find the storm ui statistic data will be cleared automatically after running for a while.
Some emitted and transferred counts are reset to zero while some components are not.
Did anyone meet the same problem?

Re: Storm UI metrics data is cleared sometimes

[Jungtaek Lim <ka...@gmail.com>]

Chen,

Upcoming Storm 1.1.0 is addressing such concerns on current metrics feature.
Async metrics consumer bolt will minimize performance affection while
metrics consumer doesn't keep up, and you can apply filter by metrics name
to get rid of metrics which you are not interested.

It uses metrics consumer, so you still need to prepare external storage DB
and relevant metrics consumer implementation. There's a popular pair on
Graphite and verisign/storm-graphite, and upcoming Ambari 2.4 will have
Storm metrics consumer for Ambari Metrics System.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 8월 1일 (월) 오후 7:06, Chen Junfeng <k-...@hotmail.com>님이 작성:

> Thanks, Jungtaek.
>
>
>
>
>
> And do you have any good suggestion to measure the performance or data
> volume of processed data in long term without influencing the performance ?
>
>
>
>
>
>
>
> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>
>
>
> *发件人: *Jungtaek Lim <ka...@gmail.com>
> *发送时间: *2016年7月31日 14:44
> *收件人: *user@storm.apache.org
> *主题: *Re: Storm UI metrics data is cleared sometimes
>
>
> Hi Chen,
>
> Storm's built-in metrics are actually task heartbeat stored at ZK, and these
> values are calculated from each task and task writes them to ZK via
> replacing (I mean overwrite).
> If worker is killed and restarted, all metrics for tasks in worker are
> reset to 0, and it overwrites previous values when first heartbeat from
> that worker is made
> This is a limitation of current metrics since we don't store it to any
> storages or centralized in-memory. We'd like to address it.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 7월 28일 (목) 오후 3:58, Chen Junfeng <k-...@hotmail.com>님이 작성:
>
>> Hi buddies,
>>
>>
>>
>> I find the storm ui statistic data will be cleared automatically after
>> running for a while.
>>
>> Some emitted and transferred counts are reset to zero while some
>> components are not.
>>
>> Did anyone meet the same problem?
>>
>>
>>
>>
>>
>>
>>
>

答复: Storm UI metrics data is cleared sometimes

[Chen Junfeng <k-...@hotmail.com>]

Thanks, Jungtaek.


And do you have any good suggestion to measure the performance or data volume of processed data in long term without influencing the performance ?



发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

发件人: Jungtaek Lim<ma...@gmail.com>
发送时间: 2016年7月31日 14:44
收件人: user@storm.apache.org<ma...@storm.apache.org>
主题: Re: Storm UI metrics data is cleared sometimes

Hi Chen,

Storm's built-in metrics are actually task heartbeat stored at ZK, and these values are calculated from each task and task writes them to ZK via replacing (I mean overwrite).
If worker is killed and restarted, all metrics for tasks in worker are reset to 0, and it overwrites previous values when first heartbeat from that worker is made
This is a limitation of current metrics since we don't store it to any storages or centralized in-memory. We'd like to address it.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016? 7? 28? (?) ?? 3:58, Chen Junfeng <k-...@hotmail.com>>?? ??:
Hi buddies,

I find the storm ui statistic data will be cleared automatically after running for a while.
Some emitted and transferred counts are reset to zero while some components are not.
Did anyone meet the same problem?

Re: Storm UI metrics data is cleared sometimes

[Jungtaek Lim <ka...@gmail.com>]

Hi Chen,

Storm's built-in metrics are actually task heartbeat stored at ZK, and these
values are calculated from each task and task writes them to ZK via
replacing (I mean overwrite).
If worker is killed and restarted, all metrics for tasks in worker are
reset to 0, and it overwrites previous values when first heartbeat from
that worker is made
This is a limitation of current metrics since we don't store it to any
storages or centralized in-memory. We'd like to address it.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 7월 28일 (목) 오후 3:58, Chen Junfeng <k-...@hotmail.com>님이 작성:

> Hi buddies,
>
>
>
> I find the storm ui statistic data will be cleared automatically after
> running for a while.
>
> Some emitted and transferred counts are reset to zero while some
> components are not.
>
> Did anyone meet the same problem?
>
>
>
>
>
>
>