You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by "Liangfei.Su" <su...@gmail.com> on 2016/02/03 14:42:58 UTC

[DISCUSS] Generic topology for hadoop metric monitoring

Eagle community

By working on https://issues.apache.org/jira/browse/EAGLE-149 to provide
hadoop metric monitoring. Current eagle way to support this kind of metric
monitoring seems could be abstracted.
I'm creating an early draft to provide a generic topology proposal for more
elegant hadoop metric monitoring
https://cwiki.apache.org/confluence/display/EAG/Generic+Topology+For+Hadoop+Metric+Monitoring
.

The fundamental idea is to introduce
0. The assumption that metric tuple would contain some descriptive
information like "metricName"
1. Stream multiplex with storm stream emit to hadoop metric to given policy
executor
2. Introduce policy executor which recognize the metric, and then could
naturally do stream join (Say the case of identify the metric on different
name node and make join for alert. Check NameNodeLagPolicy in PR #81).
3. Use the metadata to help the multiplexer/executor so that we could
define generic topology for metric monitoring.


Please comment & suggest.

Thanks,
Ralph

Re: [DISCUSS] Generic topology for hadoop metric monitoring

Posted by "Liangfei.Su" <su...@gmail.com>.
Yes,

I'm making the metric metadata (stream metadata) with reference to topic,
so that for a given metric, eagle could get the info of kafka topic from
the definition.

On Thu, Feb 4, 2016 at 5:52 AM, Zhang, Edward (GDI Hadoop) <
yonzhang@ebay.com> wrote:

> Some more suggestions
> 1. metric can belong to any Kafka topic. Once after metric monitoring is
> modeled in UI, Eagle knows which topic to consume for this metric
> 2. metric can be differentiated by field ³metricName² or by any field
> which can be specified while metric monitoring is modeled. Eagle knows how
> to map one metric from one topic into one metric stream
> 3. if 2 or more metrics are to be joined, that should be modeled in UI and
> Eagle knows how to send multiple metrics into related policies.
>
> Thanks
> Edward
>
> On 2/3/16, 5:42, "Liangfei.Su" <su...@gmail.com> wrote:
>
> >Eagle community
> >
> >By working on https://issues.apache.org/jira/browse/EAGLE-149 to provide
> >hadoop metric monitoring. Current eagle way to support this kind of metric
> >monitoring seems could be abstracted.
> >I'm creating an early draft to provide a generic topology proposal for
> >more
> >elegant hadoop metric monitoring
> >
> https://cwiki.apache.org/confluence/display/EAG/Generic+Topology+For+Hadoo
> >p+Metric+Monitoring
> >.
> >
> >The fundamental idea is to introduce
> >0. The assumption that metric tuple would contain some descriptive
> >information like "metricName"
> >1. Stream multiplex with storm stream emit to hadoop metric to given
> >policy
> >executor
> >2. Introduce policy executor which recognize the metric, and then could
> >naturally do stream join (Say the case of identify the metric on different
> >name node and make join for alert. Check NameNodeLagPolicy in PR #81).
> >3. Use the metadata to help the multiplexer/executor so that we could
> >define generic topology for metric monitoring.
> >
> >
> >Please comment & suggest.
> >
> >Thanks,
> >Ralph
>
>

Re: [DISCUSS] Generic topology for hadoop metric monitoring

Posted by "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>.
Some more suggestions
1. metric can belong to any Kafka topic. Once after metric monitoring is
modeled in UI, Eagle knows which topic to consume for this metric
2. metric can be differentiated by field ³metricName² or by any field
which can be specified while metric monitoring is modeled. Eagle knows how
to map one metric from one topic into one metric stream
3. if 2 or more metrics are to be joined, that should be modeled in UI and
Eagle knows how to send multiple metrics into related policies.

Thanks
Edward

On 2/3/16, 5:42, "Liangfei.Su" <su...@gmail.com> wrote:

>Eagle community
>
>By working on https://issues.apache.org/jira/browse/EAGLE-149 to provide
>hadoop metric monitoring. Current eagle way to support this kind of metric
>monitoring seems could be abstracted.
>I'm creating an early draft to provide a generic topology proposal for
>more
>elegant hadoop metric monitoring
>https://cwiki.apache.org/confluence/display/EAG/Generic+Topology+For+Hadoo
>p+Metric+Monitoring
>.
>
>The fundamental idea is to introduce
>0. The assumption that metric tuple would contain some descriptive
>information like "metricName"
>1. Stream multiplex with storm stream emit to hadoop metric to given
>policy
>executor
>2. Introduce policy executor which recognize the metric, and then could
>naturally do stream join (Say the case of identify the metric on different
>name node and make join for alert. Check NameNodeLagPolicy in PR #81).
>3. Use the metadata to help the multiplexer/executor so that we could
>define generic topology for metric monitoring.
>
>
>Please comment & suggest.
>
>Thanks,
>Ralph