You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Mostafa Elhemali <mo...@gmail.com> on 2012/11/27 04:41:58 UTC
Refactor MetricsSystemImpl to allow for an on-demand publish system (HADOOP-9090)
Hi all,
Yesterday I filed a JIRA
(HADOOP-9090<https://issues.apache.org/jira/browse/HADOOP-9090>)
to propose a refactoring of the MetricsSystemImpl class - the default
(only?) implementation of the Metrics2 system - to factor out some common
code in a base class and have another simple implementation that just does
on-demand publishing of metrics instead of the default periodic publishing.
The main motivation for filing this JIRA and the attached patch is that we
(Microsoft) have a need to publish metrics out of short-lived processes
(think "hadoop fs -ls") and the periodic behavior of the default
implementation doesn't really work well for those. We could write our own
metrics system implementation (and we'll probably do that in the short
term) but that would mean duplicating a lot of great code that's already in
the MetricsSystemImpl class, hence the proposal to factor out the common
code into a base class.
Does that sound reasonable? Please comment on the JIRA directly or reply
here - if the proposal sounds awful (or great) or there's something I'm
fundamentally missing I'd love to hear that feedback.
Thanks!
Mostafa
Re: Refactor MetricsSystemImpl to allow for an on-demand publish
system (HADOOP-9090)
Posted by Mostafa Elhemali <mo...@gmail.com>.
I've attached an alternative patch to the JIRA with your suggestions -
thanks again for the feedback! I think it's much simpler now.
On Tue, Nov 27, 2012 at 10:22 AM, Mostafa Elhemali <
mostafa.elhemaly@gmail.com> wrote:
> Thanks for the feedback Luke. You make a good case - to be honest I shyed
> away from modifying the default implementation since I figured it's better
> to leave existing functionality as is, but you're right that being forced
> to choose an implementation (on-demand vs. periodic) can get annoying. Let
> me try out that approach and either post an alternative patch to the JIRA
> or report back on why I failed.
>
>
> On Tue, Nov 27, 2012 at 9:37 AM, Luke Lu <ll...@vicaya.com> wrote:
>
>> Thanks for the heads up Mostafa. I've added commment/questions on the
>> JIRA.
>> The main concern is that now you have to decide what metrics system to use
>> depending on the process is long running or not. Why not just enhance the
>> existing metrics system itself to support on-demand publishing? So that
>> you
>> can use the same metrics system for all processes? We all know a
>> short-lived process can live longer than expected and periodic metrics
>> from
>> that process can provide insights on what's going on, which is the main
>> purpose of the metrics system.
>>
>> __Luke
>>
>>
>> On Mon, Nov 26, 2012 at 7:41 PM, Mostafa Elhemali <
>> mostafa.elhemaly@gmail.com> wrote:
>>
>> > Hi all,
>> > Yesterday I filed a JIRA
>> > (HADOOP-9090<https://issues.apache.org/jira/browse/HADOOP-9090>)
>> > to propose a refactoring of the MetricsSystemImpl class - the default
>> > (only?) implementation of the Metrics2 system - to factor out some
>> common
>> > code in a base class and have another simple implementation that just
>> does
>> > on-demand publishing of metrics instead of the default periodic
>> publishing.
>> > The main motivation for filing this JIRA and the attached patch is that
>> we
>> > (Microsoft) have a need to publish metrics out of short-lived processes
>> > (think "hadoop fs -ls") and the periodic behavior of the default
>> > implementation doesn't really work well for those. We could write our
>> own
>> > metrics system implementation (and we'll probably do that in the short
>> > term) but that would mean duplicating a lot of great code that's
>> already in
>> > the MetricsSystemImpl class, hence the proposal to factor out the common
>> > code into a base class.
>> >
>> > Does that sound reasonable? Please comment on the JIRA directly or reply
>> > here - if the proposal sounds awful (or great) or there's something I'm
>> > fundamentally missing I'd love to hear that feedback.
>> >
>> > Thanks!
>> > Mostafa
>> >
>>
>
>
Re: Refactor MetricsSystemImpl to allow for an on-demand publish
system (HADOOP-9090)
Posted by Mostafa Elhemali <mo...@gmail.com>.
Thanks for the feedback Luke. You make a good case - to be honest I shyed
away from modifying the default implementation since I figured it's better
to leave existing functionality as is, but you're right that being forced
to choose an implementation (on-demand vs. periodic) can get annoying. Let
me try out that approach and either post an alternative patch to the JIRA
or report back on why I failed.
On Tue, Nov 27, 2012 at 9:37 AM, Luke Lu <ll...@vicaya.com> wrote:
> Thanks for the heads up Mostafa. I've added commment/questions on the JIRA.
> The main concern is that now you have to decide what metrics system to use
> depending on the process is long running or not. Why not just enhance the
> existing metrics system itself to support on-demand publishing? So that you
> can use the same metrics system for all processes? We all know a
> short-lived process can live longer than expected and periodic metrics from
> that process can provide insights on what's going on, which is the main
> purpose of the metrics system.
>
> __Luke
>
>
> On Mon, Nov 26, 2012 at 7:41 PM, Mostafa Elhemali <
> mostafa.elhemaly@gmail.com> wrote:
>
> > Hi all,
> > Yesterday I filed a JIRA
> > (HADOOP-9090<https://issues.apache.org/jira/browse/HADOOP-9090>)
> > to propose a refactoring of the MetricsSystemImpl class - the default
> > (only?) implementation of the Metrics2 system - to factor out some common
> > code in a base class and have another simple implementation that just
> does
> > on-demand publishing of metrics instead of the default periodic
> publishing.
> > The main motivation for filing this JIRA and the attached patch is that
> we
> > (Microsoft) have a need to publish metrics out of short-lived processes
> > (think "hadoop fs -ls") and the periodic behavior of the default
> > implementation doesn't really work well for those. We could write our own
> > metrics system implementation (and we'll probably do that in the short
> > term) but that would mean duplicating a lot of great code that's already
> in
> > the MetricsSystemImpl class, hence the proposal to factor out the common
> > code into a base class.
> >
> > Does that sound reasonable? Please comment on the JIRA directly or reply
> > here - if the proposal sounds awful (or great) or there's something I'm
> > fundamentally missing I'd love to hear that feedback.
> >
> > Thanks!
> > Mostafa
> >
>
Re: Refactor MetricsSystemImpl to allow for an on-demand publish
system (HADOOP-9090)
Posted by Luke Lu <ll...@vicaya.com>.
Thanks for the heads up Mostafa. I've added commment/questions on the JIRA.
The main concern is that now you have to decide what metrics system to use
depending on the process is long running or not. Why not just enhance the
existing metrics system itself to support on-demand publishing? So that you
can use the same metrics system for all processes? We all know a
short-lived process can live longer than expected and periodic metrics from
that process can provide insights on what's going on, which is the main
purpose of the metrics system.
__Luke
On Mon, Nov 26, 2012 at 7:41 PM, Mostafa Elhemali <
mostafa.elhemaly@gmail.com> wrote:
> Hi all,
> Yesterday I filed a JIRA
> (HADOOP-9090<https://issues.apache.org/jira/browse/HADOOP-9090>)
> to propose a refactoring of the MetricsSystemImpl class - the default
> (only?) implementation of the Metrics2 system - to factor out some common
> code in a base class and have another simple implementation that just does
> on-demand publishing of metrics instead of the default periodic publishing.
> The main motivation for filing this JIRA and the attached patch is that we
> (Microsoft) have a need to publish metrics out of short-lived processes
> (think "hadoop fs -ls") and the periodic behavior of the default
> implementation doesn't really work well for those. We could write our own
> metrics system implementation (and we'll probably do that in the short
> term) but that would mean duplicating a lot of great code that's already in
> the MetricsSystemImpl class, hence the proposal to factor out the common
> code into a base class.
>
> Does that sound reasonable? Please comment on the JIRA directly or reply
> here - if the proposal sounds awful (or great) or there's something I'm
> fundamentally missing I'd love to hear that feedback.
>
> Thanks!
> Mostafa
>