plugging in to hadoop

[tejas <te...@oracle.com>]

Hi

I am trying to develop a UI and display metrics of mapreduce and HDFS 
and also provide some kind of monitoring.

1. With version 0.20, I was able to extend the 
org.apache.JobTrackerPlugin class by configuring it in the 
mapred-site.properties. Is it possible to make a similar configuration 
in version 0.23? Can i plugin my class to fetch metrics through 
JobTrackerPlugin?
I see that some of the classes in JobTracker class have been depricated? 
So would i have to plugin to the ResourceManager and the NodeManagers to 
get the metrics? Is there any api for that?

2. How does hadoop expose traps and events? If i would have to capture 
events at unfaourable states (like tracker coming down, datanode going 
low on space etc) does Hadoop send out events? How can they be trapped?

Thanks

Tejas

Re: plugging in to hadoop

[tejas <te...@oracle.com>]

I am actually looking at implementing my own UI. I am looking to develop 
monitoring and control framework for hadoop
I am mainly looking at hadoo 0.23

On 2/1/2012 8:36 AM, real great.. wrote:
> I think there are a couple of projects already looking into UI. did you
> check them?
>
> On Wed, Feb 1, 2012 at 7:57 AM, tejas<te...@oracle.com>wrote:
>
>> Hi
>>
>> I am trying to develop a UI and display metrics of mapreduce and HDFS and
>> also provide some kind of monitoring.
>>
>> 1. With version 0.20, I was able to extend the org.apache.JobTrackerPlugin
>> class by configuring it in the mapred-site.properties. Is it possible to
>> make a similar configuration in version 0.23? Can i plugin my class to
>> fetch metrics through JobTrackerPlugin?
>> I see that some of the classes in JobTracker class have been depricated?
>> So would i have to plugin to the ResourceManager and the NodeManagers to
>> get the metrics? Is there any api for that?
>>
>> 2. How does hadoop expose traps and events? If i would have to capture
>> events at unfaourable states (like tracker coming down, datanode going low
>> on space etc) does Hadoop send out events? How can they be trapped?
>>
>> Thanks
>>
>> Tejas
>>
>
>

Re: plugging in to hadoop

["real great.." <gr...@gmail.com>]

I think there are a couple of projects already looking into UI. did you
check them?

On Wed, Feb 1, 2012 at 7:57 AM, tejas <te...@oracle.com>wrote:

> Hi
>
> I am trying to develop a UI and display metrics of mapreduce and HDFS and
> also provide some kind of monitoring.
>
> 1. With version 0.20, I was able to extend the org.apache.JobTrackerPlugin
> class by configuring it in the mapred-site.properties. Is it possible to
> make a similar configuration in version 0.23? Can i plugin my class to
> fetch metrics through JobTrackerPlugin?
> I see that some of the classes in JobTracker class have been depricated?
> So would i have to plugin to the ResourceManager and the NodeManagers to
> get the metrics? Is there any api for that?
>
> 2. How does hadoop expose traps and events? If i would have to capture
> events at unfaourable states (like tracker coming down, datanode going low
> on space etc) does Hadoop send out events? How can they be trapped?
>
> Thanks
>
> Tejas
>



-- 
Regards,
R.V.