You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Anfernee Xu <an...@gmail.com> on 2014/02/04 05:07:44 UTC
what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Hi,
I'm migrating my application from MR v.1 to YARN, I do not know how to
convert my class which extends
org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to run
as TaskTracker starts up, if I migrate to YARN, I want it to run as
NodeManager starts up, so how can I make this happen?
--
--Anfernee
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Anfernee Xu <an...@gmail.com>.
Thanks Harsh,
My target is that I want to monitor disk space usage on each node in the
cluster, if space usage reaches to a certain threshold on one node, my
scheduler will not select slot on that node, in order to do that, I have a
instrumentation class on each TT to report disk space usage to JobTracker
periodically. Now I want to move to YARN, I also notice YARN metric system:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html,
do you think I can use it?
For example,
1) I implement a metric source to collect disk space usage
2) write a filter to only collect this metric
3) write a sink to publish the metric to RM
Please comment.
Thanks a lot
On Tue, Feb 4, 2014 at 5:02 AM, Harsh J <ha...@cloudera.com> wrote:
> Unfortunately there isn't a direct equivalent available for plugging
> in instrumentation classes in YARN's NodeManagers currently. The
> closest I can locate is a comment hint in the NodeManagerMetrics class
> that toys with the same idea:
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
>
> If you'll absolutely need this in YARN, can you please file a YARN
> JIRA with the motivation on why?
>
> On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> > Hi,
> >
> > I'm migrating my application from MR v.1 to YARN, I do not know how to
> > convert my class which extends
> > org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to
> run as
> > TaskTracker starts up, if I migrate to YARN, I want it to run as
> NodeManager
> > starts up, so how can I make this happen?
> >
> > --
> > --Anfernee
>
>
>
> --
> Harsh J
>
--
--Anfernee
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Anfernee Xu <an...@gmail.com>.
Thanks Harsh,
My target is that I want to monitor disk space usage on each node in the
cluster, if space usage reaches to a certain threshold on one node, my
scheduler will not select slot on that node, in order to do that, I have a
instrumentation class on each TT to report disk space usage to JobTracker
periodically. Now I want to move to YARN, I also notice YARN metric system:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html,
do you think I can use it?
For example,
1) I implement a metric source to collect disk space usage
2) write a filter to only collect this metric
3) write a sink to publish the metric to RM
Please comment.
Thanks a lot
On Tue, Feb 4, 2014 at 5:02 AM, Harsh J <ha...@cloudera.com> wrote:
> Unfortunately there isn't a direct equivalent available for plugging
> in instrumentation classes in YARN's NodeManagers currently. The
> closest I can locate is a comment hint in the NodeManagerMetrics class
> that toys with the same idea:
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
>
> If you'll absolutely need this in YARN, can you please file a YARN
> JIRA with the motivation on why?
>
> On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> > Hi,
> >
> > I'm migrating my application from MR v.1 to YARN, I do not know how to
> > convert my class which extends
> > org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to
> run as
> > TaskTracker starts up, if I migrate to YARN, I want it to run as
> NodeManager
> > starts up, so how can I make this happen?
> >
> > --
> > --Anfernee
>
>
>
> --
> Harsh J
>
--
--Anfernee
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Anfernee Xu <an...@gmail.com>.
Thanks Harsh,
My target is that I want to monitor disk space usage on each node in the
cluster, if space usage reaches to a certain threshold on one node, my
scheduler will not select slot on that node, in order to do that, I have a
instrumentation class on each TT to report disk space usage to JobTracker
periodically. Now I want to move to YARN, I also notice YARN metric system:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html,
do you think I can use it?
For example,
1) I implement a metric source to collect disk space usage
2) write a filter to only collect this metric
3) write a sink to publish the metric to RM
Please comment.
Thanks a lot
On Tue, Feb 4, 2014 at 5:02 AM, Harsh J <ha...@cloudera.com> wrote:
> Unfortunately there isn't a direct equivalent available for plugging
> in instrumentation classes in YARN's NodeManagers currently. The
> closest I can locate is a comment hint in the NodeManagerMetrics class
> that toys with the same idea:
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
>
> If you'll absolutely need this in YARN, can you please file a YARN
> JIRA with the motivation on why?
>
> On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> > Hi,
> >
> > I'm migrating my application from MR v.1 to YARN, I do not know how to
> > convert my class which extends
> > org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to
> run as
> > TaskTracker starts up, if I migrate to YARN, I want it to run as
> NodeManager
> > starts up, so how can I make this happen?
> >
> > --
> > --Anfernee
>
>
>
> --
> Harsh J
>
--
--Anfernee
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Anfernee Xu <an...@gmail.com>.
Thanks Harsh,
My target is that I want to monitor disk space usage on each node in the
cluster, if space usage reaches to a certain threshold on one node, my
scheduler will not select slot on that node, in order to do that, I have a
instrumentation class on each TT to report disk space usage to JobTracker
periodically. Now I want to move to YARN, I also notice YARN metric system:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html,
do you think I can use it?
For example,
1) I implement a metric source to collect disk space usage
2) write a filter to only collect this metric
3) write a sink to publish the metric to RM
Please comment.
Thanks a lot
On Tue, Feb 4, 2014 at 5:02 AM, Harsh J <ha...@cloudera.com> wrote:
> Unfortunately there isn't a direct equivalent available for plugging
> in instrumentation classes in YARN's NodeManagers currently. The
> closest I can locate is a comment hint in the NodeManagerMetrics class
> that toys with the same idea:
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
>
> If you'll absolutely need this in YARN, can you please file a YARN
> JIRA with the motivation on why?
>
> On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> > Hi,
> >
> > I'm migrating my application from MR v.1 to YARN, I do not know how to
> > convert my class which extends
> > org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to
> run as
> > TaskTracker starts up, if I migrate to YARN, I want it to run as
> NodeManager
> > starts up, so how can I make this happen?
> >
> > --
> > --Anfernee
>
>
>
> --
> Harsh J
>
--
--Anfernee
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Harsh J <ha...@cloudera.com>.
Unfortunately there isn't a direct equivalent available for plugging
in instrumentation classes in YARN's NodeManagers currently. The
closest I can locate is a comment hint in the NodeManagerMetrics class
that toys with the same idea:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
If you'll absolutely need this in YARN, can you please file a YARN
JIRA with the motivation on why?
On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> I'm migrating my application from MR v.1 to YARN, I do not know how to
> convert my class which extends
> org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to run as
> TaskTracker starts up, if I migrate to YARN, I want it to run as NodeManager
> starts up, so how can I make this happen?
>
> --
> --Anfernee
--
Harsh J
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Harsh J <ha...@cloudera.com>.
Unfortunately there isn't a direct equivalent available for plugging
in instrumentation classes in YARN's NodeManagers currently. The
closest I can locate is a comment hint in the NodeManagerMetrics class
that toys with the same idea:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
If you'll absolutely need this in YARN, can you please file a YARN
JIRA with the motivation on why?
On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> I'm migrating my application from MR v.1 to YARN, I do not know how to
> convert my class which extends
> org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to run as
> TaskTracker starts up, if I migrate to YARN, I want it to run as NodeManager
> starts up, so how can I make this happen?
>
> --
> --Anfernee
--
Harsh J
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Harsh J <ha...@cloudera.com>.
Unfortunately there isn't a direct equivalent available for plugging
in instrumentation classes in YARN's NodeManagers currently. The
closest I can locate is a comment hint in the NodeManagerMetrics class
that toys with the same idea:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
If you'll absolutely need this in YARN, can you please file a YARN
JIRA with the motivation on why?
On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> I'm migrating my application from MR v.1 to YARN, I do not know how to
> convert my class which extends
> org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to run as
> TaskTracker starts up, if I migrate to YARN, I want it to run as NodeManager
> starts up, so how can I make this happen?
>
> --
> --Anfernee
--
Harsh J
Re: what's counterpart for org.apache.hadoop.mapred.TaskTrackerMetricsInst
in YARN
Posted by Harsh J <ha...@cloudera.com>.
Unfortunately there isn't a direct equivalent available for plugging
in instrumentation classes in YARN's NodeManagers currently. The
closest I can locate is a comment hint in the NodeManagerMetrics class
that toys with the same idea:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java#L53
If you'll absolutely need this in YARN, can you please file a YARN
JIRA with the motivation on why?
On Tue, Feb 4, 2014 at 9:37 AM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> I'm migrating my application from MR v.1 to YARN, I do not know how to
> convert my class which extends
> org.apache.hadoop.mapred.TaskTrackerMetricsInst, I want this class to run as
> TaskTracker starts up, if I migrate to YARN, I want it to run as NodeManager
> starts up, so how can I make this happen?
>
> --
> --Anfernee
--
Harsh J