You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Nan Zhu <zh...@gmail.com> on 2012/10/30 04:49:04 UTC
Access Hadoop Counters
Hi, all
I noticed that in FileInputFormat and FileOutputFormat classes, there are counters(Counter's name?) for input and output data size,
my question is that given a JobInProgress object, e.g. jip, after I get the corresponding object by
FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
FileOutputFormat fof = (FileOutputFormat) jip.getConf().getOutputputFormat();
How can I access the values of the counters?
Best,
--
Nan Zhu
School of Computer Science,
McGill University
Re: Access Hadoop Counters
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Harsh
thanks for your reply,
yes, I'm modifying the source code of MR,
I have dig it out, just as jip.getMapCounters().getCounter(FileInputFormat.Counter.BYTES_READ);
Thx
--
Nan Zhu
School of Computer Science,
McGill University
On Tuesday, October 30, 2012 at 12:51 AM, Harsh J wrote:
> Hi,
>
> Are you attempting to modify MR sources? JobInProgress is a system
> class, not to be used by users directly. If you seek MR project-side
> development help, its better to reach the mapreduce-dev@ lists
> instead.
>
> That said, if you want the file/hdfs bytes counters, you can access
> them via the RunningJob or Job APIs directly (just give the group and
> counter names).
>
> On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zhunansjtu@gmail.com (mailto:zhunansjtu@gmail.com)> wrote:
> > Hi, all
> >
> > I noticed that in FileInputFormat and FileOutputFormat classes, there are
> > counters(Counter's name?) for input and output data size,
> >
> > my question is that given a JobInProgress object, e.g. jip, after I get the
> > corresponding object by
> >
> > FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> > FileOutputFormat fof = (FileOutputFormat)
> > jip.getConf().getOutputputFormat();
> >
> > How can I access the values of the counters?
> >
> > Best,
> >
> > --
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >
>
>
>
>
> --
> Harsh J
>
>
Re: Access Hadoop Counters
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Harsh
thanks for your reply,
yes, I'm modifying the source code of MR,
I have dig it out, just as jip.getMapCounters().getCounter(FileInputFormat.Counter.BYTES_READ);
Thx
--
Nan Zhu
School of Computer Science,
McGill University
On Tuesday, October 30, 2012 at 12:51 AM, Harsh J wrote:
> Hi,
>
> Are you attempting to modify MR sources? JobInProgress is a system
> class, not to be used by users directly. If you seek MR project-side
> development help, its better to reach the mapreduce-dev@ lists
> instead.
>
> That said, if you want the file/hdfs bytes counters, you can access
> them via the RunningJob or Job APIs directly (just give the group and
> counter names).
>
> On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zhunansjtu@gmail.com (mailto:zhunansjtu@gmail.com)> wrote:
> > Hi, all
> >
> > I noticed that in FileInputFormat and FileOutputFormat classes, there are
> > counters(Counter's name?) for input and output data size,
> >
> > my question is that given a JobInProgress object, e.g. jip, after I get the
> > corresponding object by
> >
> > FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> > FileOutputFormat fof = (FileOutputFormat)
> > jip.getConf().getOutputputFormat();
> >
> > How can I access the values of the counters?
> >
> > Best,
> >
> > --
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >
>
>
>
>
> --
> Harsh J
>
>
Re: Access Hadoop Counters
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Harsh
thanks for your reply,
yes, I'm modifying the source code of MR,
I have dig it out, just as jip.getMapCounters().getCounter(FileInputFormat.Counter.BYTES_READ);
Thx
--
Nan Zhu
School of Computer Science,
McGill University
On Tuesday, October 30, 2012 at 12:51 AM, Harsh J wrote:
> Hi,
>
> Are you attempting to modify MR sources? JobInProgress is a system
> class, not to be used by users directly. If you seek MR project-side
> development help, its better to reach the mapreduce-dev@ lists
> instead.
>
> That said, if you want the file/hdfs bytes counters, you can access
> them via the RunningJob or Job APIs directly (just give the group and
> counter names).
>
> On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zhunansjtu@gmail.com (mailto:zhunansjtu@gmail.com)> wrote:
> > Hi, all
> >
> > I noticed that in FileInputFormat and FileOutputFormat classes, there are
> > counters(Counter's name?) for input and output data size,
> >
> > my question is that given a JobInProgress object, e.g. jip, after I get the
> > corresponding object by
> >
> > FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> > FileOutputFormat fof = (FileOutputFormat)
> > jip.getConf().getOutputputFormat();
> >
> > How can I access the values of the counters?
> >
> > Best,
> >
> > --
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >
>
>
>
>
> --
> Harsh J
>
>
Re: Access Hadoop Counters
Posted by Nan Zhu <zh...@gmail.com>.
Hi, Harsh
thanks for your reply,
yes, I'm modifying the source code of MR,
I have dig it out, just as jip.getMapCounters().getCounter(FileInputFormat.Counter.BYTES_READ);
Thx
--
Nan Zhu
School of Computer Science,
McGill University
On Tuesday, October 30, 2012 at 12:51 AM, Harsh J wrote:
> Hi,
>
> Are you attempting to modify MR sources? JobInProgress is a system
> class, not to be used by users directly. If you seek MR project-side
> development help, its better to reach the mapreduce-dev@ lists
> instead.
>
> That said, if you want the file/hdfs bytes counters, you can access
> them via the RunningJob or Job APIs directly (just give the group and
> counter names).
>
> On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zhunansjtu@gmail.com (mailto:zhunansjtu@gmail.com)> wrote:
> > Hi, all
> >
> > I noticed that in FileInputFormat and FileOutputFormat classes, there are
> > counters(Counter's name?) for input and output data size,
> >
> > my question is that given a JobInProgress object, e.g. jip, after I get the
> > corresponding object by
> >
> > FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> > FileOutputFormat fof = (FileOutputFormat)
> > jip.getConf().getOutputputFormat();
> >
> > How can I access the values of the counters?
> >
> > Best,
> >
> > --
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >
>
>
>
>
> --
> Harsh J
>
>
Re: Access Hadoop Counters
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Are you attempting to modify MR sources? JobInProgress is a system
class, not to be used by users directly. If you seek MR project-side
development help, its better to reach the mapreduce-dev@ lists
instead.
That said, if you want the file/hdfs bytes counters, you can access
them via the RunningJob or Job APIs directly (just give the group and
counter names).
On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> I noticed that in FileInputFormat and FileOutputFormat classes, there are
> counters(Counter's name?) for input and output data size,
>
> my question is that given a JobInProgress object, e.g. jip, after I get the
> corresponding object by
>
> FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> FileOutputFormat fof = (FileOutputFormat)
> jip.getConf().getOutputputFormat();
>
> How can I access the values of the counters?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
--
Harsh J
Re: Access Hadoop Counters
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Are you attempting to modify MR sources? JobInProgress is a system
class, not to be used by users directly. If you seek MR project-side
development help, its better to reach the mapreduce-dev@ lists
instead.
That said, if you want the file/hdfs bytes counters, you can access
them via the RunningJob or Job APIs directly (just give the group and
counter names).
On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> I noticed that in FileInputFormat and FileOutputFormat classes, there are
> counters(Counter's name?) for input and output data size,
>
> my question is that given a JobInProgress object, e.g. jip, after I get the
> corresponding object by
>
> FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> FileOutputFormat fof = (FileOutputFormat)
> jip.getConf().getOutputputFormat();
>
> How can I access the values of the counters?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
--
Harsh J
Re: Access Hadoop Counters
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Are you attempting to modify MR sources? JobInProgress is a system
class, not to be used by users directly. If you seek MR project-side
development help, its better to reach the mapreduce-dev@ lists
instead.
That said, if you want the file/hdfs bytes counters, you can access
them via the RunningJob or Job APIs directly (just give the group and
counter names).
On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> I noticed that in FileInputFormat and FileOutputFormat classes, there are
> counters(Counter's name?) for input and output data size,
>
> my question is that given a JobInProgress object, e.g. jip, after I get the
> corresponding object by
>
> FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> FileOutputFormat fof = (FileOutputFormat)
> jip.getConf().getOutputputFormat();
>
> How can I access the values of the counters?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
--
Harsh J
Re: Access Hadoop Counters
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Are you attempting to modify MR sources? JobInProgress is a system
class, not to be used by users directly. If you seek MR project-side
development help, its better to reach the mapreduce-dev@ lists
instead.
That said, if you want the file/hdfs bytes counters, you can access
them via the RunningJob or Job APIs directly (just give the group and
counter names).
On Tue, Oct 30, 2012 at 9:19 AM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> I noticed that in FileInputFormat and FileOutputFormat classes, there are
> counters(Counter's name?) for input and output data size,
>
> my question is that given a JobInProgress object, e.g. jip, after I get the
> corresponding object by
>
> FileInputFormat fif = (FileInputFormat) jip.getConf().getInputFormat();
> FileOutputFormat fof = (FileOutputFormat)
> jip.getConf().getOutputputFormat();
>
> How can I access the values of the counters?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
--
Harsh J