You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ophir Cohen <op...@gmail.com> on 2011/05/18 17:21:52 UTC
Mapreduce counters
Hi All,
Currently MR job spilled his counters into file at the end of the run.
Is there any built-in configuration/plug-in to make it store these counters
into HBase as well?
Sounds to me like a great feature!
Does anybody did something similar?
If you did, how did you do it? Run on directory and load it into HBase?
Thanks,
Ophir
Re: Mapreduce counters
Posted by Ophir Cohen <op...@gmail.com>.
For jobs you create it sounds as a great idea.
What about other jobs such as Hive/Pig jobs?
Can anyone as any idea how it can be done on all MR jobs in a cluster no
matter the triggering of the jobs?
Ophir
On Wed, May 18, 2011 at 7:09 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> Hi Ophir,
>
> That sounds like a useful feature, maybe file a jira?
>
> I've never tried to save counters form the MR job into HBase, but you
> could pull it from the file as you said or from the Job object after
> waitForCompletion() returns by calling getCounters().
>
> -Joey
>
> On Wed, May 18, 2011 at 8:21 AM, Ophir Cohen <op...@gmail.com> wrote:
> > Hi All,
> > Currently MR job spilled his counters into file at the end of the run.
> >
> > Is there any built-in configuration/plug-in to make it store these
> counters
> > into HBase as well?
> >
> > Sounds to me like a great feature!
> >
> > Does anybody did something similar?
> > If you did, how did you do it? Run on directory and load it into HBase?
> >
> > Thanks,
> > Ophir
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
Re: Mapreduce counters
Posted by Joey Echeverria <jo...@cloudera.com>.
Hi Ophir,
That sounds like a useful feature, maybe file a jira?
I've never tried to save counters form the MR job into HBase, but you
could pull it from the file as you said or from the Job object after
waitForCompletion() returns by calling getCounters().
-Joey
On Wed, May 18, 2011 at 8:21 AM, Ophir Cohen <op...@gmail.com> wrote:
> Hi All,
> Currently MR job spilled his counters into file at the end of the run.
>
> Is there any built-in configuration/plug-in to make it store these counters
> into HBase as well?
>
> Sounds to me like a great feature!
>
> Does anybody did something similar?
> If you did, how did you do it? Run on directory and load it into HBase?
>
> Thanks,
> Ophir
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434