You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Henry Liu <hl...@attinteractive.com> on 2011/04/07 19:28:15 UTC

job counters in customized hive map/reduce script.

I'd like to report some counters in my customized map and reduce ruby scripts for hive, just like hadoop streaming does by writing to stderr.

Such as.



          #increment hadoop counters
   $stderr.puts("reporter:counter:Impression,Total,1")
   if first_click == 0 && click_details.empty?
       $stderr.puts("reporter:counter:Impression,NoClick,1")
   end

   $stderr.puts ("reporter:status: status rid #{rid}")
   $stderr.puts ("test debug message: rid #{rid}")


These are all logged as regular messages, no counters/status recognized at all.

Anyone has any ideas?

Thanks.

Re: job counters in customized hive map/reduce script.

Posted by Carl Steinbach <ca...@cloudera.com>.

Right now Hive's TRANSFORM mechanism does not allow you set counters.
There's a ticket open for adding this feature:

https://issues.apache.org/jira/browse/HIVE-1649

Thanks.

Carl

On Thu, Apr 7, 2011 at 10:35 AM, Avram Aelony <Av...@eharmony.com>wrote:

>
> Can you try using using streaming from Hive?
>
>        http://wiki.apache.org/hadoop/Hive/LanguageManual/Transform
>
> Cheers,
> ~Avram
>
>
>
> On Apr 7, 2011, at 10:28 AM, Henry Liu wrote:
>
> >
> > I'd like to report some counters in my customized map and reduce ruby
> scripts for hive, just like hadoop streaming does by writing to stderr.
> >
> > Such as.
> >
> >
> >
> >          #increment hadoop counters
> >   $stderr.puts("reporter:counter:Impression,Total,1")
> >   if first_click == 0 && click_details.empty?
> >       $stderr.puts("reporter:counter:Impression,NoClick,1")
> >   end
> >
> >   $stderr.puts ("reporter:status: status rid #{rid}")
> >   $stderr.puts ("test debug message: rid #{rid}")
> >
> >
> > These are all logged as regular messages, no counters/status recognized
> at all.
> >
> > Anyone has any ideas?
> >
> > Thanks.
> >
>
>

Re: job counters in customized hive map/reduce script.

Posted by Avram Aelony <Av...@eharmony.com>.

Can you try using using streaming from Hive?

	http://wiki.apache.org/hadoop/Hive/LanguageManual/Transform

Cheers,
~Avram



On Apr 7, 2011, at 10:28 AM, Henry Liu wrote:

> 
> I'd like to report some counters in my customized map and reduce ruby scripts for hive, just like hadoop streaming does by writing to stderr.
> 
> Such as.
> 
> 
> 
>          #increment hadoop counters
>   $stderr.puts("reporter:counter:Impression,Total,1")
>   if first_click == 0 && click_details.empty?
>       $stderr.puts("reporter:counter:Impression,NoClick,1")
>   end
> 
>   $stderr.puts ("reporter:status: status rid #{rid}")
>   $stderr.puts ("test debug message: rid #{rid}")
> 
> 
> These are all logged as regular messages, no counters/status recognized at all.
> 
> Anyone has any ideas?
> 
> Thanks.
>