How do I log from my map/reduce application?

["W.P. McNeill" <bi...@gmail.com>]

I would like to use Hadoop's Log4j infrastructure to do logging from my
map/reduce application.  I think I've got everything set up correctly, but I
am still unable to specify the logging level I want.

By default Hadoop is set up to log at level INFO.  The first line of its
log4j.properties file looks like this:

hadoop.root.logger=INFO,console


I have an application whose reducer looks like this:

package com.me;

public class MyReducer<...> extends Reducer<...> {
   private static Logger logger =
Logger.getLogger(MyReducer.class.getName());

   ...
   protected void reduce(...) {
       logger.debug("My message");
       ...
   }
}


I've added the following line to the Hadoop log4j.properties file:

log4j.logger.com.me.MyReducer=DEBUG


I expect the Hadoop system to log at level INFO, but my application to log
at level DEBUG, so that I see "My message" in the logs for the reducer task.
 However, my application does not produce any log4j output.  If I change the
line in my reducer to read logger.info("My message") the message does get
logged, so somehow I'm failing to specify that log level for this class.

I've also tried changing the log4j line for my app to
read log4j.logger.com.me.MyReducer=DEBUG,console and get the same result.

I've been through the Hadoop and log4j documentation and I can't figure out
what I'm doing wrong.  Any suggestions?

Thanks.

Re: How do I log from my map/reduce application?

[Steve Loughran <st...@apache.org>]

On 13/12/10 17:29, W.P. McNeill wrote:
> I would like to use Hadoop's Log4j infrastructure to do logging from my
> map/reduce application.  I think I've got everything set up correctly, but I
> am still unable to specify the logging level I want.
>
> By default Hadoop is set up to log at level INFO.  The first line of its
> log4j.properties file looks like this:
>
> hadoop.root.logger=INFO,console
>
>
> I have an application whose reducer looks like this:
>
> package com.me;
>
> public class MyReducer<...>  extends Reducer<...>  {
>     private static Logger logger =
> Logger.getLogger(MyReducer.class.getName());
>
>     ...
>     protected void reduce(...) {
>         logger.debug("My message");
>         ...
>     }
> }
>
>
> I've added the following line to the Hadoop log4j.properties file:
>
> log4j.logger.com.me.MyReducer=DEBUG
>
>
> I expect the Hadoop system to log at level INFO, but my application to log
> at level DEBUG, so that I see "My message" in the logs for the reducer task.
>   However, my application does not produce any log4j output.  If I change the
> line in my reducer to read logger.info("My message") the message does get
> logged, so somehow I'm failing to specify that log level for this class.
>
> I've also tried changing the log4j line for my app to
> read log4j.logger.com.me.MyReducer=DEBUG,console and get the same result.
>
> I've been through the Hadoop and log4j documentation and I can't figure out
> what I'm doing wrong.  Any suggestions?

I'd start with your logger logging @info level which log4j.properties 
file it's finding on the classpath. The code I use to find a resource 
looks like this:


    public URL findResource(String resource) {
         return getClass().getClassLoader().getResource(resource);
     }

Once you know which log4j file is being used to drive your code, then 
you can start tweaking it...