You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Mathijs Homminga <ma...@knowlogy.nl> on 2007/04/20 21:01:58 UTC

logging on Hadoop

Hi all,

I'm a bit confused by the way logging works on Hadoop.
In short, my question is: where does the log from my Nutch plugins end 
up when running on Hadoop?

I'm running Nutch 0.9 on Hadoop 0.12.2.
When I run my code on a single machine I can see that the log ends up in 
${hadoop.log.dir}/${hadoop.log.file}, as defined in the log4j.properties 
file (Nutch and my plugins use commons-logging).
But when I use Hadoop, I can't find any logfile which contains log 
entries generated by the (map|reduce) tasks. However, I do find logfiles 
which contain log from Hadoop-related classes (like the tasktracker, 
jobtracker etc).

I first thought it had something to do with HADOOP-406 
(http://issues.apache.org/jira/browse/HADOOP-406) which is about the 
fact that environment parameters passed to the parent JVM (like 
'hadoop.log.file') are not passed to the child JVM's. But even when I 
specify my log file explicitly (without the use of environment vars) in 
the log4j.properties, I still see no log entries other than the Hadoop 
classes.

Any clues?
Mathijs

Re: logging on Hadoop

Posted by Mathijs Homminga <ma...@knowlogy.nl>.

Thanks!

Arun C Murthy wrote:
> Hi Mathijs,
>
> On Fri, Apr 20, 2007 at 09:01:58PM +0200, Mathijs Homminga wrote:
>   
>> Hi all,
>>
>> I'm a bit confused by the way logging works on Hadoop.
>> In short, my question is: where does the log from my Nutch plugins end 
>> up when running on Hadoop?
>>
>> I'm running Nutch 0.9 on Hadoop 0.12.2.
>> When I run my code on a single machine I can see that the log ends up in 
>> ${hadoop.log.dir}/${hadoop.log.file}, as defined in the log4j.properties 
>> file (Nutch and my plugins use commons-logging).
>> But when I use Hadoop, I can't find any logfile which contains log 
>> entries generated by the (map|reduce) tasks. However, I do find logfiles 
>> which contain log from Hadoop-related classes (like the tasktracker, 
>> jobtracker etc).
>>
>>     
>
> The logs from the map/reduce tasks go into the ${hadoop.log.dir}/userlogs/${taskid} directory.
> They are in a specific format which aids browsing through the web-ui (webui steps: JT -> job -> task -> tasklogs).
>
> hth,
> Arun
>
>   
>> I first thought it had something to do with HADOOP-406 
>> (http://issues.apache.org/jira/browse/HADOOP-406) which is about the 
>> fact that environment parameters passed to the parent JVM (like 
>> 'hadoop.log.file') are not passed to the child JVM's. But even when I 
>> specify my log file explicitly (without the use of environment vars) in 
>> the log4j.properties, I still see no log entries other than the Hadoop 
>> classes.
>>
>> Any clues?
>> Mathijs
>>     


-- 
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.homminga@knowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

Re: logging on Hadoop

Posted by Arun C Murthy <ar...@yahoo-inc.com>.

Hi Mathijs,

On Fri, Apr 20, 2007 at 09:01:58PM +0200, Mathijs Homminga wrote:
>Hi all,
>
>I'm a bit confused by the way logging works on Hadoop.
>In short, my question is: where does the log from my Nutch plugins end 
>up when running on Hadoop?
>
>I'm running Nutch 0.9 on Hadoop 0.12.2.
>When I run my code on a single machine I can see that the log ends up in 
>${hadoop.log.dir}/${hadoop.log.file}, as defined in the log4j.properties 
>file (Nutch and my plugins use commons-logging).
>But when I use Hadoop, I can't find any logfile which contains log 
>entries generated by the (map|reduce) tasks. However, I do find logfiles 
>which contain log from Hadoop-related classes (like the tasktracker, 
>jobtracker etc).
>

The logs from the map/reduce tasks go into the ${hadoop.log.dir}/userlogs/${taskid} directory.
They are in a specific format which aids browsing through the web-ui (webui steps: JT -> job -> task -> tasklogs).

hth,
Arun

>I first thought it had something to do with HADOOP-406 
>(http://issues.apache.org/jira/browse/HADOOP-406) which is about the 
>fact that environment parameters passed to the parent JVM (like 
>'hadoop.log.file') are not passed to the child JVM's. But even when I 
>specify my log file explicitly (without the use of environment vars) in 
>the log4j.properties, I still see no log entries other than the Hadoop 
>classes.
>
>Any clues?
>Mathijs