Logging

[Sandeep Tata <sa...@gmail.com>]

Hi,

I'm running nutch 0.9 to crawl a few sites. I notice that the fetcher logs
to hadoop.log when I run nutch over the local filesystem. However, It
doesn't log to hadoop.log when I'm running on top of hadoop. (Cluster of 3
machines, I've checked to see if the crawler is working, and it is, and
seems to have crawled several thousand pages.) There is no log file in HDFS
either. Is there a property I can set to make it log to hadoop.log even when
using hadoop and HDFS ? How can I observe what the fetcher (fetch2) is doing
when running on top of hadoop/hdfs?

Thanks,
Sandeep

Re: Logging

[Dennis Kubes <ku...@apache.org>]

It depends on how your log4j.properties files in conf is setup.  If it 
is set to commandout then the logs will appear in userlogs under the 
task stdout file.  If set to hadoop.log as the log file then it should 
appear in the logs dir in hadoop.log.

Dennis Kubes

Sandeep Tata wrote:
> Hi,
> 
> I'm running nutch 0.9 to crawl a few sites. I notice that the fetcher logs
> to hadoop.log when I run nutch over the local filesystem. However, It
> doesn't log to hadoop.log when I'm running on top of hadoop. (Cluster of 3
> machines, I've checked to see if the crawler is working, and it is, and
> seems to have crawled several thousand pages.) There is no log file in HDFS
> either. Is there a property I can set to make it log to hadoop.log even when
> using hadoop and HDFS ? How can I observe what the fetcher (fetch2) is doing
> when running on top of hadoop/hdfs?
> 
> Thanks,
> Sandeep
>