Re: Application creating huge amount of logs : Nutch 2.3.1 + Hadoop 2.7.1

["shubham.gupta" <sh...@orkash.com>]

Hey,

I have changed the user.log_retain size to 10 MB still it is creating a 
huge size of logs. This leads to the failure of datanode and the job 
fails. And, if the logs are deleted periodically then the fetch phase 
takes a lot of time and it is uncertain that whether it will complete or 
not.

Shubham Gupta

On Wednesday 24 August 2016 05:20 PM, Markus Jelsma wrote:
> If it is Nutch logging, change its level in conf/log4j.properties. It can also be Hadoop logging.
> M.
>   
> -----Original message-----
>> From:shubham.gupta <sh...@orkash.com>
>> Sent: Tuesday 23rd August 2016 8:15
>> To: user@nutch.apache.org
>> Subject: Application creating huge amount of logs : Nutch 2.3.1 + Hadoop 2.7.1
>>
>> Hey
>>
>> I have integrated Nutch 2.3.1 with Hadoop 2.7.1, and the fetcher.parse
>> property is set TRUE and the database used is MongoDB. While the map job
>> of nutch runs, it creates a huge size of nodelogs over 13GB in size. And
>> the cause of such huge amount of files in unknown. Any suggestion would
>> help.
>>
>> Thanks in advance.
>>
>> Shubham Gupta
>>