hadoop can't remove logs automatically

[shangan <sh...@corp.kaixin001.com>]

hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?


<property>
  <name>hadoop.logfile.size</name>
  <value>10000000</value>
  <description>The max size of each log file</description>
</property>
<property>
  <name>hadoop.logfile.count</name>
  <value>10</value>
  <description>The max number of log files</description>
</property>


2010-10-18 



shangan 

Re: hadoop can't remove logs automatically

[Allen Wittenauer <aw...@linkedin.com>]

On Oct 17, 2010, at 10:32 PM, Allen Wittenauer wrote:

> 
> On Oct 17, 2010, at 10:29 PM, shangan wrote:
> 
>> then you just write a shell to remove the logs periodically as a workaround? or better ideas ?
> 
> We basically have a cron job that does a few things as part of our maintenance.  We have it rigged up such that it runs on the namenode and then, over ssh, runs each of these on the slave nodes:
> 
> 	- purge old logs
> 	- purge old files out of mapred temp space
> 	- kill stale/stuck tasks
> 
> Hadoop really should manage this stuff on its own, but well, Hadoop should do a lot of things to be more operable. ;)


Of course, it is worth mentioning you could also tie these logs to logadm/logwatch/rotatelogs/etc.  But I like to have them centrally managed so that they are consistent across the grid.

Re: hadoop can't remove logs automatically

[Allen Wittenauer <aw...@linkedin.com>]

On Oct 17, 2010, at 10:29 PM, shangan wrote:

> then you just write a shell to remove the logs periodically as a workaround? or better ideas ?

We basically have a cron job that does a few things as part of our maintenance.  We have it rigged up such that it runs on the namenode and then, over ssh, runs each of these on the slave nodes:

	- purge old logs
	- purge old files out of mapred temp space
	- kill stale/stuck tasks

Hadoop really should manage this stuff on its own, but well, Hadoop should do a lot of things to be more operable. ;)

Re: Re: hadoop can't remove logs automatically

[shangan <sh...@corp.kaixin001.com>]

then you just write a shell to remove the logs periodically as a workaround? or better ideas ?


2010-10-18 



shangan




发件人： Allen Wittenauer 
发送时间： 2010-10-18  13:23:34 
收件人： 
抄送： 
主题： Re: hadoop can't remove logs automatically 
 
On Oct 17, 2010, at 10:05 PM, shangan wrote:
> hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?
> 
I don't think hadoop.logfile.count is actually tied to anything in the source.  Likely a property from a bygone era.
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5540 (20101017) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com

Re: hadoop can't remove logs automatically

[Allen Wittenauer <aw...@linkedin.com>]

On Oct 17, 2010, at 10:05 PM, shangan wrote:

> hadoop logs keep increasing and not removed automatically according to the logging property. I didn't change the default logging property but it never works as it should. The default logging property is as follows, anyone know why or it is a bug not fixed ?
> 

I don't think hadoop.logfile.count is actually tied to anything in the source.  Likely a property from a bygone era.