You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sahana Bhat <sa...@gmail.com> on 2011/12/07 09:23:38 UTC
Regarding Hadoop Log Retention for Hadoop 0.20.2
Hi,
We use Hadoop 0.20.2 version.The log4j.properties file has a
property *hadoop.tasklog.logsRetainHours *(mentioned as 24 hours by default)
* *which we have set to 12.Despite this property being set we still have
old userlogs in the system.Any idea why the attempt directories (that have
the syserr/sysout files) have not been cleaned up? Are we missing any other
configuration?
Any help on this topic would be great.
Regards,
Sahana
Re: Regarding Hadoop Log Retention for Hadoop 0.20.2
Posted by Harsh J <ha...@cloudera.com>.
I'm using CDH3 here and I do not see that option in my
log4j.properties. Probably something remaining from an older
mechanism?
Good to know the actual property worked for you.
On Wed, Dec 7, 2011 at 4:18 PM, Sahana Bhat <sa...@gmail.com> wrote:
> Hi Harsh,
>
> Thanks for the reply. I realized that the property you mentioned
> was missing in my mapred-site.xml.
> I added the entry and it works just fine.
>
> Was my assumption that "hadoop.tasklog.logsRetainHours " in log4j.properties
> will do the same wrong? What is this property for in that case?
>
> Regards,
> Sahana
>
>
>
>
> On Wed, Dec 7, 2011 at 2:28 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Sahana,
>>
>> IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
>> environment and it seems to work fine on my cluster. Are you sure you
>> have tasklogs older than 24h lying around? It might even be a bug that
>> may have been fixed in the subsequent 0.20 releases that went out
>> recently.
>>
>> On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat <sa...@gmail.com> wrote:
>> > Hi,
>> >
>> > We use Hadoop 0.20.2 version.The log4j.properties file has a
>> > property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
>> > default) which we have set to 12.Despite this property being set we
>> > still
>> > have old userlogs in the system.Any idea why the attempt directories
>> > (that
>> > have the syserr/sysout files) have not been cleaned up? Are we missing
>> > any
>> > other configuration?
>> >
>> > Any help on this topic would be great.
>> >
>> > Regards,
>> > Sahana
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Regarding Hadoop Log Retention for Hadoop 0.20.2
Posted by Sahana Bhat <sa...@gmail.com>.
Hi Vinod,
Thanks for that info :).
Regards,
Sahana
On Thu, Dec 8, 2011 at 3:30 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
>
> That is a stale property in log4j.properties that isn't used for anything.
> There are other such things like noKeepSplits, purgeLogSplits etc which
> seem to be stale too.
>
> mapred.userlog.retain.hours should work just fine as Harsh correctly
> mentioned.
>
> HTH,
> +Vinod
>
>
> On Dec 7, 2011, at 2:48 AM, Sahana Bhat wrote:
>
> Was my assumption that "*hadoop.tasklog.logsRetainHours " *in
> log4j.properties will do the same wrong? What is this property for in that
> case?
>
> Regards,
> Sahana
>
>
>
Re: Regarding Hadoop Log Retention for Hadoop 0.20.2
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
That is a stale property in log4j.properties that isn't used for anything. There are other such things like noKeepSplits, purgeLogSplits etc which seem to be stale too.
mapred.userlog.retain.hours should work just fine as Harsh correctly mentioned.
HTH,
+Vinod
On Dec 7, 2011, at 2:48 AM, Sahana Bhat wrote:
> Was my assumption that "hadoop.tasklog.logsRetainHours " in log4j.properties will do the same wrong? What is this property for in that case?
>
> Regards,
> Sahana
>
Re: Regarding Hadoop Log Retention for Hadoop 0.20.2
Posted by Sahana Bhat <sa...@gmail.com>.
Hi Harsh,
Thanks for the reply. I realized that the property you mentioned
was missing in my mapred-site.xml.
I added the entry and it works just fine.
Was my assumption that "*hadoop.tasklog.logsRetainHours " *in
log4j.properties will do the same wrong? What is this property for in that
case?
Regards,
Sahana
On Wed, Dec 7, 2011 at 2:28 PM, Harsh J <ha...@cloudera.com> wrote:
> Sahana,
>
> IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
> environment and it seems to work fine on my cluster. Are you sure you
> have tasklogs older than 24h lying around? It might even be a bug that
> may have been fixed in the subsequent 0.20 releases that went out
> recently.
>
> On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat <sa...@gmail.com> wrote:
> > Hi,
> >
> > We use Hadoop 0.20.2 version.The log4j.properties file has a
> > property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
> > default) which we have set to 12.Despite this property being set we still
> > have old userlogs in the system.Any idea why the attempt directories
> (that
> > have the syserr/sysout files) have not been cleaned up? Are we missing
> any
> > other configuration?
> >
> > Any help on this topic would be great.
> >
> > Regards,
> > Sahana
>
>
>
> --
> Harsh J
>
Re: Regarding Hadoop Log Retention for Hadoop 0.20.2
Posted by Harsh J <ha...@cloudera.com>.
Sahana,
IIRC, "mapred.userlog.retain.hours" (24h default) controls this in my
environment and it seems to work fine on my cluster. Are you sure you
have tasklogs older than 24h lying around? It might even be a bug that
may have been fixed in the subsequent 0.20 releases that went out
recently.
On Wed, Dec 7, 2011 at 1:53 PM, Sahana Bhat <sa...@gmail.com> wrote:
> Hi,
>
> We use Hadoop 0.20.2 version.The log4j.properties file has a
> property hadoop.tasklog.logsRetainHours (mentioned as 24 hours by
> default) which we have set to 12.Despite this property being set we still
> have old userlogs in the system.Any idea why the attempt directories (that
> have the syserr/sysout files) have not been cleaned up? Are we missing any
> other configuration?
>
> Any help on this topic would be great.
>
> Regards,
> Sahana
--
Harsh J