how to delete logs automatically from hadoop yarn

[Sunil Garg <su...@cumulus-systems.com>]

How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Mich Talebzadeh <mi...@peridale.co.uk>]

I don’t think there is such configuration to trigger automatic removal of older files say more than 7 days through cron or Control M

 

 

# Clean up directory - get rid of old files

 

for i in $LOGDIR:14 $TMPDIR:4 $ETCDIR:7

do

        THE_DIR=`echo $i|awk -F":" '{print $1}'`

        NO_DAYS=`echo $i|awk -F":" '{print $2}'`

        find $THE_DIR -mtime +${NO_DAYS} -exec rm -f {} \;

done

#

 

HTH

 

 

Mich Talebzadeh

 

http://talebzadehmich.wordpress.com

 

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.

 

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com] 
Sent: 20 April 2015 05:53
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi Rohith,

Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.

Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.

During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that? 

 

Thanks,

Smita

 

 

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com] 
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi 

 

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.

“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

 

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?

  <property>

    <description>

      Number of seconds after an application finishes before the nodemanager's 

      DeletionService will delete the application's localized file directory

      and log directory.

      

      To diagnose Yarn application problems, set this property's value large

      enough (for example, to 600 = 10 minutes) to permit examination of these

      directories. After changing the property's value, you must restart the 

      nodemanager in order for it to have an effect.

 

      The roots of Yarn applications' work directories is configurable with

      the yarn.nodemanager.local-dirs property (see below), and the roots

      of the Yarn applications' log directories is configurable with the 

      yarn.nodemanager.log-dirs property (see also below).

    </description>

    <name>yarn.nodemanager.delete.debug-delay-sec</name>

    <value>0</value>

  </property>

 

 

Thanks & Regards

Rohith Sharma K S

From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com] 
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn

 

 

How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working 

Is there any other way we can do this or am I doing something wrong !!

 

<property>

<name>yarn.log-aggregation-enable</name>

<value>false</value>

</property>

 

<property>

<name>yarn.nodemanager.log.retain-seconds</name>

<value>3600</value>

</property>

 

Thanks

Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
                In our application, there were around 3,62,738 containers which ran successfully before we encountered this issue. So under userLogs/applicationId/ we had 3,62,738 directories, each directory having container’s stdout and stderr file. We are not expecting to rotate these stdout and stderr file as its mentioned in jira 2443. These logs are of no use after certain time, for a week we may need those in case we need to troubleshoot why container failed or so.

Thanks,
Smita

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 11:02 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
                In our application, there were around 3,62,738 containers which ran successfully before we encountered this issue. So under userLogs/applicationId/ we had 3,62,738 directories, each directory having container’s stdout and stderr file. We are not expecting to rotate these stdout and stderr file as its mentioned in jira 2443. These logs are of no use after certain time, for a week we may need those in case we need to troubleshoot why container failed or so.

Thanks,
Smita

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 11:02 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
                In our application, there were around 3,62,738 containers which ran successfully before we encountered this issue. So under userLogs/applicationId/ we had 3,62,738 directories, each directory having container’s stdout and stderr file. We are not expecting to rotate these stdout and stderr file as its mentioned in jira 2443. These logs are of no use after certain time, for a week we may need those in case we need to troubleshoot why container failed or so.

Thanks,
Smita

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 11:02 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
                In our application, there were around 3,62,738 containers which ran successfully before we encountered this issue. So under userLogs/applicationId/ we had 3,62,738 directories, each directory having container’s stdout and stderr file. We are not expecting to rotate these stdout and stderr file as its mentioned in jira 2443. These logs are of no use after certain time, for a week we may need those in case we need to troubleshoot why container failed or so.

Thanks,
Smita

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 11:02 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Mich Talebzadeh <mi...@peridale.co.uk>]

I don’t think there is such configuration to trigger automatic removal of older files say more than 7 days through cron or Control M

 

 

# Clean up directory - get rid of old files

 

for i in $LOGDIR:14 $TMPDIR:4 $ETCDIR:7

do

        THE_DIR=`echo $i|awk -F":" '{print $1}'`

        NO_DAYS=`echo $i|awk -F":" '{print $2}'`

        find $THE_DIR -mtime +${NO_DAYS} -exec rm -f {} \;

done

#

 

HTH

 

 

Mich Talebzadeh

 

http://talebzadehmich.wordpress.com

 

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.

 

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com] 
Sent: 20 April 2015 05:53
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi Rohith,

Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.

Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.

During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that? 

 

Thanks,

Smita

 

 

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com] 
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi 

 

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.

“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

 

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?

  <property>

    <description>

      Number of seconds after an application finishes before the nodemanager's 

      DeletionService will delete the application's localized file directory

      and log directory.

      

      To diagnose Yarn application problems, set this property's value large

      enough (for example, to 600 = 10 minutes) to permit examination of these

      directories. After changing the property's value, you must restart the 

      nodemanager in order for it to have an effect.

 

      The roots of Yarn applications' work directories is configurable with

      the yarn.nodemanager.local-dirs property (see below), and the roots

      of the Yarn applications' log directories is configurable with the 

      yarn.nodemanager.log-dirs property (see also below).

    </description>

    <name>yarn.nodemanager.delete.debug-delay-sec</name>

    <value>0</value>

  </property>

 

 

Thanks & Regards

Rohith Sharma K S

From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com] 
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn

 

 

How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working 

Is there any other way we can do this or am I doing something wrong !!

 

<property>

<name>yarn.log-aggregation-enable</name>

<value>false</value>

</property>

 

<property>

<name>yarn.nodemanager.log.retain-seconds</name>

<value>3600</value>

</property>

 

Thanks

Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Mich Talebzadeh <mi...@peridale.co.uk>]

I don’t think there is such configuration to trigger automatic removal of older files say more than 7 days through cron or Control M

 

 

# Clean up directory - get rid of old files

 

for i in $LOGDIR:14 $TMPDIR:4 $ETCDIR:7

do

        THE_DIR=`echo $i|awk -F":" '{print $1}'`

        NO_DAYS=`echo $i|awk -F":" '{print $2}'`

        find $THE_DIR -mtime +${NO_DAYS} -exec rm -f {} \;

done

#

 

HTH

 

 

Mich Talebzadeh

 

http://talebzadehmich.wordpress.com

 

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.

 

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com] 
Sent: 20 April 2015 05:53
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi Rohith,

Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.

Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.

During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that? 

 

Thanks,

Smita

 

 

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com] 
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi 

 

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.

“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

 

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?

  <property>

    <description>

      Number of seconds after an application finishes before the nodemanager's 

      DeletionService will delete the application's localized file directory

      and log directory.

      

      To diagnose Yarn application problems, set this property's value large

      enough (for example, to 600 = 10 minutes) to permit examination of these

      directories. After changing the property's value, you must restart the 

      nodemanager in order for it to have an effect.

 

      The roots of Yarn applications' work directories is configurable with

      the yarn.nodemanager.local-dirs property (see below), and the roots

      of the Yarn applications' log directories is configurable with the 

      yarn.nodemanager.log-dirs property (see also below).

    </description>

    <name>yarn.nodemanager.delete.debug-delay-sec</name>

    <value>0</value>

  </property>

 

 

Thanks & Regards

Rohith Sharma K S

From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com] 
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn

 

 

How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working 

Is there any other way we can do this or am I doing something wrong !!

 

<property>

<name>yarn.log-aggregation-enable</name>

<value>false</value>

</property>

 

<property>

<name>yarn.nodemanager.log.retain-seconds</name>

<value>3600</value>

</property>

 

Thanks

Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Mich Talebzadeh <mi...@peridale.co.uk>]

I don’t think there is such configuration to trigger automatic removal of older files say more than 7 days through cron or Control M

 

 

# Clean up directory - get rid of old files

 

for i in $LOGDIR:14 $TMPDIR:4 $ETCDIR:7

do

        THE_DIR=`echo $i|awk -F":" '{print $1}'`

        NO_DAYS=`echo $i|awk -F":" '{print $2}'`

        find $THE_DIR -mtime +${NO_DAYS} -exec rm -f {} \;

done

#

 

HTH

 

 

Mich Talebzadeh

 

http://talebzadehmich.wordpress.com

 

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.

 

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com] 
Sent: 20 April 2015 05:53
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi Rohith,

Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.

Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.

During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that? 

 

Thanks,

Smita

 

 

From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com] 
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

 

Hi 

 

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.

“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

 

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?

  <property>

    <description>

      Number of seconds after an application finishes before the nodemanager's 

      DeletionService will delete the application's localized file directory

      and log directory.

      

      To diagnose Yarn application problems, set this property's value large

      enough (for example, to 600 = 10 minutes) to permit examination of these

      directories. After changing the property's value, you must restart the 

      nodemanager in order for it to have an effect.

 

      The roots of Yarn applications' work directories is configurable with

      the yarn.nodemanager.local-dirs property (see below), and the roots

      of the Yarn applications' log directories is configurable with the 

      yarn.nodemanager.log-dirs property (see also below).

    </description>

    <name>yarn.nodemanager.delete.debug-delay-sec</name>

    <value>0</value>

  </property>

 

 

Thanks & Regards

Rohith Sharma K S

From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com] 
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn

 

 

How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working 

Is there any other way we can do this or am I doing something wrong !!

 

<property>

<name>yarn.log-aggregation-enable</name>

<value>false</value>

</property>

 

<property>

<name>yarn.nodemanager.log.retain-seconds</name>

<value>3600</value>

</property>

 

Thanks

Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

That’s  interesting use-case!!

>>>> let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?
I don’t think there is such configuration exist in the YARN currently. I think it should be able to handle from log4j properties.

But enabling log-aggregation, disk filling issue can be overcome. I think in the Hadoop-2.6 or later(yet to release)handling long running services on yarn is done in JIRA https://issues.apache.org/jira/i#browse/YARN-2443 .

>>> Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory
I could not get how continuous logs causing exceeding Linux resource limit.  How many containers are running in cluster and per machine? If I think, each containers holds one resource for logging.


Thanks & Regards
Rohith Sharma K S

From: Smita Deshpande [mailto:smita.deshpande@cumulus-systems.com]
Sent: 20 April 2015 10:23
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Smita Deshpande <sm...@cumulus-systems.com>]

Hi Rohith,
Thanks for your solution. The actual problem we are looking at is : We have a lifelong running application, so configurations by which logs will be deleted right after application is finished will not help us.
Because of these continuous logs, we are running out of Linux file limit and thereafter containers are not launched because of exception while creating log directory inside application ID directory.
During the job execution itself, let’s say I want to delete container logs which are older than week or so. So is there any configuration to do that?

Thanks,
Smita


From: Rohith Sharma K S [mailto:rohithsharmaks@huawei.com]
Sent: Monday, April 20, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: RE: how to delete logs automatically from hadoop yarn

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg

RE: how to delete logs automatically from hadoop yarn

[Rohith Sharma K S <ro...@huawei.com>]

Hi

With below configuration , log deletion should be triggered.  You can see from the log that deletion has been set to 3600 sec in NM like below. May be you can check NM logs for the below log that give debug information.
“INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1428298081702_0008, with delay of 10800 seconds”

But there is another configuration which affect deletion task is “yarn.nodemanager.delete.debug-delay-sec”, default value is zero. It means immediately deletion will be triggered. Check is this is configured?
  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.

      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.

      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>


Thanks & Regards
Rohith Sharma K S
From: Sunil Garg [mailto:sunil.garg@cumulus-systems.com]
Sent: 20 April 2015 09:52
To: user@hadoop.apache.org
Subject: how to delete logs automatically from hadoop yarn


How to delete logs from Hadoop yarn automatically, I Have tried following settings but it is not working
Is there any other way we can do this or am I doing something wrong !!

<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

Thanks
Sunil Garg