You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Wayne (JIRA)" <ji...@apache.org> on 2011/01/30 03:33:43 UTC
[jira] Created: (HBASE-3489) .oldlogs not being cleaned out
.oldlogs not being cleaned out
------------------------------
Key: HBASE-3489
URL: https://issues.apache.org/jira/browse/HBASE-3489
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Environment: 10 Nodes Write Heavy Cluster
Reporter: Wayne
The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3489) .oldlogs not being cleaned out
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112327#comment-13112327 ]
Todd Lipcon commented on HBASE-3489:
------------------------------------
Anyone reproduced this, recently?
> .oldlogs not being cleaned out
> ------------------------------
>
> Key: HBASE-3489
> URL: https://issues.apache.org/jira/browse/HBASE-3489
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Environment: 10 Nodes Write Heavy Cluster
> Reporter: Wayne
> Attachments: oldlog.txt
>
>
> The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3489) .oldlogs not being cleaned out
Posted by "Wayne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wayne updated HBASE-3489:
-------------------------
Attachment: oldlog.txt
Attached are the name node entries with .oldlog. After running on .90 for 5 days I had more than 18+ TB of old logs. The data size was only ~2TB. Having not even known about this until 2 days ago I had mis-calculated our required production cluster size (good news). I think these logs should be deleted normally after 1 day and at a pace able to keep up with heavy writes.
There is I believe a bug from replication in .90 as no logs seemed to get deleted, but even if it worked as designed I would have to wait until 7 days and at the rate I was load testing the 40TB limit of our test cluster might have been reached with actual data of only 10% of that. I question the 7 day limit as a good default. It causes novices like myself to think the data size is a lot bigger than it is. I was even convinced lzo compression was not working due to the spike in disk usage.
> .oldlogs not being cleaned out
> ------------------------------
>
> Key: HBASE-3489
> URL: https://issues.apache.org/jira/browse/HBASE-3489
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Environment: 10 Nodes Write Heavy Cluster
> Reporter: Wayne
> Attachments: oldlog.txt
>
>
> The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HBASE-3489) .oldlogs not being cleaned out
Posted by "Josh Wymer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188014#comment-13188014 ]
Josh Wymer commented on HBASE-3489:
-----------------------------------
After turning replication off on the slave cluster, the .oldlogs were cleaned up. So it appears as if hbase thinks that the slave cluster intends to replicate as well and doesn't clean the logs.
> .oldlogs not being cleaned out
> ------------------------------
>
> Key: HBASE-3489
> URL: https://issues.apache.org/jira/browse/HBASE-3489
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Environment: 10 Nodes Write Heavy Cluster
> Reporter: Wayne
> Attachments: oldlog.txt
>
>
> The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3489) .oldlogs not being cleaned out
Posted by "Josh Wymer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187172#comment-13187172 ]
Josh Wymer commented on HBASE-3489:
-----------------------------------
We are seeing this on our replication cluster using 0.90.4. The /hbase/.oldlogs is filled with logs that are ~ 1 month old.
> .oldlogs not being cleaned out
> ------------------------------
>
> Key: HBASE-3489
> URL: https://issues.apache.org/jira/browse/HBASE-3489
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Environment: 10 Nodes Write Heavy Cluster
> Reporter: Wayne
> Attachments: oldlog.txt
>
>
> The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3489) .oldlogs not being cleaned out
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988535#action_12988535 ]
Jean-Daniel Cryans commented on HBASE-3489:
-------------------------------------------
Hey Wayne,
Could you grep the Namenode's log for "oldlogs" and attach the result in this jira? I'd like to see what kind of log archival rate you have and if logs are really deleted at all. Thanks!
> .oldlogs not being cleaned out
> ------------------------------
>
> Key: HBASE-3489
> URL: https://issues.apache.org/jira/browse/HBASE-3489
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Environment: 10 Nodes Write Heavy Cluster
> Reporter: Wayne
>
> The .oldlogs folder is never being cleaned up. The hbase.master.logcleaner.ttl has been set to clean up the old logs but the clean up is never kicking in. The limit of 10 files is not the problem. After running for 5 days not a single log file has ever been deleted and the logcleaner is set to 2 days (from the default of 7 days). It is assumed that the replication changes that want to be sure to keep these logs around if needed have caused the cleanup to be blocked. There is no replication defined (knowingly).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.