You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/12/23 22:42:29 UTC
[jira] Created: (HBASE-2070) Collect HLogs and delete them after a
period of time
Collect HLogs and delete them after a period of time
----------------------------------------------------
Key: HBASE-2070
URL: https://issues.apache.org/jira/browse/HBASE-2070
Project: Hadoop HBase
Issue Type: New Feature
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.21.0
For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
I plan the following changes:
- Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
- Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
- Create a new thread in the master that checks for logs older than configured time and that deletes them.
I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794294#action_12794294 ]
Andrew Purtell commented on HBASE-2070:
---------------------------------------
Subtask of HBASE-1295?
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Status: Patch Available (was: Open)
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834506#action_12834506 ]
Jean-Daniel Cryans commented on HBASE-2070:
-------------------------------------------
bq. We should make a new issue for this
bq. Yeah too to the logs shouldn't be cleared if replication is down. Can we put up a gate in zk?
I was planning on doing that in the scope of HBASE-2223.
bq. It must be dumbest name ever given a file since the epoch began? (We should do that in another patch.....another issue)
Yeah, another issue.
bq. Want to make a regex to verify expected file name rather than:
Will do
bq. Do you have to put a timestamp on it? Doesn't HDFS tell you its last-modified time? (There may be caveats to this but IIRC, for something this basic should be fine).
I wanted to avoid 2 logs created at the same time having the same name. It can still happen, but the chance is very very low.
Thanks for the review!
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834419#action_12834419 ]
stack commented on HBASE-2070:
------------------------------
.bq this seems like a good first cut, but we should probably be tracking logfiles in ZK
HT does this. Makes some kinda sense in that who owns them is clear in case of HRS failure. Master could take ownership before starts splitting them. We should make a new issue for this (Hopefully we won't have as many wals going forward with working flush).
Yeah too to the logs shouldn't be cleared if replication is down. Can we put up a gate in zk?
On the patch:
Can we change this?
{code}
static final String HREGION_OLDLOGFILE_NAME = "oldlogfile.log";
{code}
It must be dumbest name ever given a file since the epoch began? (We should do that in another patch.....another issue).
Want to make a regex to verify expected file name rather than:
{code}
+ String[] parts = filePath.getName().split("\\.");
{code}
Do you have to put a timestamp on it? Doesn't HDFS tell you its last-modified time? (There may be caveats to this but IIRC, for something this basic should be fine).
Otherwise patch looks good to me.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: HBASE-2070-v3.patch
New patch retargeted on latest trunk. Adds fixes for replication's HLog.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk with Stack's comments integrated.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: HBASE-2070-v4.patch
Cleaned up patch, just minor nits. I need a review for the core side of the patch.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: HBASE-2070-v4.patch
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833216#action_12833216 ]
Jean-Daniel Cryans commented on HBASE-2070:
-------------------------------------------
@Ryan
I agree, once this case is solved what you described will be in the scope of HBASE-2223.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833986#action_12833986 ]
ryan rawson commented on HBASE-2070:
------------------------------------
this seems like a good first cut, but we should probably be tracking logfiles in ZK. They are precious materials and really dont want to lose them.
I think the BT paper in fact says something about keeping logfile status in chubby.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: HBASE-2070-v2.patch
New patch with a new test and it passes all the other tests. Good for a review.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794296#action_12794296 ]
Jean-Daniel Cryans commented on HBASE-2070:
-------------------------------------------
Yes sorry, it's under 1295.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Issue Type: Sub-task (was: New Feature)
Parent: HBASE-1295
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: (was: HBASE-2070-v4.patch)
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2070) Collect HLogs and delete them after a
period of time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2070:
--------------------------------------
Attachment: HBASE-2070.patch
First cut. Implements the three bullet points of the case description and works on a pseudo-distributed setup. Probably fails some tests.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2070) Collect HLogs and delete them after
a period of time
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833211#action_12833211 ]
ryan rawson commented on HBASE-2070:
------------------------------------
if a replication stream is delayed, we should never delete logfiles unless the disk space situation is critical. Replication sending clusters should have plenty of disk space to buffer past all foreseeable disconnection operations. This might mean buffering 5-10TB of edits...
the alternative is to reset the slave cluster and rebuild from scratch once you lose the sync. Otherwise you end up with duplicate edits that are not removable.
> Collect HLogs and delete them after a period of time
> ----------------------------------------------------
>
> Key: HBASE-2070
> URL: https://issues.apache.org/jira/browse/HBASE-2070
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2070-v2.patch, HBASE-2070-v3.patch, HBASE-2070-v4.patch, HBASE-2070.patch
>
>
> For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.
> I plan the following changes:
> - Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
> - Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
> - Create a new thread in the master that checks for logs older than configured time and that deletes them.
> I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.