You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Vladimir Rodionov (JIRA)" <ji...@apache.org> on 2015/07/18 00:28:04 UTC
[jira] [Resolved] (HBASE-14036) HBase Backup/Restore Phase 1:
Custom WAL archive cleaner
[ https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladimir Rodionov resolved HBASE-14036.
---------------------------------------
Resolution: Implemented
This feature is part of patch v4. See parent JIRA.
> HBase Backup/Restore Phase 1: Custom WAL archive cleaner
> --------------------------------------------------------
>
> Key: HBASE-14036
> URL: https://issues.apache.org/jira/browse/HBASE-14036
> Project: HBase
> Issue Type: Task
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
>
> Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL files in archive until they either get copied over to backup destination during an incremental backup or full backup (for ALL tables) happens. This is tricky, but is doable. Backup-aware WAL archiver cleaner should consult hbase:backup to determine if WAL file is safe to purge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Resolved] (HBASE-14036) HBase Backup/Restore Phase 1:
Custom WAL archive cleaner
Posted by Vladimir Rodionov <vl...@gmail.com>.
I agree that it could be done more efficiently. Instead of sending files
one-by-one, plugins
could process WAL archive directory content (all files). For very large
clusters this would save
a lot of time for BackupLogCleaner, but because it is a background thread,
the performance benefit is not clear
for client applications. If we run cleaners once a day, it does not matter
will they finish in 10 min or in 10 hours.
-Vlad
On Fri, Jul 17, 2015 at 11:38 PM, rahul gidwani <ra...@gmail.com>
wrote:
> Do you think its time to stop sending files one by one to the cleaners and
> have all the cleaners take batches of files (like replication and snapshot
> cleaners)? As we add more cleaners for large clusters and certain
> scenarios the cleaner chain comes to a crawl?
>
> On Fri, Jul 17, 2015 at 3:28 PM, Vladimir Rodionov (JIRA) <jira@apache.org
> >
> wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > ]
> >
> > Vladimir Rodionov resolved HBASE-14036.
> > ---------------------------------------
> > Resolution: Implemented
> >
> > This feature is part of patch v4. See parent JIRA.
> >
> > > HBase Backup/Restore Phase 1: Custom WAL archive cleaner
> > > --------------------------------------------------------
> > >
> > > Key: HBASE-14036
> > > URL: https://issues.apache.org/jira/browse/HBASE-14036
> > > Project: HBase
> > > Issue Type: Task
> > > Reporter: Vladimir Rodionov
> > > Assignee: Vladimir Rodionov
> > > Fix For: 2.0.0
> > >
> > >
> > > Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL
> > files in archive until they either get copied over to backup destination
> > during an incremental backup or full backup (for ALL tables) happens.
> This
> > is tricky, but is doable. Backup-aware WAL archiver cleaner should
> consult
> > hbase:backup to determine if WAL file is safe to purge.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
Re: [jira] [Resolved] (HBASE-14036) HBase Backup/Restore Phase 1:
Custom WAL archive cleaner
Posted by rahul gidwani <ra...@gmail.com>.
Do you think its time to stop sending files one by one to the cleaners and
have all the cleaners take batches of files (like replication and snapshot
cleaners)? As we add more cleaners for large clusters and certain
scenarios the cleaner chain comes to a crawl?
On Fri, Jul 17, 2015 at 3:28 PM, Vladimir Rodionov (JIRA) <ji...@apache.org>
wrote:
>
> [
> https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Vladimir Rodionov resolved HBASE-14036.
> ---------------------------------------
> Resolution: Implemented
>
> This feature is part of patch v4. See parent JIRA.
>
> > HBase Backup/Restore Phase 1: Custom WAL archive cleaner
> > --------------------------------------------------------
> >
> > Key: HBASE-14036
> > URL: https://issues.apache.org/jira/browse/HBASE-14036
> > Project: HBase
> > Issue Type: Task
> > Reporter: Vladimir Rodionov
> > Assignee: Vladimir Rodionov
> > Fix For: 2.0.0
> >
> >
> > Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL
> files in archive until they either get copied over to backup destination
> during an incremental backup or full backup (for ALL tables) happens. This
> is tricky, but is doable. Backup-aware WAL archiver cleaner should consult
> hbase:backup to determine if WAL file is safe to purge.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>