You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2016/09/01 15:44:53 UTC
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Andrew:
HBASE-16255 has been resolved.
Kindly provide your feedback.
Thanks
On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <an...@gmail.com>
wrote:
> I plan to spin up a test cluster with clusterdock and try running the IT
> under a number of different scenarios. I understand snapshots have to
> function so baseline would be the calm monkey.
>
> Unless you have some other automated way for me to run the new
> functionality repeatedly, the IT is it.
>
> > On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <vl...@gmail.com>
> wrote:
> >
> > Not sure what do you mean, Andrew by "trying out the branch via the IT",
> > but we do not recommend running this with monkey enabled.
> > It has not been tested in a such scenario yet and frankly speaking it is
> > not supposed to work (snapshots will fail anyway and we depends on
> > snapshots)
> >
> > -Vladimir
> >
> > On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > wrote:
> >
> >> Let's commit the IT to the branch, if you think the v5 patch is ready
> for
> >> commit Ted.
> >>
> >> I will be able to spend some time next week trying out the branch via
> the
> >> IT, and poking around with the new tools. After that I feel like I'll be
> >> informed enough to vote on a branch merge vote.
> >>
> >>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>> IT test is provided on HBASE-16255.
> >>>
> >>> Any other comment ?
> >>>
> >>> Thanks
> >>>
> >>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <ds...@cloudera.com>
> >> wrote:
> >>>>
> >>>> Any chance for an IT test being added to the branch first? I'd love to
> >> put
> >>>> it through the paces with clusterdock to make sure it behaves well
> with
> >>>> fault injection and the like.
> >>>>
> >>>> -Dima
> >>>>
> >>>>> On Tuesday, August 2, 2016, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>> Any more comments from the community on whether the merge can be
> >>>> conducted
> >>>>> ?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
> >>>> vladrodionov@gmail.com
> >>>>> <javascript:;>>
> >>>>> wrote:
> >>>>>
> >>>>>> Carter Shanklin posted a blog article about the feature:
> >>>>>> Some use cases and examples of a command line interface usage.
> >>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
> >>>> backup-restore-apache-hbase-apache-phoenix/
> >>>>>>
> >>>>>> -Vlad
> >>>>>>
> >>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
> >>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Ok, got it.
> >>>>>>>
> >>>>>>> -Vlad
> >>>>>>>
> >>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <enis@apache.org
> >>>>> <javascript:;>> wrote:
> >>>>>>>
> >>>>>>>> We keep the WALs which can accumulate a lot if the use case is to
> >>>> only
> >>>>>> do
> >>>>>>>> backups infrequently. This will definitely cause issues since HDFS
> >>>>> space
> >>>>>>>> will get filled up. That is why we may need an option for having
> >>>>>>>> incremental backups not used, and WAL references being deleted.
> >>>>>>>>
> >>>>>>>> Enis
> >>>>>>>>
> >>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
> >>>>>>>> vladrodionov@gmail.com <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Why anyone will ever need disabling incremental backups? If you
> do
> >>>>> not
> >>>>>>>> need
> >>>>>>>>> it - just run only full backups.
> >>>>>>>>>
> >>>>>>>>> -Vlad
> >>>>>>>>>
> >>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <enis@apache.org
> >>>>> <javascript:;>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks Matteo for chiming in.
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
> >>>>>>>>> theo.bertozzi@gmail.com <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I did some review in the early beginning, but then lost track
> >>>> of
> >>>>>> the
> >>>>>>>>>>> changes.
> >>>>>>>>>>> but I'd like to give a quick review to the full code once
> >>>> people
> >>>>>>>> here
> >>>>>>>>> are
> >>>>>>>>>>> ok with getting this feature in master (2.0).
> >>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for
> >>>>> reviewing
> >>>>>>>> the
> >>>>>>>>>> full
> >>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid
> >>>>> holding
> >>>>>>>> this
> >>>>>>>>>> for
> >>>>>>>>>>> too long, but still enough time to have people that are
> >>>>> interested
> >>>>>>>> to
> >>>>>>>>>> look
> >>>>>>>>>>> at it. with did the same thing for MOB with a mega patch
> >>>>>>>>>>> https://reviews.apache.org/r/36391/)
> >>>>>>>>>>
> >>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to handle
> >>>>> the
> >>>>>>>>> merge?
> >>>>>>>>>> As a giant patch or a rebase of code in the branch and through
> >>>> git
> >>>>>>>> merge.
> >>>>>>>>>>
> >>>>>>>>>> We need to run a vote when the to-be-merged branched is ready.
> >>>> We
> >>>>>> can
> >>>>>>>>> set a
> >>>>>>>>>> vote timeout for at least 1 week.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> most of the code seemed isolated from the beginning, few
> >>>> changes
> >>>>>>>> here
> >>>>>>>>> and
> >>>>>>>>>>> there in the core.
> >>>>>>>>>>> so, this side of things seems ok to me.
> >>>>>>>>>>>
> >>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that
> >>>>>> should
> >>>>>>>> not
> >>>>>>>>>>> take long.
> >>>>>>>>>>>
> >>>>>>>>>>> I don't know if there are already docs, but that is another
> >>>>> thing
> >>>>>> we
> >>>>>>>>> may
> >>>>>>>>>>> want to get in with the merge.
> >>>>>>>>>>> a minimal coverage at least on how to use the feature, and
> >>>> maybe
> >>>>>>>>> calling
> >>>>>>>>>> it
> >>>>>>>>>>> out as experimental?
> >>>>>>>>>>>
> >>>>>>>>>>> my main concern were around incremental backups.
> >>>>>>>>>>> I'm still not convinced around the fact that because the WALs
> >>>>>>>> contain
> >>>>>>>>>>> regions of multiple tables
> >>>>>>>>>>> the incremental backup will keep around WALs with some data
> >>>> that
> >>>>>> we
> >>>>>>>>> don't
> >>>>>>>>>>> really want in the backup (for space or maybe security
> >>>> reason).
> >>>>>>>>>>>
> >>>>>>>>>>> then there was the question about for how long should I take
> >>>>>>>>>> incrementals,
> >>>>>>>>>>> before deciding that a fresh full backup is less costly in
> >>>> terms
> >>>>>> of
> >>>>>>>>>> space.
> >>>>>>>>>>> but I think this incremental merge/compaction was a feature on
> >>>>> the
> >>>>>>>>>> roadmap
> >>>>>>>>>>> as Phase3.
> >>>>>>>>>>> which I think is ok to get later on,
> >>>>>>>>>>> maybe just call out a lifecycle example on the docs under
> >>>> "best
> >>>>>>>>>> practices".
> >>>>>>>>>>
> >>>>>>>>>> I think this will depend on the use case, and other factors like
> >>>>>>>>> bandwidth
> >>>>>>>>>> available, how much data
> >>>>>>>>>> the user is willing to lose in case of catastrophic failure and
> >>>>> how
> >>>>>>>>>> "expensive" is full backup versus
> >>>>>>>>>> incremental one.
> >>>>>>>>>>
> >>>>>>>>>> The full backup should also be useable by default, so maybe we
> >>>> can
> >>>>>>>> make
> >>>>>>>>> an
> >>>>>>>>>> option to not even keep WAL files, and completely disable
> >>>>>> incremental
> >>>>>>>>>> backups?
> >>>>>>>>>>
> >>>>>>>>>> Enis
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> has anyone interested in using backups looked at the doc in
> >>>>>>>> HBASE-7912?
> >>>>>>>>>>> is the current design of incremental backup acceptable for
> >>>>>> everyone
> >>>>>>>>>> wanting
> >>>>>>>>>>> to use this feature?
> >>>>>>>>>>> (maybe this should be a question for the @user list and not
> >>>> dev)
> >>>>>>>>>>>
> >>>>>>>>>>> is there anyone already using this feature or it is just dev
> >>>>>> testing
> >>>>>>>>> it?
> >>>>>>>>>>> to me will be interesting having a use-case/workflow example,
> >>>>>>>>>>> to see if in the real world my concerns about incremental are
> >>>>> not
> >>>>>>>>> showing
> >>>>>>>>>>> up.
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com
> >>>>> <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Gentle ping on this subject.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The changes are mostly non-intrusive.
> >>>>>>>>>>>>
> >>>>>>>>>>>> More comments are welcome.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
> >>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
> >>>>>>>>>>>> andrew.purtell@gmail.com <javascript:;>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> How hard would it be to convert what you've been using
> >>>> to
> >>>>>> test
> >>>>>>>>> end
> >>>>>>>>>> to
> >>>>>>>>>>>> end
> >>>>>>>>>>>>>> during dev into an IT?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
> >>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If
> >>>> not,
> >>>>>> any
> >>>>>>>>> tips
> >>>>>>>>>>> on a
> >>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
> >>>>> them?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests.
> >>>> We
> >>>>>>>>> provide 2
> >>>>>>>>>>> API
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> backup:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Admin.getBackupAdmin
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. Command - line via hbase command.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Everything is straightforward.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
> >>>>>>>>>>> dspivak@cloudera.com <javascript:;>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not,
> >>>>> any
> >>>>>>>> tips
> >>>>>>>>>> on
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
> >>>> them?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Dima
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
> >>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sorry, wrong links:
> >>>>>>>>>>>>>>>>> These are the phases:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Phase 1:
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030
> >>>>>>> 14030
> >>>>>>>>>>>>>>>>> Phase 2:
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123
> >>>>>>> 14123
> >>>>>>>>>>>>>>>>> Phase 3:
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414
> >>>>>>> 14414
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
> >>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> These are the phases:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Phase 1:
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>> 14030
> >>>>>>>>>>>>>>>>>> Phase 2:
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>> 14123
> >>>>>>>>>>>>>>>>>> Phase 3:
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>> 14414
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar <
> >>>>>>>>>>> enis@apache.org <javascript:;>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir,
> >>>> Ted,
> >>>>>>>> Jerry
> >>>>>>>>> and
> >>>>>>>>>>>>> others
> >>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality
> >>>>> in
> >>>>>> a
> >>>>>>>>>> series
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1].
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a
> >>>>> 4-phase
> >>>>>>>>>> project,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are
> >>>>> now
> >>>>>>>>> working
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>>> Phase 3
> >>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think
> >>>> that
> >>>>>> the
> >>>>>>>>>> current
> >>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items,
> >>>> and
> >>>>>>>> some
> >>>>>>>>>>> Phase 3
> >>>>>>>>>>>>>>>> items
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait
> >>>> for
> >>>>>> all
> >>>>>>>> the
> >>>>>>>>>>>>>> subtickets
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as
> >>>> follow
> >>>>>> up
> >>>>>>>>>> tickets
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> mostly
> >>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements
> >>>> in
> >>>>>> the
> >>>>>>>>> works
> >>>>>>>>>>> are
> >>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff.
> >>>> Thus,
> >>>>> we
> >>>>>>>>> would
> >>>>>>>>>>> like
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into
> >>>>>>>> master.
> >>>>>>>>>> The
> >>>>>>>>>>>>> parent
> >>>>>>>>>>>>>>>>> jira
> >>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the
> >>>>>>>>>>> implementation
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>>> choices in case you are interested[2].
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and
> >>>>>>>> confined
> >>>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> backup subsystem.
> >>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs
> >>>> and
> >>>>> we
> >>>>>>>>>>>>> (hortonworks)
> >>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> been running the integration tests as well as some
> >>>>>> other
> >>>>>>>>>>>>> shell-based
> >>>>>>>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the
> >>>>>> work
> >>>>>>>> has
> >>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> reviewed
> >>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted,
> >>>> myself
> >>>>>> and
> >>>>>>>>>> Jerry).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote?
> >>>>> Any
> >>>>>>>>>> concerns
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> feedback
> >>>>>>>>>>>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1]
> >>>>> https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>>>>>>>>>>>> [2]
> >>>>> https://issues.apache.org/jira/secure/attachment/12816339/
> >>>> HBaseBackupAndRestore%20-0.91.pdf
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Enis
> >>>>
> >>>>
> >>>> --
> >>>> -Dima
> >>
>
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Posted by Andrew Purtell <an...@gmail.com>.
Fine I'll cast a vote as -0.
If I find time to test that could easily change to +1. Perhaps my vote won't be needed. I don't wish to block you.
> On Sep 6, 2016, at 6:55 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Andrew:
> Do you think you would have some time this week ?
>
> Thanks
>
> On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell <an...@gmail.com>
> wrote:
>
>> Busy at work, aiming for next week.
>>
>>> On Sep 1, 2016, at 8:44 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> Andrew:
>>> HBASE-16255 has been resolved.
>>>
>>> Kindly provide your feedback.
>>>
>>> Thanks
>>>
>>> On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>>> wrote:
>>>
>>>> I plan to spin up a test cluster with clusterdock and try running the IT
>>>> under a number of different scenarios. I understand snapshots have to
>>>> function so baseline would be the calm monkey.
>>>>
>>>> Unless you have some other automated way for me to run the new
>>>> functionality repeatedly, the IT is it.
>>>>
>>>>>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <
>> vladrodionov@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Not sure what do you mean, Andrew by "trying out the branch via the
>> IT",
>>>>> but we do not recommend running this with monkey enabled.
>>>>> It has not been tested in a such scenario yet and frankly speaking it
>> is
>>>>> not supposed to work (snapshots will fail anyway and we depends on
>>>>> snapshots)
>>>>>
>>>>> -Vladimir
>>>>>
>>>>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
>>>> andrew.purtell@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Let's commit the IT to the branch, if you think the v5 patch is ready
>>>> for
>>>>>> commit Ted.
>>>>>>
>>>>>> I will be able to spend some time next week trying out the branch via
>>>> the
>>>>>> IT, and poking around with the new tools. After that I feel like I'll
>> be
>>>>>> informed enough to vote on a branch merge vote.
>>>>>>
>>>>>>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> IT test is provided on HBASE-16255.
>>>>>>>
>>>>>>> Any other comment ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <ds...@cloudera.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Any chance for an IT test being added to the branch first? I'd love
>> to
>>>>>> put
>>>>>>>> it through the paces with clusterdock to make sure it behaves well
>>>> with
>>>>>>>> fault injection and the like.
>>>>>>>>
>>>>>>>> -Dima
>>>>>>>>
>>>>>>>>> On Tuesday, August 2, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Any more comments from the community on whether the merge can be
>>>>>>>> conducted
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
>>>>>>>> vladrodionov@gmail.com
>>>>>>>>> <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Carter Shanklin posted a blog article about the feature:
>>>>>>>>>> Some use cases and examples of a command line interface usage.
>>>>>>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
>>>>>>>> backup-restore-apache-hbase-apache-phoenix/
>>>>>>>>>>
>>>>>>>>>> -Vlad
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ok, got it.
>>>>>>>>>>>
>>>>>>>>>>> -Vlad
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <enis@apache.org
>>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> We keep the WALs which can accumulate a lot if the use case is
>> to
>>>>>>>> only
>>>>>>>>>> do
>>>>>>>>>>>> backups infrequently. This will definitely cause issues since
>> HDFS
>>>>>>>>> space
>>>>>>>>>>>> will get filled up. That is why we may need an option for having
>>>>>>>>>>>> incremental backups not used, and WAL references being deleted.
>>>>>>>>>>>>
>>>>>>>>>>>> Enis
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Why anyone will ever need disabling incremental backups? If you
>>>> do
>>>>>>>>> not
>>>>>>>>>>>> need
>>>>>>>>>>>>> it - just run only full backups.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <
>> enis@apache.org
>>>>>>>>> <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Matteo for chiming in.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
>>>>>>>>>>>>> theo.bertozzi@gmail.com <javascript:;>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did some review in the early beginning, but then lost track
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>>> changes.
>>>>>>>>>>>>>>> but I'd like to give a quick review to the full code once
>>>>>>>> people
>>>>>>>>>>>> here
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> ok with getting this feature in master (2.0).
>>>>>>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for
>>>>>>>>> reviewing
>>>>>>>>>>>> the
>>>>>>>>>>>>>> full
>>>>>>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid
>>>>>>>>> holding
>>>>>>>>>>>> this
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> too long, but still enough time to have people that are
>>>>>>>>> interested
>>>>>>>>>>>> to
>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>> at it. with did the same thing for MOB with a mega patch
>>>>>>>>>>>>>>> https://reviews.apache.org/r/36391/)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to
>> handle
>>>>>>>>> the
>>>>>>>>>>>>> merge?
>>>>>>>>>>>>>> As a giant patch or a rebase of code in the branch and through
>>>>>>>> git
>>>>>>>>>>>> merge.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We need to run a vote when the to-be-merged branched is ready.
>>>>>>>> We
>>>>>>>>>> can
>>>>>>>>>>>>> set a
>>>>>>>>>>>>>> vote timeout for at least 1 week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> most of the code seemed isolated from the beginning, few
>>>>>>>> changes
>>>>>>>>>>>> here
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> there in the core.
>>>>>>>>>>>>>>> so, this side of things seems ok to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that
>>>>>>>>>> should
>>>>>>>>>>>> not
>>>>>>>>>>>>>>> take long.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't know if there are already docs, but that is another
>>>>>>>>> thing
>>>>>>>>>> we
>>>>>>>>>>>>> may
>>>>>>>>>>>>>>> want to get in with the merge.
>>>>>>>>>>>>>>> a minimal coverage at least on how to use the feature, and
>>>>>>>> maybe
>>>>>>>>>>>>> calling
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> out as experimental?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> my main concern were around incremental backups.
>>>>>>>>>>>>>>> I'm still not convinced around the fact that because the WALs
>>>>>>>>>>>> contain
>>>>>>>>>>>>>>> regions of multiple tables
>>>>>>>>>>>>>>> the incremental backup will keep around WALs with some data
>>>>>>>> that
>>>>>>>>>> we
>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> really want in the backup (for space or maybe security
>>>>>>>> reason).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then there was the question about for how long should I take
>>>>>>>>>>>>>> incrementals,
>>>>>>>>>>>>>>> before deciding that a fresh full backup is less costly in
>>>>>>>> terms
>>>>>>>>>> of
>>>>>>>>>>>>>> space.
>>>>>>>>>>>>>>> but I think this incremental merge/compaction was a feature
>> on
>>>>>>>>> the
>>>>>>>>>>>>>> roadmap
>>>>>>>>>>>>>>> as Phase3.
>>>>>>>>>>>>>>> which I think is ok to get later on,
>>>>>>>>>>>>>>> maybe just call out a lifecycle example on the docs under
>>>>>>>> "best
>>>>>>>>>>>>>> practices".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think this will depend on the use case, and other factors
>> like
>>>>>>>>>>>>> bandwidth
>>>>>>>>>>>>>> available, how much data
>>>>>>>>>>>>>> the user is willing to lose in case of catastrophic failure
>> and
>>>>>>>>> how
>>>>>>>>>>>>>> "expensive" is full backup versus
>>>>>>>>>>>>>> incremental one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The full backup should also be useable by default, so maybe we
>>>>>>>> can
>>>>>>>>>>>> make
>>>>>>>>>>>>> an
>>>>>>>>>>>>>> option to not even keep WAL files, and completely disable
>>>>>>>>>> incremental
>>>>>>>>>>>>>> backups?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Enis
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> has anyone interested in using backups looked at the doc in
>>>>>>>>>>>> HBASE-7912?
>>>>>>>>>>>>>>> is the current design of incremental backup acceptable for
>>>>>>>>>> everyone
>>>>>>>>>>>>>> wanting
>>>>>>>>>>>>>>> to use this feature?
>>>>>>>>>>>>>>> (maybe this should be a question for the @user list and not
>>>>>>>> dev)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> is there anyone already using this feature or it is just dev
>>>>>>>>>> testing
>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>> to me will be interesting having a use-case/workflow example,
>>>>>>>>>>>>>>> to see if in the real world my concerns about incremental are
>>>>>>>>> not
>>>>>>>>>>>>> showing
>>>>>>>>>>>>>>> up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>>>> <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Gentle ping on this subject.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The changes are mostly non-intrusive.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> More comments are welcome.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
>>>>>>>>>>>>>>>> andrew.purtell@gmail.com <javascript:;>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How hard would it be to convert what you've been using
>>>>>>>> to
>>>>>>>>>> test
>>>>>>>>>>>>> end
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>> during dev into an IT?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If
>>>>>>>> not,
>>>>>>>>>> any
>>>>>>>>>>>>> tips
>>>>>>>>>>>>>>> on a
>>>>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
>>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests.
>>>>>>>> We
>>>>>>>>>>>>> provide 2
>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> backup:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. Admin.getBackupAdmin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2. Command - line via hbase command.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Everything is straightforward.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
>>>>>>>>>>>>>>> dspivak@cloudera.com <javascript:;>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not,
>>>>>>>>> any
>>>>>>>>>>>> tips
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Dima
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sorry, wrong links:
>>>>>>>>>>>>>>>>>>>>> These are the phases:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Phase 1:
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030
>>>>>>>>>>> 14030
>>>>>>>>>>>>>>>>>>>>> Phase 2:
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123
>>>>>>>>>>> 14123
>>>>>>>>>>>>>>>>>>>>> Phase 3:
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414
>>>>>>>>>>> 14414
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> These are the phases:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Phase 1:
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>> 14030
>>>>>>>>>>>>>>>>>>>>>> Phase 2:
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>> 14123
>>>>>>>>>>>>>>>>>>>>>> Phase 3:
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>> 14414
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar <
>>>>>>>>>>>>>>> enis@apache.org <javascript:;>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir,
>>>>>>>> Ted,
>>>>>>>>>>>> Jerry
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> others
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>>>> series
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1].
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a
>>>>>>>>> 4-phase
>>>>>>>>>>>>>> project,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are
>>>>>>>>> now
>>>>>>>>>>>>> working
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> Phase 3
>>>>>>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think
>>>>>>>> that
>>>>>>>>>> the
>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items,
>>>>>>>> and
>>>>>>>>>>>> some
>>>>>>>>>>>>>>> Phase 3
>>>>>>>>>>>>>>>>>>>> items
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait
>>>>>>>> for
>>>>>>>>>> all
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> subtickets
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as
>>>>>>>> follow
>>>>>>>>>> up
>>>>>>>>>>>>>> tickets
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements
>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>> works
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff.
>>>>>>>> Thus,
>>>>>>>>> we
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into
>>>>>>>>>>>> master.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>>>> jira
>>>>>>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the
>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>>>>>>> choices in case you are interested[2].
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and
>>>>>>>>>>>> confined
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> backup subsystem.
>>>>>>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs
>>>>>>>> and
>>>>>>>>> we
>>>>>>>>>>>>>>>>> (hortonworks)
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> been running the integration tests as well as some
>>>>>>>>>> other
>>>>>>>>>>>>>>>>> shell-based
>>>>>>>>>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the
>>>>>>>>>> work
>>>>>>>>>>>> has
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>> reviewed
>>>>>>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted,
>>>>>>>> myself
>>>>>>>>>> and
>>>>>>>>>>>>>> Jerry).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote?
>>>>>>>>> Any
>>>>>>>>>>>>>> concerns
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>> https://issues.apache.org/jira/secure/attachment/12816339/
>>>>>>>> HBaseBackupAndRestore%20-0.91.pdf
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Enis
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Dima
>>
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Posted by Ted Yu <yu...@gmail.com>.
Andrew:
Do you think you would have some time this week ?
Thanks
On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell <an...@gmail.com>
wrote:
> Busy at work, aiming for next week.
>
> > On Sep 1, 2016, at 8:44 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > Andrew:
> > HBASE-16255 has been resolved.
> >
> > Kindly provide your feedback.
> >
> > Thanks
> >
> > On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > wrote:
> >
> >> I plan to spin up a test cluster with clusterdock and try running the IT
> >> under a number of different scenarios. I understand snapshots have to
> >> function so baseline would be the calm monkey.
> >>
> >> Unless you have some other automated way for me to run the new
> >> functionality repeatedly, the IT is it.
> >>
> >>>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <
> vladrodionov@gmail.com>
> >>> wrote:
> >>>
> >>> Not sure what do you mean, Andrew by "trying out the branch via the
> IT",
> >>> but we do not recommend running this with monkey enabled.
> >>> It has not been tested in a such scenario yet and frankly speaking it
> is
> >>> not supposed to work (snapshots will fail anyway and we depends on
> >>> snapshots)
> >>>
> >>> -Vladimir
> >>>
> >>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
> >> andrew.purtell@gmail.com>
> >>> wrote:
> >>>
> >>>> Let's commit the IT to the branch, if you think the v5 patch is ready
> >> for
> >>>> commit Ted.
> >>>>
> >>>> I will be able to spend some time next week trying out the branch via
> >> the
> >>>> IT, and poking around with the new tools. After that I feel like I'll
> be
> >>>> informed enough to vote on a branch merge vote.
> >>>>
> >>>>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>> IT test is provided on HBASE-16255.
> >>>>>
> >>>>> Any other comment ?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <ds...@cloudera.com>
> >>>> wrote:
> >>>>>>
> >>>>>> Any chance for an IT test being added to the branch first? I'd love
> to
> >>>> put
> >>>>>> it through the paces with clusterdock to make sure it behaves well
> >> with
> >>>>>> fault injection and the like.
> >>>>>>
> >>>>>> -Dima
> >>>>>>
> >>>>>>> On Tuesday, August 2, 2016, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Any more comments from the community on whether the merge can be
> >>>>>> conducted
> >>>>>>> ?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
> >>>>>> vladrodionov@gmail.com
> >>>>>>> <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Carter Shanklin posted a blog article about the feature:
> >>>>>>>> Some use cases and examples of a command line interface usage.
> >>>>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
> >>>>>> backup-restore-apache-hbase-apache-phoenix/
> >>>>>>>>
> >>>>>>>> -Vlad
> >>>>>>>>
> >>>>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
> >>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Ok, got it.
> >>>>>>>>>
> >>>>>>>>> -Vlad
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <enis@apache.org
> >>>>>>> <javascript:;>> wrote:
> >>>>>>>>>
> >>>>>>>>>> We keep the WALs which can accumulate a lot if the use case is
> to
> >>>>>> only
> >>>>>>>> do
> >>>>>>>>>> backups infrequently. This will definitely cause issues since
> HDFS
> >>>>>>> space
> >>>>>>>>>> will get filled up. That is why we may need an option for having
> >>>>>>>>>> incremental backups not used, and WAL references being deleted.
> >>>>>>>>>>
> >>>>>>>>>> Enis
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
> >>>>>>>>>> vladrodionov@gmail.com <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Why anyone will ever need disabling incremental backups? If you
> >> do
> >>>>>>> not
> >>>>>>>>>> need
> >>>>>>>>>>> it - just run only full backups.
> >>>>>>>>>>>
> >>>>>>>>>>> -Vlad
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <
> enis@apache.org
> >>>>>>> <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks Matteo for chiming in.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
> >>>>>>>>>>> theo.bertozzi@gmail.com <javascript:;>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I did some review in the early beginning, but then lost track
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> changes.
> >>>>>>>>>>>>> but I'd like to give a quick review to the full code once
> >>>>>> people
> >>>>>>>>>> here
> >>>>>>>>>>> are
> >>>>>>>>>>>>> ok with getting this feature in master (2.0).
> >>>>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for
> >>>>>>> reviewing
> >>>>>>>>>> the
> >>>>>>>>>>>> full
> >>>>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid
> >>>>>>> holding
> >>>>>>>>>> this
> >>>>>>>>>>>> for
> >>>>>>>>>>>>> too long, but still enough time to have people that are
> >>>>>>> interested
> >>>>>>>>>> to
> >>>>>>>>>>>> look
> >>>>>>>>>>>>> at it. with did the same thing for MOB with a mega patch
> >>>>>>>>>>>>> https://reviews.apache.org/r/36391/)
> >>>>>>>>>>>>
> >>>>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to
> handle
> >>>>>>> the
> >>>>>>>>>>> merge?
> >>>>>>>>>>>> As a giant patch or a rebase of code in the branch and through
> >>>>>> git
> >>>>>>>>>> merge.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We need to run a vote when the to-be-merged branched is ready.
> >>>>>> We
> >>>>>>>> can
> >>>>>>>>>>> set a
> >>>>>>>>>>>> vote timeout for at least 1 week.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> most of the code seemed isolated from the beginning, few
> >>>>>> changes
> >>>>>>>>>> here
> >>>>>>>>>>> and
> >>>>>>>>>>>>> there in the core.
> >>>>>>>>>>>>> so, this side of things seems ok to me.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that
> >>>>>>>> should
> >>>>>>>>>> not
> >>>>>>>>>>>>> take long.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I don't know if there are already docs, but that is another
> >>>>>>> thing
> >>>>>>>> we
> >>>>>>>>>>> may
> >>>>>>>>>>>>> want to get in with the merge.
> >>>>>>>>>>>>> a minimal coverage at least on how to use the feature, and
> >>>>>> maybe
> >>>>>>>>>>> calling
> >>>>>>>>>>>> it
> >>>>>>>>>>>>> out as experimental?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> my main concern were around incremental backups.
> >>>>>>>>>>>>> I'm still not convinced around the fact that because the WALs
> >>>>>>>>>> contain
> >>>>>>>>>>>>> regions of multiple tables
> >>>>>>>>>>>>> the incremental backup will keep around WALs with some data
> >>>>>> that
> >>>>>>>> we
> >>>>>>>>>>> don't
> >>>>>>>>>>>>> really want in the backup (for space or maybe security
> >>>>>> reason).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> then there was the question about for how long should I take
> >>>>>>>>>>>> incrementals,
> >>>>>>>>>>>>> before deciding that a fresh full backup is less costly in
> >>>>>> terms
> >>>>>>>> of
> >>>>>>>>>>>> space.
> >>>>>>>>>>>>> but I think this incremental merge/compaction was a feature
> on
> >>>>>>> the
> >>>>>>>>>>>> roadmap
> >>>>>>>>>>>>> as Phase3.
> >>>>>>>>>>>>> which I think is ok to get later on,
> >>>>>>>>>>>>> maybe just call out a lifecycle example on the docs under
> >>>>>> "best
> >>>>>>>>>>>> practices".
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think this will depend on the use case, and other factors
> like
> >>>>>>>>>>> bandwidth
> >>>>>>>>>>>> available, how much data
> >>>>>>>>>>>> the user is willing to lose in case of catastrophic failure
> and
> >>>>>>> how
> >>>>>>>>>>>> "expensive" is full backup versus
> >>>>>>>>>>>> incremental one.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The full backup should also be useable by default, so maybe we
> >>>>>> can
> >>>>>>>>>> make
> >>>>>>>>>>> an
> >>>>>>>>>>>> option to not even keep WAL files, and completely disable
> >>>>>>>> incremental
> >>>>>>>>>>>> backups?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Enis
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> has anyone interested in using backups looked at the doc in
> >>>>>>>>>> HBASE-7912?
> >>>>>>>>>>>>> is the current design of incremental backup acceptable for
> >>>>>>>> everyone
> >>>>>>>>>>>> wanting
> >>>>>>>>>>>>> to use this feature?
> >>>>>>>>>>>>> (maybe this should be a question for the @user list and not
> >>>>>> dev)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> is there anyone already using this feature or it is just dev
> >>>>>>>> testing
> >>>>>>>>>>> it?
> >>>>>>>>>>>>> to me will be interesting having a use-case/workflow example,
> >>>>>>>>>>>>> to see if in the real world my concerns about incremental are
> >>>>>>> not
> >>>>>>>>>>> showing
> >>>>>>>>>>>>> up.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com
> >>>>>>> <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Gentle ping on this subject.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The changes are mostly non-intrusive.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> More comments are welcome.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
> >>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
> >>>>>>>>>>>>>> andrew.purtell@gmail.com <javascript:;>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> How hard would it be to convert what you've been using
> >>>>>> to
> >>>>>>>> test
> >>>>>>>>>>> end
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>>> during dev into an IT?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
> >>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If
> >>>>>> not,
> >>>>>>>> any
> >>>>>>>>>>> tips
> >>>>>>>>>>>>> on a
> >>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
> >>>>>>> them?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests.
> >>>>>> We
> >>>>>>>>>>> provide 2
> >>>>>>>>>>>>> API
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> backup:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. Admin.getBackupAdmin
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. Command - line via hbase command.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Everything is straightforward.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
> >>>>>>>>>>>>> dspivak@cloudera.com <javascript:;>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not,
> >>>>>>> any
> >>>>>>>>>> tips
> >>>>>>>>>>>> on
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
> >>>>>> them?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -Dima
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
> >>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Sorry, wrong links:
> >>>>>>>>>>>>>>>>>>> These are the phases:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Phase 1:
> >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030
> >>>>>>>>> 14030
> >>>>>>>>>>>>>>>>>>> Phase 2:
> >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123
> >>>>>>>>> 14123
> >>>>>>>>>>>>>>>>>>> Phase 3:
> >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414
> >>>>>>>>> 14414
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
> >>>>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> These are the phases:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Phase 1:
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>> 14030
> >>>>>>>>>>>>>>>>>>>> Phase 2:
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>> 14123
> >>>>>>>>>>>>>>>>>>>> Phase 3:
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
> >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>> 14414
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -Vlad
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar <
> >>>>>>>>>>>>> enis@apache.org <javascript:;>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir,
> >>>>>> Ted,
> >>>>>>>>>> Jerry
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> others
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality
> >>>>>>> in
> >>>>>>>> a
> >>>>>>>>>>>> series
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1].
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a
> >>>>>>> 4-phase
> >>>>>>>>>>>> project,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are
> >>>>>>> now
> >>>>>>>>>>> working
> >>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>> Phase 3
> >>>>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think
> >>>>>> that
> >>>>>>>> the
> >>>>>>>>>>>> current
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items,
> >>>>>> and
> >>>>>>>>>> some
> >>>>>>>>>>>>> Phase 3
> >>>>>>>>>>>>>>>>>> items
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait
> >>>>>> for
> >>>>>>>> all
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> subtickets
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as
> >>>>>> follow
> >>>>>>>> up
> >>>>>>>>>>>> tickets
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>> mostly
> >>>>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> works
> >>>>>>>>>>>>> are
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff.
> >>>>>> Thus,
> >>>>>>> we
> >>>>>>>>>>> would
> >>>>>>>>>>>>> like
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into
> >>>>>>>>>> master.
> >>>>>>>>>>>> The
> >>>>>>>>>>>>>>> parent
> >>>>>>>>>>>>>>>>>>> jira
> >>>>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the
> >>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>>>>> choices in case you are interested[2].
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and
> >>>>>>>>>> confined
> >>>>>>>>>>> to
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> backup subsystem.
> >>>>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs
> >>>>>> and
> >>>>>>> we
> >>>>>>>>>>>>>>> (hortonworks)
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>> been running the integration tests as well as some
> >>>>>>>> other
> >>>>>>>>>>>>>>> shell-based
> >>>>>>>>>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the
> >>>>>>>> work
> >>>>>>>>>> has
> >>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>> reviewed
> >>>>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted,
> >>>>>> myself
> >>>>>>>> and
> >>>>>>>>>>>> Jerry).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote?
> >>>>>>> Any
> >>>>>>>>>>>> concerns
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> feedback
> >>>>>>>>>>>>>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>> https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>>>>>>>>>>>>>> [2]
> >>>>>>> https://issues.apache.org/jira/secure/attachment/12816339/
> >>>>>> HBaseBackupAndRestore%20-0.91.pdf
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Enis
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> -Dima
> >>
>
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Posted by Andrew Purtell <an...@gmail.com>.
Busy at work, aiming for next week.
> On Sep 1, 2016, at 8:44 AM, Ted Yu <yu...@gmail.com> wrote:
>
> Andrew:
> HBASE-16255 has been resolved.
>
> Kindly provide your feedback.
>
> Thanks
>
> On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <an...@gmail.com>
> wrote:
>
>> I plan to spin up a test cluster with clusterdock and try running the IT
>> under a number of different scenarios. I understand snapshots have to
>> function so baseline would be the calm monkey.
>>
>> Unless you have some other automated way for me to run the new
>> functionality repeatedly, the IT is it.
>>
>>>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <vl...@gmail.com>
>>> wrote:
>>>
>>> Not sure what do you mean, Andrew by "trying out the branch via the IT",
>>> but we do not recommend running this with monkey enabled.
>>> It has not been tested in a such scenario yet and frankly speaking it is
>>> not supposed to work (snapshots will fail anyway and we depends on
>>> snapshots)
>>>
>>> -Vladimir
>>>
>>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>>> wrote:
>>>
>>>> Let's commit the IT to the branch, if you think the v5 patch is ready
>> for
>>>> commit Ted.
>>>>
>>>> I will be able to spend some time next week trying out the branch via
>> the
>>>> IT, and poking around with the new tools. After that I feel like I'll be
>>>> informed enough to vote on a branch merge vote.
>>>>
>>>>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>> IT test is provided on HBASE-16255.
>>>>>
>>>>> Any other comment ?
>>>>>
>>>>> Thanks
>>>>>
>>>>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <ds...@cloudera.com>
>>>> wrote:
>>>>>>
>>>>>> Any chance for an IT test being added to the branch first? I'd love to
>>>> put
>>>>>> it through the paces with clusterdock to make sure it behaves well
>> with
>>>>>> fault injection and the like.
>>>>>>
>>>>>> -Dima
>>>>>>
>>>>>>> On Tuesday, August 2, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> Any more comments from the community on whether the merge can be
>>>>>> conducted
>>>>>>> ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
>>>>>> vladrodionov@gmail.com
>>>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Carter Shanklin posted a blog article about the feature:
>>>>>>>> Some use cases and examples of a command line interface usage.
>>>>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
>>>>>> backup-restore-apache-hbase-apache-phoenix/
>>>>>>>>
>>>>>>>> -Vlad
>>>>>>>>
>>>>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ok, got it.
>>>>>>>>>
>>>>>>>>> -Vlad
>>>>>>>>>
>>>>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <enis@apache.org
>>>>>>> <javascript:;>> wrote:
>>>>>>>>>
>>>>>>>>>> We keep the WALs which can accumulate a lot if the use case is to
>>>>>> only
>>>>>>>> do
>>>>>>>>>> backups infrequently. This will definitely cause issues since HDFS
>>>>>>> space
>>>>>>>>>> will get filled up. That is why we may need an option for having
>>>>>>>>>> incremental backups not used, and WAL references being deleted.
>>>>>>>>>>
>>>>>>>>>> Enis
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
>>>>>>>>>> vladrodionov@gmail.com <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Why anyone will ever need disabling incremental backups? If you
>> do
>>>>>>> not
>>>>>>>>>> need
>>>>>>>>>>> it - just run only full backups.
>>>>>>>>>>>
>>>>>>>>>>> -Vlad
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <enis@apache.org
>>>>>>> <javascript:;>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Matteo for chiming in.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
>>>>>>>>>>> theo.bertozzi@gmail.com <javascript:;>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I did some review in the early beginning, but then lost track
>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> changes.
>>>>>>>>>>>>> but I'd like to give a quick review to the full code once
>>>>>> people
>>>>>>>>>> here
>>>>>>>>>>> are
>>>>>>>>>>>>> ok with getting this feature in master (2.0).
>>>>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for
>>>>>>> reviewing
>>>>>>>>>> the
>>>>>>>>>>>> full
>>>>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid
>>>>>>> holding
>>>>>>>>>> this
>>>>>>>>>>>> for
>>>>>>>>>>>>> too long, but still enough time to have people that are
>>>>>>> interested
>>>>>>>>>> to
>>>>>>>>>>>> look
>>>>>>>>>>>>> at it. with did the same thing for MOB with a mega patch
>>>>>>>>>>>>> https://reviews.apache.org/r/36391/)
>>>>>>>>>>>>
>>>>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to handle
>>>>>>> the
>>>>>>>>>>> merge?
>>>>>>>>>>>> As a giant patch or a rebase of code in the branch and through
>>>>>> git
>>>>>>>>>> merge.
>>>>>>>>>>>>
>>>>>>>>>>>> We need to run a vote when the to-be-merged branched is ready.
>>>>>> We
>>>>>>>> can
>>>>>>>>>>> set a
>>>>>>>>>>>> vote timeout for at least 1 week.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> most of the code seemed isolated from the beginning, few
>>>>>> changes
>>>>>>>>>> here
>>>>>>>>>>> and
>>>>>>>>>>>>> there in the core.
>>>>>>>>>>>>> so, this side of things seems ok to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that
>>>>>>>> should
>>>>>>>>>> not
>>>>>>>>>>>>> take long.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know if there are already docs, but that is another
>>>>>>> thing
>>>>>>>> we
>>>>>>>>>>> may
>>>>>>>>>>>>> want to get in with the merge.
>>>>>>>>>>>>> a minimal coverage at least on how to use the feature, and
>>>>>> maybe
>>>>>>>>>>> calling
>>>>>>>>>>>> it
>>>>>>>>>>>>> out as experimental?
>>>>>>>>>>>>>
>>>>>>>>>>>>> my main concern were around incremental backups.
>>>>>>>>>>>>> I'm still not convinced around the fact that because the WALs
>>>>>>>>>> contain
>>>>>>>>>>>>> regions of multiple tables
>>>>>>>>>>>>> the incremental backup will keep around WALs with some data
>>>>>> that
>>>>>>>> we
>>>>>>>>>>> don't
>>>>>>>>>>>>> really want in the backup (for space or maybe security
>>>>>> reason).
>>>>>>>>>>>>>
>>>>>>>>>>>>> then there was the question about for how long should I take
>>>>>>>>>>>> incrementals,
>>>>>>>>>>>>> before deciding that a fresh full backup is less costly in
>>>>>> terms
>>>>>>>> of
>>>>>>>>>>>> space.
>>>>>>>>>>>>> but I think this incremental merge/compaction was a feature on
>>>>>>> the
>>>>>>>>>>>> roadmap
>>>>>>>>>>>>> as Phase3.
>>>>>>>>>>>>> which I think is ok to get later on,
>>>>>>>>>>>>> maybe just call out a lifecycle example on the docs under
>>>>>> "best
>>>>>>>>>>>> practices".
>>>>>>>>>>>>
>>>>>>>>>>>> I think this will depend on the use case, and other factors like
>>>>>>>>>>> bandwidth
>>>>>>>>>>>> available, how much data
>>>>>>>>>>>> the user is willing to lose in case of catastrophic failure and
>>>>>>> how
>>>>>>>>>>>> "expensive" is full backup versus
>>>>>>>>>>>> incremental one.
>>>>>>>>>>>>
>>>>>>>>>>>> The full backup should also be useable by default, so maybe we
>>>>>> can
>>>>>>>>>> make
>>>>>>>>>>> an
>>>>>>>>>>>> option to not even keep WAL files, and completely disable
>>>>>>>> incremental
>>>>>>>>>>>> backups?
>>>>>>>>>>>>
>>>>>>>>>>>> Enis
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> has anyone interested in using backups looked at the doc in
>>>>>>>>>> HBASE-7912?
>>>>>>>>>>>>> is the current design of incremental backup acceptable for
>>>>>>>> everyone
>>>>>>>>>>>> wanting
>>>>>>>>>>>>> to use this feature?
>>>>>>>>>>>>> (maybe this should be a question for the @user list and not
>>>>>> dev)
>>>>>>>>>>>>>
>>>>>>>>>>>>> is there anyone already using this feature or it is just dev
>>>>>>>> testing
>>>>>>>>>>> it?
>>>>>>>>>>>>> to me will be interesting having a use-case/workflow example,
>>>>>>>>>>>>> to see if in the real world my concerns about incremental are
>>>>>>> not
>>>>>>>>>>> showing
>>>>>>>>>>>>> up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>> <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Gentle ping on this subject.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The changes are mostly non-intrusive.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> More comments are welcome.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
>>>>>>>>>>>>>> andrew.purtell@gmail.com <javascript:;>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How hard would it be to convert what you've been using
>>>>>> to
>>>>>>>> test
>>>>>>>>>>> end
>>>>>>>>>>>> to
>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>> during dev into an IT?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If
>>>>>> not,
>>>>>>>> any
>>>>>>>>>>> tips
>>>>>>>>>>>>> on a
>>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
>>>>>>> them?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests.
>>>>>> We
>>>>>>>>>>> provide 2
>>>>>>>>>>>>> API
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> backup:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. Admin.getBackupAdmin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2. Command - line via hbase command.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Everything is straightforward.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
>>>>>>>>>>>>> dspivak@cloudera.com <javascript:;>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not,
>>>>>>> any
>>>>>>>>>> tips
>>>>>>>>>>>> on
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore
>>>>>> them?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Dima
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sorry, wrong links:
>>>>>>>>>>>>>>>>>>> These are the phases:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Phase 1:
>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030
>>>>>>>>> 14030
>>>>>>>>>>>>>>>>>>> Phase 2:
>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123
>>>>>>>>> 14123
>>>>>>>>>>>>>>>>>>> Phase 3:
>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414
>>>>>>>>> 14414
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
>>>>>>>>>>>>>>>>>> vladrodionov@gmail.com <javascript:;>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> These are the phases:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Phase 1:
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>> 14030
>>>>>>>>>>>>>>>>>>>> Phase 2:
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>> 14123
>>>>>>>>>>>>>>>>>>>> Phase 3:
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-
>>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>> 14414
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar <
>>>>>>>>>>>>> enis@apache.org <javascript:;>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir,
>>>>>> Ted,
>>>>>>>>>> Jerry
>>>>>>>>>>> and
>>>>>>>>>>>>>>> others
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality
>>>>>>> in
>>>>>>>> a
>>>>>>>>>>>> series
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1].
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a
>>>>>>> 4-phase
>>>>>>>>>>>> project,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are
>>>>>>> now
>>>>>>>>>>> working
>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> Phase 3
>>>>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think
>>>>>> that
>>>>>>>> the
>>>>>>>>>>>> current
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items,
>>>>>> and
>>>>>>>>>> some
>>>>>>>>>>>>> Phase 3
>>>>>>>>>>>>>>>>>> items
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait
>>>>>> for
>>>>>>>> all
>>>>>>>>>> the
>>>>>>>>>>>>>>>> subtickets
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as
>>>>>> follow
>>>>>>>> up
>>>>>>>>>>>> tickets
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements
>>>>>> in
>>>>>>>> the
>>>>>>>>>>> works
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff.
>>>>>> Thus,
>>>>>>> we
>>>>>>>>>>> would
>>>>>>>>>>>>> like
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into
>>>>>>>>>> master.
>>>>>>>>>>>> The
>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>> jira
>>>>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the
>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>>>>> choices in case you are interested[2].
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and
>>>>>>>>>> confined
>>>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> backup subsystem.
>>>>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs
>>>>>> and
>>>>>>> we
>>>>>>>>>>>>>>> (hortonworks)
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> been running the integration tests as well as some
>>>>>>>> other
>>>>>>>>>>>>>>> shell-based
>>>>>>>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the
>>>>>>>> work
>>>>>>>>>> has
>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>> reviewed
>>>>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted,
>>>>>> myself
>>>>>>>> and
>>>>>>>>>>>> Jerry).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote?
>>>>>>> Any
>>>>>>>>>>>> concerns
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>> https://issues.apache.org/jira/secure/attachment/12816339/
>>>>>> HBaseBackupAndRestore%20-0.91.pdf
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Enis
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Dima
>>