Re: DISCUSS: Remove hbase-backup from master?

["mallik.v.arjun@gmail.com" <ma...@gmail.com>]

On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:

> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> mallik.v.arjun@gmail.com> wrote:
>
> > Inline.
> >
> > On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >
> > > I'm afraid it is not easy to be moved to hbase-operator-tools.
> > >
> > > You can see the code under the org.apache.hadoop.hbase.backup.master
> > > package, we need to set up log cleaner at master side, and also,
> > > the LogRollMasterProcedureManager class needs MasterServices(not
> > > MasterRpcServices), which means it must be used in the same process
> with
> > > HMaster.
> > >
> > > And I'm OK with purging this feature, especially if there is no
> developer
> > > who wants to maintain it.
> > >
> > >
> > Is the sole reason to purge this feature is no developer volunteering to
> > maintain this?
> >
> >
> I'd suggest this feature does not belong in core at all.
>
> And some of us have spent time reviewing the feature in the past and back
> then determined it needed lots of finishing work, test, and polish -- none
> of which it seems to have gotten since commit.
>
> You are using it Mallik? Tell us more please. How is it working for you?
>
>
It has been a decent v1 release for us. As pointed out by others, feature
should be polished and matured without debating if backup should be part of
core or not. Some of notable points are

   - Limitations like serial backup set runs per deployment bothered us.
   - Minor logging enhancements for debugging, some NPE's, etc
   - Bandwidth limits, incremental backup did not work outright.

Since it is only a couple of months, we might not have seen a lot of corner
cases.


> S
>
>
> >
> > > For me, I suspect that the backup feature could be done more separately
> > > with the main cluster. We could use replication to backup the WALs, and
> > use
> > > Snapshot and ExportSnapshot to backup the HFiles. The feature could be
> > done
> > > as a separated project.
> > >
> > > Thanks.
> > >
> > > Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > >
> > > > It strikes me as work that has been abandoned with no supporting
> > > developer.
> > > > It has had no improvement and few commits other than adjustment
> > because a
> > > > backing dependency has changed since original contribution. It has
> not
> > > been
> > > > included in a release so has no users as yet. Does anyone use it or
> > want
> > > > it?
> >
> >
> > We have back ported this feature to 2.1.x and has been using for last few
> > months in few of our deployments.
> >
> >
> > > If not, I suggest we remove it.  I could file an issue for it to be
> >
> > > added to hbase-operator-tools for some gallant dev to pick up if they
> > > > wanted to use this backup work? (I could help w/ the migration).
> > > >
> > > > What do others think?
> > > >
> > > > S
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Mallikarjun <ma...@gmail.com>]

Sure. I have assigned it to myself. Will look into it.

Last time I checked, I did not find any failed tests and it was not hadoop3

---
Mallikarjun


On Fri, May 14, 2021 at 2:34 AM Nick Dimiduk <nd...@apache.org> wrote:

> Hi Mallikarjun,
>
> I just saw a bunch of backup tests fail on an unrelated PR build. I filed
> HBASE-25888 and uploaded some logs. I have the full test-logs.zip file, but
> it's too big to upload to jira. I linked it from the Jira, but the archive
> will disappear when the PR is eventually closed. I would ping you from
> Jira, but I didn't find any Jira user that seemed likely to be your
> account. Would you mind taking a look?
>
> Thanks,
> Nick
>
> On Wed, Dec 23, 2020 at 8:20 PM Mallikarjun <ma...@gmail.com>
> wrote:
>
> > Yea. I have noticed that.
> >
> > I have some patches ready and have to add unit tests. Will start raising
> in
> > a couple of weeks time.
> > ---
> > Mallikarjun
> >
> >
> > On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> > > The UTs in the backup module are easy to fail with NPE, I've seen this
> in
> > > several pre commit results.
> > >
> > > Any progress here?
> > >
> > > mallik.v.arjun@gmail.com <ma...@gmail.com> 于2020年12月3日周四
> > > 上午9:58写道：
> > >
> > > > On Tue, Dec 1, 2020 at 11:26 PM Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
> > > > > mallik.v.arjun@gmail.com> wrote:
> > > > >
> > > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > One reason for moving it out of core is that at some point we
> > will
> > > > have
> > > > > > > hbase 3 releases. Right now hbase 3 targeted things are in the
> > > master
> > > > > > > branch. If the feature is not sufficiently ready to be in a
> > release
> > > > > then
> > > > > > > when the time for HBase 3 releases comes we'll have to do work
> to
> > > > pull
> > > > > it
> > > > > > > out of the release branch. If the feature needs to stay in the
> > core
> > > > > repo
> > > > > > > that will necessitate that hbase 3 have a branch distinct from
> > the
> > > > > master
> > > > > > > branch (which may or may not happen anyways). At that point we
> > risk
> > > > > > having
> > > > > > > to do the work to remove the feature from a release branch
> again
> > > come
> > > > > > hbase
> > > > > > > 4.
> > > > > > >
> > > > > > >
> > > > > > > I think a lot of this is moot if you'd like to start providing
> > > > patches
> > > > > > for
> > > > > > > the things you've needed to use it. If there are gaps that you
> > > think
> > > > > can
> > > > > > > trip folks up maybe we could label it "experimental" to provide
> > > > better
> > > > > > > context for others.
> > > > > > >
> > > > > >
> > > > > > I will start putting effort in maintaining this feature.
> > > > > >
> > > > > >
> > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth
> > > > looking
> > > > > at.
> > > > >
> > > >
> > > > Any reference I can get. They seem fine when I run tests.
> > > >
> > > >
> > > > > Thanks,
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > > >
> > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net>
> > wrote:
> > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Inline
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We are allowed to debate if this should be in the tree
> or
> > > > not.
> > > > > > > Given
> > > > > > > > > the
> > > > > > > > > > > lack of interest, as evidenced by incomplete state,
> lack
> > of
> > > > > > > release,
> > > > > > > > > and
> > > > > > > > > > > lack of contribution, it is more than fair to discuss
> > > > removal.
> > > > > > > > > > >
> > > > > > > > > > > Here is my take: First of all, it is not released.
> There
> > is
> > > > no
> > > > > > > > implied
> > > > > > > > > > > roadmap or support. Second, there do not seem to be any
> > > > active
> > > > > > > > > > maintainers
> > > > > > > > > > > or volunteers as such. Third, unless someone shows up
> > with
> > > > more
> > > > > > > > patches
> > > > > > > > > > for
> > > > > > > > > > > it there will be no polish or maturing, there can be no
> > > > > > > expectations
> > > > > > > > of
> > > > > > > > > > > further improvement.
> > > > > > > > > > >
> > > > > > > > > > > That said, this is open source. New code contribution
> > will
> > > > > change
> > > > > > > the
> > > > > > > > > > > facts as they stand.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If there is an agreement upon the direction I am willing
> to
> > > > > provide
> > > > > > > > > patches
> > > > > > > > > > to take this feature forward.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > What about the changes you made to make it work?
> > > > > > > >
> > > > > > > >
> > > > > > > > > I'd be interested in patches that take this feature out of
> > > core.
> > > > > How
> > > > > > > hard
> > > > > > > > > do you think that would be? See the items Duo list above
> for
> > a
> > > > > start
> > > > > > on
> > > > > > > > > obstacles to tackle?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I am yet to be very familiar with the code to comment on the
> > > effort
> > > > > > > > required to take it out of the core. Backport and bug fixes
> > > > required
> > > > > > very
> > > > > > > > shallow understanding of the code and only some parts of it.
> > > > > > > >
> > > > > > > > I am not very clear on why some strong opinions on moving
> > backup
> > > > out
> > > > > of
> > > > > > > the
> > > > > > > > core? Is there something defined on what should be operator
> > > tools?
> > > > > > > >
> > > > > > > > Quoting what Duo Zhang said
> > > > > > > >
> > > > > > > > You can see the code under the
> > > > org.apache.hadoop.hbase.backup.master
> > > > > > > > > package, we need to set up log cleaner at master side, and
> > > also,
> > > > > > > > > the LogRollMasterProcedureManager class needs
> > > MasterServices(not
> > > > > > > > > MasterRpcServices), which means it must be used in the same
> > > > process
> > > > > > > with
> > > > > > > > > HMaster.
> > > > > > > >
> > > > > > > > .
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > S
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM,
> mallik.v.arjun@gmail.com
> > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <
> > stack@duboce.net
> > > >
> > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM
> > > mallik.v.arjun@gmail.com
> > > > <
> > > > > > > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Inline.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > > > > > > palomino219@gmail.com
> > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> I'm afraid it is not easy to be moved to
> > > > > > hbase-operator-tools.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> You can see the code under the
> > > > > > > > > org.apache.hadoop.hbase.backup.master
> > > > > > > > > > > >>>> package, we need to set up log cleaner at master
> > side,
> > > > and
> > > > > > > also,
> > > > > > > > > > > >>>> the LogRollMasterProcedureManager class needs
> > > > > > > MasterServices(not
> > > > > > > > > > > >>>> MasterRpcServices), which means it must be used in
> > the
> > > > > same
> > > > > > > > > process
> > > > > > > > > > > >> with
> > > > > > > > > > > >>>> HMaster.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> And I'm OK with purging this feature, especially
> if
> > > > there
> > > > > is
> > > > > > > no
> > > > > > > > > > > >> developer
> > > > > > > > > > > >>>> who wants to maintain it.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>> Is the sole reason to purge this feature is no
> > > developer
> > > > > > > > > volunteering
> > > > > > > > > > > to
> > > > > > > > > > > >>> maintain this?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >> I'd suggest this feature does not belong in core at
> > all.
> > > > > > > > > > > >>
> > > > > > > > > > > >> And some of us have spent time reviewing the feature
> > in
> > > > the
> > > > > > past
> > > > > > > > and
> > > > > > > > > > > back
> > > > > > > > > > > >> then determined it needed lots of finishing work,
> > test,
> > > > and
> > > > > > > polish
> > > > > > > > > --
> > > > > > > > > > > none
> > > > > > > > > > > >> of which it seems to have gotten since commit.
> > > > > > > > > > > >>
> > > > > > > > > > > >> You are using it Mallik? Tell us more please. How is
> > it
> > > > > > working
> > > > > > > > for
> > > > > > > > > > you?
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > > It has been a decent v1 release for us. As pointed
> out
> > by
> > > > > > others,
> > > > > > > > > > feature
> > > > > > > > > > > > should be polished and matured without debating if
> > backup
> > > > > > should
> > > > > > > be
> > > > > > > > > > part
> > > > > > > > > > > of
> > > > > > > > > > > > core or not. Some of notable points are
> > > > > > > > > > > >
> > > > > > > > > > > >   - Limitations like serial backup set runs per
> > > deployment
> > > > > > > bothered
> > > > > > > > > us.
> > > > > > > > > > > >   - Minor logging enhancements for debugging, some
> > NPE's,
> > > > etc
> > > > > > > > > > > >   - Bandwidth limits, incremental backup did not work
> > > > > outright.
> > > > > > > > > > > >
> > > > > > > > > > > > Since it is only a couple of months, we might not
> have
> > > > seen a
> > > > > > lot
> > > > > > > > of
> > > > > > > > > > > corner
> > > > > > > > > > > > cases.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >> S
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> For me, I suspect that the backup feature could be
> > > done
> > > > > more
> > > > > > > > > > > separately
> > > > > > > > > > > >>>> with the main cluster. We could use replication to
> > > > backup
> > > > > > the
> > > > > > > > > WALs,
> > > > > > > > > > > and
> > > > > > > > > > > >>> use
> > > > > > > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles.
> > The
> > > > > > feature
> > > > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > >>> done
> > > > > > > > > > > >>>> as a separated project.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Thanks.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五
> 上午10:18写道：
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> It strikes me as work that has been abandoned
> with
> > no
> > > > > > > > supporting
> > > > > > > > > > > >>>> developer.
> > > > > > > > > > > >>>>> It has had no improvement and few commits other
> > than
> > > > > > > adjustment
> > > > > > > > > > > >>> because a
> > > > > > > > > > > >>>>> backing dependency has changed since original
> > > > > contribution.
> > > > > > > It
> > > > > > > > > has
> > > > > > > > > > > >> not
> > > > > > > > > > > >>>> been
> > > > > > > > > > > >>>>> included in a release so has no users as yet.
> Does
> > > > anyone
> > > > > > use
> > > > > > > > it
> > > > > > > > > or
> > > > > > > > > > > >>> want
> > > > > > > > > > > >>>>> it?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> We have back ported this feature to 2.1.x and has
> > been
> > > > > using
> > > > > > > for
> > > > > > > > > last
> > > > > > > > > > > few
> > > > > > > > > > > >>> months in few of our deployments.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> If not, I suggest we remove it.  I could file an
> > issue
> > > > for
> > > > > > it
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> added to hbase-operator-tools for some gallant dev
> > to
> > > > pick
> > > > > > up
> > > > > > > if
> > > > > > > > > > they
> > > > > > > > > > > >>>>> wanted to use this backup work? (I could help w/
> > the
> > > > > > > > migration).
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> What do others think?
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> S
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Nick Dimiduk <nd...@apache.org>]

Hi Mallikarjun,

I just saw a bunch of backup tests fail on an unrelated PR build. I filed
HBASE-25888 and uploaded some logs. I have the full test-logs.zip file, but
it's too big to upload to jira. I linked it from the Jira, but the archive
will disappear when the PR is eventually closed. I would ping you from
Jira, but I didn't find any Jira user that seemed likely to be your
account. Would you mind taking a look?

Thanks,
Nick

On Wed, Dec 23, 2020 at 8:20 PM Mallikarjun <ma...@gmail.com>
wrote:

> Yea. I have noticed that.
>
> I have some patches ready and have to add unit tests. Will start raising in
> a couple of weeks time.
> ---
> Mallikarjun
>
>
> On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > The UTs in the backup module are easy to fail with NPE, I've seen this in
> > several pre commit results.
> >
> > Any progress here?
> >
> > mallik.v.arjun@gmail.com <ma...@gmail.com> 于2020年12月3日周四
> > 上午9:58写道：
> >
> > > On Tue, Dec 1, 2020 at 11:26 PM Stack <st...@duboce.net> wrote:
> > >
> > > > On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
> > > > mallik.v.arjun@gmail.com> wrote:
> > > >
> > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org>
> > wrote:
> > > > >
> > > > > > One reason for moving it out of core is that at some point we
> will
> > > have
> > > > > > hbase 3 releases. Right now hbase 3 targeted things are in the
> > master
> > > > > > branch. If the feature is not sufficiently ready to be in a
> release
> > > > then
> > > > > > when the time for HBase 3 releases comes we'll have to do work to
> > > pull
> > > > it
> > > > > > out of the release branch. If the feature needs to stay in the
> core
> > > > repo
> > > > > > that will necessitate that hbase 3 have a branch distinct from
> the
> > > > master
> > > > > > branch (which may or may not happen anyways). At that point we
> risk
> > > > > having
> > > > > > to do the work to remove the feature from a release branch again
> > come
> > > > > hbase
> > > > > > 4.
> > > > > >
> > > > > >
> > > > > > I think a lot of this is moot if you'd like to start providing
> > > patches
> > > > > for
> > > > > > the things you've needed to use it. If there are gaps that you
> > think
> > > > can
> > > > > > trip folks up maybe we could label it "experimental" to provide
> > > better
> > > > > > context for others.
> > > > > >
> > > > >
> > > > > I will start putting effort in maintaining this feature.
> > > > >
> > > > >
> > > > FYI, seems like backup has a bunch of flakey tests. Might be worth
> > > looking
> > > > at.
> > > >
> > >
> > > Any reference I can get. They seem fine when I run tests.
> > >
> > >
> > > > Thanks,
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > >
> > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net>
> wrote:
> > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Inline
> > > > > > > > >
> > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > > > > > andrew.purtell@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We are allowed to debate if this should be in the tree or
> > > not.
> > > > > > Given
> > > > > > > > the
> > > > > > > > > > lack of interest, as evidenced by incomplete state, lack
> of
> > > > > > release,
> > > > > > > > and
> > > > > > > > > > lack of contribution, it is more than fair to discuss
> > > removal.
> > > > > > > > > >
> > > > > > > > > > Here is my take: First of all, it is not released. There
> is
> > > no
> > > > > > > implied
> > > > > > > > > > roadmap or support. Second, there do not seem to be any
> > > active
> > > > > > > > > maintainers
> > > > > > > > > > or volunteers as such. Third, unless someone shows up
> with
> > > more
> > > > > > > patches
> > > > > > > > > for
> > > > > > > > > > it there will be no polish or maturing, there can be no
> > > > > > expectations
> > > > > > > of
> > > > > > > > > > further improvement.
> > > > > > > > > >
> > > > > > > > > > That said, this is open source. New code contribution
> will
> > > > change
> > > > > > the
> > > > > > > > > > facts as they stand.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If there is an agreement upon the direction I am willing to
> > > > provide
> > > > > > > > patches
> > > > > > > > > to take this feature forward.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > What about the changes you made to make it work?
> > > > > > >
> > > > > > >
> > > > > > > > I'd be interested in patches that take this feature out of
> > core.
> > > > How
> > > > > > hard
> > > > > > > > do you think that would be? See the items Duo list above for
> a
> > > > start
> > > > > on
> > > > > > > > obstacles to tackle?
> > > > > > > >
> > > > > > >
> > > > > > > I am yet to be very familiar with the code to comment on the
> > effort
> > > > > > > required to take it out of the core. Backport and bug fixes
> > > required
> > > > > very
> > > > > > > shallow understanding of the code and only some parts of it.
> > > > > > >
> > > > > > > I am not very clear on why some strong opinions on moving
> backup
> > > out
> > > > of
> > > > > > the
> > > > > > > core? Is there something defined on what should be operator
> > tools?
> > > > > > >
> > > > > > > Quoting what Duo Zhang said
> > > > > > >
> > > > > > > You can see the code under the
> > > org.apache.hadoop.hbase.backup.master
> > > > > > > > package, we need to set up log cleaner at master side, and
> > also,
> > > > > > > > the LogRollMasterProcedureManager class needs
> > MasterServices(not
> > > > > > > > MasterRpcServices), which means it must be used in the same
> > > process
> > > > > > with
> > > > > > > > HMaster.
> > > > > > >
> > > > > > > .
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > S
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com
> > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <
> stack@duboce.net
> > >
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM
> > mallik.v.arjun@gmail.com
> > > <
> > > > > > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> Inline.
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > > > > > palomino219@gmail.com
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> I'm afraid it is not easy to be moved to
> > > > > hbase-operator-tools.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> You can see the code under the
> > > > > > > > org.apache.hadoop.hbase.backup.master
> > > > > > > > > > >>>> package, we need to set up log cleaner at master
> side,
> > > and
> > > > > > also,
> > > > > > > > > > >>>> the LogRollMasterProcedureManager class needs
> > > > > > MasterServices(not
> > > > > > > > > > >>>> MasterRpcServices), which means it must be used in
> the
> > > > same
> > > > > > > > process
> > > > > > > > > > >> with
> > > > > > > > > > >>>> HMaster.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> And I'm OK with purging this feature, especially if
> > > there
> > > > is
> > > > > > no
> > > > > > > > > > >> developer
> > > > > > > > > > >>>> who wants to maintain it.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>> Is the sole reason to purge this feature is no
> > developer
> > > > > > > > volunteering
> > > > > > > > > > to
> > > > > > > > > > >>> maintain this?
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >> I'd suggest this feature does not belong in core at
> all.
> > > > > > > > > > >>
> > > > > > > > > > >> And some of us have spent time reviewing the feature
> in
> > > the
> > > > > past
> > > > > > > and
> > > > > > > > > > back
> > > > > > > > > > >> then determined it needed lots of finishing work,
> test,
> > > and
> > > > > > polish
> > > > > > > > --
> > > > > > > > > > none
> > > > > > > > > > >> of which it seems to have gotten since commit.
> > > > > > > > > > >>
> > > > > > > > > > >> You are using it Mallik? Tell us more please. How is
> it
> > > > > working
> > > > > > > for
> > > > > > > > > you?
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > > It has been a decent v1 release for us. As pointed out
> by
> > > > > others,
> > > > > > > > > feature
> > > > > > > > > > > should be polished and matured without debating if
> backup
> > > > > should
> > > > > > be
> > > > > > > > > part
> > > > > > > > > > of
> > > > > > > > > > > core or not. Some of notable points are
> > > > > > > > > > >
> > > > > > > > > > >   - Limitations like serial backup set runs per
> > deployment
> > > > > > bothered
> > > > > > > > us.
> > > > > > > > > > >   - Minor logging enhancements for debugging, some
> NPE's,
> > > etc
> > > > > > > > > > >   - Bandwidth limits, incremental backup did not work
> > > > outright.
> > > > > > > > > > >
> > > > > > > > > > > Since it is only a couple of months, we might not have
> > > seen a
> > > > > lot
> > > > > > > of
> > > > > > > > > > corner
> > > > > > > > > > > cases.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >> S
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>>
> > > > > > > > > > >>>> For me, I suspect that the backup feature could be
> > done
> > > > more
> > > > > > > > > > separately
> > > > > > > > > > >>>> with the main cluster. We could use replication to
> > > backup
> > > > > the
> > > > > > > > WALs,
> > > > > > > > > > and
> > > > > > > > > > >>> use
> > > > > > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles.
> The
> > > > > feature
> > > > > > > > could
> > > > > > > > > be
> > > > > > > > > > >>> done
> > > > > > > > > > >>>> as a separated project.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Thanks.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> It strikes me as work that has been abandoned with
> no
> > > > > > > supporting
> > > > > > > > > > >>>> developer.
> > > > > > > > > > >>>>> It has had no improvement and few commits other
> than
> > > > > > adjustment
> > > > > > > > > > >>> because a
> > > > > > > > > > >>>>> backing dependency has changed since original
> > > > contribution.
> > > > > > It
> > > > > > > > has
> > > > > > > > > > >> not
> > > > > > > > > > >>>> been
> > > > > > > > > > >>>>> included in a release so has no users as yet. Does
> > > anyone
> > > > > use
> > > > > > > it
> > > > > > > > or
> > > > > > > > > > >>> want
> > > > > > > > > > >>>>> it?
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> We have back ported this feature to 2.1.x and has
> been
> > > > using
> > > > > > for
> > > > > > > > last
> > > > > > > > > > few
> > > > > > > > > > >>> months in few of our deployments.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>> If not, I suggest we remove it.  I could file an
> issue
> > > for
> > > > > it
> > > > > > to
> > > > > > > > be
> > > > > > > > > > >>>
> > > > > > > > > > >>>> added to hbase-operator-tools for some gallant dev
> to
> > > pick
> > > > > up
> > > > > > if
> > > > > > > > > they
> > > > > > > > > > >>>>> wanted to use this backup work? (I could help w/
> the
> > > > > > > migration).
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> What do others think?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> S
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Mallikarjun <ma...@gmail.com>]

Yea. I have noticed that.

I have some patches ready and have to add unit tests. Will start raising in
a couple of weeks time.
---
Mallikarjun


On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> The UTs in the backup module are easy to fail with NPE, I've seen this in
> several pre commit results.
>
> Any progress here?
>
> mallik.v.arjun@gmail.com <ma...@gmail.com> 于2020年12月3日周四
> 上午9:58写道：
>
> > On Tue, Dec 1, 2020 at 11:26 PM Stack <st...@duboce.net> wrote:
> >
> > > On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
> > > mallik.v.arjun@gmail.com> wrote:
> > >
> > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org>
> wrote:
> > > >
> > > > > One reason for moving it out of core is that at some point we will
> > have
> > > > > hbase 3 releases. Right now hbase 3 targeted things are in the
> master
> > > > > branch. If the feature is not sufficiently ready to be in a release
> > > then
> > > > > when the time for HBase 3 releases comes we'll have to do work to
> > pull
> > > it
> > > > > out of the release branch. If the feature needs to stay in the core
> > > repo
> > > > > that will necessitate that hbase 3 have a branch distinct from the
> > > master
> > > > > branch (which may or may not happen anyways). At that point we risk
> > > > having
> > > > > to do the work to remove the feature from a release branch again
> come
> > > > hbase
> > > > > 4.
> > > > >
> > > > >
> > > > > I think a lot of this is moot if you'd like to start providing
> > patches
> > > > for
> > > > > the things you've needed to use it. If there are gaps that you
> think
> > > can
> > > > > trip folks up maybe we could label it "experimental" to provide
> > better
> > > > > context for others.
> > > > >
> > > >
> > > > I will start putting effort in maintaining this feature.
> > > >
> > > >
> > > FYI, seems like backup has a bunch of flakey tests. Might be worth
> > looking
> > > at.
> > >
> >
> > Any reference I can get. They seem fine when I run tests.
> >
> >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > >
> > > >
> > > > >
> > > > > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > > > > mallik.v.arjun@gmail.com> wrote:
> > > > >
> > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
> > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > > >
> > > > > > > > Inline
> > > > > > > >
> > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > > > > andrew.purtell@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We are allowed to debate if this should be in the tree or
> > not.
> > > > > Given
> > > > > > > the
> > > > > > > > > lack of interest, as evidenced by incomplete state, lack of
> > > > > release,
> > > > > > > and
> > > > > > > > > lack of contribution, it is more than fair to discuss
> > removal.
> > > > > > > > >
> > > > > > > > > Here is my take: First of all, it is not released. There is
> > no
> > > > > > implied
> > > > > > > > > roadmap or support. Second, there do not seem to be any
> > active
> > > > > > > > maintainers
> > > > > > > > > or volunteers as such. Third, unless someone shows up with
> > more
> > > > > > patches
> > > > > > > > for
> > > > > > > > > it there will be no polish or maturing, there can be no
> > > > > expectations
> > > > > > of
> > > > > > > > > further improvement.
> > > > > > > > >
> > > > > > > > > That said, this is open source. New code contribution will
> > > change
> > > > > the
> > > > > > > > > facts as they stand.
> > > > > > > >
> > > > > > > >
> > > > > > > > If there is an agreement upon the direction I am willing to
> > > provide
> > > > > > > patches
> > > > > > > > to take this feature forward.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > What about the changes you made to make it work?
> > > > > >
> > > > > >
> > > > > > > I'd be interested in patches that take this feature out of
> core.
> > > How
> > > > > hard
> > > > > > > do you think that would be? See the items Duo list above for a
> > > start
> > > > on
> > > > > > > obstacles to tackle?
> > > > > > >
> > > > > >
> > > > > > I am yet to be very familiar with the code to comment on the
> effort
> > > > > > required to take it out of the core. Backport and bug fixes
> > required
> > > > very
> > > > > > shallow understanding of the code and only some parts of it.
> > > > > >
> > > > > > I am not very clear on why some strong opinions on moving backup
> > out
> > > of
> > > > > the
> > > > > > core? Is there something defined on what should be operator
> tools?
> > > > > >
> > > > > > Quoting what Duo Zhang said
> > > > > >
> > > > > > You can see the code under the
> > org.apache.hadoop.hbase.backup.master
> > > > > > > package, we need to set up log cleaner at master side, and
> also,
> > > > > > > the LogRollMasterProcedureManager class needs
> MasterServices(not
> > > > > > > MasterRpcServices), which means it must be used in the same
> > process
> > > > > with
> > > > > > > HMaster.
> > > > > >
> > > > > > .
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com
> > wrote:
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <stack@duboce.net
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM
> mallik.v.arjun@gmail.com
> > <
> > > > > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Inline.
> > > > > > > > > >>>
> > > > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > > > > palomino219@gmail.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> I'm afraid it is not easy to be moved to
> > > > hbase-operator-tools.
> > > > > > > > > >>>>
> > > > > > > > > >>>> You can see the code under the
> > > > > > > org.apache.hadoop.hbase.backup.master
> > > > > > > > > >>>> package, we need to set up log cleaner at master side,
> > and
> > > > > also,
> > > > > > > > > >>>> the LogRollMasterProcedureManager class needs
> > > > > MasterServices(not
> > > > > > > > > >>>> MasterRpcServices), which means it must be used in the
> > > same
> > > > > > > process
> > > > > > > > > >> with
> > > > > > > > > >>>> HMaster.
> > > > > > > > > >>>>
> > > > > > > > > >>>> And I'm OK with purging this feature, especially if
> > there
> > > is
> > > > > no
> > > > > > > > > >> developer
> > > > > > > > > >>>> who wants to maintain it.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>> Is the sole reason to purge this feature is no
> developer
> > > > > > > volunteering
> > > > > > > > > to
> > > > > > > > > >>> maintain this?
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >> I'd suggest this feature does not belong in core at all.
> > > > > > > > > >>
> > > > > > > > > >> And some of us have spent time reviewing the feature in
> > the
> > > > past
> > > > > > and
> > > > > > > > > back
> > > > > > > > > >> then determined it needed lots of finishing work, test,
> > and
> > > > > polish
> > > > > > > --
> > > > > > > > > none
> > > > > > > > > >> of which it seems to have gotten since commit.
> > > > > > > > > >>
> > > > > > > > > >> You are using it Mallik? Tell us more please. How is it
> > > > working
> > > > > > for
> > > > > > > > you?
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > > It has been a decent v1 release for us. As pointed out by
> > > > others,
> > > > > > > > feature
> > > > > > > > > > should be polished and matured without debating if backup
> > > > should
> > > > > be
> > > > > > > > part
> > > > > > > > > of
> > > > > > > > > > core or not. Some of notable points are
> > > > > > > > > >
> > > > > > > > > >   - Limitations like serial backup set runs per
> deployment
> > > > > bothered
> > > > > > > us.
> > > > > > > > > >   - Minor logging enhancements for debugging, some NPE's,
> > etc
> > > > > > > > > >   - Bandwidth limits, incremental backup did not work
> > > outright.
> > > > > > > > > >
> > > > > > > > > > Since it is only a couple of months, we might not have
> > seen a
> > > > lot
> > > > > > of
> > > > > > > > > corner
> > > > > > > > > > cases.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >> S
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>>
> > > > > > > > > >>>> For me, I suspect that the backup feature could be
> done
> > > more
> > > > > > > > > separately
> > > > > > > > > >>>> with the main cluster. We could use replication to
> > backup
> > > > the
> > > > > > > WALs,
> > > > > > > > > and
> > > > > > > > > >>> use
> > > > > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The
> > > > feature
> > > > > > > could
> > > > > > > > be
> > > > > > > > > >>> done
> > > > > > > > > >>>> as a separated project.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > > > > > >>>>
> > > > > > > > > >>>>> It strikes me as work that has been abandoned with no
> > > > > > supporting
> > > > > > > > > >>>> developer.
> > > > > > > > > >>>>> It has had no improvement and few commits other than
> > > > > adjustment
> > > > > > > > > >>> because a
> > > > > > > > > >>>>> backing dependency has changed since original
> > > contribution.
> > > > > It
> > > > > > > has
> > > > > > > > > >> not
> > > > > > > > > >>>> been
> > > > > > > > > >>>>> included in a release so has no users as yet. Does
> > anyone
> > > > use
> > > > > > it
> > > > > > > or
> > > > > > > > > >>> want
> > > > > > > > > >>>>> it?
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> We have back ported this feature to 2.1.x and has been
> > > using
> > > > > for
> > > > > > > last
> > > > > > > > > few
> > > > > > > > > >>> months in few of our deployments.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>> If not, I suggest we remove it.  I could file an issue
> > for
> > > > it
> > > > > to
> > > > > > > be
> > > > > > > > > >>>
> > > > > > > > > >>>> added to hbase-operator-tools for some gallant dev to
> > pick
> > > > up
> > > > > if
> > > > > > > > they
> > > > > > > > > >>>>> wanted to use this backup work? (I could help w/ the
> > > > > > migration).
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> What do others think?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> S
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

["张铎 (Duo Zhang)" <pa...@gmail.com>]

The UTs in the backup module are easy to fail with NPE, I've seen this in
several pre commit results.

Any progress here?

mallik.v.arjun@gmail.com <ma...@gmail.com> 于2020年12月3日周四 上午9:58写道：

> On Tue, Dec 1, 2020 at 11:26 PM Stack <st...@duboce.net> wrote:
>
> > On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
> > mallik.v.arjun@gmail.com> wrote:
> >
> > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org> wrote:
> > >
> > > > One reason for moving it out of core is that at some point we will
> have
> > > > hbase 3 releases. Right now hbase 3 targeted things are in the master
> > > > branch. If the feature is not sufficiently ready to be in a release
> > then
> > > > when the time for HBase 3 releases comes we'll have to do work to
> pull
> > it
> > > > out of the release branch. If the feature needs to stay in the core
> > repo
> > > > that will necessitate that hbase 3 have a branch distinct from the
> > master
> > > > branch (which may or may not happen anyways). At that point we risk
> > > having
> > > > to do the work to remove the feature from a release branch again come
> > > hbase
> > > > 4.
> > > >
> > > >
> > > > I think a lot of this is moot if you'd like to start providing
> patches
> > > for
> > > > the things you've needed to use it. If there are gaps that you think
> > can
> > > > trip folks up maybe we could label it "experimental" to provide
> better
> > > > context for others.
> > > >
> > >
> > > I will start putting effort in maintaining this feature.
> > >
> > >
> > FYI, seems like backup has a bunch of flakey tests. Might be worth
> looking
> > at.
> >
>
> Any reference I can get. They seem fine when I run tests.
>
>
> > Thanks,
> > S
> >
> >
> >
> >
> > >
> > > >
> > > > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > > > mallik.v.arjun@gmail.com> wrote:
> > > >
> > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > > > mallik.v.arjun@gmail.com> wrote:
> > > > > >
> > > > > > > Inline
> > > > > > >
> > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > > > andrew.purtell@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We are allowed to debate if this should be in the tree or
> not.
> > > > Given
> > > > > > the
> > > > > > > > lack of interest, as evidenced by incomplete state, lack of
> > > > release,
> > > > > > and
> > > > > > > > lack of contribution, it is more than fair to discuss
> removal.
> > > > > > > >
> > > > > > > > Here is my take: First of all, it is not released. There is
> no
> > > > > implied
> > > > > > > > roadmap or support. Second, there do not seem to be any
> active
> > > > > > > maintainers
> > > > > > > > or volunteers as such. Third, unless someone shows up with
> more
> > > > > patches
> > > > > > > for
> > > > > > > > it there will be no polish or maturing, there can be no
> > > > expectations
> > > > > of
> > > > > > > > further improvement.
> > > > > > > >
> > > > > > > > That said, this is open source. New code contribution will
> > change
> > > > the
> > > > > > > > facts as they stand.
> > > > > > >
> > > > > > >
> > > > > > > If there is an agreement upon the direction I am willing to
> > provide
> > > > > > patches
> > > > > > > to take this feature forward.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > What about the changes you made to make it work?
> > > > >
> > > > >
> > > > > > I'd be interested in patches that take this feature out of core.
> > How
> > > > hard
> > > > > > do you think that would be? See the items Duo list above for a
> > start
> > > on
> > > > > > obstacles to tackle?
> > > > > >
> > > > >
> > > > > I am yet to be very familiar with the code to comment on the effort
> > > > > required to take it out of the core. Backport and bug fixes
> required
> > > very
> > > > > shallow understanding of the code and only some parts of it.
> > > > >
> > > > > I am not very clear on why some strong opinions on moving backup
> out
> > of
> > > > the
> > > > > core? Is there something defined on what should be operator tools?
> > > > >
> > > > > Quoting what Duo Zhang said
> > > > >
> > > > > You can see the code under the
> org.apache.hadoop.hbase.backup.master
> > > > > > package, we need to set up log cleaner at master side, and also,
> > > > > > the LogRollMasterProcedureManager class needs MasterServices(not
> > > > > > MasterRpcServices), which means it must be used in the same
> process
> > > > with
> > > > > > HMaster.
> > > > >
> > > > > .
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > S
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com
> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net>
> > > wrote:
> > > > > > > > >
> > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com
> <
> > > > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > > > >>>
> > > > > > > > >>> Inline.
> > > > > > > > >>>
> > > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > > > palomino219@gmail.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> I'm afraid it is not easy to be moved to
> > > hbase-operator-tools.
> > > > > > > > >>>>
> > > > > > > > >>>> You can see the code under the
> > > > > > org.apache.hadoop.hbase.backup.master
> > > > > > > > >>>> package, we need to set up log cleaner at master side,
> and
> > > > also,
> > > > > > > > >>>> the LogRollMasterProcedureManager class needs
> > > > MasterServices(not
> > > > > > > > >>>> MasterRpcServices), which means it must be used in the
> > same
> > > > > > process
> > > > > > > > >> with
> > > > > > > > >>>> HMaster.
> > > > > > > > >>>>
> > > > > > > > >>>> And I'm OK with purging this feature, especially if
> there
> > is
> > > > no
> > > > > > > > >> developer
> > > > > > > > >>>> who wants to maintain it.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>> Is the sole reason to purge this feature is no developer
> > > > > > volunteering
> > > > > > > > to
> > > > > > > > >>> maintain this?
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >> I'd suggest this feature does not belong in core at all.
> > > > > > > > >>
> > > > > > > > >> And some of us have spent time reviewing the feature in
> the
> > > past
> > > > > and
> > > > > > > > back
> > > > > > > > >> then determined it needed lots of finishing work, test,
> and
> > > > polish
> > > > > > --
> > > > > > > > none
> > > > > > > > >> of which it seems to have gotten since commit.
> > > > > > > > >>
> > > > > > > > >> You are using it Mallik? Tell us more please. How is it
> > > working
> > > > > for
> > > > > > > you?
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > > It has been a decent v1 release for us. As pointed out by
> > > others,
> > > > > > > feature
> > > > > > > > > should be polished and matured without debating if backup
> > > should
> > > > be
> > > > > > > part
> > > > > > > > of
> > > > > > > > > core or not. Some of notable points are
> > > > > > > > >
> > > > > > > > >   - Limitations like serial backup set runs per deployment
> > > > bothered
> > > > > > us.
> > > > > > > > >   - Minor logging enhancements for debugging, some NPE's,
> etc
> > > > > > > > >   - Bandwidth limits, incremental backup did not work
> > outright.
> > > > > > > > >
> > > > > > > > > Since it is only a couple of months, we might not have
> seen a
> > > lot
> > > > > of
> > > > > > > > corner
> > > > > > > > > cases.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >> S
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>>
> > > > > > > > >>>> For me, I suspect that the backup feature could be done
> > more
> > > > > > > > separately
> > > > > > > > >>>> with the main cluster. We could use replication to
> backup
> > > the
> > > > > > WALs,
> > > > > > > > and
> > > > > > > > >>> use
> > > > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The
> > > feature
> > > > > > could
> > > > > > > be
> > > > > > > > >>> done
> > > > > > > > >>>> as a separated project.
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks.
> > > > > > > > >>>>
> > > > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > > > > >>>>
> > > > > > > > >>>>> It strikes me as work that has been abandoned with no
> > > > > supporting
> > > > > > > > >>>> developer.
> > > > > > > > >>>>> It has had no improvement and few commits other than
> > > > adjustment
> > > > > > > > >>> because a
> > > > > > > > >>>>> backing dependency has changed since original
> > contribution.
> > > > It
> > > > > > has
> > > > > > > > >> not
> > > > > > > > >>>> been
> > > > > > > > >>>>> included in a release so has no users as yet. Does
> anyone
> > > use
> > > > > it
> > > > > > or
> > > > > > > > >>> want
> > > > > > > > >>>>> it?
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> We have back ported this feature to 2.1.x and has been
> > using
> > > > for
> > > > > > last
> > > > > > > > few
> > > > > > > > >>> months in few of our deployments.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>> If not, I suggest we remove it.  I could file an issue
> for
> > > it
> > > > to
> > > > > > be
> > > > > > > > >>>
> > > > > > > > >>>> added to hbase-operator-tools for some gallant dev to
> pick
> > > up
> > > > if
> > > > > > > they
> > > > > > > > >>>>> wanted to use this backup work? (I could help w/ the
> > > > > migration).
> > > > > > > > >>>>>
> > > > > > > > >>>>> What do others think?
> > > > > > > > >>>>>
> > > > > > > > >>>>> S
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

["mallik.v.arjun@gmail.com" <ma...@gmail.com>]

On Tue, Dec 1, 2020 at 11:26 PM Stack <st...@duboce.net> wrote:

> On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
> mallik.v.arjun@gmail.com> wrote:
>
> > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org> wrote:
> >
> > > One reason for moving it out of core is that at some point we will have
> > > hbase 3 releases. Right now hbase 3 targeted things are in the master
> > > branch. If the feature is not sufficiently ready to be in a release
> then
> > > when the time for HBase 3 releases comes we'll have to do work to pull
> it
> > > out of the release branch. If the feature needs to stay in the core
> repo
> > > that will necessitate that hbase 3 have a branch distinct from the
> master
> > > branch (which may or may not happen anyways). At that point we risk
> > having
> > > to do the work to remove the feature from a release branch again come
> > hbase
> > > 4.
> > >
> > >
> > > I think a lot of this is moot if you'd like to start providing patches
> > for
> > > the things you've needed to use it. If there are gaps that you think
> can
> > > trip folks up maybe we could label it "experimental" to provide better
> > > context for others.
> > >
> >
> > I will start putting effort in maintaining this feature.
> >
> >
> FYI, seems like backup has a bunch of flakey tests. Might be worth looking
> at.
>

Any reference I can get. They seem fine when I run tests.


> Thanks,
> S
>
>
>
>
> >
> > >
> > > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > > mallik.v.arjun@gmail.com> wrote:
> > >
> > > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > > mallik.v.arjun@gmail.com> wrote:
> > > > >
> > > > > > Inline
> > > > > >
> > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > > andrew.purtell@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We are allowed to debate if this should be in the tree or not.
> > > Given
> > > > > the
> > > > > > > lack of interest, as evidenced by incomplete state, lack of
> > > release,
> > > > > and
> > > > > > > lack of contribution, it is more than fair to discuss removal.
> > > > > > >
> > > > > > > Here is my take: First of all, it is not released. There is no
> > > > implied
> > > > > > > roadmap or support. Second, there do not seem to be any active
> > > > > > maintainers
> > > > > > > or volunteers as such. Third, unless someone shows up with more
> > > > patches
> > > > > > for
> > > > > > > it there will be no polish or maturing, there can be no
> > > expectations
> > > > of
> > > > > > > further improvement.
> > > > > > >
> > > > > > > That said, this is open source. New code contribution will
> change
> > > the
> > > > > > > facts as they stand.
> > > > > >
> > > > > >
> > > > > > If there is an agreement upon the direction I am willing to
> provide
> > > > > patches
> > > > > > to take this feature forward.
> > > > > >
> > > > > >
> > > > >
> > > > > What about the changes you made to make it work?
> > > >
> > > >
> > > > > I'd be interested in patches that take this feature out of core.
> How
> > > hard
> > > > > do you think that would be? See the items Duo list above for a
> start
> > on
> > > > > obstacles to tackle?
> > > > >
> > > >
> > > > I am yet to be very familiar with the code to comment on the effort
> > > > required to take it out of the core. Backport and bug fixes required
> > very
> > > > shallow understanding of the code and only some parts of it.
> > > >
> > > > I am not very clear on why some strong opinions on moving backup out
> of
> > > the
> > > > core? Is there something defined on what should be operator tools?
> > > >
> > > > Quoting what Duo Zhang said
> > > >
> > > > You can see the code under the org.apache.hadoop.hbase.backup.master
> > > > > package, we need to set up log cleaner at master side, and also,
> > > > > the LogRollMasterProcedureManager class needs MasterServices(not
> > > > > MasterRpcServices), which means it must be used in the same process
> > > with
> > > > > HMaster.
> > > >
> > > > .
> > > >
> > > >
> > > > >
> > > > > Thanks,
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net>
> > wrote:
> > > > > > > >
> > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > > >>>
> > > > > > > >>> Inline.
> > > > > > > >>>
> > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > > palomino219@gmail.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>>
> > > > > > > >>>> I'm afraid it is not easy to be moved to
> > hbase-operator-tools.
> > > > > > > >>>>
> > > > > > > >>>> You can see the code under the
> > > > > org.apache.hadoop.hbase.backup.master
> > > > > > > >>>> package, we need to set up log cleaner at master side, and
> > > also,
> > > > > > > >>>> the LogRollMasterProcedureManager class needs
> > > MasterServices(not
> > > > > > > >>>> MasterRpcServices), which means it must be used in the
> same
> > > > > process
> > > > > > > >> with
> > > > > > > >>>> HMaster.
> > > > > > > >>>>
> > > > > > > >>>> And I'm OK with purging this feature, especially if there
> is
> > > no
> > > > > > > >> developer
> > > > > > > >>>> who wants to maintain it.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>> Is the sole reason to purge this feature is no developer
> > > > > volunteering
> > > > > > > to
> > > > > > > >>> maintain this?
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >> I'd suggest this feature does not belong in core at all.
> > > > > > > >>
> > > > > > > >> And some of us have spent time reviewing the feature in the
> > past
> > > > and
> > > > > > > back
> > > > > > > >> then determined it needed lots of finishing work, test, and
> > > polish
> > > > > --
> > > > > > > none
> > > > > > > >> of which it seems to have gotten since commit.
> > > > > > > >>
> > > > > > > >> You are using it Mallik? Tell us more please. How is it
> > working
> > > > for
> > > > > > you?
> > > > > > > >>
> > > > > > > >>
> > > > > > > > It has been a decent v1 release for us. As pointed out by
> > others,
> > > > > > feature
> > > > > > > > should be polished and matured without debating if backup
> > should
> > > be
> > > > > > part
> > > > > > > of
> > > > > > > > core or not. Some of notable points are
> > > > > > > >
> > > > > > > >   - Limitations like serial backup set runs per deployment
> > > bothered
> > > > > us.
> > > > > > > >   - Minor logging enhancements for debugging, some NPE's, etc
> > > > > > > >   - Bandwidth limits, incremental backup did not work
> outright.
> > > > > > > >
> > > > > > > > Since it is only a couple of months, we might not have seen a
> > lot
> > > > of
> > > > > > > corner
> > > > > > > > cases.
> > > > > > > >
> > > > > > > >
> > > > > > > >> S
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>>
> > > > > > > >>>> For me, I suspect that the backup feature could be done
> more
> > > > > > > separately
> > > > > > > >>>> with the main cluster. We could use replication to backup
> > the
> > > > > WALs,
> > > > > > > and
> > > > > > > >>> use
> > > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The
> > feature
> > > > > could
> > > > > > be
> > > > > > > >>> done
> > > > > > > >>>> as a separated project.
> > > > > > > >>>>
> > > > > > > >>>> Thanks.
> > > > > > > >>>>
> > > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > > > >>>>
> > > > > > > >>>>> It strikes me as work that has been abandoned with no
> > > > supporting
> > > > > > > >>>> developer.
> > > > > > > >>>>> It has had no improvement and few commits other than
> > > adjustment
> > > > > > > >>> because a
> > > > > > > >>>>> backing dependency has changed since original
> contribution.
> > > It
> > > > > has
> > > > > > > >> not
> > > > > > > >>>> been
> > > > > > > >>>>> included in a release so has no users as yet. Does anyone
> > use
> > > > it
> > > > > or
> > > > > > > >>> want
> > > > > > > >>>>> it?
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> We have back ported this feature to 2.1.x and has been
> using
> > > for
> > > > > last
> > > > > > > few
> > > > > > > >>> months in few of our deployments.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>> If not, I suggest we remove it.  I could file an issue for
> > it
> > > to
> > > > > be
> > > > > > > >>>
> > > > > > > >>>> added to hbase-operator-tools for some gallant dev to pick
> > up
> > > if
> > > > > > they
> > > > > > > >>>>> wanted to use this backup work? (I could help w/ the
> > > > migration).
> > > > > > > >>>>>
> > > > > > > >>>>> What do others think?
> > > > > > > >>>>>
> > > > > > > >>>>> S
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Stack <st...@duboce.net>]

On Tue, Dec 1, 2020 at 6:38 AM mallik.v.arjun@gmail.com <
mallik.v.arjun@gmail.com> wrote:

> On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org> wrote:
>
> > One reason for moving it out of core is that at some point we will have
> > hbase 3 releases. Right now hbase 3 targeted things are in the master
> > branch. If the feature is not sufficiently ready to be in a release then
> > when the time for HBase 3 releases comes we'll have to do work to pull it
> > out of the release branch. If the feature needs to stay in the core repo
> > that will necessitate that hbase 3 have a branch distinct from the master
> > branch (which may or may not happen anyways). At that point we risk
> having
> > to do the work to remove the feature from a release branch again come
> hbase
> > 4.
> >
> >
> > I think a lot of this is moot if you'd like to start providing patches
> for
> > the things you've needed to use it. If there are gaps that you think can
> > trip folks up maybe we could label it "experimental" to provide better
> > context for others.
> >
>
> I will start putting effort in maintaining this feature.
>
>
FYI, seems like backup has a bunch of flakey tests. Might be worth looking
at.
Thanks,
S




>
> >
> > On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> > mallik.v.arjun@gmail.com> wrote:
> >
> > > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
> > >
> > > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > > mallik.v.arjun@gmail.com> wrote:
> > > >
> > > > > Inline
> > > > >
> > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> > andrew.purtell@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > We are allowed to debate if this should be in the tree or not.
> > Given
> > > > the
> > > > > > lack of interest, as evidenced by incomplete state, lack of
> > release,
> > > > and
> > > > > > lack of contribution, it is more than fair to discuss removal.
> > > > > >
> > > > > > Here is my take: First of all, it is not released. There is no
> > > implied
> > > > > > roadmap or support. Second, there do not seem to be any active
> > > > > maintainers
> > > > > > or volunteers as such. Third, unless someone shows up with more
> > > patches
> > > > > for
> > > > > > it there will be no polish or maturing, there can be no
> > expectations
> > > of
> > > > > > further improvement.
> > > > > >
> > > > > > That said, this is open source. New code contribution will change
> > the
> > > > > > facts as they stand.
> > > > >
> > > > >
> > > > > If there is an agreement upon the direction I am willing to provide
> > > > patches
> > > > > to take this feature forward.
> > > > >
> > > > >
> > > >
> > > > What about the changes you made to make it work?
> > >
> > >
> > > > I'd be interested in patches that take this feature out of core. How
> > hard
> > > > do you think that would be? See the items Duo list above for a start
> on
> > > > obstacles to tackle?
> > > >
> > >
> > > I am yet to be very familiar with the code to comment on the effort
> > > required to take it out of the core. Backport and bug fixes required
> very
> > > shallow understanding of the code and only some parts of it.
> > >
> > > I am not very clear on why some strong opinions on moving backup out of
> > the
> > > core? Is there something defined on what should be operator tools?
> > >
> > > Quoting what Duo Zhang said
> > >
> > > You can see the code under the org.apache.hadoop.hbase.backup.master
> > > > package, we need to set up log cleaner at master side, and also,
> > > > the LogRollMasterProcedureManager class needs MasterServices(not
> > > > MasterRpcServices), which means it must be used in the same process
> > with
> > > > HMaster.
> > >
> > > .
> > >
> > >
> > > >
> > > > Thanks,
> > > > S
> > > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net>
> wrote:
> > > > > > >
> > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > > >>>
> > > > > > >>> Inline.
> > > > > > >>>
> > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > > palomino219@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>>> I'm afraid it is not easy to be moved to
> hbase-operator-tools.
> > > > > > >>>>
> > > > > > >>>> You can see the code under the
> > > > org.apache.hadoop.hbase.backup.master
> > > > > > >>>> package, we need to set up log cleaner at master side, and
> > also,
> > > > > > >>>> the LogRollMasterProcedureManager class needs
> > MasterServices(not
> > > > > > >>>> MasterRpcServices), which means it must be used in the same
> > > > process
> > > > > > >> with
> > > > > > >>>> HMaster.
> > > > > > >>>>
> > > > > > >>>> And I'm OK with purging this feature, especially if there is
> > no
> > > > > > >> developer
> > > > > > >>>> who wants to maintain it.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>> Is the sole reason to purge this feature is no developer
> > > > volunteering
> > > > > > to
> > > > > > >>> maintain this?
> > > > > > >>>
> > > > > > >>>
> > > > > > >> I'd suggest this feature does not belong in core at all.
> > > > > > >>
> > > > > > >> And some of us have spent time reviewing the feature in the
> past
> > > and
> > > > > > back
> > > > > > >> then determined it needed lots of finishing work, test, and
> > polish
> > > > --
> > > > > > none
> > > > > > >> of which it seems to have gotten since commit.
> > > > > > >>
> > > > > > >> You are using it Mallik? Tell us more please. How is it
> working
> > > for
> > > > > you?
> > > > > > >>
> > > > > > >>
> > > > > > > It has been a decent v1 release for us. As pointed out by
> others,
> > > > > feature
> > > > > > > should be polished and matured without debating if backup
> should
> > be
> > > > > part
> > > > > > of
> > > > > > > core or not. Some of notable points are
> > > > > > >
> > > > > > >   - Limitations like serial backup set runs per deployment
> > bothered
> > > > us.
> > > > > > >   - Minor logging enhancements for debugging, some NPE's, etc
> > > > > > >   - Bandwidth limits, incremental backup did not work outright.
> > > > > > >
> > > > > > > Since it is only a couple of months, we might not have seen a
> lot
> > > of
> > > > > > corner
> > > > > > > cases.
> > > > > > >
> > > > > > >
> > > > > > >> S
> > > > > > >>
> > > > > > >>
> > > > > > >>>
> > > > > > >>>> For me, I suspect that the backup feature could be done more
> > > > > > separately
> > > > > > >>>> with the main cluster. We could use replication to backup
> the
> > > > WALs,
> > > > > > and
> > > > > > >>> use
> > > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The
> feature
> > > > could
> > > > > be
> > > > > > >>> done
> > > > > > >>>> as a separated project.
> > > > > > >>>>
> > > > > > >>>> Thanks.
> > > > > > >>>>
> > > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > > >>>>
> > > > > > >>>>> It strikes me as work that has been abandoned with no
> > > supporting
> > > > > > >>>> developer.
> > > > > > >>>>> It has had no improvement and few commits other than
> > adjustment
> > > > > > >>> because a
> > > > > > >>>>> backing dependency has changed since original contribution.
> > It
> > > > has
> > > > > > >> not
> > > > > > >>>> been
> > > > > > >>>>> included in a release so has no users as yet. Does anyone
> use
> > > it
> > > > or
> > > > > > >>> want
> > > > > > >>>>> it?
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> We have back ported this feature to 2.1.x and has been using
> > for
> > > > last
> > > > > > few
> > > > > > >>> months in few of our deployments.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>> If not, I suggest we remove it.  I could file an issue for
> it
> > to
> > > > be
> > > > > > >>>
> > > > > > >>>> added to hbase-operator-tools for some gallant dev to pick
> up
> > if
> > > > > they
> > > > > > >>>>> wanted to use this backup work? (I could help w/ the
> > > migration).
> > > > > > >>>>>
> > > > > > >>>>> What do others think?
> > > > > > >>>>>
> > > > > > >>>>> S
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

["mallik.v.arjun@gmail.com" <ma...@gmail.com>]

On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey <bu...@apache.org> wrote:

> One reason for moving it out of core is that at some point we will have
> hbase 3 releases. Right now hbase 3 targeted things are in the master
> branch. If the feature is not sufficiently ready to be in a release then
> when the time for HBase 3 releases comes we'll have to do work to pull it
> out of the release branch. If the feature needs to stay in the core repo
> that will necessitate that hbase 3 have a branch distinct from the master
> branch (which may or may not happen anyways). At that point we risk having
> to do the work to remove the feature from a release branch again come hbase
> 4.
>
>
> I think a lot of this is moot if you'd like to start providing patches for
> the things you've needed to use it. If there are gaps that you think can
> trip folks up maybe we could label it "experimental" to provide better
> context for others.
>

I will start putting effort in maintaining this feature.


>
> On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
> mallik.v.arjun@gmail.com> wrote:
>
> > On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
> >
> > > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > > mallik.v.arjun@gmail.com> wrote:
> > >
> > > > Inline
> > > >
> > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <
> andrew.purtell@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > We are allowed to debate if this should be in the tree or not.
> Given
> > > the
> > > > > lack of interest, as evidenced by incomplete state, lack of
> release,
> > > and
> > > > > lack of contribution, it is more than fair to discuss removal.
> > > > >
> > > > > Here is my take: First of all, it is not released. There is no
> > implied
> > > > > roadmap or support. Second, there do not seem to be any active
> > > > maintainers
> > > > > or volunteers as such. Third, unless someone shows up with more
> > patches
> > > > for
> > > > > it there will be no polish or maturing, there can be no
> expectations
> > of
> > > > > further improvement.
> > > > >
> > > > > That said, this is open source. New code contribution will change
> the
> > > > > facts as they stand.
> > > >
> > > >
> > > > If there is an agreement upon the direction I am willing to provide
> > > patches
> > > > to take this feature forward.
> > > >
> > > >
> > >
> > > What about the changes you made to make it work?
> >
> >
> > > I'd be interested in patches that take this feature out of core. How
> hard
> > > do you think that would be? See the items Duo list above for a start on
> > > obstacles to tackle?
> > >
> >
> > I am yet to be very familiar with the code to comment on the effort
> > required to take it out of the core. Backport and bug fixes required very
> > shallow understanding of the code and only some parts of it.
> >
> > I am not very clear on why some strong opinions on moving backup out of
> the
> > core? Is there something defined on what should be operator tools?
> >
> > Quoting what Duo Zhang said
> >
> > You can see the code under the org.apache.hadoop.hbase.backup.master
> > > package, we need to set up log cleaner at master side, and also,
> > > the LogRollMasterProcedureManager class needs MasterServices(not
> > > MasterRpcServices), which means it must be used in the same process
> with
> > > HMaster.
> >
> > .
> >
> >
> > >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > > >
> > > > >
> > > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > > > > >
> > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> > > > > >
> > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > > >>>
> > > > > >>> Inline.
> > > > > >>>
> > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> > palomino219@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
> > > > > >>>>
> > > > > >>>> You can see the code under the
> > > org.apache.hadoop.hbase.backup.master
> > > > > >>>> package, we need to set up log cleaner at master side, and
> also,
> > > > > >>>> the LogRollMasterProcedureManager class needs
> MasterServices(not
> > > > > >>>> MasterRpcServices), which means it must be used in the same
> > > process
> > > > > >> with
> > > > > >>>> HMaster.
> > > > > >>>>
> > > > > >>>> And I'm OK with purging this feature, especially if there is
> no
> > > > > >> developer
> > > > > >>>> who wants to maintain it.
> > > > > >>>>
> > > > > >>>>
> > > > > >>> Is the sole reason to purge this feature is no developer
> > > volunteering
> > > > > to
> > > > > >>> maintain this?
> > > > > >>>
> > > > > >>>
> > > > > >> I'd suggest this feature does not belong in core at all.
> > > > > >>
> > > > > >> And some of us have spent time reviewing the feature in the past
> > and
> > > > > back
> > > > > >> then determined it needed lots of finishing work, test, and
> polish
> > > --
> > > > > none
> > > > > >> of which it seems to have gotten since commit.
> > > > > >>
> > > > > >> You are using it Mallik? Tell us more please. How is it working
> > for
> > > > you?
> > > > > >>
> > > > > >>
> > > > > > It has been a decent v1 release for us. As pointed out by others,
> > > > feature
> > > > > > should be polished and matured without debating if backup should
> be
> > > > part
> > > > > of
> > > > > > core or not. Some of notable points are
> > > > > >
> > > > > >   - Limitations like serial backup set runs per deployment
> bothered
> > > us.
> > > > > >   - Minor logging enhancements for debugging, some NPE's, etc
> > > > > >   - Bandwidth limits, incremental backup did not work outright.
> > > > > >
> > > > > > Since it is only a couple of months, we might not have seen a lot
> > of
> > > > > corner
> > > > > > cases.
> > > > > >
> > > > > >
> > > > > >> S
> > > > > >>
> > > > > >>
> > > > > >>>
> > > > > >>>> For me, I suspect that the backup feature could be done more
> > > > > separately
> > > > > >>>> with the main cluster. We could use replication to backup the
> > > WALs,
> > > > > and
> > > > > >>> use
> > > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The feature
> > > could
> > > > be
> > > > > >>> done
> > > > > >>>> as a separated project.
> > > > > >>>>
> > > > > >>>> Thanks.
> > > > > >>>>
> > > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > > >>>>
> > > > > >>>>> It strikes me as work that has been abandoned with no
> > supporting
> > > > > >>>> developer.
> > > > > >>>>> It has had no improvement and few commits other than
> adjustment
> > > > > >>> because a
> > > > > >>>>> backing dependency has changed since original contribution.
> It
> > > has
> > > > > >> not
> > > > > >>>> been
> > > > > >>>>> included in a release so has no users as yet. Does anyone use
> > it
> > > or
> > > > > >>> want
> > > > > >>>>> it?
> > > > > >>>
> > > > > >>>
> > > > > >>> We have back ported this feature to 2.1.x and has been using
> for
> > > last
> > > > > few
> > > > > >>> months in few of our deployments.
> > > > > >>>
> > > > > >>>
> > > > > >>>> If not, I suggest we remove it.  I could file an issue for it
> to
> > > be
> > > > > >>>
> > > > > >>>> added to hbase-operator-tools for some gallant dev to pick up
> if
> > > > they
> > > > > >>>>> wanted to use this backup work? (I could help w/ the
> > migration).
> > > > > >>>>>
> > > > > >>>>> What do others think?
> > > > > >>>>>
> > > > > >>>>> S
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Sean Busbey <bu...@apache.org>]

One reason for moving it out of core is that at some point we will have
hbase 3 releases. Right now hbase 3 targeted things are in the master
branch. If the feature is not sufficiently ready to be in a release then
when the time for HBase 3 releases comes we'll have to do work to pull it
out of the release branch. If the feature needs to stay in the core repo
that will necessitate that hbase 3 have a branch distinct from the master
branch (which may or may not happen anyways). At that point we risk having
to do the work to remove the feature from a release branch again come hbase
4.


I think a lot of this is moot if you'd like to start providing patches for
the things you've needed to use it. If there are gaps that you think can
trip folks up maybe we could label it "experimental" to provide better
context for others.

On Tue, Dec 1, 2020, 07:48 mallik.v.arjun@gmail.com <
mallik.v.arjun@gmail.com> wrote:

> On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:
>
> > On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> > mallik.v.arjun@gmail.com> wrote:
> >
> > > Inline
> > >
> > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <andrew.purtell@gmail.com
> >
> > > wrote:
> > >
> > > > We are allowed to debate if this should be in the tree or not. Given
> > the
> > > > lack of interest, as evidenced by incomplete state, lack of release,
> > and
> > > > lack of contribution, it is more than fair to discuss removal.
> > > >
> > > > Here is my take: First of all, it is not released. There is no
> implied
> > > > roadmap or support. Second, there do not seem to be any active
> > > maintainers
> > > > or volunteers as such. Third, unless someone shows up with more
> patches
> > > for
> > > > it there will be no polish or maturing, there can be no expectations
> of
> > > > further improvement.
> > > >
> > > > That said, this is open source. New code contribution will change the
> > > > facts as they stand.
> > >
> > >
> > > If there is an agreement upon the direction I am willing to provide
> > patches
> > > to take this feature forward.
> > >
> > >
> >
> > What about the changes you made to make it work?
>
>
> > I'd be interested in patches that take this feature out of core. How hard
> > do you think that would be? See the items Duo list above for a start on
> > obstacles to tackle?
> >
>
> I am yet to be very familiar with the code to comment on the effort
> required to take it out of the core. Backport and bug fixes required very
> shallow understanding of the code and only some parts of it.
>
> I am not very clear on why some strong opinions on moving backup out of the
> core? Is there something defined on what should be operator tools?
>
> Quoting what Duo Zhang said
>
> You can see the code under the org.apache.hadoop.hbase.backup.master
> > package, we need to set up log cleaner at master side, and also,
> > the LogRollMasterProcedureManager class needs MasterServices(not
> > MasterRpcServices), which means it must be used in the same process with
> > HMaster.
>
> .
>
>
> >
> > Thanks,
> > S
> >
> >
> >
> > >
> > > >
> > > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > > > >
> > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> > > > >
> > > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > > > >>> mallik.v.arjun@gmail.com> wrote:
> > > > >>>
> > > > >>> Inline.
> > > > >>>
> > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <
> palomino219@gmail.com
> > >
> > > > >> wrote:
> > > > >>>
> > > > >>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
> > > > >>>>
> > > > >>>> You can see the code under the
> > org.apache.hadoop.hbase.backup.master
> > > > >>>> package, we need to set up log cleaner at master side, and also,
> > > > >>>> the LogRollMasterProcedureManager class needs MasterServices(not
> > > > >>>> MasterRpcServices), which means it must be used in the same
> > process
> > > > >> with
> > > > >>>> HMaster.
> > > > >>>>
> > > > >>>> And I'm OK with purging this feature, especially if there is no
> > > > >> developer
> > > > >>>> who wants to maintain it.
> > > > >>>>
> > > > >>>>
> > > > >>> Is the sole reason to purge this feature is no developer
> > volunteering
> > > > to
> > > > >>> maintain this?
> > > > >>>
> > > > >>>
> > > > >> I'd suggest this feature does not belong in core at all.
> > > > >>
> > > > >> And some of us have spent time reviewing the feature in the past
> and
> > > > back
> > > > >> then determined it needed lots of finishing work, test, and polish
> > --
> > > > none
> > > > >> of which it seems to have gotten since commit.
> > > > >>
> > > > >> You are using it Mallik? Tell us more please. How is it working
> for
> > > you?
> > > > >>
> > > > >>
> > > > > It has been a decent v1 release for us. As pointed out by others,
> > > feature
> > > > > should be polished and matured without debating if backup should be
> > > part
> > > > of
> > > > > core or not. Some of notable points are
> > > > >
> > > > >   - Limitations like serial backup set runs per deployment bothered
> > us.
> > > > >   - Minor logging enhancements for debugging, some NPE's, etc
> > > > >   - Bandwidth limits, incremental backup did not work outright.
> > > > >
> > > > > Since it is only a couple of months, we might not have seen a lot
> of
> > > > corner
> > > > > cases.
> > > > >
> > > > >
> > > > >> S
> > > > >>
> > > > >>
> > > > >>>
> > > > >>>> For me, I suspect that the backup feature could be done more
> > > > separately
> > > > >>>> with the main cluster. We could use replication to backup the
> > WALs,
> > > > and
> > > > >>> use
> > > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The feature
> > could
> > > be
> > > > >>> done
> > > > >>>> as a separated project.
> > > > >>>>
> > > > >>>> Thanks.
> > > > >>>>
> > > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > > >>>>
> > > > >>>>> It strikes me as work that has been abandoned with no
> supporting
> > > > >>>> developer.
> > > > >>>>> It has had no improvement and few commits other than adjustment
> > > > >>> because a
> > > > >>>>> backing dependency has changed since original contribution. It
> > has
> > > > >> not
> > > > >>>> been
> > > > >>>>> included in a release so has no users as yet. Does anyone use
> it
> > or
> > > > >>> want
> > > > >>>>> it?
> > > > >>>
> > > > >>>
> > > > >>> We have back ported this feature to 2.1.x and has been using for
> > last
> > > > few
> > > > >>> months in few of our deployments.
> > > > >>>
> > > > >>>
> > > > >>>> If not, I suggest we remove it.  I could file an issue for it to
> > be
> > > > >>>
> > > > >>>> added to hbase-operator-tools for some gallant dev to pick up if
> > > they
> > > > >>>>> wanted to use this backup work? (I could help w/ the
> migration).
> > > > >>>>>
> > > > >>>>> What do others think?
> > > > >>>>>
> > > > >>>>> S
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

["mallik.v.arjun@gmail.com" <ma...@gmail.com>]

On Tue, Dec 1, 2020 at 12:14 PM Stack <st...@duboce.net> wrote:

> On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
> mallik.v.arjun@gmail.com> wrote:
>
> > Inline
> >
> > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <an...@gmail.com>
> > wrote:
> >
> > > We are allowed to debate if this should be in the tree or not. Given
> the
> > > lack of interest, as evidenced by incomplete state, lack of release,
> and
> > > lack of contribution, it is more than fair to discuss removal.
> > >
> > > Here is my take: First of all, it is not released. There is no implied
> > > roadmap or support. Second, there do not seem to be any active
> > maintainers
> > > or volunteers as such. Third, unless someone shows up with more patches
> > for
> > > it there will be no polish or maturing, there can be no expectations of
> > > further improvement.
> > >
> > > That said, this is open source. New code contribution will change the
> > > facts as they stand.
> >
> >
> > If there is an agreement upon the direction I am willing to provide
> patches
> > to take this feature forward.
> >
> >
>
> What about the changes you made to make it work?


> I'd be interested in patches that take this feature out of core. How hard
> do you think that would be? See the items Duo list above for a start on
> obstacles to tackle?
>

I am yet to be very familiar with the code to comment on the effort
required to take it out of the core. Backport and bug fixes required very
shallow understanding of the code and only some parts of it.

I am not very clear on why some strong opinions on moving backup out of the
core? Is there something defined on what should be operator tools?

Quoting what Duo Zhang said

You can see the code under the org.apache.hadoop.hbase.backup.master
> package, we need to set up log cleaner at master side, and also,
> the LogRollMasterProcedureManager class needs MasterServices(not
> MasterRpcServices), which means it must be used in the same process with
> HMaster.

.


>
> Thanks,
> S
>
>
>
> >
> > >
> > > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > > >
> > > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> > > >
> > > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > > >>> mallik.v.arjun@gmail.com> wrote:
> > > >>>
> > > >>> Inline.
> > > >>>
> > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <palomino219@gmail.com
> >
> > > >> wrote:
> > > >>>
> > > >>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
> > > >>>>
> > > >>>> You can see the code under the
> org.apache.hadoop.hbase.backup.master
> > > >>>> package, we need to set up log cleaner at master side, and also,
> > > >>>> the LogRollMasterProcedureManager class needs MasterServices(not
> > > >>>> MasterRpcServices), which means it must be used in the same
> process
> > > >> with
> > > >>>> HMaster.
> > > >>>>
> > > >>>> And I'm OK with purging this feature, especially if there is no
> > > >> developer
> > > >>>> who wants to maintain it.
> > > >>>>
> > > >>>>
> > > >>> Is the sole reason to purge this feature is no developer
> volunteering
> > > to
> > > >>> maintain this?
> > > >>>
> > > >>>
> > > >> I'd suggest this feature does not belong in core at all.
> > > >>
> > > >> And some of us have spent time reviewing the feature in the past and
> > > back
> > > >> then determined it needed lots of finishing work, test, and polish
> --
> > > none
> > > >> of which it seems to have gotten since commit.
> > > >>
> > > >> You are using it Mallik? Tell us more please. How is it working for
> > you?
> > > >>
> > > >>
> > > > It has been a decent v1 release for us. As pointed out by others,
> > feature
> > > > should be polished and matured without debating if backup should be
> > part
> > > of
> > > > core or not. Some of notable points are
> > > >
> > > >   - Limitations like serial backup set runs per deployment bothered
> us.
> > > >   - Minor logging enhancements for debugging, some NPE's, etc
> > > >   - Bandwidth limits, incremental backup did not work outright.
> > > >
> > > > Since it is only a couple of months, we might not have seen a lot of
> > > corner
> > > > cases.
> > > >
> > > >
> > > >> S
> > > >>
> > > >>
> > > >>>
> > > >>>> For me, I suspect that the backup feature could be done more
> > > separately
> > > >>>> with the main cluster. We could use replication to backup the
> WALs,
> > > and
> > > >>> use
> > > >>>> Snapshot and ExportSnapshot to backup the HFiles. The feature
> could
> > be
> > > >>> done
> > > >>>> as a separated project.
> > > >>>>
> > > >>>> Thanks.
> > > >>>>
> > > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > > >>>>
> > > >>>>> It strikes me as work that has been abandoned with no supporting
> > > >>>> developer.
> > > >>>>> It has had no improvement and few commits other than adjustment
> > > >>> because a
> > > >>>>> backing dependency has changed since original contribution. It
> has
> > > >> not
> > > >>>> been
> > > >>>>> included in a release so has no users as yet. Does anyone use it
> or
> > > >>> want
> > > >>>>> it?
> > > >>>
> > > >>>
> > > >>> We have back ported this feature to 2.1.x and has been using for
> last
> > > few
> > > >>> months in few of our deployments.
> > > >>>
> > > >>>
> > > >>>> If not, I suggest we remove it.  I could file an issue for it to
> be
> > > >>>
> > > >>>> added to hbase-operator-tools for some gallant dev to pick up if
> > they
> > > >>>>> wanted to use this backup work? (I could help w/ the migration).
> > > >>>>>
> > > >>>>> What do others think?
> > > >>>>>
> > > >>>>> S
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>

Re: DISCUSS: Remove hbase-backup from master?

[Stack <st...@duboce.net>]

On Mon, Nov 30, 2020 at 9:30 PM mallik.v.arjun@gmail.com <
mallik.v.arjun@gmail.com> wrote:

> Inline
>
> On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <an...@gmail.com>
> wrote:
>
> > We are allowed to debate if this should be in the tree or not. Given the
> > lack of interest, as evidenced by incomplete state, lack of release, and
> > lack of contribution, it is more than fair to discuss removal.
> >
> > Here is my take: First of all, it is not released. There is no implied
> > roadmap or support. Second, there do not seem to be any active
> maintainers
> > or volunteers as such. Third, unless someone shows up with more patches
> for
> > it there will be no polish or maturing, there can be no expectations of
> > further improvement.
> >
> > That said, this is open source. New code contribution will change the
> > facts as they stand.
>
>
> If there is an agreement upon the direction I am willing to provide patches
> to take this feature forward.
>
>

What about the changes you made to make it work?

I'd be interested in patches that take this feature out of core. How hard
do you think that would be? See the items Duo list above for a start on
obstacles to tackle?

Thanks,
S



>
> >
> > > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> > >
> > > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> > >
> > >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> > >>> mallik.v.arjun@gmail.com> wrote:
> > >>>
> > >>> Inline.
> > >>>
> > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > >> wrote:
> > >>>
> > >>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
> > >>>>
> > >>>> You can see the code under the org.apache.hadoop.hbase.backup.master
> > >>>> package, we need to set up log cleaner at master side, and also,
> > >>>> the LogRollMasterProcedureManager class needs MasterServices(not
> > >>>> MasterRpcServices), which means it must be used in the same process
> > >> with
> > >>>> HMaster.
> > >>>>
> > >>>> And I'm OK with purging this feature, especially if there is no
> > >> developer
> > >>>> who wants to maintain it.
> > >>>>
> > >>>>
> > >>> Is the sole reason to purge this feature is no developer volunteering
> > to
> > >>> maintain this?
> > >>>
> > >>>
> > >> I'd suggest this feature does not belong in core at all.
> > >>
> > >> And some of us have spent time reviewing the feature in the past and
> > back
> > >> then determined it needed lots of finishing work, test, and polish --
> > none
> > >> of which it seems to have gotten since commit.
> > >>
> > >> You are using it Mallik? Tell us more please. How is it working for
> you?
> > >>
> > >>
> > > It has been a decent v1 release for us. As pointed out by others,
> feature
> > > should be polished and matured without debating if backup should be
> part
> > of
> > > core or not. Some of notable points are
> > >
> > >   - Limitations like serial backup set runs per deployment bothered us.
> > >   - Minor logging enhancements for debugging, some NPE's, etc
> > >   - Bandwidth limits, incremental backup did not work outright.
> > >
> > > Since it is only a couple of months, we might not have seen a lot of
> > corner
> > > cases.
> > >
> > >
> > >> S
> > >>
> > >>
> > >>>
> > >>>> For me, I suspect that the backup feature could be done more
> > separately
> > >>>> with the main cluster. We could use replication to backup the WALs,
> > and
> > >>> use
> > >>>> Snapshot and ExportSnapshot to backup the HFiles. The feature could
> be
> > >>> done
> > >>>> as a separated project.
> > >>>>
> > >>>> Thanks.
> > >>>>
> > >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> > >>>>
> > >>>>> It strikes me as work that has been abandoned with no supporting
> > >>>> developer.
> > >>>>> It has had no improvement and few commits other than adjustment
> > >>> because a
> > >>>>> backing dependency has changed since original contribution. It has
> > >> not
> > >>>> been
> > >>>>> included in a release so has no users as yet. Does anyone use it or
> > >>> want
> > >>>>> it?
> > >>>
> > >>>
> > >>> We have back ported this feature to 2.1.x and has been using for last
> > few
> > >>> months in few of our deployments.
> > >>>
> > >>>
> > >>>> If not, I suggest we remove it.  I could file an issue for it to be
> > >>>
> > >>>> added to hbase-operator-tools for some gallant dev to pick up if
> they
> > >>>>> wanted to use this backup work? (I could help w/ the migration).
> > >>>>>
> > >>>>> What do others think?
> > >>>>>
> > >>>>> S
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Re: DISCUSS: Remove hbase-backup from master?

["mallik.v.arjun@gmail.com" <ma...@gmail.com>]

Inline

On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell <an...@gmail.com>
wrote:

> We are allowed to debate if this should be in the tree or not. Given the
> lack of interest, as evidenced by incomplete state, lack of release, and
> lack of contribution, it is more than fair to discuss removal.
>
> Here is my take: First of all, it is not released. There is no implied
> roadmap or support. Second, there do not seem to be any active maintainers
> or volunteers as such. Third, unless someone shows up with more patches for
> it there will be no polish or maturing, there can be no expectations of
> further improvement.
>
> That said, this is open source. New code contribution will change the
> facts as they stand.


If there is an agreement upon the direction I am willing to provide patches
to take this feature forward.


>
> > On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> >
> > On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> >
> >>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
> >>> mallik.v.arjun@gmail.com> wrote:
> >>>
> >>> Inline.
> >>>
> >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >>>
> >>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
> >>>>
> >>>> You can see the code under the org.apache.hadoop.hbase.backup.master
> >>>> package, we need to set up log cleaner at master side, and also,
> >>>> the LogRollMasterProcedureManager class needs MasterServices(not
> >>>> MasterRpcServices), which means it must be used in the same process
> >> with
> >>>> HMaster.
> >>>>
> >>>> And I'm OK with purging this feature, especially if there is no
> >> developer
> >>>> who wants to maintain it.
> >>>>
> >>>>
> >>> Is the sole reason to purge this feature is no developer volunteering
> to
> >>> maintain this?
> >>>
> >>>
> >> I'd suggest this feature does not belong in core at all.
> >>
> >> And some of us have spent time reviewing the feature in the past and
> back
> >> then determined it needed lots of finishing work, test, and polish --
> none
> >> of which it seems to have gotten since commit.
> >>
> >> You are using it Mallik? Tell us more please. How is it working for you?
> >>
> >>
> > It has been a decent v1 release for us. As pointed out by others, feature
> > should be polished and matured without debating if backup should be part
> of
> > core or not. Some of notable points are
> >
> >   - Limitations like serial backup set runs per deployment bothered us.
> >   - Minor logging enhancements for debugging, some NPE's, etc
> >   - Bandwidth limits, incremental backup did not work outright.
> >
> > Since it is only a couple of months, we might not have seen a lot of
> corner
> > cases.
> >
> >
> >> S
> >>
> >>
> >>>
> >>>> For me, I suspect that the backup feature could be done more
> separately
> >>>> with the main cluster. We could use replication to backup the WALs,
> and
> >>> use
> >>>> Snapshot and ExportSnapshot to backup the HFiles. The feature could be
> >>> done
> >>>> as a separated project.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
> >>>>
> >>>>> It strikes me as work that has been abandoned with no supporting
> >>>> developer.
> >>>>> It has had no improvement and few commits other than adjustment
> >>> because a
> >>>>> backing dependency has changed since original contribution. It has
> >> not
> >>>> been
> >>>>> included in a release so has no users as yet. Does anyone use it or
> >>> want
> >>>>> it?
> >>>
> >>>
> >>> We have back ported this feature to 2.1.x and has been using for last
> few
> >>> months in few of our deployments.
> >>>
> >>>
> >>>> If not, I suggest we remove it.  I could file an issue for it to be
> >>>
> >>>> added to hbase-operator-tools for some gallant dev to pick up if they
> >>>>> wanted to use this backup work? (I could help w/ the migration).
> >>>>>
> >>>>> What do others think?
> >>>>>
> >>>>> S
> >>>>>
> >>>>
> >>>
> >>
>

Re: DISCUSS: Remove hbase-backup from master?

[Andrew Purtell <an...@gmail.com>]

We are allowed to debate if this should be in the tree or not. Given the lack of interest, as evidenced by incomplete state, lack of release, and lack of contribution, it is more than fair to discuss removal. 

Here is my take: First of all, it is not released. There is no implied roadmap or support. Second, there do not seem to be any active maintainers or volunteers as such. Third, unless someone shows up with more patches for it there will be no polish or maturing, there can be no expectations of further improvement. 

That said, this is open source. New code contribution will change the facts as they stand.  


> On Nov 30, 2020, at 5:52 PM, mallik.v.arjun@gmail.com wrote:
> 
> On Mon, Nov 30, 2020 at 9:59 PM Stack <st...@duboce.net> wrote:
> 
>>> On Mon, Nov 30, 2020 at 5:55 AM mallik.v.arjun@gmail.com <
>>> mallik.v.arjun@gmail.com> wrote:
>>> 
>>> Inline.
>>> 
>>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>> 
>>>> I'm afraid it is not easy to be moved to hbase-operator-tools.
>>>> 
>>>> You can see the code under the org.apache.hadoop.hbase.backup.master
>>>> package, we need to set up log cleaner at master side, and also,
>>>> the LogRollMasterProcedureManager class needs MasterServices(not
>>>> MasterRpcServices), which means it must be used in the same process
>> with
>>>> HMaster.
>>>> 
>>>> And I'm OK with purging this feature, especially if there is no
>> developer
>>>> who wants to maintain it.
>>>> 
>>>> 
>>> Is the sole reason to purge this feature is no developer volunteering to
>>> maintain this?
>>> 
>>> 
>> I'd suggest this feature does not belong in core at all.
>> 
>> And some of us have spent time reviewing the feature in the past and back
>> then determined it needed lots of finishing work, test, and polish -- none
>> of which it seems to have gotten since commit.
>> 
>> You are using it Mallik? Tell us more please. How is it working for you?
>> 
>> 
> It has been a decent v1 release for us. As pointed out by others, feature
> should be polished and matured without debating if backup should be part of
> core or not. Some of notable points are
> 
>   - Limitations like serial backup set runs per deployment bothered us.
>   - Minor logging enhancements for debugging, some NPE's, etc
>   - Bandwidth limits, incremental backup did not work outright.
> 
> Since it is only a couple of months, we might not have seen a lot of corner
> cases.
> 
> 
>> S
>> 
>> 
>>> 
>>>> For me, I suspect that the backup feature could be done more separately
>>>> with the main cluster. We could use replication to backup the WALs, and
>>> use
>>>> Snapshot and ExportSnapshot to backup the HFiles. The feature could be
>>> done
>>>> as a separated project.
>>>> 
>>>> Thanks.
>>>> 
>>>> Stack <st...@duboce.net> 于2020年11月20日周五 上午10:18写道：
>>>> 
>>>>> It strikes me as work that has been abandoned with no supporting
>>>> developer.
>>>>> It has had no improvement and few commits other than adjustment
>>> because a
>>>>> backing dependency has changed since original contribution. It has
>> not
>>>> been
>>>>> included in a release so has no users as yet. Does anyone use it or
>>> want
>>>>> it?
>>> 
>>> 
>>> We have back ported this feature to 2.1.x and has been using for last few
>>> months in few of our deployments.
>>> 
>>> 
>>>> If not, I suggest we remove it.  I could file an issue for it to be
>>> 
>>>> added to hbase-operator-tools for some gallant dev to pick up if they
>>>>> wanted to use this backup work? (I could help w/ the migration).
>>>>> 
>>>>> What do others think?
>>>>> 
>>>>> S
>>>>> 
>>>> 
>>> 
>>