You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jonathan Hsieh <jo...@cloudera.com> on 2016/10/06 07:07:38 UTC
Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
I'd like to get the -Dhadoop.profile=3.0 at least to compiling and passing
licensing working for the first hbase alpha (or whatever we end up calling
it)
I'll propose these items:
1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the most
recent). currently we are against 3.0.0-SNAPSHOT.
2) add precompile checks against a hadoop 3.x (HBASE-16733)
3) get 'mvn test install -Dskiptests' to succeed without licensing issues
(HBASE-16712)
4) Have a job setup in jenkins so that we can gain insight and burn down
unit tests failures against hadoop3.
These items have a good chance of landing in the next week or two.
Other related issues that are nice to have but wouldn't block an hbase
alpha include:
1) having no always failing unit tests against hadoop3 (HBASE-6581)
Thoughts?
Jon.
On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <sy...@gmail.com>
wrote:
> Hello, All,
>
> It is time to discuss about the schedule of HBase 2.0 release. HBase 2.0
> release is a big major release. When we release 1.0, we had 0.99 as dev
> preview/beta release. We should do something similar for the 2.0 release.
>
> Matteo and I talked about this. We think about that we need some
> Alpha/Beta milestones before the RC and final Release-to-Web 2.0 release.
>
> I don't know whether there is any discussion on this community about the
> Alpha/Beta release criteria. My proposal is that once we cut the branch-2,
> we should only have new features that are absolutely needed for major
> release (festures cannot be added in minor release) and those features
> should be "almost ready". For Alpha releases, we can still accept these
> kind of features; for Beta release, only bug fixes and performance
> improvement on existing features (should we also accept existing feature
> improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take).
>
> This is a big release and requires a lot of work from Release Manager. I
> asked Matteo whether I could help to be some kind of backup / hot-standby /
> assistant RM. I think he is very happy to have someone to share some RM
> duties. Thus, I will help make this 2.0 release as smooth as possible.
>
> Here is a tentative plan:
> - For now, we are thinking of creating branch-2 middle of this month and
> have 2.0-Alpha1 release at the end of this month or begin of Nov. The
> definition of Alpha1 is that we could deploy to a cluster and it can do
> some simple CRUD and table DDLs; and not crash (of course, UT passing).
>
> - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold
> higher bar. We will run some IT tests to make sure that it would
> functional.
>
> - At that time, we will lock down and not allow any new features, every 4-6
> weeks, we will have a Beta release (my realistic guess is that we would hit
> the US Christmas holiday at that time, so first Beta would take longer than
> 6 weeks). For Beta release, we would fix bugs and do performance tuning.
> Planning to have 2 Betas. (Question: in the past, do we need vote to have
> a Beta release?)
>
> - Once the code are in the stable stage, then we will have RCs and vote for
> the final release.
>
> Please let us know whether this is a reasonable approach that will make the
> release successful.
>
> Currently, we are aware of the following on-going new features for 2.0: new
> Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid Logical
> Clock, and maybe AsyncRegion / C++ client). If you have a feature that
> wants to be part of 2.0 release, please send discussion to dev community
> and we can make a call on accepting/rejecting.
>
> Thanks
> Stephen
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Posted by Andrew Purtell <ap...@apache.org>.
I assume we will have matrix testing of HBase versions against designated
upstream build targets for Hadoop 2.x and Hadoop 3.x? If not, if not too
much trouble would be a good idea. We're ad hoc about checking if a commit
to 0.98 breaks the Hadoop 1 build. Every few release candidates I have to
commit addendums to fix as part of the RC process. For what it's worth.
On Thu, Oct 6, 2016 at 11:58 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Again my goal is to have the hadoop 3 profile that already exist provide
> some signal and not be broken as it is currently. There are are other
> options but this one seems reasonable since hadoop3 releases are starting
> to show up.
>
> The milestone I propose is it compiles properly, and it passes our
> licensing enforcers. A further milestone for another release is passing
> unit tests.
>
> This is essentially no different for what we do for all the different
> hadoop version (2.6.x 2.7.x etc we support hbase 1.x on.), or the different
> builds with the different jdks.
>
> Jon.
>
> On Thu, Oct 6, 2016 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. one set of hbase binaries that will work against multiple hadoops
> >
> > I would be interested to know what tests are / will be performed
> > against 3.0.0-alpha1
> > (using artifacts built against 2.7.1).
> >
> > Thanks
> >
> >
> > On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > Yes.
> > >
> > > The goal is to produce one set of hbase binaries that will work against
> > > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but
> > > preferentially tested against and likely including binaries from a
> stable
> > > hadoop version.
> > >
> > > Up until recently, compiling against the hadoop 3 profile fails because
> > of
> > > compilation issues and licensing issues, Another issue, HBASE-16711
> has
> > > already landed which fixed compilation against hadoop2 and hadoop3.
> What
> > > remains is on the short proposed list makes sure licensing enforcers
> are
> > > satisfied correctly and getting build infrastructure precommit checks
> in
> > > place so we don't inadvertently introduce new problems.
> > >
> > > Jon.
> > >
> > > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Jon:
> > > > Once the goals you outlined below are achieved, would user be able to
> > use
> > > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed
> > with
> > > > hadoop
> > > > 3.0.0-alpha1 ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and
> > > > passing
> > > > > licensing working for the first hbase alpha (or whatever we end up
> > > > calling
> > > > > it)
> > > > >
> > > > > I'll propose these items:
> > > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is
> the
> > > > most
> > > > > recent). currently we are against 3.0.0-SNAPSHOT.
> > > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> > > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing
> > > issues
> > > > > (HBASE-16712)
> > > > > 4) Have a job setup in jenkins so that we can gain insight and burn
> > > down
> > > > > unit tests failures against hadoop3.
> > > > >
> > > > > These items have a good chance of landing in the next week or two.
> > > > >
> > > > > Other related issues that are nice to have but wouldn't block an
> > hbase
> > > > > alpha include:
> > > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581)
> > > > >
> > > > > Thoughts?
> > > > > Jon.
> > > > >
> > > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <
> > syuanjiangdev@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hello, All,
> > > > > >
> > > > > > It is time to discuss about the schedule of HBase 2.0 release.
> > HBase
> > > > 2.0
> > > > > > release is a big major release. When we release 1.0, we had 0.99
> > as
> > > > dev
> > > > > > preview/beta release. We should do something similar for the 2.0
> > > > > release.
> > > > > >
> > > > > > Matteo and I talked about this. We think about that we need
> some
> > > > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0
> > > > release.
> > > > > >
> > > > > > I don't know whether there is any discussion on this community
> > about
> > > > the
> > > > > > Alpha/Beta release criteria. My proposal is that once we cut the
> > > > > branch-2,
> > > > > > we should only have new features that are absolutely needed for
> > major
> > > > > > release (festures cannot be added in minor release) and those
> > > features
> > > > > > should be "almost ready". For Alpha releases, we can still
> accept
> > > > these
> > > > > > kind of features; for Beta release, only bug fixes and
> performance
> > > > > > improvement on existing features (should we also accept existing
> > > > feature
> > > > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my
> > take).
> > > > > >
> > > > > > This is a big release and requires a lot of work from Release
> > > > Manager. I
> > > > > > asked Matteo whether I could help to be some kind of backup /
> > > > > hot-standby /
> > > > > > assistant RM. I think he is very happy to have someone to share
> > some
> > > > RM
> > > > > > duties. Thus, I will help make this 2.0 release as smooth as
> > > possible.
> > > > > >
> > > > > > Here is a tentative plan:
> > > > > > - For now, we are thinking of creating branch-2 middle of this
> > month
> > > > and
> > > > > > have 2.0-Alpha1 release at the end of this month or begin of Nov.
> > > The
> > > > > > definition of Alpha1 is that we could deploy to a cluster and it
> > can
> > > do
> > > > > > some simple CRUD and table DDLs; and not crash (of course, UT
> > > passing).
> > > > > >
> > > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It
> would
> > > hold
> > > > > > higher bar. We will run some IT tests to make sure that it would
> > > > > > functional.
> > > > > >
> > > > > > - At that time, we will lock down and not allow any new features,
> > > every
> > > > > 4-6
> > > > > > weeks, we will have a Beta release (my realistic guess is that we
> > > would
> > > > > hit
> > > > > > the US Christmas holiday at that time, so first Beta would take
> > > longer
> > > > > than
> > > > > > 6 weeks). For Beta release, we would fix bugs and do performance
> > > > tuning.
> > > > > > Planning to have 2 Betas. (Question: in the past, do we need
> vote
> > to
> > > > > have
> > > > > > a Beta release?)
> > > > > >
> > > > > > - Once the code are in the stable stage, then we will have RCs
> and
> > > vote
> > > > > for
> > > > > > the final release.
> > > > > >
> > > > > > Please let us know whether this is a reasonable approach that
> will
> > > make
> > > > > the
> > > > > > release successful.
> > > > > >
> > > > > > Currently, we are aware of the following on-going new features
> for
> > > 2.0:
> > > > > new
> > > > > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid
> > > > Logical
> > > > > > Clock, and maybe AsyncRegion / C++ client). If you have a
> feature
> > > that
> > > > > > wants to be part of 2.0 release, please send discussion to dev
> > > > community
> > > > > > and we can make a call on accepting/rejecting.
> > > > > >
> > > > > > Thanks
> > > > > > Stephen
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > // jon@cloudera.com // @jmhsieh
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Again my goal is to have the hadoop 3 profile that already exist provide
some signal and not be broken as it is currently. There are are other
options but this one seems reasonable since hadoop3 releases are starting
to show up.
The milestone I propose is it compiles properly, and it passes our
licensing enforcers. A further milestone for another release is passing
unit tests.
This is essentially no different for what we do for all the different
hadoop version (2.6.x 2.7.x etc we support hbase 1.x on.), or the different
builds with the different jdks.
Jon.
On Thu, Oct 6, 2016 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. one set of hbase binaries that will work against multiple hadoops
>
> I would be interested to know what tests are / will be performed
> against 3.0.0-alpha1
> (using artifacts built against 2.7.1).
>
> Thanks
>
>
> On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Yes.
> >
> > The goal is to produce one set of hbase binaries that will work against
> > multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but
> > preferentially tested against and likely including binaries from a stable
> > hadoop version.
> >
> > Up until recently, compiling against the hadoop 3 profile fails because
> of
> > compilation issues and licensing issues, Another issue, HBASE-16711 has
> > already landed which fixed compilation against hadoop2 and hadoop3. What
> > remains is on the short proposed list makes sure licensing enforcers are
> > satisfied correctly and getting build infrastructure precommit checks in
> > place so we don't inadvertently introduce new problems.
> >
> > Jon.
> >
> > On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Jon:
> > > Once the goals you outlined below are achieved, would user be able to
> use
> > > build artifacts compiled against hadoop 2.7.1 on a cluster deployed
> with
> > > hadoop
> > > 3.0.0-alpha1 ?
> > >
> > > Cheers
> > >
> > > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jo...@cloudera.com>
> > wrote:
> > >
> > > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and
> > > passing
> > > > licensing working for the first hbase alpha (or whatever we end up
> > > calling
> > > > it)
> > > >
> > > > I'll propose these items:
> > > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the
> > > most
> > > > recent). currently we are against 3.0.0-SNAPSHOT.
> > > > 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> > > > 3) get 'mvn test install -Dskiptests' to succeed without licensing
> > issues
> > > > (HBASE-16712)
> > > > 4) Have a job setup in jenkins so that we can gain insight and burn
> > down
> > > > unit tests failures against hadoop3.
> > > >
> > > > These items have a good chance of landing in the next week or two.
> > > >
> > > > Other related issues that are nice to have but wouldn't block an
> hbase
> > > > alpha include:
> > > > 1) having no always failing unit tests against hadoop3 (HBASE-6581)
> > > >
> > > > Thoughts?
> > > > Jon.
> > > >
> > > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <
> syuanjiangdev@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hello, All,
> > > > >
> > > > > It is time to discuss about the schedule of HBase 2.0 release.
> HBase
> > > 2.0
> > > > > release is a big major release. When we release 1.0, we had 0.99
> as
> > > dev
> > > > > preview/beta release. We should do something similar for the 2.0
> > > > release.
> > > > >
> > > > > Matteo and I talked about this. We think about that we need some
> > > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0
> > > release.
> > > > >
> > > > > I don't know whether there is any discussion on this community
> about
> > > the
> > > > > Alpha/Beta release criteria. My proposal is that once we cut the
> > > > branch-2,
> > > > > we should only have new features that are absolutely needed for
> major
> > > > > release (festures cannot be added in minor release) and those
> > features
> > > > > should be "almost ready". For Alpha releases, we can still accept
> > > these
> > > > > kind of features; for Beta release, only bug fixes and performance
> > > > > improvement on existing features (should we also accept existing
> > > feature
> > > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my
> take).
> > > > >
> > > > > This is a big release and requires a lot of work from Release
> > > Manager. I
> > > > > asked Matteo whether I could help to be some kind of backup /
> > > > hot-standby /
> > > > > assistant RM. I think he is very happy to have someone to share
> some
> > > RM
> > > > > duties. Thus, I will help make this 2.0 release as smooth as
> > possible.
> > > > >
> > > > > Here is a tentative plan:
> > > > > - For now, we are thinking of creating branch-2 middle of this
> month
> > > and
> > > > > have 2.0-Alpha1 release at the end of this month or begin of Nov.
> > The
> > > > > definition of Alpha1 is that we could deploy to a cluster and it
> can
> > do
> > > > > some simple CRUD and table DDLs; and not crash (of course, UT
> > passing).
> > > > >
> > > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would
> > hold
> > > > > higher bar. We will run some IT tests to make sure that it would
> > > > > functional.
> > > > >
> > > > > - At that time, we will lock down and not allow any new features,
> > every
> > > > 4-6
> > > > > weeks, we will have a Beta release (my realistic guess is that we
> > would
> > > > hit
> > > > > the US Christmas holiday at that time, so first Beta would take
> > longer
> > > > than
> > > > > 6 weeks). For Beta release, we would fix bugs and do performance
> > > tuning.
> > > > > Planning to have 2 Betas. (Question: in the past, do we need vote
> to
> > > > have
> > > > > a Beta release?)
> > > > >
> > > > > - Once the code are in the stable stage, then we will have RCs and
> > vote
> > > > for
> > > > > the final release.
> > > > >
> > > > > Please let us know whether this is a reasonable approach that will
> > make
> > > > the
> > > > > release successful.
> > > > >
> > > > > Currently, we are aware of the following on-going new features for
> > 2.0:
> > > > new
> > > > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid
> > > Logical
> > > > > Clock, and maybe AsyncRegion / C++ client). If you have a feature
> > that
> > > > > wants to be part of 2.0 release, please send discussion to dev
> > > community
> > > > > and we can make a call on accepting/rejecting.
> > > > >
> > > > > Thanks
> > > > > Stephen
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > // Jonathan Hsieh (shay)
> > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > // jon@cloudera.com // @jmhsieh
> > > >
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Posted by Ted Yu <yu...@gmail.com>.
bq. one set of hbase binaries that will work against multiple hadoops
I would be interested to know what tests are / will be performed
against 3.0.0-alpha1
(using artifacts built against 2.7.1).
Thanks
On Thu, Oct 6, 2016 at 10:41 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Yes.
>
> The goal is to produce one set of hbase binaries that will work against
> multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but
> preferentially tested against and likely including binaries from a stable
> hadoop version.
>
> Up until recently, compiling against the hadoop 3 profile fails because of
> compilation issues and licensing issues, Another issue, HBASE-16711 has
> already landed which fixed compilation against hadoop2 and hadoop3. What
> remains is on the short proposed list makes sure licensing enforcers are
> satisfied correctly and getting build infrastructure precommit checks in
> place so we don't inadvertently introduce new problems.
>
> Jon.
>
> On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Jon:
> > Once the goals you outlined below are achieved, would user be able to use
> > build artifacts compiled against hadoop 2.7.1 on a cluster deployed with
> > hadoop
> > 3.0.0-alpha1 ?
> >
> > Cheers
> >
> > On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and
> > passing
> > > licensing working for the first hbase alpha (or whatever we end up
> > calling
> > > it)
> > >
> > > I'll propose these items:
> > > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the
> > most
> > > recent). currently we are against 3.0.0-SNAPSHOT.
> > > 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> > > 3) get 'mvn test install -Dskiptests' to succeed without licensing
> issues
> > > (HBASE-16712)
> > > 4) Have a job setup in jenkins so that we can gain insight and burn
> down
> > > unit tests failures against hadoop3.
> > >
> > > These items have a good chance of landing in the next week or two.
> > >
> > > Other related issues that are nice to have but wouldn't block an hbase
> > > alpha include:
> > > 1) having no always failing unit tests against hadoop3 (HBASE-6581)
> > >
> > > Thoughts?
> > > Jon.
> > >
> > > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <syuanjiangdev@gmail.com
> >
> > > wrote:
> > >
> > > > Hello, All,
> > > >
> > > > It is time to discuss about the schedule of HBase 2.0 release. HBase
> > 2.0
> > > > release is a big major release. When we release 1.0, we had 0.99 as
> > dev
> > > > preview/beta release. We should do something similar for the 2.0
> > > release.
> > > >
> > > > Matteo and I talked about this. We think about that we need some
> > > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0
> > release.
> > > >
> > > > I don't know whether there is any discussion on this community about
> > the
> > > > Alpha/Beta release criteria. My proposal is that once we cut the
> > > branch-2,
> > > > we should only have new features that are absolutely needed for major
> > > > release (festures cannot be added in minor release) and those
> features
> > > > should be "almost ready". For Alpha releases, we can still accept
> > these
> > > > kind of features; for Beta release, only bug fixes and performance
> > > > improvement on existing features (should we also accept existing
> > feature
> > > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take).
> > > >
> > > > This is a big release and requires a lot of work from Release
> > Manager. I
> > > > asked Matteo whether I could help to be some kind of backup /
> > > hot-standby /
> > > > assistant RM. I think he is very happy to have someone to share some
> > RM
> > > > duties. Thus, I will help make this 2.0 release as smooth as
> possible.
> > > >
> > > > Here is a tentative plan:
> > > > - For now, we are thinking of creating branch-2 middle of this month
> > and
> > > > have 2.0-Alpha1 release at the end of this month or begin of Nov.
> The
> > > > definition of Alpha1 is that we could deploy to a cluster and it can
> do
> > > > some simple CRUD and table DDLs; and not crash (of course, UT
> passing).
> > > >
> > > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would
> hold
> > > > higher bar. We will run some IT tests to make sure that it would
> > > > functional.
> > > >
> > > > - At that time, we will lock down and not allow any new features,
> every
> > > 4-6
> > > > weeks, we will have a Beta release (my realistic guess is that we
> would
> > > hit
> > > > the US Christmas holiday at that time, so first Beta would take
> longer
> > > than
> > > > 6 weeks). For Beta release, we would fix bugs and do performance
> > tuning.
> > > > Planning to have 2 Betas. (Question: in the past, do we need vote to
> > > have
> > > > a Beta release?)
> > > >
> > > > - Once the code are in the stable stage, then we will have RCs and
> vote
> > > for
> > > > the final release.
> > > >
> > > > Please let us know whether this is a reasonable approach that will
> make
> > > the
> > > > release successful.
> > > >
> > > > Currently, we are aware of the following on-going new features for
> 2.0:
> > > new
> > > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid
> > Logical
> > > > Clock, and maybe AsyncRegion / C++ client). If you have a feature
> that
> > > > wants to be part of 2.0 release, please send discussion to dev
> > community
> > > > and we can make a call on accepting/rejecting.
> > > >
> > > > Thanks
> > > > Stephen
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Yes.
The goal is to produce one set of hbase binaries that will work against
multiple hadoops such as the 2.7 and 3.0.0-alpha1 versions, but
preferentially tested against and likely including binaries from a stable
hadoop version.
Up until recently, compiling against the hadoop 3 profile fails because of
compilation issues and licensing issues, Another issue, HBASE-16711 has
already landed which fixed compilation against hadoop2 and hadoop3. What
remains is on the short proposed list makes sure licensing enforcers are
satisfied correctly and getting build infrastructure precommit checks in
place so we don't inadvertently introduce new problems.
Jon.
On Thu, Oct 6, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> Jon:
> Once the goals you outlined below are achieved, would user be able to use
> build artifacts compiled against hadoop 2.7.1 on a cluster deployed with
> hadoop
> 3.0.0-alpha1 ?
>
> Cheers
>
> On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > I'd like to get the -Dhadoop.profile=3.0 at least to compiling and
> passing
> > licensing working for the first hbase alpha (or whatever we end up
> calling
> > it)
> >
> > I'll propose these items:
> > 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the
> most
> > recent). currently we are against 3.0.0-SNAPSHOT.
> > 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> > 3) get 'mvn test install -Dskiptests' to succeed without licensing issues
> > (HBASE-16712)
> > 4) Have a job setup in jenkins so that we can gain insight and burn down
> > unit tests failures against hadoop3.
> >
> > These items have a good chance of landing in the next week or two.
> >
> > Other related issues that are nice to have but wouldn't block an hbase
> > alpha include:
> > 1) having no always failing unit tests against hadoop3 (HBASE-6581)
> >
> > Thoughts?
> > Jon.
> >
> > On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <sy...@gmail.com>
> > wrote:
> >
> > > Hello, All,
> > >
> > > It is time to discuss about the schedule of HBase 2.0 release. HBase
> 2.0
> > > release is a big major release. When we release 1.0, we had 0.99 as
> dev
> > > preview/beta release. We should do something similar for the 2.0
> > release.
> > >
> > > Matteo and I talked about this. We think about that we need some
> > > Alpha/Beta milestones before the RC and final Release-to-Web 2.0
> release.
> > >
> > > I don't know whether there is any discussion on this community about
> the
> > > Alpha/Beta release criteria. My proposal is that once we cut the
> > branch-2,
> > > we should only have new features that are absolutely needed for major
> > > release (festures cannot be added in minor release) and those features
> > > should be "almost ready". For Alpha releases, we can still accept
> these
> > > kind of features; for Beta release, only bug fixes and performance
> > > improvement on existing features (should we also accept existing
> feature
> > > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take).
> > >
> > > This is a big release and requires a lot of work from Release
> Manager. I
> > > asked Matteo whether I could help to be some kind of backup /
> > hot-standby /
> > > assistant RM. I think he is very happy to have someone to share some
> RM
> > > duties. Thus, I will help make this 2.0 release as smooth as possible.
> > >
> > > Here is a tentative plan:
> > > - For now, we are thinking of creating branch-2 middle of this month
> and
> > > have 2.0-Alpha1 release at the end of this month or begin of Nov. The
> > > definition of Alpha1 is that we could deploy to a cluster and it can do
> > > some simple CRUD and table DDLs; and not crash (of course, UT passing).
> > >
> > > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold
> > > higher bar. We will run some IT tests to make sure that it would
> > > functional.
> > >
> > > - At that time, we will lock down and not allow any new features, every
> > 4-6
> > > weeks, we will have a Beta release (my realistic guess is that we would
> > hit
> > > the US Christmas holiday at that time, so first Beta would take longer
> > than
> > > 6 weeks). For Beta release, we would fix bugs and do performance
> tuning.
> > > Planning to have 2 Betas. (Question: in the past, do we need vote to
> > have
> > > a Beta release?)
> > >
> > > - Once the code are in the stable stage, then we will have RCs and vote
> > for
> > > the final release.
> > >
> > > Please let us know whether this is a reasonable approach that will make
> > the
> > > release successful.
> > >
> > > Currently, we are aware of the following on-going new features for 2.0:
> > new
> > > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid
> Logical
> > > Clock, and maybe AsyncRegion / C++ client). If you have a feature that
> > > wants to be part of 2.0 release, please send discussion to dev
> community
> > > and we can make a call on accepting/rejecting.
> > >
> > > Thanks
> > > Stephen
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: Hadoop 3.x profile working for hbase 2.0 [Re: HBASE 2.0]
Posted by Ted Yu <yu...@gmail.com>.
Jon:
Once the goals you outlined below are achieved, would user be able to use
build artifacts compiled against hadoop 2.7.1 on a cluster deployed with hadoop
3.0.0-alpha1 ?
Cheers
On Thu, Oct 6, 2016 at 12:07 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> I'd like to get the -Dhadoop.profile=3.0 at least to compiling and passing
> licensing working for the first hbase alpha (or whatever we end up calling
> it)
>
> I'll propose these items:
> 1) peg to one of the recent hadoop alphas (hadoop 3.0.0-alpha1 is the most
> recent). currently we are against 3.0.0-SNAPSHOT.
> 2) add precompile checks against a hadoop 3.x (HBASE-16733)
> 3) get 'mvn test install -Dskiptests' to succeed without licensing issues
> (HBASE-16712)
> 4) Have a job setup in jenkins so that we can gain insight and burn down
> unit tests failures against hadoop3.
>
> These items have a good chance of landing in the next week or two.
>
> Other related issues that are nice to have but wouldn't block an hbase
> alpha include:
> 1) having no always failing unit tests against hadoop3 (HBASE-6581)
>
> Thoughts?
> Jon.
>
> On Mon, Oct 3, 2016 at 3:27 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
>
> > Hello, All,
> >
> > It is time to discuss about the schedule of HBase 2.0 release. HBase 2.0
> > release is a big major release. When we release 1.0, we had 0.99 as dev
> > preview/beta release. We should do something similar for the 2.0
> release.
> >
> > Matteo and I talked about this. We think about that we need some
> > Alpha/Beta milestones before the RC and final Release-to-Web 2.0 release.
> >
> > I don't know whether there is any discussion on this community about the
> > Alpha/Beta release criteria. My proposal is that once we cut the
> branch-2,
> > we should only have new features that are absolutely needed for major
> > release (festures cannot be added in minor release) and those features
> > should be "almost ready". For Alpha releases, we can still accept these
> > kind of features; for Beta release, only bug fixes and performance
> > improvement on existing features (should we also accept existing feature
> > improvement in Beta? Maybe Beta 1, Not in Beta 2 - that is my take).
> >
> > This is a big release and requires a lot of work from Release Manager. I
> > asked Matteo whether I could help to be some kind of backup /
> hot-standby /
> > assistant RM. I think he is very happy to have someone to share some RM
> > duties. Thus, I will help make this 2.0 release as smooth as possible.
> >
> > Here is a tentative plan:
> > - For now, we are thinking of creating branch-2 middle of this month and
> > have 2.0-Alpha1 release at the end of this month or begin of Nov. The
> > definition of Alpha1 is that we could deploy to a cluster and it can do
> > some simple CRUD and table DDLs; and not crash (of course, UT passing).
> >
> > - Then we will have 2.0-Alpha2 in 4-6 weeks after Apha1. It would hold
> > higher bar. We will run some IT tests to make sure that it would
> > functional.
> >
> > - At that time, we will lock down and not allow any new features, every
> 4-6
> > weeks, we will have a Beta release (my realistic guess is that we would
> hit
> > the US Christmas holiday at that time, so first Beta would take longer
> than
> > 6 weeks). For Beta release, we would fix bugs and do performance tuning.
> > Planning to have 2 Betas. (Question: in the past, do we need vote to
> have
> > a Beta release?)
> >
> > - Once the code are in the stable stage, then we will have RCs and vote
> for
> > the final release.
> >
> > Please let us know whether this is a reasonable approach that will make
> the
> > release successful.
> >
> > Currently, we are aware of the following on-going new features for 2.0:
> new
> > Assignment Manager, backup/restore, off-heap, protobuff 3, Hybrid Logical
> > Clock, and maybe AsyncRegion / C++ client). If you have a feature that
> > wants to be part of 2.0 release, please send discussion to dev community
> > and we can make a call on accepting/rejecting.
> >
> > Thanks
> > Stephen
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>