[DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

Hi Hadoopers!

Over in HBase we've been discussing the impact of our dependencies on our
downstream users. As our most fundamental dependency, Hadoop plays a big
role in the operational cost of running an HBase instance.

Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1].
We don't drop Hadoop minor release lines in minor releases so we are
unlikely remove anything from this set until HBase 2.0, probably at the end
of 2015 / start of 2016 (and currently we plan to continue supporting at
least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
shipped binaries to Hadoop 2.6, following some stability testing by part of
our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
that could destroy HBase clusters should users decide to turn on HDFS
encryption[4]. Our installation instructions tell folks to replace these
jars with the version of Hadoop they are actually running, but not all
users follow those instructions so we want to minimize the pain for them.

Regular maintenance releases are key to keeping operational burdens low for
our downstream users; we don't want them to be forced to choose between
living with broken systems and stomaching the risk of upgrades across
minor/major version numbers. Looking back over the three aforementioned
Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov
2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a
year without a release[5]. On our discussion of shipping Hadoop 2.6
binaries, one of your PMC members mentioned that with continued work on the
2.7 line y'all weren't planning any additional releases of the earlier
minor versions[6].

The HBase community requests that Hadoop pick up making bug-fix-only patch
releases again on a regular schedule[7]. Preferably on the 2.6 line and
preferably monthly. We realize that given the time gap since 2.6.0 it will
likely take a big to get 2.6.1 together, but after that it should take much
less effort to continue.

[1]: http://hbase.apache.org/book.html#hadoop
[2]: http://s.apache.org/ReP
[3]: HBASE-13339
[4]: HADOOP-11674 and HADOOP-11710
[5]: http://hadoop.apache.org/releases.html
[6]: http://s.apache.org/MTY
[7]: http://s.apache.org/ViP

-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

Why not just have the discussion here? It seems integral to the matter of
having more maintenance releases on those versions.

On Wed, Jul 15, 2015 at 11:39 AM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> I believe there was general consensus to do more maintenance releases, as
> witnessed in the other thread.
>
> There have been discussions on what should go into 2.x.1, 2.x.2, etc., but
> I don't think we have a clear proposal. It would be nice to put that
> together, so committers know where all to commit patches to. Otherwise,
> release managers will have to look through branch-2 and pull in the fixes.
> Either approach is fine by me, but would be nice to start a [DISCUSS]
> thread on how to go about this.
>
> Any takers?
>
> On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>
> > Strong +1 for having a 2.6.1 release. I understand Vinod has been trying
> to
> > get that effort going but it's been stalled a little bit. It would be
> good
> > to rekindle that effort.
> >
> > Companies with big hadoop 2.x deployments (including mine) have always
> > tried to stabilize a 2.x release by testing/collecting/researching
> critical
> > issues on the release. Each would come up with its own set of fixes to
> > backport. We would also communicate it via offline channels. During the
> > hadoop summit, we thought it would be great if we all came together and
> > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> > example) with all the critical issues fixed.
> >
> > Thanks,
> > Sangjin
> >
> >
> > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> wrote:
> >
> > > Thank you for the notification. Trying to back port bug fixes.
> > >
> > > - Tsuyoshi
> > >
> > > On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > > > Hi Hadoopers!
> > > >
> > > > Over in HBase we've been discussing the impact of our dependencies on
> > our
> > > > downstream users. As our most fundamental dependency, Hadoop plays a
> > big
> > > > role in the operational cost of running an HBase instance.
> > > >
> > > > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > > 2.6[1].
> > > > We don't drop Hadoop minor release lines in minor releases so we are
> > > > unlikely remove anything from this set until HBase 2.0, probably at
> the
> > > end
> > > > of 2015 / start of 2016 (and currently we plan to continue supporting
> > at
> > > > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> our
> > > > shipped binaries to Hadoop 2.6, following some stability testing by
> > part
> > > of
> > > > our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> > bugs
> > > > that could destroy HBase clusters should users decide to turn on HDFS
> > > > encryption[4]. Our installation instructions tell folks to replace
> > these
> > > > jars with the version of Hadoop they are actually running, but not
> all
> > > > users follow those instructions so we want to minimize the pain for
> > them.
> > > >
> > > > Regular maintenance releases are key to keeping operational burdens
> low
> > > for
> > > > our downstream users; we don't want them to be forced to choose
> between
> > > > living with broken systems and stomaching the risk of upgrades across
> > > > minor/major version numbers. Looking back over the three
> aforementioned
> > > > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
> in
> > > Nov
> > > > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> to
> > > be a
> > > > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > > binaries, one of your PMC members mentioned that with continued work
> on
> > > the
> > > > 2.7 line y'all weren't planning any additional releases of the
> earlier
> > > > minor versions[6].
> > > >
> > > > The HBase community requests that Hadoop pick up making bug-fix-only
> > > patch
> > > > releases again on a regular schedule[7]. Preferably on the 2.6 line
> and
> > > > preferably monthly. We realize that given the time gap since 2.6.0 it
> > > will
> > > > likely take a big to get 2.6.1 together, but after that it should
> take
> > > much
> > > > less effort to continue.
> > > >
> > > > [1]: http://hbase.apache.org/book.html#hadoop
> > > > [2]: http://s.apache.org/ReP
> > > > [3]: HBASE-13339
> > > > [4]: HADOOP-11674 and HADOOP-11710
> > > > [5]: http://hadoop.apache.org/releases.html
> > > > [6]: http://s.apache.org/MTY
> > > > [7]: http://s.apache.org/ViP
> > > >
> > > > --
> > > > Sean
> > >
> >
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>



-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

Why not just have the discussion here? It seems integral to the matter of
having more maintenance releases on those versions.

On Wed, Jul 15, 2015 at 11:39 AM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> I believe there was general consensus to do more maintenance releases, as
> witnessed in the other thread.
>
> There have been discussions on what should go into 2.x.1, 2.x.2, etc., but
> I don't think we have a clear proposal. It would be nice to put that
> together, so committers know where all to commit patches to. Otherwise,
> release managers will have to look through branch-2 and pull in the fixes.
> Either approach is fine by me, but would be nice to start a [DISCUSS]
> thread on how to go about this.
>
> Any takers?
>
> On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>
> > Strong +1 for having a 2.6.1 release. I understand Vinod has been trying
> to
> > get that effort going but it's been stalled a little bit. It would be
> good
> > to rekindle that effort.
> >
> > Companies with big hadoop 2.x deployments (including mine) have always
> > tried to stabilize a 2.x release by testing/collecting/researching
> critical
> > issues on the release. Each would come up with its own set of fixes to
> > backport. We would also communicate it via offline channels. During the
> > hadoop summit, we thought it would be great if we all came together and
> > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> > example) with all the critical issues fixed.
> >
> > Thanks,
> > Sangjin
> >
> >
> > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> wrote:
> >
> > > Thank you for the notification. Trying to back port bug fixes.
> > >
> > > - Tsuyoshi
> > >
> > > On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > > > Hi Hadoopers!
> > > >
> > > > Over in HBase we've been discussing the impact of our dependencies on
> > our
> > > > downstream users. As our most fundamental dependency, Hadoop plays a
> > big
> > > > role in the operational cost of running an HBase instance.
> > > >
> > > > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > > 2.6[1].
> > > > We don't drop Hadoop minor release lines in minor releases so we are
> > > > unlikely remove anything from this set until HBase 2.0, probably at
> the
> > > end
> > > > of 2015 / start of 2016 (and currently we plan to continue supporting
> > at
> > > > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> our
> > > > shipped binaries to Hadoop 2.6, following some stability testing by
> > part
> > > of
> > > > our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> > bugs
> > > > that could destroy HBase clusters should users decide to turn on HDFS
> > > > encryption[4]. Our installation instructions tell folks to replace
> > these
> > > > jars with the version of Hadoop they are actually running, but not
> all
> > > > users follow those instructions so we want to minimize the pain for
> > them.
> > > >
> > > > Regular maintenance releases are key to keeping operational burdens
> low
> > > for
> > > > our downstream users; we don't want them to be forced to choose
> between
> > > > living with broken systems and stomaching the risk of upgrades across
> > > > minor/major version numbers. Looking back over the three
> aforementioned
> > > > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
> in
> > > Nov
> > > > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> to
> > > be a
> > > > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > > binaries, one of your PMC members mentioned that with continued work
> on
> > > the
> > > > 2.7 line y'all weren't planning any additional releases of the
> earlier
> > > > minor versions[6].
> > > >
> > > > The HBase community requests that Hadoop pick up making bug-fix-only
> > > patch
> > > > releases again on a regular schedule[7]. Preferably on the 2.6 line
> and
> > > > preferably monthly. We realize that given the time gap since 2.6.0 it
> > > will
> > > > likely take a big to get 2.6.1 together, but after that it should
> take
> > > much
> > > > less effort to continue.
> > > >
> > > > [1]: http://hbase.apache.org/book.html#hadoop
> > > > [2]: http://s.apache.org/ReP
> > > > [3]: HBASE-13339
> > > > [4]: HADOOP-11674 and HADOOP-11710
> > > > [5]: http://hadoop.apache.org/releases.html
> > > > [6]: http://s.apache.org/MTY
> > > > [7]: http://s.apache.org/ViP
> > > >
> > > > --
> > > > Sean
> > >
> >
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>



-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

I believe there was general consensus to do more maintenance releases, as
witnessed in the other thread.

There have been discussions on what should go into 2.x.1, 2.x.2, etc., but
I don't think we have a clear proposal. It would be nice to put that
together, so committers know where all to commit patches to. Otherwise,
release managers will have to look through branch-2 and pull in the fixes.
Either approach is fine by me, but would be nice to start a [DISCUSS]
thread on how to go about this.

Any takers?

On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:

> Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
> get that effort going but it's been stalled a little bit. It would be good
> to rekindle that effort.
>
> Companies with big hadoop 2.x deployments (including mine) have always
> tried to stabilize a 2.x release by testing/collecting/researching critical
> issues on the release. Each would come up with its own set of fixes to
> backport. We would also communicate it via offline channels. During the
> hadoop summit, we thought it would be great if we all came together and
> create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> example) with all the critical issues fixed.
>
> Thanks,
> Sangjin
>
>
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> > Thank you for the notification. Trying to back port bug fixes.
> >
> > - Tsuyoshi
> >
> > On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> > > Hi Hadoopers!
> > >
> > > Over in HBase we've been discussing the impact of our dependencies on
> our
> > > downstream users. As our most fundamental dependency, Hadoop plays a
> big
> > > role in the operational cost of running an HBase instance.
> > >
> > > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > 2.6[1].
> > > We don't drop Hadoop minor release lines in minor releases so we are
> > > unlikely remove anything from this set until HBase 2.0, probably at the
> > end
> > > of 2015 / start of 2016 (and currently we plan to continue supporting
> at
> > > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> > > shipped binaries to Hadoop 2.6, following some stability testing by
> part
> > of
> > > our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> bugs
> > > that could destroy HBase clusters should users decide to turn on HDFS
> > > encryption[4]. Our installation instructions tell folks to replace
> these
> > > jars with the version of Hadoop they are actually running, but not all
> > > users follow those instructions so we want to minimize the pain for
> them.
> > >
> > > Regular maintenance releases are key to keeping operational burdens low
> > for
> > > our downstream users; we don't want them to be forced to choose between
> > > living with broken systems and stomaching the risk of upgrades across
> > > minor/major version numbers. Looking back over the three aforementioned
> > > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> > Nov
> > > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> > be a
> > > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > binaries, one of your PMC members mentioned that with continued work on
> > the
> > > 2.7 line y'all weren't planning any additional releases of the earlier
> > > minor versions[6].
> > >
> > > The HBase community requests that Hadoop pick up making bug-fix-only
> > patch
> > > releases again on a regular schedule[7]. Preferably on the 2.6 line and
> > > preferably monthly. We realize that given the time gap since 2.6.0 it
> > will
> > > likely take a big to get 2.6.1 together, but after that it should take
> > much
> > > less effort to continue.
> > >
> > > [1]: http://hbase.apache.org/book.html#hadoop
> > > [2]: http://s.apache.org/ReP
> > > [3]: HBASE-13339
> > > [4]: HADOOP-11674 and HADOOP-11710
> > > [5]: http://hadoop.apache.org/releases.html
> > > [6]: http://s.apache.org/MTY
> > > [7]: http://s.apache.org/ViP
> > >
> > > --
> > > Sean
> >
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Elliott Clark <ec...@apache.org>]

If people are concerned about regression then just don't install new
versions, or install a vendor tested stable version. Giving users choices
is a good thing for stability.

On Wed, Jul 15, 2015 at 2:17 PM, Sangjin Lee <sj...@gmail.com> wrote:

> I think the bar for making the maintenance releases should be set
> reasonably high, and the main reason is the concern for
> stability/regression. Unfortunately there have been cases where a seemingly
> innocuous bug fix introduced regressions, small or large. And that defeats
> the purpose of a maintenance release.
>
> On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Every new patch potentially brings in new bugs. So, if we want to limit
> the
> > kinds of potential bugs introduced in point releases, we might want to
> > limit what gets in.
> >
> > Would be nice to make sure a point release is more stable than a previous
> > point release in that line.
> >
> > On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Why not just include all backwards compatible bug fixes?
> > >
> > > Alternatively, why not appoint a Release Manager for the minor release
> > line
> > > and then allow them to arbitrate when there's disagreement about
> > inclusion?
> > > This has worked well in the HBase community.
> > >
> > > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > As I proposed in the other thread, how about we adopting the
> following
> > > > model:
> > > >
> > > > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> > the
> > > > next minor release.
> > > > x.y.2 releases have all Blocker, Critical bug fixes applied to the
> next
> > > > minor release.
> > > > x.y.3 releases have all Blocker bug fixes applied to next minor
> > release.
> > > >
> > > > Here I am assuming there are no security-fix-only or other urgent
> > > releases.
> > > >
> > > > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > > > release.
> > > >
> > > > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > > > vinodkv@hortonworks.com> wrote:
> > > >
> > > > > Yeah, I started a thread while back on this one (
> > > > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > > > discussions re 2.6.1.
> > > > >
> > > > > The biggest problem I found offline was about what bug-fixes are
> > > > > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> > Given
> > > > the
> > > > > number of bug-fixes that went into 2.7.x and into branch-2.8,
> > figuring
> > > > out
> > > > > a set of patches that is acceptable for everyone is a huge
> challenge
> > > > which
> > > > > kind of stalled my attempts.
> > > > >
> > > > > Thanks
> > > > > +Vinod
> > > > >
> > > > >
> > > > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com>
> wrote:
> > > > > >
> > > > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > > > trying
> > > > > to
> > > > > > get that effort going but it's been stalled a little bit. It
> would
> > be
> > > > > good
> > > > > > to rekindle that effort.
> > > > > >
> > > > > > Companies with big hadoop 2.x deployments (including mine) have
> > > always
> > > > > > tried to stabilize a 2.x release by
> testing/collecting/researching
> > > > > critical
> > > > > > issues on the release. Each would come up with its own set of
> fixes
> > > to
> > > > > > backport. We would also communicate it via offline channels.
> During
> > > the
> > > > > > hadoop summit, we thought it would be great if we all came
> together
> > > and
> > > > > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> > 2.6
> > > > for
> > > > > > example) with all the critical issues fixed.
> > > > > >
> > > > > > Thanks,
> > > > > > Sangjin
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <
> ozawa@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Thank you for the notification. Trying to back port bug fixes.
> > > > > >>
> > > > > >> - Tsuyoshi
> > > > > >>
> > > > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <
> busbey@cloudera.com
> > >
> > > > > wrote:
> > > > > >>> Hi Hadoopers!
> > > > > >>>
> > > > > >>> Over in HBase we've been discussing the impact of our
> > dependencies
> > > on
> > > > > our
> > > > > >>> downstream users. As our most fundamental dependency, Hadoop
> > plays
> > > a
> > > > > big
> > > > > >>> role in the operational cost of running an HBase instance.
> > > > > >>>
> > > > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
> > and
> > > > > >> 2.6[1].
> > > > > >>> We don't drop Hadoop minor release lines in minor releases so
> we
> > > are
> > > > > >>> unlikely remove anything from this set until HBase 2.0,
> probably
> > at
> > > > the
> > > > > >> end
> > > > > >>> of 2015 / start of 2016 (and currently we plan to continue
> > > supporting
> > > > > at
> > > > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> > updating
> > > > our
> > > > > >>> shipped binaries to Hadoop 2.6, following some stability
> testing
> > by
> > > > > part
> > > > > >> of
> > > > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a
> couple
> > > of
> > > > > bugs
> > > > > >>> that could destroy HBase clusters should users decide to turn
> on
> > > HDFS
> > > > > >>> encryption[4]. Our installation instructions tell folks to
> > replace
> > > > > these
> > > > > >>> jars with the version of Hadoop they are actually running, but
> > not
> > > > all
> > > > > >>> users follow those instructions so we want to minimize the pain
> > for
> > > > > them.
> > > > > >>>
> > > > > >>> Regular maintenance releases are key to keeping operational
> > burdens
> > > > low
> > > > > >> for
> > > > > >>> our downstream users; we don't want them to be forced to choose
> > > > between
> > > > > >>> living with broken systems and stomaching the risk of upgrades
> > > across
> > > > > >>> minor/major version numbers. Looking back over the three
> > > > aforementioned
> > > > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0
> came
> > > out
> > > > in
> > > > > >> Nov
> > > > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> > looks
> > > > to
> > > > > >> be a
> > > > > >>> year without a release[5]. On our discussion of shipping Hadoop
> > 2.6
> > > > > >>> binaries, one of your PMC members mentioned that with continued
> > > work
> > > > on
> > > > > >> the
> > > > > >>> 2.7 line y'all weren't planning any additional releases of the
> > > > earlier
> > > > > >>> minor versions[6].
> > > > > >>>
> > > > > >>> The HBase community requests that Hadoop pick up making
> > > bug-fix-only
> > > > > >> patch
> > > > > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> > line
> > > > and
> > > > > >>> preferably monthly. We realize that given the time gap since
> > 2.6.0
> > > it
> > > > > >> will
> > > > > >>> likely take a big to get 2.6.1 together, but after that it
> should
> > > > take
> > > > > >> much
> > > > > >>> less effort to continue.
> > > > > >>>
> > > > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > > > >>> [2]: http://s.apache.org/ReP
> > > > > >>> [3]: HBASE-13339
> > > > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > > > >>> [5]: http://hadoop.apache.org/releases.html
> > > > > >>> [6]: http://s.apache.org/MTY
> > > > > >>> [7]: http://s.apache.org/ViP
> > > > > >>>
> > > > > >>> --
> > > > > >>> Sean
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Karthik Kambatla
> > > > Software Engineer, Cloudera Inc.
> > > > --------------------------------------------
> > > > http://five.sentenc.es
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Karthik Kambatla
> > Software Engineer, Cloudera Inc.
> > --------------------------------------------
> > http://five.sentenc.es
> >
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Elliott Clark <ec...@apache.org>]

If people are concerned about regression then just don't install new
versions, or install a vendor tested stable version. Giving users choices
is a good thing for stability.

On Wed, Jul 15, 2015 at 2:17 PM, Sangjin Lee <sj...@gmail.com> wrote:

> I think the bar for making the maintenance releases should be set
> reasonably high, and the main reason is the concern for
> stability/regression. Unfortunately there have been cases where a seemingly
> innocuous bug fix introduced regressions, small or large. And that defeats
> the purpose of a maintenance release.
>
> On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > Every new patch potentially brings in new bugs. So, if we want to limit
> the
> > kinds of potential bugs introduced in point releases, we might want to
> > limit what gets in.
> >
> > Would be nice to make sure a point release is more stable than a previous
> > point release in that line.
> >
> > On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Why not just include all backwards compatible bug fixes?
> > >
> > > Alternatively, why not appoint a Release Manager for the minor release
> > line
> > > and then allow them to arbitrate when there's disagreement about
> > inclusion?
> > > This has worked well in the HBase community.
> > >
> > > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > As I proposed in the other thread, how about we adopting the
> following
> > > > model:
> > > >
> > > > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> > the
> > > > next minor release.
> > > > x.y.2 releases have all Blocker, Critical bug fixes applied to the
> next
> > > > minor release.
> > > > x.y.3 releases have all Blocker bug fixes applied to next minor
> > release.
> > > >
> > > > Here I am assuming there are no security-fix-only or other urgent
> > > releases.
> > > >
> > > > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > > > release.
> > > >
> > > > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > > > vinodkv@hortonworks.com> wrote:
> > > >
> > > > > Yeah, I started a thread while back on this one (
> > > > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > > > discussions re 2.6.1.
> > > > >
> > > > > The biggest problem I found offline was about what bug-fixes are
> > > > > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> > Given
> > > > the
> > > > > number of bug-fixes that went into 2.7.x and into branch-2.8,
> > figuring
> > > > out
> > > > > a set of patches that is acceptable for everyone is a huge
> challenge
> > > > which
> > > > > kind of stalled my attempts.
> > > > >
> > > > > Thanks
> > > > > +Vinod
> > > > >
> > > > >
> > > > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com>
> wrote:
> > > > > >
> > > > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > > > trying
> > > > > to
> > > > > > get that effort going but it's been stalled a little bit. It
> would
> > be
> > > > > good
> > > > > > to rekindle that effort.
> > > > > >
> > > > > > Companies with big hadoop 2.x deployments (including mine) have
> > > always
> > > > > > tried to stabilize a 2.x release by
> testing/collecting/researching
> > > > > critical
> > > > > > issues on the release. Each would come up with its own set of
> fixes
> > > to
> > > > > > backport. We would also communicate it via offline channels.
> During
> > > the
> > > > > > hadoop summit, we thought it would be great if we all came
> together
> > > and
> > > > > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> > 2.6
> > > > for
> > > > > > example) with all the critical issues fixed.
> > > > > >
> > > > > > Thanks,
> > > > > > Sangjin
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <
> ozawa@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Thank you for the notification. Trying to back port bug fixes.
> > > > > >>
> > > > > >> - Tsuyoshi
> > > > > >>
> > > > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <
> busbey@cloudera.com
> > >
> > > > > wrote:
> > > > > >>> Hi Hadoopers!
> > > > > >>>
> > > > > >>> Over in HBase we've been discussing the impact of our
> > dependencies
> > > on
> > > > > our
> > > > > >>> downstream users. As our most fundamental dependency, Hadoop
> > plays
> > > a
> > > > > big
> > > > > >>> role in the operational cost of running an HBase instance.
> > > > > >>>
> > > > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
> > and
> > > > > >> 2.6[1].
> > > > > >>> We don't drop Hadoop minor release lines in minor releases so
> we
> > > are
> > > > > >>> unlikely remove anything from this set until HBase 2.0,
> probably
> > at
> > > > the
> > > > > >> end
> > > > > >>> of 2015 / start of 2016 (and currently we plan to continue
> > > supporting
> > > > > at
> > > > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> > updating
> > > > our
> > > > > >>> shipped binaries to Hadoop 2.6, following some stability
> testing
> > by
> > > > > part
> > > > > >> of
> > > > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a
> couple
> > > of
> > > > > bugs
> > > > > >>> that could destroy HBase clusters should users decide to turn
> on
> > > HDFS
> > > > > >>> encryption[4]. Our installation instructions tell folks to
> > replace
> > > > > these
> > > > > >>> jars with the version of Hadoop they are actually running, but
> > not
> > > > all
> > > > > >>> users follow those instructions so we want to minimize the pain
> > for
> > > > > them.
> > > > > >>>
> > > > > >>> Regular maintenance releases are key to keeping operational
> > burdens
> > > > low
> > > > > >> for
> > > > > >>> our downstream users; we don't want them to be forced to choose
> > > > between
> > > > > >>> living with broken systems and stomaching the risk of upgrades
> > > across
> > > > > >>> minor/major version numbers. Looking back over the three
> > > > aforementioned
> > > > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0
> came
> > > out
> > > > in
> > > > > >> Nov
> > > > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> > looks
> > > > to
> > > > > >> be a
> > > > > >>> year without a release[5]. On our discussion of shipping Hadoop
> > 2.6
> > > > > >>> binaries, one of your PMC members mentioned that with continued
> > > work
> > > > on
> > > > > >> the
> > > > > >>> 2.7 line y'all weren't planning any additional releases of the
> > > > earlier
> > > > > >>> minor versions[6].
> > > > > >>>
> > > > > >>> The HBase community requests that Hadoop pick up making
> > > bug-fix-only
> > > > > >> patch
> > > > > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> > line
> > > > and
> > > > > >>> preferably monthly. We realize that given the time gap since
> > 2.6.0
> > > it
> > > > > >> will
> > > > > >>> likely take a big to get 2.6.1 together, but after that it
> should
> > > > take
> > > > > >> much
> > > > > >>> less effort to continue.
> > > > > >>>
> > > > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > > > >>> [2]: http://s.apache.org/ReP
> > > > > >>> [3]: HBASE-13339
> > > > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > > > >>> [5]: http://hadoop.apache.org/releases.html
> > > > > >>> [6]: http://s.apache.org/MTY
> > > > > >>> [7]: http://s.apache.org/ViP
> > > > > >>>
> > > > > >>> --
> > > > > >>> Sean
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Karthik Kambatla
> > > > Software Engineer, Cloudera Inc.
> > > > --------------------------------------------
> > > > http://five.sentenc.es
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Karthik Kambatla
> > Software Engineer, Cloudera Inc.
> > --------------------------------------------
> > http://five.sentenc.es
> >
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sangjin Lee <sj...@gmail.com>]

I think the bar for making the maintenance releases should be set
reasonably high, and the main reason is the concern for
stability/regression. Unfortunately there have been cases where a seemingly
innocuous bug fix introduced regressions, small or large. And that defeats
the purpose of a maintenance release.

On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Every new patch potentially brings in new bugs. So, if we want to limit the
> kinds of potential bugs introduced in point releases, we might want to
> limit what gets in.
>
> Would be nice to make sure a point release is more stable than a previous
> point release in that line.
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > > As I proposed in the other thread, how about we adopting the following
> > > model:
> > >
> > > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> > > next minor release.
> > > x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> > > minor release.
> > > x.y.3 releases have all Blocker bug fixes applied to next minor
> release.
> > >
> > > Here I am assuming there are no security-fix-only or other urgent
> > releases.
> > >
> > > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > > release.
> > >
> > > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@hortonworks.com> wrote:
> > >
> > > > Yeah, I started a thread while back on this one (
> > > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > > discussions re 2.6.1.
> > > >
> > > > The biggest problem I found offline was about what bug-fixes are
> > > > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> > > the
> > > > number of bug-fixes that went into 2.7.x and into branch-2.8,
> figuring
> > > out
> > > > a set of patches that is acceptable for everyone is a huge challenge
> > > which
> > > > kind of stalled my attempts.
> > > >
> > > > Thanks
> > > > +Vinod
> > > >
> > > >
> > > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > > > >
> > > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > > trying
> > > > to
> > > > > get that effort going but it's been stalled a little bit. It would
> be
> > > > good
> > > > > to rekindle that effort.
> > > > >
> > > > > Companies with big hadoop 2.x deployments (including mine) have
> > always
> > > > > tried to stabilize a 2.x release by testing/collecting/researching
> > > > critical
> > > > > issues on the release. Each would come up with its own set of fixes
> > to
> > > > > backport. We would also communicate it via offline channels. During
> > the
> > > > > hadoop summit, we thought it would be great if we all came together
> > and
> > > > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> > > for
> > > > > example) with all the critical issues fixed.
> > > > >
> > > > > Thanks,
> > > > > Sangjin
> > > > >
> > > > >
> > > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org
> >
> > > > wrote:
> > > > >
> > > > >> Thank you for the notification. Trying to back port bug fixes.
> > > > >>
> > > > >> - Tsuyoshi
> > > > >>
> > > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com
> >
> > > > wrote:
> > > > >>> Hi Hadoopers!
> > > > >>>
> > > > >>> Over in HBase we've been discussing the impact of our
> dependencies
> > on
> > > > our
> > > > >>> downstream users. As our most fundamental dependency, Hadoop
> plays
> > a
> > > > big
> > > > >>> role in the operational cost of running an HBase instance.
> > > > >>>
> > > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
> and
> > > > >> 2.6[1].
> > > > >>> We don't drop Hadoop minor release lines in minor releases so we
> > are
> > > > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> > > the
> > > > >> end
> > > > >>> of 2015 / start of 2016 (and currently we plan to continue
> > supporting
> > > > at
> > > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> > > our
> > > > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> > > > part
> > > > >> of
> > > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> > of
> > > > bugs
> > > > >>> that could destroy HBase clusters should users decide to turn on
> > HDFS
> > > > >>> encryption[4]. Our installation instructions tell folks to
> replace
> > > > these
> > > > >>> jars with the version of Hadoop they are actually running, but
> not
> > > all
> > > > >>> users follow those instructions so we want to minimize the pain
> for
> > > > them.
> > > > >>>
> > > > >>> Regular maintenance releases are key to keeping operational
> burdens
> > > low
> > > > >> for
> > > > >>> our downstream users; we don't want them to be forced to choose
> > > between
> > > > >>> living with broken systems and stomaching the risk of upgrades
> > across
> > > > >>> minor/major version numbers. Looking back over the three
> > > aforementioned
> > > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> > out
> > > in
> > > > >> Nov
> > > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> > > to
> > > > >> be a
> > > > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> > > > >>> binaries, one of your PMC members mentioned that with continued
> > work
> > > on
> > > > >> the
> > > > >>> 2.7 line y'all weren't planning any additional releases of the
> > > earlier
> > > > >>> minor versions[6].
> > > > >>>
> > > > >>> The HBase community requests that Hadoop pick up making
> > bug-fix-only
> > > > >> patch
> > > > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> > > and
> > > > >>> preferably monthly. We realize that given the time gap since
> 2.6.0
> > it
> > > > >> will
> > > > >>> likely take a big to get 2.6.1 together, but after that it should
> > > take
> > > > >> much
> > > > >>> less effort to continue.
> > > > >>>
> > > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > > >>> [2]: http://s.apache.org/ReP
> > > > >>> [3]: HBASE-13339
> > > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > > >>> [5]: http://hadoop.apache.org/releases.html
> > > > >>> [6]: http://s.apache.org/MTY
> > > > >>> [7]: http://s.apache.org/ViP
> > > > >>>
> > > > >>> --
> > > > >>> Sean
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Karthik Kambatla
> > > Software Engineer, Cloudera Inc.
> > > --------------------------------------------
> > > http://five.sentenc.es
> > >
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sangjin Lee <sj...@gmail.com>]

I think the bar for making the maintenance releases should be set
reasonably high, and the main reason is the concern for
stability/regression. Unfortunately there have been cases where a seemingly
innocuous bug fix introduced regressions, small or large. And that defeats
the purpose of a maintenance release.

On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> Every new patch potentially brings in new bugs. So, if we want to limit the
> kinds of potential bugs introduced in point releases, we might want to
> limit what gets in.
>
> Would be nice to make sure a point release is more stable than a previous
> point release in that line.
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > > As I proposed in the other thread, how about we adopting the following
> > > model:
> > >
> > > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> > > next minor release.
> > > x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> > > minor release.
> > > x.y.3 releases have all Blocker bug fixes applied to next minor
> release.
> > >
> > > Here I am assuming there are no security-fix-only or other urgent
> > releases.
> > >
> > > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > > release.
> > >
> > > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@hortonworks.com> wrote:
> > >
> > > > Yeah, I started a thread while back on this one (
> > > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > > discussions re 2.6.1.
> > > >
> > > > The biggest problem I found offline was about what bug-fixes are
> > > > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> > > the
> > > > number of bug-fixes that went into 2.7.x and into branch-2.8,
> figuring
> > > out
> > > > a set of patches that is acceptable for everyone is a huge challenge
> > > which
> > > > kind of stalled my attempts.
> > > >
> > > > Thanks
> > > > +Vinod
> > > >
> > > >
> > > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > > > >
> > > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > > trying
> > > > to
> > > > > get that effort going but it's been stalled a little bit. It would
> be
> > > > good
> > > > > to rekindle that effort.
> > > > >
> > > > > Companies with big hadoop 2.x deployments (including mine) have
> > always
> > > > > tried to stabilize a 2.x release by testing/collecting/researching
> > > > critical
> > > > > issues on the release. Each would come up with its own set of fixes
> > to
> > > > > backport. We would also communicate it via offline channels. During
> > the
> > > > > hadoop summit, we thought it would be great if we all came together
> > and
> > > > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> > > for
> > > > > example) with all the critical issues fixed.
> > > > >
> > > > > Thanks,
> > > > > Sangjin
> > > > >
> > > > >
> > > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org
> >
> > > > wrote:
> > > > >
> > > > >> Thank you for the notification. Trying to back port bug fixes.
> > > > >>
> > > > >> - Tsuyoshi
> > > > >>
> > > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com
> >
> > > > wrote:
> > > > >>> Hi Hadoopers!
> > > > >>>
> > > > >>> Over in HBase we've been discussing the impact of our
> dependencies
> > on
> > > > our
> > > > >>> downstream users. As our most fundamental dependency, Hadoop
> plays
> > a
> > > > big
> > > > >>> role in the operational cost of running an HBase instance.
> > > > >>>
> > > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
> and
> > > > >> 2.6[1].
> > > > >>> We don't drop Hadoop minor release lines in minor releases so we
> > are
> > > > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> > > the
> > > > >> end
> > > > >>> of 2015 / start of 2016 (and currently we plan to continue
> > supporting
> > > > at
> > > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> > > our
> > > > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> > > > part
> > > > >> of
> > > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> > of
> > > > bugs
> > > > >>> that could destroy HBase clusters should users decide to turn on
> > HDFS
> > > > >>> encryption[4]. Our installation instructions tell folks to
> replace
> > > > these
> > > > >>> jars with the version of Hadoop they are actually running, but
> not
> > > all
> > > > >>> users follow those instructions so we want to minimize the pain
> for
> > > > them.
> > > > >>>
> > > > >>> Regular maintenance releases are key to keeping operational
> burdens
> > > low
> > > > >> for
> > > > >>> our downstream users; we don't want them to be forced to choose
> > > between
> > > > >>> living with broken systems and stomaching the risk of upgrades
> > across
> > > > >>> minor/major version numbers. Looking back over the three
> > > aforementioned
> > > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> > out
> > > in
> > > > >> Nov
> > > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> > > to
> > > > >> be a
> > > > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> > > > >>> binaries, one of your PMC members mentioned that with continued
> > work
> > > on
> > > > >> the
> > > > >>> 2.7 line y'all weren't planning any additional releases of the
> > > earlier
> > > > >>> minor versions[6].
> > > > >>>
> > > > >>> The HBase community requests that Hadoop pick up making
> > bug-fix-only
> > > > >> patch
> > > > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> > > and
> > > > >>> preferably monthly. We realize that given the time gap since
> 2.6.0
> > it
> > > > >> will
> > > > >>> likely take a big to get 2.6.1 together, but after that it should
> > > take
> > > > >> much
> > > > >>> less effort to continue.
> > > > >>>
> > > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > > >>> [2]: http://s.apache.org/ReP
> > > > >>> [3]: HBASE-13339
> > > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > > >>> [5]: http://hadoop.apache.org/releases.html
> > > > >>> [6]: http://s.apache.org/MTY
> > > > >>> [7]: http://s.apache.org/ViP
> > > > >>>
> > > > >>> --
> > > > >>> Sean
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Karthik Kambatla
> > > Software Engineer, Cloudera Inc.
> > > --------------------------------------------
> > > http://five.sentenc.es
> > >
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

Every new patch potentially brings in new bugs. So, if we want to limit the
kinds of potential bugs introduced in point releases, we might want to
limit what gets in.

Would be nice to make sure a point release is more stable than a previous
point release in that line.

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:

> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > As I proposed in the other thread, how about we adopting the following
> > model:
> >
> > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
> > next minor release.
> > x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> > minor release.
> > x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >
> > Here I am assuming there are no security-fix-only or other urgent
> releases.
> >
> > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > release.
> >
> > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > vinodkv@hortonworks.com> wrote:
> >
> > > Yeah, I started a thread while back on this one (
> > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > discussions re 2.6.1.
> > >
> > > The biggest problem I found offline was about what bug-fixes are
> > > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
> > the
> > > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> > out
> > > a set of patches that is acceptable for everyone is a huge challenge
> > which
> > > kind of stalled my attempts.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > > >
> > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > trying
> > > to
> > > > get that effort going but it's been stalled a little bit. It would be
> > > good
> > > > to rekindle that effort.
> > > >
> > > > Companies with big hadoop 2.x deployments (including mine) have
> always
> > > > tried to stabilize a 2.x release by testing/collecting/researching
> > > critical
> > > > issues on the release. Each would come up with its own set of fixes
> to
> > > > backport. We would also communicate it via offline channels. During
> the
> > > > hadoop summit, we thought it would be great if we all came together
> and
> > > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
> > for
> > > > example) with all the critical issues fixed.
> > > >
> > > > Thanks,
> > > > Sangjin
> > > >
> > > >
> > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> > > wrote:
> > > >
> > > >> Thank you for the notification. Trying to back port bug fixes.
> > > >>
> > > >> - Tsuyoshi
> > > >>
> > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > > wrote:
> > > >>> Hi Hadoopers!
> > > >>>
> > > >>> Over in HBase we've been discussing the impact of our dependencies
> on
> > > our
> > > >>> downstream users. As our most fundamental dependency, Hadoop plays
> a
> > > big
> > > >>> role in the operational cost of running an HBase instance.
> > > >>>
> > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > > >> 2.6[1].
> > > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> > > >>> unlikely remove anything from this set until HBase 2.0, probably at
> > the
> > > >> end
> > > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> > > at
> > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> > our
> > > >>> shipped binaries to Hadoop 2.6, following some stability testing by
> > > part
> > > >> of
> > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> > > bugs
> > > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> > > >>> encryption[4]. Our installation instructions tell folks to replace
> > > these
> > > >>> jars with the version of Hadoop they are actually running, but not
> > all
> > > >>> users follow those instructions so we want to minimize the pain for
> > > them.
> > > >>>
> > > >>> Regular maintenance releases are key to keeping operational burdens
> > low
> > > >> for
> > > >>> our downstream users; we don't want them to be forced to choose
> > between
> > > >>> living with broken systems and stomaching the risk of upgrades
> across
> > > >>> minor/major version numbers. Looking back over the three
> > aforementioned
> > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> > in
> > > >> Nov
> > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> > to
> > > >> be a
> > > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > >>> binaries, one of your PMC members mentioned that with continued
> work
> > on
> > > >> the
> > > >>> 2.7 line y'all weren't planning any additional releases of the
> > earlier
> > > >>> minor versions[6].
> > > >>>
> > > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> > > >> patch
> > > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
> > and
> > > >>> preferably monthly. We realize that given the time gap since 2.6.0
> it
> > > >> will
> > > >>> likely take a big to get 2.6.1 together, but after that it should
> > take
> > > >> much
> > > >>> less effort to continue.
> > > >>>
> > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > >>> [2]: http://s.apache.org/ReP
> > > >>> [3]: HBASE-13339
> > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > >>> [5]: http://hadoop.apache.org/releases.html
> > > >>> [6]: http://s.apache.org/MTY
> > > >>> [7]: http://s.apache.org/ViP
> > > >>>
> > > >>> --
> > > >>> Sean
> > > >>
> > >
> > >
> >
> >
> > --
> > Karthik Kambatla
> > Software Engineer, Cloudera Inc.
> > --------------------------------------------
> > http://five.sentenc.es
> >
>
>
>
> --
> Sean
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Andrew Purtell <ap...@apache.org>]

Over on HBase, committers volunteer to be release runners for a whole
release line. I wouldn't use the word 'appoint' necessarily because the
arrangement is an informal communal practice, not written down anywhere as
policy or codified into bylaws.

If it is helpful to have a data point from another community, I am the RM
for the 0.98.x line of HBase releases. I think it helps us that one
committer and PMC is able to specialize on a particular release line. I
know the back-porting nits in and out. Back port decisions are predicated
on user demand. Sometimes users ask for changes to come back. Sometimes I
will find something relevant - especially bug fixes, or solutions for
operational warts - and proactively pick it back (I run 0.98 in
production). Sometimes contributions are from users or devs running 0.98
and of course they need their fixes to ultimately arrive in the next 0.98
release. I help out with that process where possible. I may spend a couple
of days every couple of weeks carefully picking back and porting changes.
Every month I spin a new release candidate and ask our PMC to judge it
worthy.

We have other volunteer RMs for 1.0.x, 1.1.x, and 1.2.x. This of course
doesn't infinitely scale. It could be argued as a good side effect we won't
have many active release lines, only what we are capable of focusing on as
a PMC. For example, should I want to throw my hat in the ring for 1.3.x,
when the time arrives, I may ask for consensus on retiring 0.98 first.


On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org> wrote:

> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> >> As I proposed in the other thread, how about we adopting the following
> >> model:
> >>
> >> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> >> next minor release.
> >> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> >> minor release.
> >> x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >>
> >> Here I am assuming there are no security-fix-only or other urgent
> releases.
> >>
> >> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> >> release.
> >>
> >> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> >> vinodkv@hortonworks.com> wrote:
> >>
> >> > Yeah, I started a thread while back on this one (
> >> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> >> > discussions re 2.6.1.
> >> >
> >> > The biggest problem I found offline was about what bug-fixes are
> >> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> >> the
> >> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> >> out
> >> > a set of patches that is acceptable for everyone is a huge challenge
> >> which
> >> > kind of stalled my attempts.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> >
> >> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >> > >
> >> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> >> trying
> >> > to
> >> > > get that effort going but it's been stalled a little bit. It would
> be
> >> > good
> >> > > to rekindle that effort.
> >> > >
> >> > > Companies with big hadoop 2.x deployments (including mine) have
> always
> >> > > tried to stabilize a 2.x release by testing/collecting/researching
> >> > critical
> >> > > issues on the release. Each would come up with its own set of fixes
> to
> >> > > backport. We would also communicate it via offline channels. During
> the
> >> > > hadoop summit, we thought it would be great if we all came together
> and
> >> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> >> for
> >> > > example) with all the critical issues fixed.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> > >
> >> > >
> >> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> >> > wrote:
> >> > >
> >> > >> Thank you for the notification. Trying to back port bug fixes.
> >> > >>
> >> > >> - Tsuyoshi
> >> > >>
> >> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > >>> Hi Hadoopers!
> >> > >>>
> >> > >>> Over in HBase we've been discussing the impact of our
> dependencies on
> >> > our
> >> > >>> downstream users. As our most fundamental dependency, Hadoop
> plays a
> >> > big
> >> > >>> role in the operational cost of running an HBase instance.
> >> > >>>
> >> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> > >> 2.6[1].
> >> > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> >> > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> >> the
> >> > >> end
> >> > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> >> > at
> >> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> >> our
> >> > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> >> > part
> >> > >> of
> >> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> >> > bugs
> >> > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> >> > >>> encryption[4]. Our installation instructions tell folks to replace
> >> > these
> >> > >>> jars with the version of Hadoop they are actually running, but not
> >> all
> >> > >>> users follow those instructions so we want to minimize the pain
> for
> >> > them.
> >> > >>>
> >> > >>> Regular maintenance releases are key to keeping operational
> burdens
> >> low
> >> > >> for
> >> > >>> our downstream users; we don't want them to be forced to choose
> >> between
> >> > >>> living with broken systems and stomaching the risk of upgrades
> across
> >> > >>> minor/major version numbers. Looking back over the three
> >> aforementioned
> >> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> >> in
> >> > >> Nov
> >> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> >> to
> >> > >> be a
> >> > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> >> > >>> binaries, one of your PMC members mentioned that with continued
> work
> >> on
> >> > >> the
> >> > >>> 2.7 line y'all weren't planning any additional releases of the
> >> earlier
> >> > >>> minor versions[6].
> >> > >>>
> >> > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> >> > >> patch
> >> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> >> and
> >> > >>> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> >> > >> will
> >> > >>> likely take a big to get 2.6.1 together, but after that it should
> >> take
> >> > >> much
> >> > >>> less effort to continue.
> >> > >>>
> >> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> >> > >>> [2]: http://s.apache.org/ReP
> >> > >>> [3]: HBASE-13339
> >> > >>> [4]: HADOOP-11674 and HADOOP-11710
> >> > >>> [5]: http://hadoop.apache.org/releases.html
> >> > >>> [6]: http://s.apache.org/MTY
> >> > >>> [7]: http://s.apache.org/ViP
> >> > >>>
> >> > >>> --
> >> > >>> Sean
> >> > >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Karthik Kambatla
> >> Software Engineer, Cloudera Inc.
> >> --------------------------------------------
> >> http://five.sentenc.es
> >>
> >
> >
> >
> > --
> > Sean
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Andrew Purtell <ap...@apache.org>]

Over on HBase, committers volunteer to be release runners for a whole
release line. I wouldn't use the word 'appoint' necessarily because the
arrangement is an informal communal practice, not written down anywhere as
policy or codified into bylaws.

If it is helpful to have a data point from another community, I am the RM
for the 0.98.x line of HBase releases. I think it helps us that one
committer and PMC is able to specialize on a particular release line. I
know the back-porting nits in and out. Back port decisions are predicated
on user demand. Sometimes users ask for changes to come back. Sometimes I
will find something relevant - especially bug fixes, or solutions for
operational warts - and proactively pick it back (I run 0.98 in
production). Sometimes contributions are from users or devs running 0.98
and of course they need their fixes to ultimately arrive in the next 0.98
release. I help out with that process where possible. I may spend a couple
of days every couple of weeks carefully picking back and porting changes.
Every month I spin a new release candidate and ask our PMC to judge it
worthy.

We have other volunteer RMs for 1.0.x, 1.1.x, and 1.2.x. This of course
doesn't infinitely scale. It could be argued as a good side effect we won't
have many active release lines, only what we are capable of focusing on as
a PMC. For example, should I want to throw my hat in the ring for 1.3.x,
when the time arrives, I may ask for consensus on retiring 0.98 first.


On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org> wrote:

> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> >> As I proposed in the other thread, how about we adopting the following
> >> model:
> >>
> >> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> >> next minor release.
> >> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> >> minor release.
> >> x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >>
> >> Here I am assuming there are no security-fix-only or other urgent
> releases.
> >>
> >> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> >> release.
> >>
> >> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> >> vinodkv@hortonworks.com> wrote:
> >>
> >> > Yeah, I started a thread while back on this one (
> >> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> >> > discussions re 2.6.1.
> >> >
> >> > The biggest problem I found offline was about what bug-fixes are
> >> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> >> the
> >> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> >> out
> >> > a set of patches that is acceptable for everyone is a huge challenge
> >> which
> >> > kind of stalled my attempts.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> >
> >> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >> > >
> >> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> >> trying
> >> > to
> >> > > get that effort going but it's been stalled a little bit. It would
> be
> >> > good
> >> > > to rekindle that effort.
> >> > >
> >> > > Companies with big hadoop 2.x deployments (including mine) have
> always
> >> > > tried to stabilize a 2.x release by testing/collecting/researching
> >> > critical
> >> > > issues on the release. Each would come up with its own set of fixes
> to
> >> > > backport. We would also communicate it via offline channels. During
> the
> >> > > hadoop summit, we thought it would be great if we all came together
> and
> >> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> >> for
> >> > > example) with all the critical issues fixed.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> > >
> >> > >
> >> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> >> > wrote:
> >> > >
> >> > >> Thank you for the notification. Trying to back port bug fixes.
> >> > >>
> >> > >> - Tsuyoshi
> >> > >>
> >> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > >>> Hi Hadoopers!
> >> > >>>
> >> > >>> Over in HBase we've been discussing the impact of our
> dependencies on
> >> > our
> >> > >>> downstream users. As our most fundamental dependency, Hadoop
> plays a
> >> > big
> >> > >>> role in the operational cost of running an HBase instance.
> >> > >>>
> >> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> > >> 2.6[1].
> >> > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> >> > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> >> the
> >> > >> end
> >> > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> >> > at
> >> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> >> our
> >> > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> >> > part
> >> > >> of
> >> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> >> > bugs
> >> > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> >> > >>> encryption[4]. Our installation instructions tell folks to replace
> >> > these
> >> > >>> jars with the version of Hadoop they are actually running, but not
> >> all
> >> > >>> users follow those instructions so we want to minimize the pain
> for
> >> > them.
> >> > >>>
> >> > >>> Regular maintenance releases are key to keeping operational
> burdens
> >> low
> >> > >> for
> >> > >>> our downstream users; we don't want them to be forced to choose
> >> between
> >> > >>> living with broken systems and stomaching the risk of upgrades
> across
> >> > >>> minor/major version numbers. Looking back over the three
> >> aforementioned
> >> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> >> in
> >> > >> Nov
> >> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> >> to
> >> > >> be a
> >> > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> >> > >>> binaries, one of your PMC members mentioned that with continued
> work
> >> on
> >> > >> the
> >> > >>> 2.7 line y'all weren't planning any additional releases of the
> >> earlier
> >> > >>> minor versions[6].
> >> > >>>
> >> > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> >> > >> patch
> >> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> >> and
> >> > >>> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> >> > >> will
> >> > >>> likely take a big to get 2.6.1 together, but after that it should
> >> take
> >> > >> much
> >> > >>> less effort to continue.
> >> > >>>
> >> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> >> > >>> [2]: http://s.apache.org/ReP
> >> > >>> [3]: HBASE-13339
> >> > >>> [4]: HADOOP-11674 and HADOOP-11710
> >> > >>> [5]: http://hadoop.apache.org/releases.html
> >> > >>> [6]: http://s.apache.org/MTY
> >> > >>> [7]: http://s.apache.org/ViP
> >> > >>>
> >> > >>> --
> >> > >>> Sean
> >> > >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Karthik Kambatla
> >> Software Engineer, Cloudera Inc.
> >> --------------------------------------------
> >> http://five.sentenc.es
> >>
> >
> >
> >
> > --
> > Sean
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

+1 Chris is right.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++





-----Original Message-----
From: Chris Douglas <cd...@apache.org>
Reply-To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Date: Wednesday, July 15, 2015 at 3:07 PM
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Cc: dev <de...@hbase.apache.org>
Subject: Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y
versions

>On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>> Alternatively, why not appoint a Release Manager for the minor release
>>line
>> and then allow them to arbitrate when there's disagreement about
>>inclusion?
>> This has worked well in the HBase community.
>
>
>Release managers aren't appointed in Hadoop. Any committer can RM a
>release branch and encourage others to help with it. An RM can set the
>bar arbitrarily, but an RC only becomes a release when a majority of
>PMC votes approve it in a VOTE. -C
>
>On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>> Why not just include all backwards compatible bug fixes?
>>
>> Alternatively, why not appoint a Release Manager for the minor release
>>line
>> and then allow them to arbitrate when there's disagreement about
>>inclusion?
>> This has worked well in the HBase community.
>>
>> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> As I proposed in the other thread, how about we adopting the following
>>> model:
>>>
>>> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
>>>the
>>> next minor release.
>>> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
>>> minor release.
>>> x.y.3 releases have all Blocker bug fixes applied to next minor
>>>release.
>>>
>>> Here I am assuming there are no security-fix-only or other urgent
>>>releases.
>>>
>>> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
>>> release.
>>>
>>> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>> > Yeah, I started a thread while back on this one (
>>> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
>>> > discussions re 2.6.1.
>>> >
>>> > The biggest problem I found offline was about what bug-fixes are
>>> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
>>>Given
>>> the
>>> > number of bug-fixes that went into 2.7.x and into branch-2.8,
>>>figuring
>>> out
>>> > a set of patches that is acceptable for everyone is a huge challenge
>>> which
>>> > kind of stalled my attempts.
>>> >
>>> > Thanks
>>> > +Vinod
>>> >
>>> >
>>> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>>> > >
>>> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
>>> trying
>>> > to
>>> > > get that effort going but it's been stalled a little bit. It would
>>>be
>>> > good
>>> > > to rekindle that effort.
>>> > >
>>> > > Companies with big hadoop 2.x deployments (including mine) have
>>>always
>>> > > tried to stabilize a 2.x release by testing/collecting/researching
>>> > critical
>>> > > issues on the release. Each would come up with its own set of
>>>fixes to
>>> > > backport. We would also communicate it via offline channels.
>>>During the
>>> > > hadoop summit, we thought it would be great if we all came
>>>together and
>>> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
>>>2.6
>>> for
>>> > > example) with all the critical issues fixed.
>>> > >
>>> > > Thanks,
>>> > > Sangjin
>>> > >
>>> > >
>>> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
>>> > wrote:
>>> > >
>>> > >> Thank you for the notification. Trying to back port bug fixes.
>>> > >>
>>> > >> - Tsuyoshi
>>> > >>
>>> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
>>> > wrote:
>>> > >>> Hi Hadoopers!
>>> > >>>
>>> > >>> Over in HBase we've been discussing the impact of our
>>>dependencies on
>>> > our
>>> > >>> downstream users. As our most fundamental dependency, Hadoop
>>>plays a
>>> > big
>>> > >>> role in the operational cost of running an HBase instance.
>>> > >>>
>>> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
>>>and
>>> > >> 2.6[1].
>>> > >>> We don't drop Hadoop minor release lines in minor releases so we
>>>are
>>> > >>> unlikely remove anything from this set until HBase 2.0, probably
>>>at
>>> the
>>> > >> end
>>> > >>> of 2015 / start of 2016 (and currently we plan to continue
>>>supporting
>>> > at
>>> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
>>>updating
>>> our
>>> > >>> shipped binaries to Hadoop 2.6, following some stability testing
>>>by
>>> > part
>>> > >> of
>>> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a
>>>couple of
>>> > bugs
>>> > >>> that could destroy HBase clusters should users decide to turn on
>>>HDFS
>>> > >>> encryption[4]. Our installation instructions tell folks to
>>>replace
>>> > these
>>> > >>> jars with the version of Hadoop they are actually running, but
>>>not
>>> all
>>> > >>> users follow those instructions so we want to minimize the pain
>>>for
>>> > them.
>>> > >>>
>>> > >>> Regular maintenance releases are key to keeping operational
>>>burdens
>>> low
>>> > >> for
>>> > >>> our downstream users; we don't want them to be forced to choose
>>> between
>>> > >>> living with broken systems and stomaching the risk of upgrades
>>>across
>>> > >>> minor/major version numbers. Looking back over the three
>>> aforementioned
>>> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
>>>out
>>> in
>>> > >> Nov
>>> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
>>>looks
>>> to
>>> > >> be a
>>> > >>> year without a release[5]. On our discussion of shipping Hadoop
>>>2.6
>>> > >>> binaries, one of your PMC members mentioned that with continued
>>>work
>>> on
>>> > >> the
>>> > >>> 2.7 line y'all weren't planning any additional releases of the
>>> earlier
>>> > >>> minor versions[6].
>>> > >>>
>>> > >>> The HBase community requests that Hadoop pick up making
>>>bug-fix-only
>>> > >> patch
>>> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
>>>line
>>> and
>>> > >>> preferably monthly. We realize that given the time gap since
>>>2.6.0 it
>>> > >> will
>>> > >>> likely take a big to get 2.6.1 together, but after that it should
>>> take
>>> > >> much
>>> > >>> less effort to continue.
>>> > >>>
>>> > >>> [1]: http://hbase.apache.org/book.html#hadoop
>>> > >>> [2]: http://s.apache.org/ReP
>>> > >>> [3]: HBASE-13339
>>> > >>> [4]: HADOOP-11674 and HADOOP-11710
>>> > >>> [5]: http://hadoop.apache.org/releases.html
>>> > >>> [6]: http://s.apache.org/MTY
>>> > >>> [7]: http://s.apache.org/ViP
>>> > >>>
>>> > >>> --
>>> > >>> Sean
>>> > >>
>>> >
>>> >
>>>
>>>
>>> --
>>> Karthik Kambatla
>>> Software Engineer, Cloudera Inc.
>>> --------------------------------------------
>>> http://five.sentenc.es
>>>
>>
>>
>>
>> --
>> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinayakumar B <vi...@apache.org>]

+1 for steve's suggestion.
I agree completely that everything should be finalized before committing in.

-Vinay
On Jul 16, 2015 2:29 PM, "Steve Loughran" <st...@hortonworks.com> wrote:

>
> 1. I agree that the bar for patches going in should be very high: there's
> always the risk of some subtle regression. The more patches, the higher the
> risk, the more traumatic the update
>
> 2. I like the idea of having a list of proposed candidate patches, all of
> which can be reviewed and discussed before going in.
>
> > On 16 Jul 2015, at 02:43, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
> >
> >
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>
>
> Link is
> https://issues.apache.org/jira/browse/YARN-3575?jql=labels%20%3D%202.6.1-candidate
>
> 3. Maybe we should have some guidelines of what isn't going to get in
> except in very, very special cases
>
> -any change to classpath/dependencies
> -any change to the signature of an API, including exception types & text
> -changes to wire formats
>
> 4. We could also consider driving patches based on those that downstream
> redistributors of Hadoop felt were important enough to backport. That's
> cloudera as well as us, Amazon if they filed JIRAs, Microsoft, + others.
> Ideally patches that have been tested and released, so there's a high
> chance regressions would have surfaced already.
>
> 5. Then there's the "these broke HBase changes"; vinod already has
> HADOOP-11710 in there, as an example.
>
> 6. And of course, any security issue patch should go in.
>
> Overall then: the expectation should be that patches won't go in by
> default, unless viewed as critical. We have to be ruthless, and people
> shouldn't commit things without getting approval from others.
>
> -Steve
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

On Fri, Jul 17, 2015 at 6:04 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> > On 16 Jul 2015, at 15:56, Karthik Kambatla <ka...@cloudera.com> wrote:
> >
> > On Thu, Jul 16, 2015 at 10:42 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> >> On Thu, Jul 16, 2015 at 9:17 AM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> On Thu, Jul 16, 2015 at 4:59 AM, Steve Loughran <
> stevel@hortonworks.com>
> >>> wrote:
> >>>
> >>>
> >>>> -any change to the signature of an API, including exception types &
> >> text
> >>>> -changes to wire formats
> >>>>
> >>>
> >>> These two should hold for minor releases also, no?
> >>>
> >>>
> >> At the risk of derailing this thread, no definitely not. "any change"
> would
> >> include backwards compatible additions / changes. Using this stricter
> >> restriction is great for patch releases, since it means that a user can
> >> safely move onto a newer patch release with the assurance that if some
> >> regression should show up they can move back to an earlier patch release
> >> without risk that changes in their application since upgrading won't
> work
> >> due to reliance on an addition.
> >>
> >
> > I am not sure I understand the need for restriction for source and binary
> > backwards-compatible API changes.
>
>
> Here's an example: YARN-3477, TimelineClientImpl swallows exceptions
>
> https://issues.apache.org/jira/browse/YARN-3477
>
> I want to change it so that when retries time out, it rethrows the
> original exception, and convert InterruptedException to
> InterruptedIOException then throw
>
> These don't change the explicit binary signature of the code, but they do
> change what gets thrown on a timeout, as well as the text in the message.
>
> I'd rate that as a -1 for a 2.6.x patch, or 2.7.x, but happy to put into
> 2.8
>

I now see the distinction. Do we want to add this to our compatibility
guide? The more up to date that is, the higher the chance of we actually
sticking to it. :)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Steve Loughran <st...@hortonworks.com>]

> On 16 Jul 2015, at 15:56, Karthik Kambatla <ka...@cloudera.com> wrote:
> 
> On Thu, Jul 16, 2015 at 10:42 AM, Sean Busbey <bu...@cloudera.com> wrote:
> 
>> On Thu, Jul 16, 2015 at 9:17 AM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>> 
>>> On Thu, Jul 16, 2015 at 4:59 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>> 
>>> 
>>>> -any change to the signature of an API, including exception types &
>> text
>>>> -changes to wire formats
>>>> 
>>> 
>>> These two should hold for minor releases also, no?
>>> 
>>> 
>> At the risk of derailing this thread, no definitely not. "any change" would
>> include backwards compatible additions / changes. Using this stricter
>> restriction is great for patch releases, since it means that a user can
>> safely move onto a newer patch release with the assurance that if some
>> regression should show up they can move back to an earlier patch release
>> without risk that changes in their application since upgrading won't work
>> due to reliance on an addition.
>> 
> 
> I am not sure I understand the need for restriction for source and binary
> backwards-compatible API changes.


Here's an example: YARN-3477, TimelineClientImpl swallows exceptions

https://issues.apache.org/jira/browse/YARN-3477

I want to change it so that when retries time out, it rethrows the original exception, and convert InterruptedException to InterruptedIOException then throw

These don't change the explicit binary signature of the code, but they do change what gets thrown on a timeout, as well as the text in the message.

I'd rate that as a -1 for a 2.6.x patch, or 2.7.x, but happy to put into 2.8

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

On Thu, Jul 16, 2015 at 10:42 AM, Sean Busbey <bu...@cloudera.com> wrote:

> On Thu, Jul 16, 2015 at 9:17 AM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > On Thu, Jul 16, 2015 at 4:59 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> >
> > > -any change to the signature of an API, including exception types &
> text
> > > -changes to wire formats
> > >
> >
> > These two should hold for minor releases also, no?
> >
> >
> At the risk of derailing this thread, no definitely not. "any change" would
> include backwards compatible additions / changes. Using this stricter
> restriction is great for patch releases, since it means that a user can
> safely move onto a newer patch release with the assurance that if some
> regression should show up they can move back to an earlier patch release
> without risk that changes in their application since upgrading won't work
> due to reliance on an addition.
>

I am not sure I understand the need for restriction for source and binary
backwards-compatible API changes.


>
>
> --
> Sean
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

On Thu, Jul 16, 2015 at 9:17 AM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> On Thu, Jul 16, 2015 at 4:59 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
> > -any change to the signature of an API, including exception types & text
> > -changes to wire formats
> >
>
> These two should hold for minor releases also, no?
>
>
At the risk of derailing this thread, no definitely not. "any change" would
include backwards compatible additions / changes. Using this stricter
restriction is great for patch releases, since it means that a user can
safely move onto a newer patch release with the assurance that if some
regression should show up they can move back to an earlier patch release
without risk that changes in their application since upgrading won't work
due to reliance on an addition.


-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

On Thu, Jul 16, 2015 at 4:59 AM, Steve Loughran <st...@hortonworks.com>
wrote:

>
> 1. I agree that the bar for patches going in should be very high: there's
> always the risk of some subtle regression. The more patches, the higher the
> risk, the more traumatic the update
>
> 2. I like the idea of having a list of proposed candidate patches, all of
> which can be reviewed and discussed before going in.
>

+1 from me too.


>
> > On 16 Jul 2015, at 02:43, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
> >
> >
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>
>
> Link is
> https://issues.apache.org/jira/browse/YARN-3575?jql=labels%20%3D%202.6.1-candidate
>
> 3. Maybe we should have some guidelines of what isn't going to get in
> except in very, very special cases
>
> -any change to classpath/dependencies
>

Sounds good. We should work towards making this a requirement within a
major release, at least for dependencies on the client side.


> -any change to the signature of an API, including exception types & text
> -changes to wire formats
>

These two should hold for minor releases also, no?


>
> 4. We could also consider driving patches based on those that downstream
> redistributors of Hadoop felt were important enough to backport. That's
> cloudera as well as us, Amazon if they filed JIRAs, Microsoft, + others.
> Ideally patches that have been tested and released, so there's a high
> chance regressions would have surfaced already.
>
> 5. Then there's the "these broke HBase changes"; vinod already has
> HADOOP-11710 in there, as an example.
>
> 6. And of course, any security issue patch should go in.
>
> Overall then: the expectation should be that patches won't go in by
> default, unless viewed as critical. We have to be ruthless, and people
> shouldn't commit things without getting approval from others.
>
> -Steve
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Steve Loughran <st...@hortonworks.com>]

1. I agree that the bar for patches going in should be very high: there's always the risk of some subtle regression. The more patches, the higher the risk, the more traumatic the update

2. I like the idea of having a list of proposed candidate patches, all of which can be reviewed and discussed before going in. 

> On 16 Jul 2015, at 02:43, Vinod Kumar Vavilapalli <vi...@hortonworks.com> wrote:
> 
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>


Link is https://issues.apache.org/jira/browse/YARN-3575?jql=labels%20%3D%202.6.1-candidate

3. Maybe we should have some guidelines of what isn't going to get in except in very, very special cases

-any change to classpath/dependencies
-any change to the signature of an API, including exception types & text
-changes to wire formats

4. We could also consider driving patches based on those that downstream redistributors of Hadoop felt were important enough to backport. That's cloudera as well as us, Amazon if they filed JIRAs, Microsoft, + others. Ideally patches that have been tested and released, so there's a high chance regressions would have surfaced already.

5. Then there's the "these broke HBase changes"; vinod already has HADOOP-11710 in there, as an example.

6. And of course, any security issue patch should go in.

Overall then: the expectation should be that patches won't go in by default, unless viewed as critical. We have to be ruthless, and people shouldn't commit things without getting approval from others.

-Steve

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

To close the loop on this one, I created a label for the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>, in order to make progress while the issue is still hot.

The discussion is continuing on the 2.6.1 release planning thread.

Thanks
+Vinod

On Jul 15, 2015, at 6:09 PM, Andrew Purtell <ap...@apache.org>> wrote:

Inline

On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

I can understand these (sort of newish) questions from hbase-dev. We
already have a well laid-out release-management process. If people want to
learn more about how it works, please head over to
http://hadoop.apache.org/bylaws.html.

In terms of 2.6.1 release management, it wasn’t stuck for lack of
volunteers for RM. I originally was driving it [1], and then Akira recently
volunteered too in a separate thread [2] (which I totally missed
participating in before), so we have enough help.


​Right, but in that thread it looks like you were stymied sorting through
the potential set of updates to apply to the patch release. That was the
motivation behind my sharing a bit about HBase branch RMs. Totally fine if
that's not something Hadoop wants to explore. I only offered it as a data
point where another project avoids that problem with a different kind of RM
focus. And, so, that's that FWIW




It is clearly acknowledged there is a demand for 2.6.1, we now have to
figure out the mechanics, agreeing on a reasonable & common list of fixes
to start with.


​Thank you very much for this consideration!​



Tx for the feedback so far @hbase-dev! I agree with Karthik earlier -
let’s continue the discussion in hadoop-dev in the release thread [1] for
2.6.1.

Thanks
+Vinod

[1] Planning Hadoop 2.6.1 release
http://markmail.org/message/sbykjn5xgnksh6wg
[2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR

On Jul 15, 2015, at 4:07 PM, Stack <stack@duboce.net<mailto:
stack@duboce.net>> wrote:

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cdouglas@apache.org
<ma...@apache.org>> wrote:

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>> wrote:
Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>> wrote:
Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <kasha@cloudera.com
<ma...@cloudera.com>>
wrote:

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent
releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

Yeah, I started a thread while back on this one (
http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are
acceptable and what aren’t for everyone wishing to consume 2.6.1.
Given
the
number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
out
a set of patches that is acceptable for everyone is a huge challenge
which
kind of stalled my attempts.

Thanks
+Vinod


On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sjlee0@gmail.com<mailto:
sjlee0@gmail.com>> wrote:

Strong +1 for having a 2.6.1 release. I understand Vinod has been
trying
to
get that effort going but it's been stalled a little bit. It would
be
good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have
always
tried to stabilize a 2.x release by testing/collecting/researching
critical
issues on the release. Each would come up with its own set of fixes
to
backport. We would also communicate it via offline channels. During
the
hadoop summit, we thought it would be great if we all came together
and
create a public stability/bugfix release on top of 2.x (2.6.1 for
2.6
for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org<mailto:
ozawa@apache.org>>
wrote:

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>>
wrote:
Hi Hadoopers!

Over in HBase we've been discussing the impact of our
dependencies on
our
downstream users. As our most fundamental dependency, Hadoop
plays a
big
role in the operational cost of running an HBase instance.

Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
2.6[1].
We don't drop Hadoop minor release lines in minor releases so we
are
unlikely remove anything from this set until HBase 2.0, probably
at
the
end
of 2015 / start of 2016 (and currently we plan to continue
supporting
at
least 2.4 for HBase 2.0 [2]). Lately we've been discussing
updating
our
shipped binaries to Hadoop 2.6, following some stability testing
by
part
of
our community[3]. Unfortunately, 2.6.0 in particular has a couple
of
bugs
that could destroy HBase clusters should users decide to turn on
HDFS
encryption[4]. Our installation instructions tell folks to replace
these
jars with the version of Hadoop they are actually running, but not
all
users follow those instructions so we want to minimize the pain
for
them.

Regular maintenance releases are key to keeping operational
burdens
low
for
our downstream users; we don't want them to be forced to choose
between
living with broken systems and stomaching the risk of upgrades
across
minor/major version numbers. Looking back over the three
aforementioned
Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
out
in
Nov
2014, when 2.5 had its last patch release as well. Hadoop 2.4
looks
to
be a
year without a release[5]. On our discussion of shipping Hadoop
2.6
binaries, one of your PMC members mentioned that with continued
work
on
the
2.7 line y'all weren't planning any additional releases of the
earlier
minor versions[6].

The HBase community requests that Hadoop pick up making
bug-fix-only
patch
releases again on a regular schedule[7]. Preferably on the 2.6
line
and
preferably monthly. We realize that given the time gap since
2.6.0 it
will
likely take a big to get 2.6.1 together, but after that it should
take
much
less effort to continue.

[1]: http://hbase.apache.org/book.html#hadoop
[2]: http://s.apache.org/ReP
[3]: HBASE-13339
[4]: HADOOP-11674 and HADOOP-11710
[5]: http://hadoop.apache.org/releases.html
[6]: http://s.apache.org/MTY
[7]: http://s.apache.org/ViP

--
Sean





--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es




--
Sean





--
Best regards,

  - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

To close the loop on this one, I created a label for the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>, in order to make progress while the issue is still hot.

The discussion is continuing on the 2.6.1 release planning thread.

Thanks
+Vinod

On Jul 15, 2015, at 6:09 PM, Andrew Purtell <ap...@apache.org>> wrote:

Inline

On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

I can understand these (sort of newish) questions from hbase-dev. We
already have a well laid-out release-management process. If people want to
learn more about how it works, please head over to
http://hadoop.apache.org/bylaws.html.

In terms of 2.6.1 release management, it wasn’t stuck for lack of
volunteers for RM. I originally was driving it [1], and then Akira recently
volunteered too in a separate thread [2] (which I totally missed
participating in before), so we have enough help.


​Right, but in that thread it looks like you were stymied sorting through
the potential set of updates to apply to the patch release. That was the
motivation behind my sharing a bit about HBase branch RMs. Totally fine if
that's not something Hadoop wants to explore. I only offered it as a data
point where another project avoids that problem with a different kind of RM
focus. And, so, that's that FWIW




It is clearly acknowledged there is a demand for 2.6.1, we now have to
figure out the mechanics, agreeing on a reasonable & common list of fixes
to start with.


​Thank you very much for this consideration!​



Tx for the feedback so far @hbase-dev! I agree with Karthik earlier -
let’s continue the discussion in hadoop-dev in the release thread [1] for
2.6.1.

Thanks
+Vinod

[1] Planning Hadoop 2.6.1 release
http://markmail.org/message/sbykjn5xgnksh6wg
[2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR

On Jul 15, 2015, at 4:07 PM, Stack <stack@duboce.net<mailto:
stack@duboce.net>> wrote:

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cdouglas@apache.org
<ma...@apache.org>> wrote:

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>> wrote:
Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>> wrote:
Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <kasha@cloudera.com
<ma...@cloudera.com>>
wrote:

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent
releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

Yeah, I started a thread while back on this one (
http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are
acceptable and what aren’t for everyone wishing to consume 2.6.1.
Given
the
number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
out
a set of patches that is acceptable for everyone is a huge challenge
which
kind of stalled my attempts.

Thanks
+Vinod


On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sjlee0@gmail.com<mailto:
sjlee0@gmail.com>> wrote:

Strong +1 for having a 2.6.1 release. I understand Vinod has been
trying
to
get that effort going but it's been stalled a little bit. It would
be
good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have
always
tried to stabilize a 2.x release by testing/collecting/researching
critical
issues on the release. Each would come up with its own set of fixes
to
backport. We would also communicate it via offline channels. During
the
hadoop summit, we thought it would be great if we all came together
and
create a public stability/bugfix release on top of 2.x (2.6.1 for
2.6
for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org<mailto:
ozawa@apache.org>>
wrote:

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com<mailto:
busbey@cloudera.com>>
wrote:
Hi Hadoopers!

Over in HBase we've been discussing the impact of our
dependencies on
our
downstream users. As our most fundamental dependency, Hadoop
plays a
big
role in the operational cost of running an HBase instance.

Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
2.6[1].
We don't drop Hadoop minor release lines in minor releases so we
are
unlikely remove anything from this set until HBase 2.0, probably
at
the
end
of 2015 / start of 2016 (and currently we plan to continue
supporting
at
least 2.4 for HBase 2.0 [2]). Lately we've been discussing
updating
our
shipped binaries to Hadoop 2.6, following some stability testing
by
part
of
our community[3]. Unfortunately, 2.6.0 in particular has a couple
of
bugs
that could destroy HBase clusters should users decide to turn on
HDFS
encryption[4]. Our installation instructions tell folks to replace
these
jars with the version of Hadoop they are actually running, but not
all
users follow those instructions so we want to minimize the pain
for
them.

Regular maintenance releases are key to keeping operational
burdens
low
for
our downstream users; we don't want them to be forced to choose
between
living with broken systems and stomaching the risk of upgrades
across
minor/major version numbers. Looking back over the three
aforementioned
Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
out
in
Nov
2014, when 2.5 had its last patch release as well. Hadoop 2.4
looks
to
be a
year without a release[5]. On our discussion of shipping Hadoop
2.6
binaries, one of your PMC members mentioned that with continued
work
on
the
2.7 line y'all weren't planning any additional releases of the
earlier
minor versions[6].

The HBase community requests that Hadoop pick up making
bug-fix-only
patch
releases again on a regular schedule[7]. Preferably on the 2.6
line
and
preferably monthly. We realize that given the time gap since
2.6.0 it
will
likely take a big to get 2.6.1 together, but after that it should
take
much
less effort to continue.

[1]: http://hbase.apache.org/book.html#hadoop
[2]: http://s.apache.org/ReP
[3]: HBASE-13339
[4]: HADOOP-11674 and HADOOP-11710
[5]: http://hadoop.apache.org/releases.html
[6]: http://s.apache.org/MTY
[7]: http://s.apache.org/ViP

--
Sean





--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es




--
Sean





--
Best regards,

  - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Andrew Purtell <ap...@apache.org>]

Inline

On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> I can understand these (sort of newish) questions from hbase-dev. We
> already have a well laid-out release-management process. If people want to
> learn more about how it works, please head over to
> http://hadoop.apache.org/bylaws.html.
>
> In terms of 2.6.1 release management, it wasn’t stuck for lack of
> volunteers for RM. I originally was driving it [1], and then Akira recently
> volunteered too in a separate thread [2] (which I totally missed
> participating in before), so we have enough help.
>

​Right, but in that thread it looks like you were stymied sorting through
the potential set of updates to apply to the patch release. That was the
motivation behind my sharing a bit about HBase branch RMs. Totally fine if
that's not something Hadoop wants to explore. I only offered it as a data
point where another project avoids that problem with a different kind of RM
focus. And, so, that's that FWIW



>
> It is clearly acknowledged there is a demand for 2.6.1, we now have to
> figure out the mechanics, agreeing on a reasonable & common list of fixes
> to start with.
>
>
​Thank you very much for this consideration!​



> Tx for the feedback so far @hbase-dev! I agree with Karthik earlier -
> let’s continue the discussion in hadoop-dev in the release thread [1] for
> 2.6.1.
>
> Thanks
> +Vinod
>
> [1] Planning Hadoop 2.6.1 release
> http://markmail.org/message/sbykjn5xgnksh6wg
> [2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR
>
> On Jul 15, 2015, at 4:07 PM, Stack <stack@duboce.net<mailto:
> stack@duboce.net>> wrote:
>
> Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
> You'd get some help (especially if the bar is set high and only critical
> bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
> updates, and so on).
>
> St.Ack
>
> On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cdouglas@apache.org
> <ma...@apache.org>> wrote:
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>> wrote:
> Alternatively, why not appoint a Release Manager for the minor release
> line
> and then allow them to arbitrate when there's disagreement about
> inclusion?
> This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>> wrote:
> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release
> line
> and then allow them to arbitrate when there's disagreement about
> inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <kasha@cloudera.com
> <ma...@cloudera.com>>
> wrote:
>
> As I proposed in the other thread, how about we adopting the following
> model:
>
> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> next minor release.
> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> minor release.
> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>
> Here I am assuming there are no security-fix-only or other urgent
> releases.
>
> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> release.
>
> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:
>
> Yeah, I started a thread while back on this one (
> http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> discussions re 2.6.1.
>
> The biggest problem I found offline was about what bug-fixes are
> acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> the
> number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> out
> a set of patches that is acceptable for everyone is a huge challenge
> which
> kind of stalled my attempts.
>
> Thanks
> +Vinod
>
>
> On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sjlee0@gmail.com<mailto:
> sjlee0@gmail.com>> wrote:
>
> Strong +1 for having a 2.6.1 release. I understand Vinod has been
> trying
> to
> get that effort going but it's been stalled a little bit. It would
> be
> good
> to rekindle that effort.
>
> Companies with big hadoop 2.x deployments (including mine) have
> always
> tried to stabilize a 2.x release by testing/collecting/researching
> critical
> issues on the release. Each would come up with its own set of fixes
> to
> backport. We would also communicate it via offline channels. During
> the
> hadoop summit, we thought it would be great if we all came together
> and
> create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> for
> example) with all the critical issues fixed.
>
> Thanks,
> Sangjin
>
>
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org<mailto:
> ozawa@apache.org>>
> wrote:
>
> Thank you for the notification. Trying to back port bug fixes.
>
> - Tsuyoshi
>
> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>>
> wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our
> dependencies on
> our
> downstream users. As our most fundamental dependency, Hadoop
> plays a
> big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we
> are
> unlikely remove anything from this set until HBase 2.0, probably
> at
> the
> end
> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> our
> shipped binaries to Hadoop 2.6, following some stability testing
> by
> part
> of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> bugs
> that could destroy HBase clusters should users decide to turn on
> HDFS
> encryption[4]. Our installation instructions tell folks to replace
> these
> jars with the version of Hadoop they are actually running, but not
> all
> users follow those instructions so we want to minimize the pain
> for
> them.
>
> Regular maintenance releases are key to keeping operational
> burdens
> low
> for
> our downstream users; we don't want them to be forced to choose
> between
> living with broken systems and stomaching the risk of upgrades
> across
> minor/major version numbers. Looking back over the three
> aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> in
> Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> to
> be a
> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> binaries, one of your PMC members mentioned that with continued
> work
> on
> the
> 2.7 line y'all weren't planning any additional releases of the
> earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making
> bug-fix-only
> patch
> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> and
> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> will
> likely take a big to get 2.6.1 together, but after that it should
> take
> much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>
> --
> Sean
>
>
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>
>
>
>
> --
> Sean
>
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Andrew Purtell <ap...@apache.org>]

Inline

On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> I can understand these (sort of newish) questions from hbase-dev. We
> already have a well laid-out release-management process. If people want to
> learn more about how it works, please head over to
> http://hadoop.apache.org/bylaws.html.
>
> In terms of 2.6.1 release management, it wasn’t stuck for lack of
> volunteers for RM. I originally was driving it [1], and then Akira recently
> volunteered too in a separate thread [2] (which I totally missed
> participating in before), so we have enough help.
>

​Right, but in that thread it looks like you were stymied sorting through
the potential set of updates to apply to the patch release. That was the
motivation behind my sharing a bit about HBase branch RMs. Totally fine if
that's not something Hadoop wants to explore. I only offered it as a data
point where another project avoids that problem with a different kind of RM
focus. And, so, that's that FWIW



>
> It is clearly acknowledged there is a demand for 2.6.1, we now have to
> figure out the mechanics, agreeing on a reasonable & common list of fixes
> to start with.
>
>
​Thank you very much for this consideration!​



> Tx for the feedback so far @hbase-dev! I agree with Karthik earlier -
> let’s continue the discussion in hadoop-dev in the release thread [1] for
> 2.6.1.
>
> Thanks
> +Vinod
>
> [1] Planning Hadoop 2.6.1 release
> http://markmail.org/message/sbykjn5xgnksh6wg
> [2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR
>
> On Jul 15, 2015, at 4:07 PM, Stack <stack@duboce.net<mailto:
> stack@duboce.net>> wrote:
>
> Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
> You'd get some help (especially if the bar is set high and only critical
> bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
> updates, and so on).
>
> St.Ack
>
> On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cdouglas@apache.org
> <ma...@apache.org>> wrote:
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>> wrote:
> Alternatively, why not appoint a Release Manager for the minor release
> line
> and then allow them to arbitrate when there's disagreement about
> inclusion?
> This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>> wrote:
> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release
> line
> and then allow them to arbitrate when there's disagreement about
> inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <kasha@cloudera.com
> <ma...@cloudera.com>>
> wrote:
>
> As I proposed in the other thread, how about we adopting the following
> model:
>
> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> next minor release.
> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> minor release.
> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>
> Here I am assuming there are no security-fix-only or other urgent
> releases.
>
> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> release.
>
> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:
>
> Yeah, I started a thread while back on this one (
> http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> discussions re 2.6.1.
>
> The biggest problem I found offline was about what bug-fixes are
> acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> the
> number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> out
> a set of patches that is acceptable for everyone is a huge challenge
> which
> kind of stalled my attempts.
>
> Thanks
> +Vinod
>
>
> On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sjlee0@gmail.com<mailto:
> sjlee0@gmail.com>> wrote:
>
> Strong +1 for having a 2.6.1 release. I understand Vinod has been
> trying
> to
> get that effort going but it's been stalled a little bit. It would
> be
> good
> to rekindle that effort.
>
> Companies with big hadoop 2.x deployments (including mine) have
> always
> tried to stabilize a 2.x release by testing/collecting/researching
> critical
> issues on the release. Each would come up with its own set of fixes
> to
> backport. We would also communicate it via offline channels. During
> the
> hadoop summit, we thought it would be great if we all came together
> and
> create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> for
> example) with all the critical issues fixed.
>
> Thanks,
> Sangjin
>
>
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <ozawa@apache.org<mailto:
> ozawa@apache.org>>
> wrote:
>
> Thank you for the notification. Trying to back port bug fixes.
>
> - Tsuyoshi
>
> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <busbey@cloudera.com<mailto:
> busbey@cloudera.com>>
> wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our
> dependencies on
> our
> downstream users. As our most fundamental dependency, Hadoop
> plays a
> big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we
> are
> unlikely remove anything from this set until HBase 2.0, probably
> at
> the
> end
> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> our
> shipped binaries to Hadoop 2.6, following some stability testing
> by
> part
> of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> bugs
> that could destroy HBase clusters should users decide to turn on
> HDFS
> encryption[4]. Our installation instructions tell folks to replace
> these
> jars with the version of Hadoop they are actually running, but not
> all
> users follow those instructions so we want to minimize the pain
> for
> them.
>
> Regular maintenance releases are key to keeping operational
> burdens
> low
> for
> our downstream users; we don't want them to be forced to choose
> between
> living with broken systems and stomaching the risk of upgrades
> across
> minor/major version numbers. Looking back over the three
> aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> in
> Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> to
> be a
> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> binaries, one of your PMC members mentioned that with continued
> work
> on
> the
> 2.7 line y'all weren't planning any additional releases of the
> earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making
> bug-fix-only
> patch
> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> and
> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> will
> likely take a big to get 2.6.1 together, but after that it should
> take
> much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>
> --
> Sean
>
>
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>
>
>
>
> --
> Sean
>
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to http://hadoop.apache.org/bylaws.html.

In terms of 2.6.1 release management, it wasn’t stuck for lack of volunteers for RM. I originally was driving it [1], and then Akira recently volunteered too in a separate thread [2] (which I totally missed participating in before), so we have enough help.

It is clearly acknowledged there is a demand for 2.6.1, we now have to figure out the mechanics, agreeing on a reasonable & common list of fixes to start with.

Tx for the feedback so far @hbase-dev! I agree with Karthik earlier - let’s continue the discussion in hadoop-dev in the release thread [1] for 2.6.1.

Thanks
+Vinod

[1] Planning Hadoop 2.6.1 release http://markmail.org/message/sbykjn5xgnksh6wg
[2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR

On Jul 15, 2015, at 4:07 PM, Stack <st...@duboce.net>> wrote:

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org>> wrote:

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>>
wrote:

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent
releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

Yeah, I started a thread while back on this one (
http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are
acceptable and what aren’t for everyone wishing to consume 2.6.1.
Given
the
number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
out
a set of patches that is acceptable for everyone is a huge challenge
which
kind of stalled my attempts.

Thanks
+Vinod


On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com>> wrote:

Strong +1 for having a 2.6.1 release. I understand Vinod has been
trying
to
get that effort going but it's been stalled a little bit. It would
be
good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have
always
tried to stabilize a 2.x release by testing/collecting/researching
critical
issues on the release. Each would come up with its own set of fixes
to
backport. We would also communicate it via offline channels. During
the
hadoop summit, we thought it would be great if we all came together
and
create a public stability/bugfix release on top of 2.x (2.6.1 for
2.6
for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>>
wrote:

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>>
wrote:
Hi Hadoopers!

Over in HBase we've been discussing the impact of our
dependencies on
our
downstream users. As our most fundamental dependency, Hadoop
plays a
big
role in the operational cost of running an HBase instance.

Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
2.6[1].
We don't drop Hadoop minor release lines in minor releases so we
are
unlikely remove anything from this set until HBase 2.0, probably
at
the
end
of 2015 / start of 2016 (and currently we plan to continue
supporting
at
least 2.4 for HBase 2.0 [2]). Lately we've been discussing
updating
our
shipped binaries to Hadoop 2.6, following some stability testing
by
part
of
our community[3]. Unfortunately, 2.6.0 in particular has a couple
of
bugs
that could destroy HBase clusters should users decide to turn on
HDFS
encryption[4]. Our installation instructions tell folks to replace
these
jars with the version of Hadoop they are actually running, but not
all
users follow those instructions so we want to minimize the pain
for
them.

Regular maintenance releases are key to keeping operational
burdens
low
for
our downstream users; we don't want them to be forced to choose
between
living with broken systems and stomaching the risk of upgrades
across
minor/major version numbers. Looking back over the three
aforementioned
Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
out
in
Nov
2014, when 2.5 had its last patch release as well. Hadoop 2.4
looks
to
be a
year without a release[5]. On our discussion of shipping Hadoop
2.6
binaries, one of your PMC members mentioned that with continued
work
on
the
2.7 line y'all weren't planning any additional releases of the
earlier
minor versions[6].

The HBase community requests that Hadoop pick up making
bug-fix-only
patch
releases again on a regular schedule[7]. Preferably on the 2.6
line
and
preferably monthly. We realize that given the time gap since
2.6.0 it
will
likely take a big to get 2.6.1 together, but after that it should
take
much
less effort to continue.

[1]: http://hbase.apache.org/book.html#hadoop
[2]: http://s.apache.org/ReP
[3]: HBASE-13339
[4]: HADOOP-11674 and HADOOP-11710
[5]: http://hadoop.apache.org/releases.html
[6]: http://s.apache.org/MTY
[7]: http://s.apache.org/ViP

--
Sean





--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es




--
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to http://hadoop.apache.org/bylaws.html.

In terms of 2.6.1 release management, it wasn’t stuck for lack of volunteers for RM. I originally was driving it [1], and then Akira recently volunteered too in a separate thread [2] (which I totally missed participating in before), so we have enough help.

It is clearly acknowledged there is a demand for 2.6.1, we now have to figure out the mechanics, agreeing on a reasonable & common list of fixes to start with.

Tx for the feedback so far @hbase-dev! I agree with Karthik earlier - let’s continue the discussion in hadoop-dev in the release thread [1] for 2.6.1.

Thanks
+Vinod

[1] Planning Hadoop 2.6.1 release http://markmail.org/message/sbykjn5xgnksh6wg
[2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR

On Jul 15, 2015, at 4:07 PM, Stack <st...@duboce.net>> wrote:

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org>> wrote:

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release
line
and then allow them to arbitrate when there's disagreement about
inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>>
wrote:

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent
releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com<ma...@hortonworks.com>> wrote:

Yeah, I started a thread while back on this one (
http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are
acceptable and what aren’t for everyone wishing to consume 2.6.1.
Given
the
number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
out
a set of patches that is acceptable for everyone is a huge challenge
which
kind of stalled my attempts.

Thanks
+Vinod


On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com>> wrote:

Strong +1 for having a 2.6.1 release. I understand Vinod has been
trying
to
get that effort going but it's been stalled a little bit. It would
be
good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have
always
tried to stabilize a 2.x release by testing/collecting/researching
critical
issues on the release. Each would come up with its own set of fixes
to
backport. We would also communicate it via offline channels. During
the
hadoop summit, we thought it would be great if we all came together
and
create a public stability/bugfix release on top of 2.x (2.6.1 for
2.6
for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>>
wrote:

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>>
wrote:
Hi Hadoopers!

Over in HBase we've been discussing the impact of our
dependencies on
our
downstream users. As our most fundamental dependency, Hadoop
plays a
big
role in the operational cost of running an HBase instance.

Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
2.6[1].
We don't drop Hadoop minor release lines in minor releases so we
are
unlikely remove anything from this set until HBase 2.0, probably
at
the
end
of 2015 / start of 2016 (and currently we plan to continue
supporting
at
least 2.4 for HBase 2.0 [2]). Lately we've been discussing
updating
our
shipped binaries to Hadoop 2.6, following some stability testing
by
part
of
our community[3]. Unfortunately, 2.6.0 in particular has a couple
of
bugs
that could destroy HBase clusters should users decide to turn on
HDFS
encryption[4]. Our installation instructions tell folks to replace
these
jars with the version of Hadoop they are actually running, but not
all
users follow those instructions so we want to minimize the pain
for
them.

Regular maintenance releases are key to keeping operational
burdens
low
for
our downstream users; we don't want them to be forced to choose
between
living with broken systems and stomaching the risk of upgrades
across
minor/major version numbers. Looking back over the three
aforementioned
Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
out
in
Nov
2014, when 2.5 had its last patch release as well. Hadoop 2.4
looks
to
be a
year without a release[5]. On our discussion of shipping Hadoop
2.6
binaries, one of your PMC members mentioned that with continued
work
on
the
2.7 line y'all weren't planning any additional releases of the
earlier
minor versions[6].

The HBase community requests that Hadoop pick up making
bug-fix-only
patch
releases again on a regular schedule[7]. Preferably on the 2.6
line
and
preferably monthly. We realize that given the time gap since
2.6.0 it
will
likely take a big to get 2.6.1 together, but after that it should
take
much
less effort to continue.

[1]: http://hbase.apache.org/book.html#hadoop
[2]: http://s.apache.org/ReP
[3]: HBASE-13339
[4]: HADOOP-11674 and HADOOP-11710
[5]: http://hadoop.apache.org/releases.html
[6]: http://s.apache.org/MTY
[7]: http://s.apache.org/ViP

--
Sean





--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es




--
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Stack <st...@duboce.net>]

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org> wrote:

> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> >> As I proposed in the other thread, how about we adopting the following
> >> model:
> >>
> >> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> >> next minor release.
> >> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> >> minor release.
> >> x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >>
> >> Here I am assuming there are no security-fix-only or other urgent
> releases.
> >>
> >> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> >> release.
> >>
> >> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> >> vinodkv@hortonworks.com> wrote:
> >>
> >> > Yeah, I started a thread while back on this one (
> >> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> >> > discussions re 2.6.1.
> >> >
> >> > The biggest problem I found offline was about what bug-fixes are
> >> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> >> the
> >> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> >> out
> >> > a set of patches that is acceptable for everyone is a huge challenge
> >> which
> >> > kind of stalled my attempts.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> >
> >> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >> > >
> >> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> >> trying
> >> > to
> >> > > get that effort going but it's been stalled a little bit. It would
> be
> >> > good
> >> > > to rekindle that effort.
> >> > >
> >> > > Companies with big hadoop 2.x deployments (including mine) have
> always
> >> > > tried to stabilize a 2.x release by testing/collecting/researching
> >> > critical
> >> > > issues on the release. Each would come up with its own set of fixes
> to
> >> > > backport. We would also communicate it via offline channels. During
> the
> >> > > hadoop summit, we thought it would be great if we all came together
> and
> >> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> >> for
> >> > > example) with all the critical issues fixed.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> > >
> >> > >
> >> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> >> > wrote:
> >> > >
> >> > >> Thank you for the notification. Trying to back port bug fixes.
> >> > >>
> >> > >> - Tsuyoshi
> >> > >>
> >> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > >>> Hi Hadoopers!
> >> > >>>
> >> > >>> Over in HBase we've been discussing the impact of our
> dependencies on
> >> > our
> >> > >>> downstream users. As our most fundamental dependency, Hadoop
> plays a
> >> > big
> >> > >>> role in the operational cost of running an HBase instance.
> >> > >>>
> >> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> > >> 2.6[1].
> >> > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> >> > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> >> the
> >> > >> end
> >> > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> >> > at
> >> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> >> our
> >> > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> >> > part
> >> > >> of
> >> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> >> > bugs
> >> > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> >> > >>> encryption[4]. Our installation instructions tell folks to replace
> >> > these
> >> > >>> jars with the version of Hadoop they are actually running, but not
> >> all
> >> > >>> users follow those instructions so we want to minimize the pain
> for
> >> > them.
> >> > >>>
> >> > >>> Regular maintenance releases are key to keeping operational
> burdens
> >> low
> >> > >> for
> >> > >>> our downstream users; we don't want them to be forced to choose
> >> between
> >> > >>> living with broken systems and stomaching the risk of upgrades
> across
> >> > >>> minor/major version numbers. Looking back over the three
> >> aforementioned
> >> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> >> in
> >> > >> Nov
> >> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> >> to
> >> > >> be a
> >> > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> >> > >>> binaries, one of your PMC members mentioned that with continued
> work
> >> on
> >> > >> the
> >> > >>> 2.7 line y'all weren't planning any additional releases of the
> >> earlier
> >> > >>> minor versions[6].
> >> > >>>
> >> > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> >> > >> patch
> >> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> >> and
> >> > >>> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> >> > >> will
> >> > >>> likely take a big to get 2.6.1 together, but after that it should
> >> take
> >> > >> much
> >> > >>> less effort to continue.
> >> > >>>
> >> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> >> > >>> [2]: http://s.apache.org/ReP
> >> > >>> [3]: HBASE-13339
> >> > >>> [4]: HADOOP-11674 and HADOOP-11710
> >> > >>> [5]: http://hadoop.apache.org/releases.html
> >> > >>> [6]: http://s.apache.org/MTY
> >> > >>> [7]: http://s.apache.org/ViP
> >> > >>>
> >> > >>> --
> >> > >>> Sean
> >> > >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Karthik Kambatla
> >> Software Engineer, Cloudera Inc.
> >> --------------------------------------------
> >> http://five.sentenc.es
> >>
> >
> >
> >
> > --
> > Sean
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

+1 Chris is right.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++





-----Original Message-----
From: Chris Douglas <cd...@apache.org>
Reply-To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Date: Wednesday, July 15, 2015 at 3:07 PM
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Cc: dev <de...@hbase.apache.org>
Subject: Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y
versions

>On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>> Alternatively, why not appoint a Release Manager for the minor release
>>line
>> and then allow them to arbitrate when there's disagreement about
>>inclusion?
>> This has worked well in the HBase community.
>
>
>Release managers aren't appointed in Hadoop. Any committer can RM a
>release branch and encourage others to help with it. An RM can set the
>bar arbitrarily, but an RC only becomes a release when a majority of
>PMC votes approve it in a VOTE. -C
>
>On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
>> Why not just include all backwards compatible bug fixes?
>>
>> Alternatively, why not appoint a Release Manager for the minor release
>>line
>> and then allow them to arbitrate when there's disagreement about
>>inclusion?
>> This has worked well in the HBase community.
>>
>> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> As I proposed in the other thread, how about we adopting the following
>>> model:
>>>
>>> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
>>>the
>>> next minor release.
>>> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
>>> minor release.
>>> x.y.3 releases have all Blocker bug fixes applied to next minor
>>>release.
>>>
>>> Here I am assuming there are no security-fix-only or other urgent
>>>releases.
>>>
>>> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
>>> release.
>>>
>>> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>> > Yeah, I started a thread while back on this one (
>>> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
>>> > discussions re 2.6.1.
>>> >
>>> > The biggest problem I found offline was about what bug-fixes are
>>> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
>>>Given
>>> the
>>> > number of bug-fixes that went into 2.7.x and into branch-2.8,
>>>figuring
>>> out
>>> > a set of patches that is acceptable for everyone is a huge challenge
>>> which
>>> > kind of stalled my attempts.
>>> >
>>> > Thanks
>>> > +Vinod
>>> >
>>> >
>>> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>>> > >
>>> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
>>> trying
>>> > to
>>> > > get that effort going but it's been stalled a little bit. It would
>>>be
>>> > good
>>> > > to rekindle that effort.
>>> > >
>>> > > Companies with big hadoop 2.x deployments (including mine) have
>>>always
>>> > > tried to stabilize a 2.x release by testing/collecting/researching
>>> > critical
>>> > > issues on the release. Each would come up with its own set of
>>>fixes to
>>> > > backport. We would also communicate it via offline channels.
>>>During the
>>> > > hadoop summit, we thought it would be great if we all came
>>>together and
>>> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
>>>2.6
>>> for
>>> > > example) with all the critical issues fixed.
>>> > >
>>> > > Thanks,
>>> > > Sangjin
>>> > >
>>> > >
>>> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
>>> > wrote:
>>> > >
>>> > >> Thank you for the notification. Trying to back port bug fixes.
>>> > >>
>>> > >> - Tsuyoshi
>>> > >>
>>> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
>>> > wrote:
>>> > >>> Hi Hadoopers!
>>> > >>>
>>> > >>> Over in HBase we've been discussing the impact of our
>>>dependencies on
>>> > our
>>> > >>> downstream users. As our most fundamental dependency, Hadoop
>>>plays a
>>> > big
>>> > >>> role in the operational cost of running an HBase instance.
>>> > >>>
>>> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5,
>>>and
>>> > >> 2.6[1].
>>> > >>> We don't drop Hadoop minor release lines in minor releases so we
>>>are
>>> > >>> unlikely remove anything from this set until HBase 2.0, probably
>>>at
>>> the
>>> > >> end
>>> > >>> of 2015 / start of 2016 (and currently we plan to continue
>>>supporting
>>> > at
>>> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
>>>updating
>>> our
>>> > >>> shipped binaries to Hadoop 2.6, following some stability testing
>>>by
>>> > part
>>> > >> of
>>> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a
>>>couple of
>>> > bugs
>>> > >>> that could destroy HBase clusters should users decide to turn on
>>>HDFS
>>> > >>> encryption[4]. Our installation instructions tell folks to
>>>replace
>>> > these
>>> > >>> jars with the version of Hadoop they are actually running, but
>>>not
>>> all
>>> > >>> users follow those instructions so we want to minimize the pain
>>>for
>>> > them.
>>> > >>>
>>> > >>> Regular maintenance releases are key to keeping operational
>>>burdens
>>> low
>>> > >> for
>>> > >>> our downstream users; we don't want them to be forced to choose
>>> between
>>> > >>> living with broken systems and stomaching the risk of upgrades
>>>across
>>> > >>> minor/major version numbers. Looking back over the three
>>> aforementioned
>>> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
>>>out
>>> in
>>> > >> Nov
>>> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
>>>looks
>>> to
>>> > >> be a
>>> > >>> year without a release[5]. On our discussion of shipping Hadoop
>>>2.6
>>> > >>> binaries, one of your PMC members mentioned that with continued
>>>work
>>> on
>>> > >> the
>>> > >>> 2.7 line y'all weren't planning any additional releases of the
>>> earlier
>>> > >>> minor versions[6].
>>> > >>>
>>> > >>> The HBase community requests that Hadoop pick up making
>>>bug-fix-only
>>> > >> patch
>>> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
>>>line
>>> and
>>> > >>> preferably monthly. We realize that given the time gap since
>>>2.6.0 it
>>> > >> will
>>> > >>> likely take a big to get 2.6.1 together, but after that it should
>>> take
>>> > >> much
>>> > >>> less effort to continue.
>>> > >>>
>>> > >>> [1]: http://hbase.apache.org/book.html#hadoop
>>> > >>> [2]: http://s.apache.org/ReP
>>> > >>> [3]: HBASE-13339
>>> > >>> [4]: HADOOP-11674 and HADOOP-11710
>>> > >>> [5]: http://hadoop.apache.org/releases.html
>>> > >>> [6]: http://s.apache.org/MTY
>>> > >>> [7]: http://s.apache.org/ViP
>>> > >>>
>>> > >>> --
>>> > >>> Sean
>>> > >>
>>> >
>>> >
>>>
>>>
>>> --
>>> Karthik Kambatla
>>> Software Engineer, Cloudera Inc.
>>> --------------------------------------------
>>> http://five.sentenc.es
>>>
>>
>>
>>
>> --
>> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Stack <st...@duboce.net>]

Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)?
You'd get some help (especially if the bar is set high and only critical
bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar
updates, and so on).

St.Ack

On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas <cd...@apache.org> wrote:

> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
>
>
> Release managers aren't appointed in Hadoop. Any committer can RM a
> release branch and encourage others to help with it. An RM can set the
> bar arbitrarily, but an RC only becomes a release when a majority of
> PMC votes approve it in a VOTE. -C
>
> On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Why not just include all backwards compatible bug fixes?
> >
> > Alternatively, why not appoint a Release Manager for the minor release
> line
> > and then allow them to arbitrate when there's disagreement about
> inclusion?
> > This has worked well in the HBase community.
> >
> > On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> >> As I proposed in the other thread, how about we adopting the following
> >> model:
> >>
> >> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to
> the
> >> next minor release.
> >> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> >> minor release.
> >> x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >>
> >> Here I am assuming there are no security-fix-only or other urgent
> releases.
> >>
> >> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> >> release.
> >>
> >> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> >> vinodkv@hortonworks.com> wrote:
> >>
> >> > Yeah, I started a thread while back on this one (
> >> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> >> > discussions re 2.6.1.
> >> >
> >> > The biggest problem I found offline was about what bug-fixes are
> >> > acceptable and what aren’t for everyone wishing to consume 2.6.1.
> Given
> >> the
> >> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> >> out
> >> > a set of patches that is acceptable for everyone is a huge challenge
> >> which
> >> > kind of stalled my attempts.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> >
> >> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >> > >
> >> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> >> trying
> >> > to
> >> > > get that effort going but it's been stalled a little bit. It would
> be
> >> > good
> >> > > to rekindle that effort.
> >> > >
> >> > > Companies with big hadoop 2.x deployments (including mine) have
> always
> >> > > tried to stabilize a 2.x release by testing/collecting/researching
> >> > critical
> >> > > issues on the release. Each would come up with its own set of fixes
> to
> >> > > backport. We would also communicate it via offline channels. During
> the
> >> > > hadoop summit, we thought it would be great if we all came together
> and
> >> > > create a public stability/bugfix release on top of 2.x (2.6.1 for
> 2.6
> >> for
> >> > > example) with all the critical issues fixed.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> > >
> >> > >
> >> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> >> > wrote:
> >> > >
> >> > >> Thank you for the notification. Trying to back port bug fixes.
> >> > >>
> >> > >> - Tsuyoshi
> >> > >>
> >> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> >> > wrote:
> >> > >>> Hi Hadoopers!
> >> > >>>
> >> > >>> Over in HBase we've been discussing the impact of our
> dependencies on
> >> > our
> >> > >>> downstream users. As our most fundamental dependency, Hadoop
> plays a
> >> > big
> >> > >>> role in the operational cost of running an HBase instance.
> >> > >>>
> >> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> > >> 2.6[1].
> >> > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> >> > >>> unlikely remove anything from this set until HBase 2.0, probably
> at
> >> the
> >> > >> end
> >> > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> >> > at
> >> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing
> updating
> >> our
> >> > >>> shipped binaries to Hadoop 2.6, following some stability testing
> by
> >> > part
> >> > >> of
> >> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> >> > bugs
> >> > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> >> > >>> encryption[4]. Our installation instructions tell folks to replace
> >> > these
> >> > >>> jars with the version of Hadoop they are actually running, but not
> >> all
> >> > >>> users follow those instructions so we want to minimize the pain
> for
> >> > them.
> >> > >>>
> >> > >>> Regular maintenance releases are key to keeping operational
> burdens
> >> low
> >> > >> for
> >> > >>> our downstream users; we don't want them to be forced to choose
> >> between
> >> > >>> living with broken systems and stomaching the risk of upgrades
> across
> >> > >>> minor/major version numbers. Looking back over the three
> >> aforementioned
> >> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> >> in
> >> > >> Nov
> >> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4
> looks
> >> to
> >> > >> be a
> >> > >>> year without a release[5]. On our discussion of shipping Hadoop
> 2.6
> >> > >>> binaries, one of your PMC members mentioned that with continued
> work
> >> on
> >> > >> the
> >> > >>> 2.7 line y'all weren't planning any additional releases of the
> >> earlier
> >> > >>> minor versions[6].
> >> > >>>
> >> > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> >> > >> patch
> >> > >>> releases again on a regular schedule[7]. Preferably on the 2.6
> line
> >> and
> >> > >>> preferably monthly. We realize that given the time gap since
> 2.6.0 it
> >> > >> will
> >> > >>> likely take a big to get 2.6.1 together, but after that it should
> >> take
> >> > >> much
> >> > >>> less effort to continue.
> >> > >>>
> >> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> >> > >>> [2]: http://s.apache.org/ReP
> >> > >>> [3]: HBASE-13339
> >> > >>> [4]: HADOOP-11674 and HADOOP-11710
> >> > >>> [5]: http://hadoop.apache.org/releases.html
> >> > >>> [6]: http://s.apache.org/MTY
> >> > >>> [7]: http://s.apache.org/ViP
> >> > >>>
> >> > >>> --
> >> > >>> Sean
> >> > >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Karthik Kambatla
> >> Software Engineer, Cloudera Inc.
> >> --------------------------------------------
> >> http://five.sentenc.es
> >>
> >
> >
> >
> > --
> > Sean
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Chris Douglas <cd...@apache.org>]

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
>> As I proposed in the other thread, how about we adopting the following
>> model:
>>
>> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
>> next minor release.
>> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
>> minor release.
>> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>>
>> Here I am assuming there are no security-fix-only or other urgent releases.
>>
>> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
>> release.
>>
>> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>> > Yeah, I started a thread while back on this one (
>> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
>> > discussions re 2.6.1.
>> >
>> > The biggest problem I found offline was about what bug-fixes are
>> > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
>> the
>> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
>> out
>> > a set of patches that is acceptable for everyone is a huge challenge
>> which
>> > kind of stalled my attempts.
>> >
>> > Thanks
>> > +Vinod
>> >
>> >
>> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>> > >
>> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
>> trying
>> > to
>> > > get that effort going but it's been stalled a little bit. It would be
>> > good
>> > > to rekindle that effort.
>> > >
>> > > Companies with big hadoop 2.x deployments (including mine) have always
>> > > tried to stabilize a 2.x release by testing/collecting/researching
>> > critical
>> > > issues on the release. Each would come up with its own set of fixes to
>> > > backport. We would also communicate it via offline channels. During the
>> > > hadoop summit, we thought it would be great if we all came together and
>> > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
>> for
>> > > example) with all the critical issues fixed.
>> > >
>> > > Thanks,
>> > > Sangjin
>> > >
>> > >
>> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
>> > wrote:
>> > >
>> > >> Thank you for the notification. Trying to back port bug fixes.
>> > >>
>> > >> - Tsuyoshi
>> > >>
>> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >>> Hi Hadoopers!
>> > >>>
>> > >>> Over in HBase we've been discussing the impact of our dependencies on
>> > our
>> > >>> downstream users. As our most fundamental dependency, Hadoop plays a
>> > big
>> > >>> role in the operational cost of running an HBase instance.
>> > >>>
>> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
>> > >> 2.6[1].
>> > >>> We don't drop Hadoop minor release lines in minor releases so we are
>> > >>> unlikely remove anything from this set until HBase 2.0, probably at
>> the
>> > >> end
>> > >>> of 2015 / start of 2016 (and currently we plan to continue supporting
>> > at
>> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
>> our
>> > >>> shipped binaries to Hadoop 2.6, following some stability testing by
>> > part
>> > >> of
>> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
>> > bugs
>> > >>> that could destroy HBase clusters should users decide to turn on HDFS
>> > >>> encryption[4]. Our installation instructions tell folks to replace
>> > these
>> > >>> jars with the version of Hadoop they are actually running, but not
>> all
>> > >>> users follow those instructions so we want to minimize the pain for
>> > them.
>> > >>>
>> > >>> Regular maintenance releases are key to keeping operational burdens
>> low
>> > >> for
>> > >>> our downstream users; we don't want them to be forced to choose
>> between
>> > >>> living with broken systems and stomaching the risk of upgrades across
>> > >>> minor/major version numbers. Looking back over the three
>> aforementioned
>> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
>> in
>> > >> Nov
>> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
>> to
>> > >> be a
>> > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
>> > >>> binaries, one of your PMC members mentioned that with continued work
>> on
>> > >> the
>> > >>> 2.7 line y'all weren't planning any additional releases of the
>> earlier
>> > >>> minor versions[6].
>> > >>>
>> > >>> The HBase community requests that Hadoop pick up making bug-fix-only
>> > >> patch
>> > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
>> and
>> > >>> preferably monthly. We realize that given the time gap since 2.6.0 it
>> > >> will
>> > >>> likely take a big to get 2.6.1 together, but after that it should
>> take
>> > >> much
>> > >>> less effort to continue.
>> > >>>
>> > >>> [1]: http://hbase.apache.org/book.html#hadoop
>> > >>> [2]: http://s.apache.org/ReP
>> > >>> [3]: HBASE-13339
>> > >>> [4]: HADOOP-11674 and HADOOP-11710
>> > >>> [5]: http://hadoop.apache.org/releases.html
>> > >>> [6]: http://s.apache.org/MTY
>> > >>> [7]: http://s.apache.org/ViP
>> > >>>
>> > >>> --
>> > >>> Sean
>> > >>
>> >
>> >
>>
>>
>> --
>> Karthik Kambatla
>> Software Engineer, Cloudera Inc.
>> --------------------------------------------
>> http://five.sentenc.es
>>
>
>
>
> --
> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Chris Douglas <cd...@apache.org>]

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.


Release managers aren't appointed in Hadoop. Any committer can RM a
release branch and encourage others to help with it. An RM can set the
bar arbitrarily, but an RC only becomes a release when a majority of
PMC votes approve it in a VOTE. -C

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
>> As I proposed in the other thread, how about we adopting the following
>> model:
>>
>> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
>> next minor release.
>> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
>> minor release.
>> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>>
>> Here I am assuming there are no security-fix-only or other urgent releases.
>>
>> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
>> release.
>>
>> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>> > Yeah, I started a thread while back on this one (
>> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
>> > discussions re 2.6.1.
>> >
>> > The biggest problem I found offline was about what bug-fixes are
>> > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
>> the
>> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
>> out
>> > a set of patches that is acceptable for everyone is a huge challenge
>> which
>> > kind of stalled my attempts.
>> >
>> > Thanks
>> > +Vinod
>> >
>> >
>> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
>> > >
>> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
>> trying
>> > to
>> > > get that effort going but it's been stalled a little bit. It would be
>> > good
>> > > to rekindle that effort.
>> > >
>> > > Companies with big hadoop 2.x deployments (including mine) have always
>> > > tried to stabilize a 2.x release by testing/collecting/researching
>> > critical
>> > > issues on the release. Each would come up with its own set of fixes to
>> > > backport. We would also communicate it via offline channels. During the
>> > > hadoop summit, we thought it would be great if we all came together and
>> > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
>> for
>> > > example) with all the critical issues fixed.
>> > >
>> > > Thanks,
>> > > Sangjin
>> > >
>> > >
>> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
>> > wrote:
>> > >
>> > >> Thank you for the notification. Trying to back port bug fixes.
>> > >>
>> > >> - Tsuyoshi
>> > >>
>> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >>> Hi Hadoopers!
>> > >>>
>> > >>> Over in HBase we've been discussing the impact of our dependencies on
>> > our
>> > >>> downstream users. As our most fundamental dependency, Hadoop plays a
>> > big
>> > >>> role in the operational cost of running an HBase instance.
>> > >>>
>> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
>> > >> 2.6[1].
>> > >>> We don't drop Hadoop minor release lines in minor releases so we are
>> > >>> unlikely remove anything from this set until HBase 2.0, probably at
>> the
>> > >> end
>> > >>> of 2015 / start of 2016 (and currently we plan to continue supporting
>> > at
>> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
>> our
>> > >>> shipped binaries to Hadoop 2.6, following some stability testing by
>> > part
>> > >> of
>> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
>> > bugs
>> > >>> that could destroy HBase clusters should users decide to turn on HDFS
>> > >>> encryption[4]. Our installation instructions tell folks to replace
>> > these
>> > >>> jars with the version of Hadoop they are actually running, but not
>> all
>> > >>> users follow those instructions so we want to minimize the pain for
>> > them.
>> > >>>
>> > >>> Regular maintenance releases are key to keeping operational burdens
>> low
>> > >> for
>> > >>> our downstream users; we don't want them to be forced to choose
>> between
>> > >>> living with broken systems and stomaching the risk of upgrades across
>> > >>> minor/major version numbers. Looking back over the three
>> aforementioned
>> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
>> in
>> > >> Nov
>> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
>> to
>> > >> be a
>> > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
>> > >>> binaries, one of your PMC members mentioned that with continued work
>> on
>> > >> the
>> > >>> 2.7 line y'all weren't planning any additional releases of the
>> earlier
>> > >>> minor versions[6].
>> > >>>
>> > >>> The HBase community requests that Hadoop pick up making bug-fix-only
>> > >> patch
>> > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
>> and
>> > >>> preferably monthly. We realize that given the time gap since 2.6.0 it
>> > >> will
>> > >>> likely take a big to get 2.6.1 together, but after that it should
>> take
>> > >> much
>> > >>> less effort to continue.
>> > >>>
>> > >>> [1]: http://hbase.apache.org/book.html#hadoop
>> > >>> [2]: http://s.apache.org/ReP
>> > >>> [3]: HBASE-13339
>> > >>> [4]: HADOOP-11674 and HADOOP-11710
>> > >>> [5]: http://hadoop.apache.org/releases.html
>> > >>> [6]: http://s.apache.org/MTY
>> > >>> [7]: http://s.apache.org/ViP
>> > >>>
>> > >>> --
>> > >>> Sean
>> > >>
>> >
>> >
>>
>>
>> --
>> Karthik Kambatla
>> Software Engineer, Cloudera Inc.
>> --------------------------------------------
>> http://five.sentenc.es
>>
>
>
>
> --
> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

Every new patch potentially brings in new bugs. So, if we want to limit the
kinds of potential bugs introduced in point releases, we might want to
limit what gets in.

Would be nice to make sure a point release is more stable than a previous
point release in that line.

On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey <bu...@cloudera.com> wrote:

> Why not just include all backwards compatible bug fixes?
>
> Alternatively, why not appoint a Release Manager for the minor release line
> and then allow them to arbitrate when there's disagreement about inclusion?
> This has worked well in the HBase community.
>
> On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > As I proposed in the other thread, how about we adopting the following
> > model:
> >
> > x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
> > next minor release.
> > x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> > minor release.
> > x.y.3 releases have all Blocker bug fixes applied to next minor release.
> >
> > Here I am assuming there are no security-fix-only or other urgent
> releases.
> >
> > We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> > release.
> >
> > On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> > vinodkv@hortonworks.com> wrote:
> >
> > > Yeah, I started a thread while back on this one (
> > > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > > discussions re 2.6.1.
> > >
> > > The biggest problem I found offline was about what bug-fixes are
> > > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
> > the
> > > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> > out
> > > a set of patches that is acceptable for everyone is a huge challenge
> > which
> > > kind of stalled my attempts.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > > >
> > > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> > trying
> > > to
> > > > get that effort going but it's been stalled a little bit. It would be
> > > good
> > > > to rekindle that effort.
> > > >
> > > > Companies with big hadoop 2.x deployments (including mine) have
> always
> > > > tried to stabilize a 2.x release by testing/collecting/researching
> > > critical
> > > > issues on the release. Each would come up with its own set of fixes
> to
> > > > backport. We would also communicate it via offline channels. During
> the
> > > > hadoop summit, we thought it would be great if we all came together
> and
> > > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
> > for
> > > > example) with all the critical issues fixed.
> > > >
> > > > Thanks,
> > > > Sangjin
> > > >
> > > >
> > > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> > > wrote:
> > > >
> > > >> Thank you for the notification. Trying to back port bug fixes.
> > > >>
> > > >> - Tsuyoshi
> > > >>
> > > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > > wrote:
> > > >>> Hi Hadoopers!
> > > >>>
> > > >>> Over in HBase we've been discussing the impact of our dependencies
> on
> > > our
> > > >>> downstream users. As our most fundamental dependency, Hadoop plays
> a
> > > big
> > > >>> role in the operational cost of running an HBase instance.
> > > >>>
> > > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > > >> 2.6[1].
> > > >>> We don't drop Hadoop minor release lines in minor releases so we
> are
> > > >>> unlikely remove anything from this set until HBase 2.0, probably at
> > the
> > > >> end
> > > >>> of 2015 / start of 2016 (and currently we plan to continue
> supporting
> > > at
> > > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> > our
> > > >>> shipped binaries to Hadoop 2.6, following some stability testing by
> > > part
> > > >> of
> > > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple
> of
> > > bugs
> > > >>> that could destroy HBase clusters should users decide to turn on
> HDFS
> > > >>> encryption[4]. Our installation instructions tell folks to replace
> > > these
> > > >>> jars with the version of Hadoop they are actually running, but not
> > all
> > > >>> users follow those instructions so we want to minimize the pain for
> > > them.
> > > >>>
> > > >>> Regular maintenance releases are key to keeping operational burdens
> > low
> > > >> for
> > > >>> our downstream users; we don't want them to be forced to choose
> > between
> > > >>> living with broken systems and stomaching the risk of upgrades
> across
> > > >>> minor/major version numbers. Looking back over the three
> > aforementioned
> > > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came
> out
> > in
> > > >> Nov
> > > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> > to
> > > >> be a
> > > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > >>> binaries, one of your PMC members mentioned that with continued
> work
> > on
> > > >> the
> > > >>> 2.7 line y'all weren't planning any additional releases of the
> > earlier
> > > >>> minor versions[6].
> > > >>>
> > > >>> The HBase community requests that Hadoop pick up making
> bug-fix-only
> > > >> patch
> > > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
> > and
> > > >>> preferably monthly. We realize that given the time gap since 2.6.0
> it
> > > >> will
> > > >>> likely take a big to get 2.6.1 together, but after that it should
> > take
> > > >> much
> > > >>> less effort to continue.
> > > >>>
> > > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > > >>> [2]: http://s.apache.org/ReP
> > > >>> [3]: HBASE-13339
> > > >>> [4]: HADOOP-11674 and HADOOP-11710
> > > >>> [5]: http://hadoop.apache.org/releases.html
> > > >>> [6]: http://s.apache.org/MTY
> > > >>> [7]: http://s.apache.org/ViP
> > > >>>
> > > >>> --
> > > >>> Sean
> > > >>
> > >
> > >
> >
> >
> > --
> > Karthik Kambatla
> > Software Engineer, Cloudera Inc.
> > --------------------------------------------
> > http://five.sentenc.es
> >
>
>
>
> --
> Sean
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release line
and then allow them to arbitrate when there's disagreement about inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> As I proposed in the other thread, how about we adopting the following
> model:
>
> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
> next minor release.
> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> minor release.
> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>
> Here I am assuming there are no security-fix-only or other urgent releases.
>
> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> release.
>
> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
> > Yeah, I started a thread while back on this one (
> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > discussions re 2.6.1.
> >
> > The biggest problem I found offline was about what bug-fixes are
> > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
> the
> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> out
> > a set of patches that is acceptable for everyone is a huge challenge
> which
> > kind of stalled my attempts.
> >
> > Thanks
> > +Vinod
> >
> >
> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > >
> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> trying
> > to
> > > get that effort going but it's been stalled a little bit. It would be
> > good
> > > to rekindle that effort.
> > >
> > > Companies with big hadoop 2.x deployments (including mine) have always
> > > tried to stabilize a 2.x release by testing/collecting/researching
> > critical
> > > issues on the release. Each would come up with its own set of fixes to
> > > backport. We would also communicate it via offline channels. During the
> > > hadoop summit, we thought it would be great if we all came together and
> > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
> for
> > > example) with all the critical issues fixed.
> > >
> > > Thanks,
> > > Sangjin
> > >
> > >
> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> > wrote:
> > >
> > >> Thank you for the notification. Trying to back port bug fixes.
> > >>
> > >> - Tsuyoshi
> > >>
> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >>> Hi Hadoopers!
> > >>>
> > >>> Over in HBase we've been discussing the impact of our dependencies on
> > our
> > >>> downstream users. As our most fundamental dependency, Hadoop plays a
> > big
> > >>> role in the operational cost of running an HBase instance.
> > >>>
> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > >> 2.6[1].
> > >>> We don't drop Hadoop minor release lines in minor releases so we are
> > >>> unlikely remove anything from this set until HBase 2.0, probably at
> the
> > >> end
> > >>> of 2015 / start of 2016 (and currently we plan to continue supporting
> > at
> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> our
> > >>> shipped binaries to Hadoop 2.6, following some stability testing by
> > part
> > >> of
> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> > bugs
> > >>> that could destroy HBase clusters should users decide to turn on HDFS
> > >>> encryption[4]. Our installation instructions tell folks to replace
> > these
> > >>> jars with the version of Hadoop they are actually running, but not
> all
> > >>> users follow those instructions so we want to minimize the pain for
> > them.
> > >>>
> > >>> Regular maintenance releases are key to keeping operational burdens
> low
> > >> for
> > >>> our downstream users; we don't want them to be forced to choose
> between
> > >>> living with broken systems and stomaching the risk of upgrades across
> > >>> minor/major version numbers. Looking back over the three
> aforementioned
> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
> in
> > >> Nov
> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> to
> > >> be a
> > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> > >>> binaries, one of your PMC members mentioned that with continued work
> on
> > >> the
> > >>> 2.7 line y'all weren't planning any additional releases of the
> earlier
> > >>> minor versions[6].
> > >>>
> > >>> The HBase community requests that Hadoop pick up making bug-fix-only
> > >> patch
> > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
> and
> > >>> preferably monthly. We realize that given the time gap since 2.6.0 it
> > >> will
> > >>> likely take a big to get 2.6.1 together, but after that it should
> take
> > >> much
> > >>> less effort to continue.
> > >>>
> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > >>> [2]: http://s.apache.org/ReP
> > >>> [3]: HBASE-13339
> > >>> [4]: HADOOP-11674 and HADOOP-11710
> > >>> [5]: http://hadoop.apache.org/releases.html
> > >>> [6]: http://s.apache.org/MTY
> > >>> [7]: http://s.apache.org/ViP
> > >>>
> > >>> --
> > >>> Sean
> > >>
> >
> >
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>



-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sean Busbey <bu...@cloudera.com>]

Why not just include all backwards compatible bug fixes?

Alternatively, why not appoint a Release Manager for the minor release line
and then allow them to arbitrate when there's disagreement about inclusion?
This has worked well in the HBase community.

On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla <ka...@cloudera.com>
wrote:

> As I proposed in the other thread, how about we adopting the following
> model:
>
> x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
> next minor release.
> x.y.2 releases have all Blocker, Critical bug fixes applied to the next
> minor release.
> x.y.3 releases have all Blocker bug fixes applied to next minor release.
>
> Here I am assuming there are no security-fix-only or other urgent releases.
>
> We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
> release.
>
> On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
> > Yeah, I started a thread while back on this one (
> > http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> > discussions re 2.6.1.
> >
> > The biggest problem I found offline was about what bug-fixes are
> > acceptable and what aren’t for everyone wishing to consume 2.6.1. Given
> the
> > number of bug-fixes that went into 2.7.x and into branch-2.8, figuring
> out
> > a set of patches that is acceptable for everyone is a huge challenge
> which
> > kind of stalled my attempts.
> >
> > Thanks
> > +Vinod
> >
> >
> > > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> > >
> > > Strong +1 for having a 2.6.1 release. I understand Vinod has been
> trying
> > to
> > > get that effort going but it's been stalled a little bit. It would be
> > good
> > > to rekindle that effort.
> > >
> > > Companies with big hadoop 2.x deployments (including mine) have always
> > > tried to stabilize a 2.x release by testing/collecting/researching
> > critical
> > > issues on the release. Each would come up with its own set of fixes to
> > > backport. We would also communicate it via offline channels. During the
> > > hadoop summit, we thought it would be great if we all came together and
> > > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6
> for
> > > example) with all the critical issues fixed.
> > >
> > > Thanks,
> > > Sangjin
> > >
> > >
> > > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> > wrote:
> > >
> > >> Thank you for the notification. Trying to back port bug fixes.
> > >>
> > >> - Tsuyoshi
> > >>
> > >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >>> Hi Hadoopers!
> > >>>
> > >>> Over in HBase we've been discussing the impact of our dependencies on
> > our
> > >>> downstream users. As our most fundamental dependency, Hadoop plays a
> > big
> > >>> role in the operational cost of running an HBase instance.
> > >>>
> > >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > >> 2.6[1].
> > >>> We don't drop Hadoop minor release lines in minor releases so we are
> > >>> unlikely remove anything from this set until HBase 2.0, probably at
> the
> > >> end
> > >>> of 2015 / start of 2016 (and currently we plan to continue supporting
> > at
> > >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating
> our
> > >>> shipped binaries to Hadoop 2.6, following some stability testing by
> > part
> > >> of
> > >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> > bugs
> > >>> that could destroy HBase clusters should users decide to turn on HDFS
> > >>> encryption[4]. Our installation instructions tell folks to replace
> > these
> > >>> jars with the version of Hadoop they are actually running, but not
> all
> > >>> users follow those instructions so we want to minimize the pain for
> > them.
> > >>>
> > >>> Regular maintenance releases are key to keeping operational burdens
> low
> > >> for
> > >>> our downstream users; we don't want them to be forced to choose
> between
> > >>> living with broken systems and stomaching the risk of upgrades across
> > >>> minor/major version numbers. Looking back over the three
> aforementioned
> > >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out
> in
> > >> Nov
> > >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks
> to
> > >> be a
> > >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> > >>> binaries, one of your PMC members mentioned that with continued work
> on
> > >> the
> > >>> 2.7 line y'all weren't planning any additional releases of the
> earlier
> > >>> minor versions[6].
> > >>>
> > >>> The HBase community requests that Hadoop pick up making bug-fix-only
> > >> patch
> > >>> releases again on a regular schedule[7]. Preferably on the 2.6 line
> and
> > >>> preferably monthly. We realize that given the time gap since 2.6.0 it
> > >> will
> > >>> likely take a big to get 2.6.1 together, but after that it should
> take
> > >> much
> > >>> less effort to continue.
> > >>>
> > >>> [1]: http://hbase.apache.org/book.html#hadoop
> > >>> [2]: http://s.apache.org/ReP
> > >>> [3]: HBASE-13339
> > >>> [4]: HADOOP-11674 and HADOOP-11710
> > >>> [5]: http://hadoop.apache.org/releases.html
> > >>> [6]: http://s.apache.org/MTY
> > >>> [7]: http://s.apache.org/ViP
> > >>>
> > >>> --
> > >>> Sean
> > >>
> >
> >
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> --------------------------------------------
> http://five.sentenc.es
>



-- 
Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> Yeah, I started a thread while back on this one (
> http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> discussions re 2.6.1.
>
> The biggest problem I found offline was about what bug-fixes are
> acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the
> number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out
> a set of patches that is acceptable for everyone is a huge challenge which
> kind of stalled my attempts.
>
> Thanks
> +Vinod
>
>
> > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >
> > Strong +1 for having a 2.6.1 release. I understand Vinod has been trying
> to
> > get that effort going but it's been stalled a little bit. It would be
> good
> > to rekindle that effort.
> >
> > Companies with big hadoop 2.x deployments (including mine) have always
> > tried to stabilize a 2.x release by testing/collecting/researching
> critical
> > issues on the release. Each would come up with its own set of fixes to
> > backport. We would also communicate it via offline channels. During the
> > hadoop summit, we thought it would be great if we all came together and
> > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> > example) with all the critical issues fixed.
> >
> > Thanks,
> > Sangjin
> >
> >
> > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> wrote:
> >
> >> Thank you for the notification. Trying to back port bug fixes.
> >>
> >> - Tsuyoshi
> >>
> >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>> Hi Hadoopers!
> >>>
> >>> Over in HBase we've been discussing the impact of our dependencies on
> our
> >>> downstream users. As our most fundamental dependency, Hadoop plays a
> big
> >>> role in the operational cost of running an HBase instance.
> >>>
> >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> 2.6[1].
> >>> We don't drop Hadoop minor release lines in minor releases so we are
> >>> unlikely remove anything from this set until HBase 2.0, probably at the
> >> end
> >>> of 2015 / start of 2016 (and currently we plan to continue supporting
> at
> >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> >>> shipped binaries to Hadoop 2.6, following some stability testing by
> part
> >> of
> >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> bugs
> >>> that could destroy HBase clusters should users decide to turn on HDFS
> >>> encryption[4]. Our installation instructions tell folks to replace
> these
> >>> jars with the version of Hadoop they are actually running, but not all
> >>> users follow those instructions so we want to minimize the pain for
> them.
> >>>
> >>> Regular maintenance releases are key to keeping operational burdens low
> >> for
> >>> our downstream users; we don't want them to be forced to choose between
> >>> living with broken systems and stomaching the risk of upgrades across
> >>> minor/major version numbers. Looking back over the three aforementioned
> >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> >> Nov
> >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> >> be a
> >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> >>> binaries, one of your PMC members mentioned that with continued work on
> >> the
> >>> 2.7 line y'all weren't planning any additional releases of the earlier
> >>> minor versions[6].
> >>>
> >>> The HBase community requests that Hadoop pick up making bug-fix-only
> >> patch
> >>> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> >>> preferably monthly. We realize that given the time gap since 2.6.0 it
> >> will
> >>> likely take a big to get 2.6.1 together, but after that it should take
> >> much
> >>> less effort to continue.
> >>>
> >>> [1]: http://hbase.apache.org/book.html#hadoop
> >>> [2]: http://s.apache.org/ReP
> >>> [3]: HBASE-13339
> >>> [4]: HADOOP-11674 and HADOOP-11710
> >>> [5]: http://hadoop.apache.org/releases.html
> >>> [6]: http://s.apache.org/MTY
> >>> [7]: http://s.apache.org/ViP
> >>>
> >>> --
> >>> Sean
> >>
>
>


-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

As I proposed in the other thread, how about we adopting the following
model:

x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the
next minor release.
x.y.2 releases have all Blocker, Critical bug fixes applied to the next
minor release.
x.y.3 releases have all Blocker bug fixes applied to next minor release.

Here I am assuming there are no security-fix-only or other urgent releases.

We could apply this approach for 2.7.x onwards, and do an adhoc 2.6
release.

On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> Yeah, I started a thread while back on this one (
> http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline
> discussions re 2.6.1.
>
> The biggest problem I found offline was about what bug-fixes are
> acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the
> number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out
> a set of patches that is acceptable for everyone is a huge challenge which
> kind of stalled my attempts.
>
> Thanks
> +Vinod
>
>
> > On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> >
> > Strong +1 for having a 2.6.1 release. I understand Vinod has been trying
> to
> > get that effort going but it's been stalled a little bit. It would be
> good
> > to rekindle that effort.
> >
> > Companies with big hadoop 2.x deployments (including mine) have always
> > tried to stabilize a 2.x release by testing/collecting/researching
> critical
> > issues on the release. Each would come up with its own set of fixes to
> > backport. We would also communicate it via offline channels. During the
> > hadoop summit, we thought it would be great if we all came together and
> > create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> > example) with all the critical issues fixed.
> >
> > Thanks,
> > Sangjin
> >
> >
> > On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org>
> wrote:
> >
> >> Thank you for the notification. Trying to back port bug fixes.
> >>
> >> - Tsuyoshi
> >>
> >> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>> Hi Hadoopers!
> >>>
> >>> Over in HBase we've been discussing the impact of our dependencies on
> our
> >>> downstream users. As our most fundamental dependency, Hadoop plays a
> big
> >>> role in the operational cost of running an HBase instance.
> >>>
> >>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> >> 2.6[1].
> >>> We don't drop Hadoop minor release lines in minor releases so we are
> >>> unlikely remove anything from this set until HBase 2.0, probably at the
> >> end
> >>> of 2015 / start of 2016 (and currently we plan to continue supporting
> at
> >>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> >>> shipped binaries to Hadoop 2.6, following some stability testing by
> part
> >> of
> >>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> bugs
> >>> that could destroy HBase clusters should users decide to turn on HDFS
> >>> encryption[4]. Our installation instructions tell folks to replace
> these
> >>> jars with the version of Hadoop they are actually running, but not all
> >>> users follow those instructions so we want to minimize the pain for
> them.
> >>>
> >>> Regular maintenance releases are key to keeping operational burdens low
> >> for
> >>> our downstream users; we don't want them to be forced to choose between
> >>> living with broken systems and stomaching the risk of upgrades across
> >>> minor/major version numbers. Looking back over the three aforementioned
> >>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> >> Nov
> >>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> >> be a
> >>> year without a release[5]. On our discussion of shipping Hadoop 2.6
> >>> binaries, one of your PMC members mentioned that with continued work on
> >> the
> >>> 2.7 line y'all weren't planning any additional releases of the earlier
> >>> minor versions[6].
> >>>
> >>> The HBase community requests that Hadoop pick up making bug-fix-only
> >> patch
> >>> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> >>> preferably monthly. We realize that given the time gap since 2.6.0 it
> >> will
> >>> likely take a big to get 2.6.1 together, but after that it should take
> >> much
> >>> less effort to continue.
> >>>
> >>> [1]: http://hbase.apache.org/book.html#hadoop
> >>> [2]: http://s.apache.org/ReP
> >>> [3]: HBASE-13339
> >>> [4]: HADOOP-11674 and HADOOP-11710
> >>> [5]: http://hadoop.apache.org/releases.html
> >>> [6]: http://s.apache.org/MTY
> >>> [7]: http://s.apache.org/ViP
> >>>
> >>> --
> >>> Sean
> >>
>
>


-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

Yeah, I started a thread while back on this one (http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts.

Thanks
+Vinod


> On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> 
> Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
> get that effort going but it's been stalled a little bit. It would be good
> to rekindle that effort.
> 
> Companies with big hadoop 2.x deployments (including mine) have always
> tried to stabilize a 2.x release by testing/collecting/researching critical
> issues on the release. Each would come up with its own set of fixes to
> backport. We would also communicate it via offline channels. During the
> hadoop summit, we thought it would be great if we all came together and
> create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> example) with all the critical issues fixed.
> 
> Thanks,
> Sangjin
> 
> 
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> 
>> Thank you for the notification. Trying to back port bug fixes.
>> 
>> - Tsuyoshi
>> 
>> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>> Hi Hadoopers!
>>> 
>>> Over in HBase we've been discussing the impact of our dependencies on our
>>> downstream users. As our most fundamental dependency, Hadoop plays a big
>>> role in the operational cost of running an HBase instance.
>>> 
>>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
>> 2.6[1].
>>> We don't drop Hadoop minor release lines in minor releases so we are
>>> unlikely remove anything from this set until HBase 2.0, probably at the
>> end
>>> of 2015 / start of 2016 (and currently we plan to continue supporting at
>>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
>>> shipped binaries to Hadoop 2.6, following some stability testing by part
>> of
>>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
>>> that could destroy HBase clusters should users decide to turn on HDFS
>>> encryption[4]. Our installation instructions tell folks to replace these
>>> jars with the version of Hadoop they are actually running, but not all
>>> users follow those instructions so we want to minimize the pain for them.
>>> 
>>> Regular maintenance releases are key to keeping operational burdens low
>> for
>>> our downstream users; we don't want them to be forced to choose between
>>> living with broken systems and stomaching the risk of upgrades across
>>> minor/major version numbers. Looking back over the three aforementioned
>>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
>> Nov
>>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
>> be a
>>> year without a release[5]. On our discussion of shipping Hadoop 2.6
>>> binaries, one of your PMC members mentioned that with continued work on
>> the
>>> 2.7 line y'all weren't planning any additional releases of the earlier
>>> minor versions[6].
>>> 
>>> The HBase community requests that Hadoop pick up making bug-fix-only
>> patch
>>> releases again on a regular schedule[7]. Preferably on the 2.6 line and
>>> preferably monthly. We realize that given the time gap since 2.6.0 it
>> will
>>> likely take a big to get 2.6.1 together, but after that it should take
>> much
>>> less effort to continue.
>>> 
>>> [1]: http://hbase.apache.org/book.html#hadoop
>>> [2]: http://s.apache.org/ReP
>>> [3]: HBASE-13339
>>> [4]: HADOOP-11674 and HADOOP-11710
>>> [5]: http://hadoop.apache.org/releases.html
>>> [6]: http://s.apache.org/MTY
>>> [7]: http://s.apache.org/ViP
>>> 
>>> --
>>> Sean
>> 

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Vinod Kumar Vavilapalli <vi...@hortonworks.com>]

Yeah, I started a thread while back on this one (http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts.

Thanks
+Vinod


> On Jul 15, 2015, at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:
> 
> Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
> get that effort going but it's been stalled a little bit. It would be good
> to rekindle that effort.
> 
> Companies with big hadoop 2.x deployments (including mine) have always
> tried to stabilize a 2.x release by testing/collecting/researching critical
> issues on the release. Each would come up with its own set of fixes to
> backport. We would also communicate it via offline channels. During the
> hadoop summit, we thought it would be great if we all came together and
> create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> example) with all the critical issues fixed.
> 
> Thanks,
> Sangjin
> 
> 
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> 
>> Thank you for the notification. Trying to back port bug fixes.
>> 
>> - Tsuyoshi
>> 
>> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>> Hi Hadoopers!
>>> 
>>> Over in HBase we've been discussing the impact of our dependencies on our
>>> downstream users. As our most fundamental dependency, Hadoop plays a big
>>> role in the operational cost of running an HBase instance.
>>> 
>>> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
>> 2.6[1].
>>> We don't drop Hadoop minor release lines in minor releases so we are
>>> unlikely remove anything from this set until HBase 2.0, probably at the
>> end
>>> of 2015 / start of 2016 (and currently we plan to continue supporting at
>>> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
>>> shipped binaries to Hadoop 2.6, following some stability testing by part
>> of
>>> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
>>> that could destroy HBase clusters should users decide to turn on HDFS
>>> encryption[4]. Our installation instructions tell folks to replace these
>>> jars with the version of Hadoop they are actually running, but not all
>>> users follow those instructions so we want to minimize the pain for them.
>>> 
>>> Regular maintenance releases are key to keeping operational burdens low
>> for
>>> our downstream users; we don't want them to be forced to choose between
>>> living with broken systems and stomaching the risk of upgrades across
>>> minor/major version numbers. Looking back over the three aforementioned
>>> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
>> Nov
>>> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
>> be a
>>> year without a release[5]. On our discussion of shipping Hadoop 2.6
>>> binaries, one of your PMC members mentioned that with continued work on
>> the
>>> 2.7 line y'all weren't planning any additional releases of the earlier
>>> minor versions[6].
>>> 
>>> The HBase community requests that Hadoop pick up making bug-fix-only
>> patch
>>> releases again on a regular schedule[7]. Preferably on the 2.6 line and
>>> preferably monthly. We realize that given the time gap since 2.6.0 it
>> will
>>> likely take a big to get 2.6.1 together, but after that it should take
>> much
>>> less effort to continue.
>>> 
>>> [1]: http://hbase.apache.org/book.html#hadoop
>>> [2]: http://s.apache.org/ReP
>>> [3]: HBASE-13339
>>> [4]: HADOOP-11674 and HADOOP-11710
>>> [5]: http://hadoop.apache.org/releases.html
>>> [6]: http://s.apache.org/MTY
>>> [7]: http://s.apache.org/ViP
>>> 
>>> --
>>> Sean
>> 

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Karthik Kambatla <ka...@cloudera.com>]

I believe there was general consensus to do more maintenance releases, as
witnessed in the other thread.

There have been discussions on what should go into 2.x.1, 2.x.2, etc., but
I don't think we have a clear proposal. It would be nice to put that
together, so committers know where all to commit patches to. Otherwise,
release managers will have to look through branch-2 and pull in the fixes.
Either approach is fine by me, but would be nice to start a [DISCUSS]
thread on how to go about this.

Any takers?

On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee <sj...@gmail.com> wrote:

> Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
> get that effort going but it's been stalled a little bit. It would be good
> to rekindle that effort.
>
> Companies with big hadoop 2.x deployments (including mine) have always
> tried to stabilize a 2.x release by testing/collecting/researching critical
> issues on the release. Each would come up with its own set of fixes to
> backport. We would also communicate it via offline channels. During the
> hadoop summit, we thought it would be great if we all came together and
> create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
> example) with all the critical issues fixed.
>
> Thanks,
> Sangjin
>
>
> On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> > Thank you for the notification. Trying to back port bug fixes.
> >
> > - Tsuyoshi
> >
> > On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> > > Hi Hadoopers!
> > >
> > > Over in HBase we've been discussing the impact of our dependencies on
> our
> > > downstream users. As our most fundamental dependency, Hadoop plays a
> big
> > > role in the operational cost of running an HBase instance.
> > >
> > > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> > 2.6[1].
> > > We don't drop Hadoop minor release lines in minor releases so we are
> > > unlikely remove anything from this set until HBase 2.0, probably at the
> > end
> > > of 2015 / start of 2016 (and currently we plan to continue supporting
> at
> > > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> > > shipped binaries to Hadoop 2.6, following some stability testing by
> part
> > of
> > > our community[3]. Unfortunately, 2.6.0 in particular has a couple of
> bugs
> > > that could destroy HBase clusters should users decide to turn on HDFS
> > > encryption[4]. Our installation instructions tell folks to replace
> these
> > > jars with the version of Hadoop they are actually running, but not all
> > > users follow those instructions so we want to minimize the pain for
> them.
> > >
> > > Regular maintenance releases are key to keeping operational burdens low
> > for
> > > our downstream users; we don't want them to be forced to choose between
> > > living with broken systems and stomaching the risk of upgrades across
> > > minor/major version numbers. Looking back over the three aforementioned
> > > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> > Nov
> > > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> > be a
> > > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > > binaries, one of your PMC members mentioned that with continued work on
> > the
> > > 2.7 line y'all weren't planning any additional releases of the earlier
> > > minor versions[6].
> > >
> > > The HBase community requests that Hadoop pick up making bug-fix-only
> > patch
> > > releases again on a regular schedule[7]. Preferably on the 2.6 line and
> > > preferably monthly. We realize that given the time gap since 2.6.0 it
> > will
> > > likely take a big to get 2.6.1 together, but after that it should take
> > much
> > > less effort to continue.
> > >
> > > [1]: http://hbase.apache.org/book.html#hadoop
> > > [2]: http://s.apache.org/ReP
> > > [3]: HBASE-13339
> > > [4]: HADOOP-11674 and HADOOP-11710
> > > [5]: http://hadoop.apache.org/releases.html
> > > [6]: http://s.apache.org/MTY
> > > [7]: http://s.apache.org/ViP
> > >
> > > --
> > > Sean
> >
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sangjin Lee <sj...@gmail.com>]

Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
get that effort going but it's been stalled a little bit. It would be good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have always
tried to stabilize a 2.x release by testing/collecting/researching critical
issues on the release. Each would come up with its own set of fixes to
backport. We would also communicate it via offline channels. During the
hadoop summit, we thought it would be great if we all came together and
create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:

> Thank you for the notification. Trying to back port bug fixes.
>
> - Tsuyoshi
>
> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
> > Hi Hadoopers!
> >
> > Over in HBase we've been discussing the impact of our dependencies on our
> > downstream users. As our most fundamental dependency, Hadoop plays a big
> > role in the operational cost of running an HBase instance.
> >
> > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> 2.6[1].
> > We don't drop Hadoop minor release lines in minor releases so we are
> > unlikely remove anything from this set until HBase 2.0, probably at the
> end
> > of 2015 / start of 2016 (and currently we plan to continue supporting at
> > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> > shipped binaries to Hadoop 2.6, following some stability testing by part
> of
> > our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> > that could destroy HBase clusters should users decide to turn on HDFS
> > encryption[4]. Our installation instructions tell folks to replace these
> > jars with the version of Hadoop they are actually running, but not all
> > users follow those instructions so we want to minimize the pain for them.
> >
> > Regular maintenance releases are key to keeping operational burdens low
> for
> > our downstream users; we don't want them to be forced to choose between
> > living with broken systems and stomaching the risk of upgrades across
> > minor/major version numbers. Looking back over the three aforementioned
> > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> Nov
> > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> be a
> > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > binaries, one of your PMC members mentioned that with continued work on
> the
> > 2.7 line y'all weren't planning any additional releases of the earlier
> > minor versions[6].
> >
> > The HBase community requests that Hadoop pick up making bug-fix-only
> patch
> > releases again on a regular schedule[7]. Preferably on the 2.6 line and
> > preferably monthly. We realize that given the time gap since 2.6.0 it
> will
> > likely take a big to get 2.6.1 together, but after that it should take
> much
> > less effort to continue.
> >
> > [1]: http://hbase.apache.org/book.html#hadoop
> > [2]: http://s.apache.org/ReP
> > [3]: HBASE-13339
> > [4]: HADOOP-11674 and HADOOP-11710
> > [5]: http://hadoop.apache.org/releases.html
> > [6]: http://s.apache.org/MTY
> > [7]: http://s.apache.org/ViP
> >
> > --
> > Sean
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Sangjin Lee <sj...@gmail.com>]

Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to
get that effort going but it's been stalled a little bit. It would be good
to rekindle that effort.

Companies with big hadoop 2.x deployments (including mine) have always
tried to stabilize a 2.x release by testing/collecting/researching critical
issues on the release. Each would come up with its own set of fixes to
backport. We would also communicate it via offline channels. During the
hadoop summit, we thought it would be great if we all came together and
create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for
example) with all the critical issues fixed.

Thanks,
Sangjin


On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:

> Thank you for the notification. Trying to back port bug fixes.
>
> - Tsuyoshi
>
> On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
> > Hi Hadoopers!
> >
> > Over in HBase we've been discussing the impact of our dependencies on our
> > downstream users. As our most fundamental dependency, Hadoop plays a big
> > role in the operational cost of running an HBase instance.
> >
> > Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and
> 2.6[1].
> > We don't drop Hadoop minor release lines in minor releases so we are
> > unlikely remove anything from this set until HBase 2.0, probably at the
> end
> > of 2015 / start of 2016 (and currently we plan to continue supporting at
> > least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> > shipped binaries to Hadoop 2.6, following some stability testing by part
> of
> > our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> > that could destroy HBase clusters should users decide to turn on HDFS
> > encryption[4]. Our installation instructions tell folks to replace these
> > jars with the version of Hadoop they are actually running, but not all
> > users follow those instructions so we want to minimize the pain for them.
> >
> > Regular maintenance releases are key to keeping operational burdens low
> for
> > our downstream users; we don't want them to be forced to choose between
> > living with broken systems and stomaching the risk of upgrades across
> > minor/major version numbers. Looking back over the three aforementioned
> > Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in
> Nov
> > 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to
> be a
> > year without a release[5]. On our discussion of shipping Hadoop 2.6
> > binaries, one of your PMC members mentioned that with continued work on
> the
> > 2.7 line y'all weren't planning any additional releases of the earlier
> > minor versions[6].
> >
> > The HBase community requests that Hadoop pick up making bug-fix-only
> patch
> > releases again on a regular schedule[7]. Preferably on the 2.6 line and
> > preferably monthly. We realize that given the time gap since 2.6.0 it
> will
> > likely take a big to get 2.6.1 together, but after that it should take
> much
> > less effort to continue.
> >
> > [1]: http://hbase.apache.org/book.html#hadoop
> > [2]: http://s.apache.org/ReP
> > [3]: HBASE-13339
> > [4]: HADOOP-11674 and HADOOP-11710
> > [5]: http://hadoop.apache.org/releases.html
> > [6]: http://s.apache.org/MTY
> > [7]: http://s.apache.org/ViP
> >
> > --
> > Sean
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Tsuyoshi Ozawa <oz...@apache.org>]

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our dependencies on our
> downstream users. As our most fundamental dependency, Hadoop plays a big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we are
> unlikely remove anything from this set until HBase 2.0, probably at the end
> of 2015 / start of 2016 (and currently we plan to continue supporting at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> shipped binaries to Hadoop 2.6, following some stability testing by part of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> that could destroy HBase clusters should users decide to turn on HDFS
> encryption[4]. Our installation instructions tell folks to replace these
> jars with the version of Hadoop they are actually running, but not all
> users follow those instructions so we want to minimize the pain for them.
>
> Regular maintenance releases are key to keeping operational burdens low for
> our downstream users; we don't want them to be forced to choose between
> living with broken systems and stomaching the risk of upgrades across
> minor/major version numbers. Looking back over the three aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a
> year without a release[5]. On our discussion of shipping Hadoop 2.6
> binaries, one of your PMC members mentioned that with continued work on the
> 2.7 line y'all weren't planning any additional releases of the earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making bug-fix-only patch
> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> preferably monthly. We realize that given the time gap since 2.6.0 it will
> likely take a big to get 2.6.1 together, but after that it should take much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>
> --
> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thanks Sean for starting this thread.

I started almost the same discussion[1] before, so I'm obviously +1 for 
creating 2.6.1 release. I'd like to start the work ASAP.

[1] http://s.apache.org/MMR

Regards,
Akira

On 7/15/15 03:45, Sean Busbey wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our dependencies on our
> downstream users. As our most fundamental dependency, Hadoop plays a big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we are
> unlikely remove anything from this set until HBase 2.0, probably at the end
> of 2015 / start of 2016 (and currently we plan to continue supporting at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> shipped binaries to Hadoop 2.6, following some stability testing by part of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> that could destroy HBase clusters should users decide to turn on HDFS
> encryption[4]. Our installation instructions tell folks to replace these
> jars with the version of Hadoop they are actually running, but not all
> users follow those instructions so we want to minimize the pain for them.
>
> Regular maintenance releases are key to keeping operational burdens low for
> our downstream users; we don't want them to be forced to choose between
> living with broken systems and stomaching the risk of upgrades across
> minor/major version numbers. Looking back over the three aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a
> year without a release[5]. On our discussion of shipping Hadoop 2.6
> binaries, one of your PMC members mentioned that with continued work on the
> 2.7 line y'all weren't planning any additional releases of the earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making bug-fix-only patch
> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> preferably monthly. We realize that given the time gap since 2.6.0 it will
> likely take a big to get 2.6.1 together, but after that it should take much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Tsuyoshi Ozawa <oz...@apache.org>]

Thank you for the notification. Trying to back port bug fixes.

- Tsuyoshi

On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey <bu...@cloudera.com> wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our dependencies on our
> downstream users. As our most fundamental dependency, Hadoop plays a big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we are
> unlikely remove anything from this set until HBase 2.0, probably at the end
> of 2015 / start of 2016 (and currently we plan to continue supporting at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> shipped binaries to Hadoop 2.6, following some stability testing by part of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> that could destroy HBase clusters should users decide to turn on HDFS
> encryption[4]. Our installation instructions tell folks to replace these
> jars with the version of Hadoop they are actually running, but not all
> users follow those instructions so we want to minimize the pain for them.
>
> Regular maintenance releases are key to keeping operational burdens low for
> our downstream users; we don't want them to be forced to choose between
> living with broken systems and stomaching the risk of upgrades across
> minor/major version numbers. Looking back over the three aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a
> year without a release[5]. On our discussion of shipping Hadoop 2.6
> binaries, one of your PMC members mentioned that with continued work on the
> 2.7 line y'all weren't planning any additional releases of the earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making bug-fix-only patch
> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> preferably monthly. We realize that given the time gap since 2.6.0 it will
> likely take a big to get 2.6.1 together, but after that it should take much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>
> --
> Sean

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thanks Sean for starting this thread.

I started almost the same discussion[1] before, so I'm obviously +1 for 
creating 2.6.1 release. I'd like to start the work ASAP.

[1] http://s.apache.org/MMR

Regards,
Akira

On 7/15/15 03:45, Sean Busbey wrote:
> Hi Hadoopers!
>
> Over in HBase we've been discussing the impact of our dependencies on our
> downstream users. As our most fundamental dependency, Hadoop plays a big
> role in the operational cost of running an HBase instance.
>
> Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1].
> We don't drop Hadoop minor release lines in minor releases so we are
> unlikely remove anything from this set until HBase 2.0, probably at the end
> of 2015 / start of 2016 (and currently we plan to continue supporting at
> least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our
> shipped binaries to Hadoop 2.6, following some stability testing by part of
> our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs
> that could destroy HBase clusters should users decide to turn on HDFS
> encryption[4]. Our installation instructions tell folks to replace these
> jars with the version of Hadoop they are actually running, but not all
> users follow those instructions so we want to minimize the pain for them.
>
> Regular maintenance releases are key to keeping operational burdens low for
> our downstream users; we don't want them to be forced to choose between
> living with broken systems and stomaching the risk of upgrades across
> minor/major version numbers. Looking back over the three aforementioned
> Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov
> 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a
> year without a release[5]. On our discussion of shipping Hadoop 2.6
> binaries, one of your PMC members mentioned that with continued work on the
> 2.7 line y'all weren't planning any additional releases of the earlier
> minor versions[6].
>
> The HBase community requests that Hadoop pick up making bug-fix-only patch
> releases again on a regular schedule[7]. Preferably on the 2.6 line and
> preferably monthly. We realize that given the time gap since 2.6.0 it will
> likely take a big to get 2.6.1 together, but after that it should take much
> less effort to continue.
>
> [1]: http://hbase.apache.org/book.html#hadoop
> [2]: http://s.apache.org/ReP
> [3]: HBASE-13339
> [4]: HADOOP-11674 and HADOOP-11710
> [5]: http://hadoop.apache.org/releases.html
> [6]: http://s.apache.org/MTY
> [7]: http://s.apache.org/ViP
>