2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Hi all,

Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.

Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.

I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.

Thanks,
+Vinod

[1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343

Re: 2.7.3 release plan

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Hi payam,

Please send an e-mail from your address to 
common-dev-unsubscribe@hadoop.apache.org

On 4/1/16 15:00, payam rastogi wrote:
> Please unsubscribe me from this group.
>
> ________________________________________
> From: Akira AJISAKA <aj...@oss.nttdata.co.jp>
> Sent: Thursday, March 31, 2016 12:15 PM
> To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>>
>

Re: 2.7.3 release plan

[payam rastogi <pa...@outlook.com>]

Please unsubscribe me from this group.

________________________________________
From: Akira AJISAKA <aj...@oss.nttdata.co.jp>
Sent: Thursday, March 31, 2016 12:15 PM
To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Colin McCabe <co...@cmccabe.xyz>]

Sure.  We will continue this discussion on HADOOP-12893.  It now appears
that binary tarballs may need their own license and notice files-- see
the jira for details.

Best,
C.

On Thu, Apr 7, 2016, at 09:28, Vinod Kumar Vavilapalli wrote:
> Colin / Sean, appreciate moving your feedback (and copy-pasting your
> current comments) on this issue to the JIRA: HADOOP-12893
> 
> Thanks
> +Vinod
> 
> > On Apr 7, 2016, at 7:43 AM, Sean Busbey <bu...@cloudera.com> wrote:
> > 
> > On Wed, Apr 6, 2016 at 6:26 PM, Colin McCabe <cmccabe@apache.org <ma...@apache.org>> wrote:
> >> In general, the only bundled native component I can see is lz4.  I guess
> >> debatably we should add tree.h to the NOTICE file as well, since it came
> >> from BSD and is licensed under that license.
> >> 
> >> Please keep in mind bundling means "included in the source tree", NOT
> >> "downloaded during the build process."  "mvn package" dumps a ton of
> >> jars in the build directory, but these dependencies aren't considered
> >> bundled since their source does not appear in our git repo.  Similarly,
> >> linking against a library doesn't make it "bundled", nor does dlopening
> >> symbols in that library.
> >> 
> >> The big omission is that we have a lot of Javascript source files in our
> >> source tree that do not appear in LICENSE or NOTICE.  I agree that we
> >> should address those before making a new release.
> >> 
> >> best,
> >> 
> >> 
> > 
> > Each artifact that the PMC publishes must abide by the ASF licensing
> > policy. That includes
> > 
> > * Source release artifact
> > * any convenience binary artifacts places on dist.apache
> > * any convenience jars put into the ASF Nexus repository
> > 
> > That likely means that we bundle much more than just what's in the source tree.
> > 
> > (Though this sounds like we're getting off topic for the 2.7.3 release plan.)
> > 
> > -- 
> > busbey
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Colin / Sean, appreciate moving your feedback (and copy-pasting your current comments) on this issue to the JIRA: HADOOP-12893

Thanks
+Vinod

> On Apr 7, 2016, at 7:43 AM, Sean Busbey <bu...@cloudera.com> wrote:
> 
> On Wed, Apr 6, 2016 at 6:26 PM, Colin McCabe <cmccabe@apache.org <ma...@apache.org>> wrote:
>> In general, the only bundled native component I can see is lz4.  I guess
>> debatably we should add tree.h to the NOTICE file as well, since it came
>> from BSD and is licensed under that license.
>> 
>> Please keep in mind bundling means "included in the source tree", NOT
>> "downloaded during the build process."  "mvn package" dumps a ton of
>> jars in the build directory, but these dependencies aren't considered
>> bundled since their source does not appear in our git repo.  Similarly,
>> linking against a library doesn't make it "bundled", nor does dlopening
>> symbols in that library.
>> 
>> The big omission is that we have a lot of Javascript source files in our
>> source tree that do not appear in LICENSE or NOTICE.  I agree that we
>> should address those before making a new release.
>> 
>> best,
>> 
>> 
> 
> Each artifact that the PMC publishes must abide by the ASF licensing
> policy. That includes
> 
> * Source release artifact
> * any convenience binary artifacts places on dist.apache
> * any convenience jars put into the ASF Nexus repository
> 
> That likely means that we bundle much more than just what's in the source tree.
> 
> (Though this sounds like we're getting off topic for the 2.7.3 release plan.)
> 
> -- 
> busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

On Wed, Apr 6, 2016 at 6:26 PM, Colin McCabe <cm...@apache.org> wrote:
> In general, the only bundled native component I can see is lz4.  I guess
> debatably we should add tree.h to the NOTICE file as well, since it came
> from BSD and is licensed under that license.
>
> Please keep in mind bundling means "included in the source tree", NOT
> "downloaded during the build process."  "mvn package" dumps a ton of
> jars in the build directory, but these dependencies aren't considered
> bundled since their source does not appear in our git repo.  Similarly,
> linking against a library doesn't make it "bundled", nor does dlopening
> symbols in that library.
>
> The big omission is that we have a lot of Javascript source files in our
> source tree that do not appear in LICENSE or NOTICE.  I agree that we
> should address those before making a new release.
>
> best,
>
>

Each artifact that the PMC publishes must abide by the ASF licensing
policy. That includes

* Source release artifact
* any convenience binary artifacts places on dist.apache
* any convenience jars put into the ASF Nexus repository

That likely means that we bundle much more than just what's in the source tree.

(Though this sounds like we're getting off topic for the 2.7.3 release plan.)

-- 
busbey

Re: 2.7.3 release plan

[Colin McCabe <cm...@apache.org>]

In general, the only bundled native component I can see is lz4.  I guess
debatably we should add tree.h to the NOTICE file as well, since it came
from BSD and is licensed under that license.

Please keep in mind bundling means "included in the source tree", NOT
"downloaded during the build process."  "mvn package" dumps a ton of
jars in the build directory, but these dependencies aren't considered
bundled since their source does not appear in our git repo.  Similarly,
linking against a library doesn't make it "bundled", nor does dlopening
symbols in that library.

The big omission is that we have a lot of Javascript source files in our
source tree that do not appear in LICENSE or NOTICE.  I agree that we
should address those before making a new release.

best,


On Wed, Apr 6, 2016, at 14:13, Allen Wittenauer wrote:
> 
> 	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….
> 
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Allen,

I see you marked it for 2.7.3, so it’s there on my radar.

That said, I’ll definitely need help from you (and likely others) in getting it fixed.

Thanks
+Vinod

> On Apr 6, 2016, at 2:13 PM, Allen Wittenauer <al...@yahoo.com.INVALID> wrote:
> 
> 
> 	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Allen,

I see you marked it for 2.7.3, so it’s there on my radar.

That said, I’ll definitely need help from you (and likely others) in getting it fixed.

Thanks
+Vinod

> On Apr 6, 2016, at 2:13 PM, Allen Wittenauer <al...@yahoo.com.INVALID> wrote:
> 
> 
> 	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Allen,

I see you marked it for 2.7.3, so it’s there on my radar.

That said, I’ll definitely need help from you (and likely others) in getting it fixed.

Thanks
+Vinod

> On Apr 6, 2016, at 2:13 PM, Allen Wittenauer <al...@yahoo.com.INVALID> wrote:
> 
> 
> 	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Allen,

I see you marked it for 2.7.3, so it’s there on my radar.

That said, I’ll definitely need help from you (and likely others) in getting it fixed.

Thanks
+Vinod

> On Apr 6, 2016, at 2:13 PM, Allen Wittenauer <al...@yahoo.com.INVALID> wrote:
> 
> 
> 	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Allen Wittenauer <al...@yahoo.com.INVALID>]

	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Allen Wittenauer <al...@yahoo.com.INVALID>]

	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Allen Wittenauer <al...@yahoo.com.INVALID>]

	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Allen Wittenauer <al...@yahoo.com.INVALID>]

	This is probably a good time to remind/point folks to HADOOP-12893.  Apache Hadoop's binary artifacts (with or without native code) and source artifacts are not complying with the licenses of bundled components.  I fairly confident this means releases are off the table until someone audits the entire codebase given how many have been added without any thought to actually following the terms of the license….

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

I'm +1 for ctrezzo's proposal, happy to do the revert from branch-2.7 if
this is acceptable to Vinod.

There's some additional discussion on the HDFS-8791 JIRA for those who are
only following this email thread.

Best,
Andrew

On Tue, Apr 5, 2016 at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:

> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
>
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
>
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
>
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
>
>
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> > I commented on the JIRA way back (see
> >
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666
> ),
> > saying what I said below. Unfortunately, I haven’t followed the patch
> along
> > after my initial comment.
> >
> > This isn’t about any specific release - starting 2.6 we declared support
> > for rolling upgrades and downgrades. Any patch that breaks this should
> not
> > be in branch-2.
> >
> > Two options from where I stand
> >  (1) For folks who worked on the patch: Is there a way to make (a) the
> > upgrade-downgrade seamless for people who don’t care about this (b) and
> > have explicit documentation for people who care to switch this behavior
> on
> > and are willing to risk not having downgrades. If this means a new
> > configuration property, so be it. It’s a necessary evil.
> >  (2) Just let specific users backport this into specific 2.x branches
> they
> > need and leave it only on trunk.
> >
> > Unless this behavior stops breaking rolling upgrades/downgrades, I think
> > we should just revert it from branch-2 and definitely 2.7.3 as it stands
> > today.
> >
> > +Vinod
> >
> >
> > > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> > >
> > > A few thoughts:
> > >
> > > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > > prerequisite for HDFS-8791. Without that patch, upgrades can be very
> slow
> > > for data nodes depending on your setup.
> > >
> > > 2. We have already deployed this patch internally so, with my Twitter
> hat
> > > on, I would be perfectly happy as long as it makes it into trunk and
> 2.8.
> > > That being said, I would be hesitant to deploy the current 2.7.x or
> 2.6.x
> > > releases on a large production cluster that has a diverse set of block
> > ids
> > > without this patch, especially if your data nodes have a large number
> of
> > > disks or you are using federation. To be clear though: this highly
> > depends
> > > on your setup and at a minimum you should verify that this regression
> > will
> > > not affect you. The current block-id based layout in 2.6.x and 2.7.2
> has
> > a
> > > performance regression that gets worse over time. When you see it
> > happening
> > > on a live cluster, it is one of the harder issues to identify a root
> > cause
> > > and debug. I do understand that this is currently only affecting a
> > smaller
> > > number of users, but I also think this number has potential to increase
> > as
> > > time goes on. Maybe we can issue a warning in the release notes for
> > future
> > > 2.7.x and 2.6.x releases?
> > >
> > > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> > to
> > > this proposal on this thread) would be to cut a 2.8 release off of the
> > > 2.7.3 release with the new layout. What people currently think of as
> 2.8
> > > would then become 2.9. This would give customers a stable release that
> > they
> > > could deploy with the new layout and would not break upgrade and
> > downgrade
> > > expectations.
> > >
> > > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> > would
> > >> patch the release to revert HDFS-8791 before pushing it out to
> > production.
> > >> For what it's worth.
> > >>
> > >>
> > >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > >> wrote:
> > >>
> > >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> > branch-2.6.
> > >>> HDFS-8578 is a very important related fix since otherwise upgrade
> will
> > be
> > >>> very slow.
> > >>>
> > >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > >>>> maintenance release. I've only seen it crop up on a handful of our
> > >>>> customer's clusters, and large users like Twitter and Yahoo that
> seem
> > >> to
> > >>> be
> > >>>> more affected are also the most able to patch this change in
> > >> themselves.
> > >>>>
> > >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> > >>>> breaking upgrade and downgrade expectations when it doesn't affect
> the
> > >>> (in
> > >>>> my experience) vast majority of users.
> > >>>>
> > >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> > but
> > >>>> will let him elaborate.
> > >>>>
> > >>>> Best,
> > >>>> Andrew
> > >>>>
> > >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > >>> wrote:
> > >>>>
> > >>>>> As of 2 days ago, there were already 135 jiras associated with
> 2.7.3,
> > >>>>> if *any* of them end up introducing a regression the inclusion of
> > >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> > back
> > >>>>> things out. If that happens to any substantial number of downstream
> > >>>>> folks, or any particularly vocal downstream folks, then it is very
> > >>>>> likely we'll lose the remaining trust of operators for rolling out
> > >>>>> maintenance releases. That's a pretty steep cost.
> > >>>>>
> > >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having
> to
> > >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >>>>> unreasonable burden.
> > >>>>>
> > >>>>> I agree that this fix is important, I just think we should either
> cut
> > >>>>> a version of 2.8 that includes it or find a way to do it that gives
> > an
> > >>>>> operational path for rolling downgrade.
> > >>>>>
> > >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > >>> wrote:
> > >>>>>> Thanks for bringing up this topic, Sean.
> > >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> > >>> HDFS-8791
> > >>>>> haven't been committed in so that's why we didn't discuss this
> > >> earlier.
> > >>>>>> I remember in JIRA discussion, we treated this layout change as a
> > >>>>> Blocker bug that fixing a significant performance regression before
> > >> but
> > >>> not
> > >>>>> a normal performance improvement. And I believe HDFS community
> > already
> > >>> did
> > >>>>> their best with careful and patient to deliver the fix and other
> > >> related
> > >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> > HDFS-8578,
> > >>> you
> > >>>>> can see 30+ rounds patch review back and forth by senior
> committers,
> > >>> not to
> > >>>>> mention the outstanding performance test data in HDFS-8791.
> > >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> > RM
> > >>> for
> > >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> > >> into
> > >>>>> 2.6.5 now.
> > >>>>>> Just my 2 cents.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Junping
> > >>>>>>
> > >>>>>> ________________________________________
> > >>>>>> From: Sean Busbey <bu...@cloudera.com>
> > >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> > >>>>>> To: hdfs-dev@hadoop.apache.org
> > >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >>>>> mapreduce-dev@hadoop.apache.org
> > >>>>>> Subject: Re: 2.7.3 release plan
> > >>>>>>
> > >>>>>> A layout change in a maintenance release sounds very risky. I saw
> > >> some
> > >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> > >> to
> > >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> > >> thought
> > >>>>>> we did RMs per release rather than per branch? No one claiming to
> > >> be a
> > >>>>>> release manager ever spoke up AFAICT.
> > >>>>>>
> > >>>>>> Should this change be included? Should it go into a special 2.8
> > >>>>>> release as mentioned in the ticket?
> > >>>>>>
> > >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> > >>>>>>> Thank you Vinod!
> > >>>>>>>
> > >>>>>>> FYI: 2.7.3 will be a bit special release.
> > >>>>>>>
> > >>>>>>> HDFS-8791 bumped up the datanode layout version,
> > >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >>>>>>> is impossible. We can rollback instead.
> > >>>>>>>
> > >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> > >>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Akira
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >>>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go
> out
> > >>>>> (which
> > >>>>>>>> did go out mid February). Got a little busy since.
> > >>>>>>>>
> > >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> > >> towards a
> > >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> > >> bug-fixes
> > >>>>> and *no*
> > >>>>>>>> features / improvements.
> > >>>>>>>>
> > >>>>>>>> I hope to cut an RC in a week - giving enough time for
> outstanding
> > >>>>> blocker
> > >>>>>>>> / critical issues. Will start moving out any tickets that are
> not
> > >>>>> blockers
> > >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> > >> critical
> > >>>>> tickets
> > >>>>>>>> outstanding as of now.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> +Vinod
> > >>>>>>>>
> > >>>>>>>> [1] 2.7.3 release blockers:
> > >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> busbey
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> busbey
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
> >
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

I'm +1 for ctrezzo's proposal, happy to do the revert from branch-2.7 if
this is acceptable to Vinod.

There's some additional discussion on the HDFS-8791 JIRA for those who are
only following this email thread.

Best,
Andrew

On Tue, Apr 5, 2016 at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:

> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
>
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
>
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
>
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
>
>
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> > I commented on the JIRA way back (see
> >
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666
> ),
> > saying what I said below. Unfortunately, I haven’t followed the patch
> along
> > after my initial comment.
> >
> > This isn’t about any specific release - starting 2.6 we declared support
> > for rolling upgrades and downgrades. Any patch that breaks this should
> not
> > be in branch-2.
> >
> > Two options from where I stand
> >  (1) For folks who worked on the patch: Is there a way to make (a) the
> > upgrade-downgrade seamless for people who don’t care about this (b) and
> > have explicit documentation for people who care to switch this behavior
> on
> > and are willing to risk not having downgrades. If this means a new
> > configuration property, so be it. It’s a necessary evil.
> >  (2) Just let specific users backport this into specific 2.x branches
> they
> > need and leave it only on trunk.
> >
> > Unless this behavior stops breaking rolling upgrades/downgrades, I think
> > we should just revert it from branch-2 and definitely 2.7.3 as it stands
> > today.
> >
> > +Vinod
> >
> >
> > > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> > >
> > > A few thoughts:
> > >
> > > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > > prerequisite for HDFS-8791. Without that patch, upgrades can be very
> slow
> > > for data nodes depending on your setup.
> > >
> > > 2. We have already deployed this patch internally so, with my Twitter
> hat
> > > on, I would be perfectly happy as long as it makes it into trunk and
> 2.8.
> > > That being said, I would be hesitant to deploy the current 2.7.x or
> 2.6.x
> > > releases on a large production cluster that has a diverse set of block
> > ids
> > > without this patch, especially if your data nodes have a large number
> of
> > > disks or you are using federation. To be clear though: this highly
> > depends
> > > on your setup and at a minimum you should verify that this regression
> > will
> > > not affect you. The current block-id based layout in 2.6.x and 2.7.2
> has
> > a
> > > performance regression that gets worse over time. When you see it
> > happening
> > > on a live cluster, it is one of the harder issues to identify a root
> > cause
> > > and debug. I do understand that this is currently only affecting a
> > smaller
> > > number of users, but I also think this number has potential to increase
> > as
> > > time goes on. Maybe we can issue a warning in the release notes for
> > future
> > > 2.7.x and 2.6.x releases?
> > >
> > > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> > to
> > > this proposal on this thread) would be to cut a 2.8 release off of the
> > > 2.7.3 release with the new layout. What people currently think of as
> 2.8
> > > would then become 2.9. This would give customers a stable release that
> > they
> > > could deploy with the new layout and would not break upgrade and
> > downgrade
> > > expectations.
> > >
> > > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> > would
> > >> patch the release to revert HDFS-8791 before pushing it out to
> > production.
> > >> For what it's worth.
> > >>
> > >>
> > >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > >> wrote:
> > >>
> > >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> > branch-2.6.
> > >>> HDFS-8578 is a very important related fix since otherwise upgrade
> will
> > be
> > >>> very slow.
> > >>>
> > >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > >>>> maintenance release. I've only seen it crop up on a handful of our
> > >>>> customer's clusters, and large users like Twitter and Yahoo that
> seem
> > >> to
> > >>> be
> > >>>> more affected are also the most able to patch this change in
> > >> themselves.
> > >>>>
> > >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> > >>>> breaking upgrade and downgrade expectations when it doesn't affect
> the
> > >>> (in
> > >>>> my experience) vast majority of users.
> > >>>>
> > >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> > but
> > >>>> will let him elaborate.
> > >>>>
> > >>>> Best,
> > >>>> Andrew
> > >>>>
> > >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > >>> wrote:
> > >>>>
> > >>>>> As of 2 days ago, there were already 135 jiras associated with
> 2.7.3,
> > >>>>> if *any* of them end up introducing a regression the inclusion of
> > >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> > back
> > >>>>> things out. If that happens to any substantial number of downstream
> > >>>>> folks, or any particularly vocal downstream folks, then it is very
> > >>>>> likely we'll lose the remaining trust of operators for rolling out
> > >>>>> maintenance releases. That's a pretty steep cost.
> > >>>>>
> > >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having
> to
> > >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >>>>> unreasonable burden.
> > >>>>>
> > >>>>> I agree that this fix is important, I just think we should either
> cut
> > >>>>> a version of 2.8 that includes it or find a way to do it that gives
> > an
> > >>>>> operational path for rolling downgrade.
> > >>>>>
> > >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > >>> wrote:
> > >>>>>> Thanks for bringing up this topic, Sean.
> > >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> > >>> HDFS-8791
> > >>>>> haven't been committed in so that's why we didn't discuss this
> > >> earlier.
> > >>>>>> I remember in JIRA discussion, we treated this layout change as a
> > >>>>> Blocker bug that fixing a significant performance regression before
> > >> but
> > >>> not
> > >>>>> a normal performance improvement. And I believe HDFS community
> > already
> > >>> did
> > >>>>> their best with careful and patient to deliver the fix and other
> > >> related
> > >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> > HDFS-8578,
> > >>> you
> > >>>>> can see 30+ rounds patch review back and forth by senior
> committers,
> > >>> not to
> > >>>>> mention the outstanding performance test data in HDFS-8791.
> > >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> > RM
> > >>> for
> > >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> > >> into
> > >>>>> 2.6.5 now.
> > >>>>>> Just my 2 cents.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Junping
> > >>>>>>
> > >>>>>> ________________________________________
> > >>>>>> From: Sean Busbey <bu...@cloudera.com>
> > >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> > >>>>>> To: hdfs-dev@hadoop.apache.org
> > >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >>>>> mapreduce-dev@hadoop.apache.org
> > >>>>>> Subject: Re: 2.7.3 release plan
> > >>>>>>
> > >>>>>> A layout change in a maintenance release sounds very risky. I saw
> > >> some
> > >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> > >> to
> > >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> > >> thought
> > >>>>>> we did RMs per release rather than per branch? No one claiming to
> > >> be a
> > >>>>>> release manager ever spoke up AFAICT.
> > >>>>>>
> > >>>>>> Should this change be included? Should it go into a special 2.8
> > >>>>>> release as mentioned in the ticket?
> > >>>>>>
> > >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> > >>>>>>> Thank you Vinod!
> > >>>>>>>
> > >>>>>>> FYI: 2.7.3 will be a bit special release.
> > >>>>>>>
> > >>>>>>> HDFS-8791 bumped up the datanode layout version,
> > >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >>>>>>> is impossible. We can rollback instead.
> > >>>>>>>
> > >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> > >>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Akira
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >>>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go
> out
> > >>>>> (which
> > >>>>>>>> did go out mid February). Got a little busy since.
> > >>>>>>>>
> > >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> > >> towards a
> > >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> > >> bug-fixes
> > >>>>> and *no*
> > >>>>>>>> features / improvements.
> > >>>>>>>>
> > >>>>>>>> I hope to cut an RC in a week - giving enough time for
> outstanding
> > >>>>> blocker
> > >>>>>>>> / critical issues. Will start moving out any tickets that are
> not
> > >>>>> blockers
> > >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> > >> critical
> > >>>>> tickets
> > >>>>>>>> outstanding as of now.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> +Vinod
> > >>>>>>>>
> > >>>>>>>> [1] 2.7.3 release blockers:
> > >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> busbey
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> busbey
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
> >
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks Chris.

+1 for reverting form 2.7. This is the least we should do. Can you help doing the needful?

I personally am not completely sold on a release with *only* the layout changes. Like I was saying before, we can let specific users backport this into specific 2.x branches they need and leave it only on trunk / branch-2. That said, I would love to hear others’ thoughts on this, but let’s fork that discussion off from this 2.7.3 thread. Re a fresh 2.8, I have renewed my efforts on 2.8 with a view of cutting an RC in weeks. Not sure if that does or doesn’t help this discussion.

Thanks
+Vinod

> On Apr 5, 2016, at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
> 
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
> 
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
> 
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
> 
> 
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
> 
>> I commented on the JIRA way back (see
>> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
>> saying what I said below. Unfortunately, I haven’t followed the patch along
>> after my initial comment.
>> 
>> This isn’t about any specific release - starting 2.6 we declared support
>> for rolling upgrades and downgrades. Any patch that breaks this should not
>> be in branch-2.
>> 
>> Two options from where I stand
>> (1) For folks who worked on the patch: Is there a way to make (a) the
>> upgrade-downgrade seamless for people who don’t care about this (b) and
>> have explicit documentation for people who care to switch this behavior on
>> and are willing to risk not having downgrades. If this means a new
>> configuration property, so be it. It’s a necessary evil.
>> (2) Just let specific users backport this into specific 2.x branches they
>> need and leave it only on trunk.
>> 
>> Unless this behavior stops breaking rolling upgrades/downgrades, I think
>> we should just revert it from branch-2 and definitely 2.7.3 as it stands
>> today.
>> 
>> +Vinod
>> 
>> 
>>> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
>>> 
>>> A few thoughts:
>>> 
>>> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
>>> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
>>> for data nodes depending on your setup.
>>> 
>>> 2. We have already deployed this patch internally so, with my Twitter hat
>>> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
>>> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
>>> releases on a large production cluster that has a diverse set of block
>> ids
>>> without this patch, especially if your data nodes have a large number of
>>> disks or you are using federation. To be clear though: this highly
>> depends
>>> on your setup and at a minimum you should verify that this regression
>> will
>>> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
>> a
>>> performance regression that gets worse over time. When you see it
>> happening
>>> on a live cluster, it is one of the harder issues to identify a root
>> cause
>>> and debug. I do understand that this is currently only affecting a
>> smaller
>>> number of users, but I also think this number has potential to increase
>> as
>>> time goes on. Maybe we can issue a warning in the release notes for
>> future
>>> 2.7.x and 2.6.x releases?
>>> 
>>> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
>> to
>>> this proposal on this thread) would be to cut a 2.8 release off of the
>>> 2.7.3 release with the new layout. What people currently think of as 2.8
>>> would then become 2.9. This would give customers a stable release that
>> they
>>> could deploy with the new layout and would not break upgrade and
>> downgrade
>>> expectations.
>>> 
>>> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>> 
>>>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
>> would
>>>> patch the release to revert HDFS-8791 before pushing it out to
>> production.
>>>> For what it's worth.
>>>> 
>>>> 
>>>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>>>> backported the parallel DN upgrade improvement (HDFS-8578) to
>> branch-2.6.
>>>>> HDFS-8578 is a very important related fix since otherwise upgrade will
>> be
>>>>> very slow.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>>>> maintenance release. I've only seen it crop up on a handful of our
>>>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>>>> to
>>>>> be
>>>>>> more affected are also the most able to patch this change in
>>>> themselves.
>>>>>> 
>>>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>>>> (in
>>>>>> my experience) vast majority of users.
>>>>>> 
>>>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
>> but
>>>>>> will let him elaborate.
>>>>>> 
>>>>>> Best,
>>>>>> Andrew
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>>>> wrote:
>>>>>> 
>>>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>>>> HDFS-8791 means that folks will have cluster downtime in order to
>> back
>>>>>>> things out. If that happens to any substantial number of downstream
>>>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>>>> maintenance releases. That's a pretty steep cost.
>>>>>>> 
>>>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>>>> unreasonable burden.
>>>>>>> 
>>>>>>> I agree that this fix is important, I just think we should either cut
>>>>>>> a version of 2.8 that includes it or find a way to do it that gives
>> an
>>>>>>> operational path for rolling downgrade.
>>>>>>> 
>>>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>>>> wrote:
>>>>>>>> Thanks for bringing up this topic, Sean.
>>>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>>>> HDFS-8791
>>>>>>> haven't been committed in so that's why we didn't discuss this
>>>> earlier.
>>>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>>>> Blocker bug that fixing a significant performance regression before
>>>> but
>>>>> not
>>>>>>> a normal performance improvement. And I believe HDFS community
>> already
>>>>> did
>>>>>>> their best with careful and patient to deliver the fix and other
>>>> related
>>>>>>> patches (like upgrade fix in HDFS-8578). Take an example of
>> HDFS-8578,
>>>>> you
>>>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>>>> not to
>>>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
>> RM
>>>>> for
>>>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>>>> into
>>>>>>> 2.6.5 now.
>>>>>>>> Just my 2 cents.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Junping
>>>>>>>> 
>>>>>>>> ________________________________________
>>>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> Subject: Re: 2.7.3 release plan
>>>>>>>> 
>>>>>>>> A layout change in a maintenance release sounds very risky. I saw
>>>> some
>>>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>>>> to
>>>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>>>> thought
>>>>>>>> we did RMs per release rather than per branch? No one claiming to
>>>> be a
>>>>>>>> release manager ever spoke up AFAICT.
>>>>>>>> 
>>>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>>>> release as mentioned in the ticket?
>>>>>>>> 
>>>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>>>> Thank you Vinod!
>>>>>>>>> 
>>>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>>>> 
>>>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>>>> is impossible. We can rollback instead.
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Akira
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>>> (which
>>>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>>>> 
>>>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>>>> towards a
>>>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>>>> bug-fixes
>>>>>>> and *no*
>>>>>>>>>> features / improvements.
>>>>>>>>>> 
>>>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>>> blocker
>>>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>>>> blockers
>>>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>>>> critical
>>>>>>> tickets
>>>>>>>>>> outstanding as of now.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> +Vinod
>>>>>>>>>> 
>>>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> busbey
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> busbey
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>>  - Andy
>>>> 
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>> (via Tom White)
>>>> 
>> 
>> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks Chris.

+1 for reverting form 2.7. This is the least we should do. Can you help doing the needful?

I personally am not completely sold on a release with *only* the layout changes. Like I was saying before, we can let specific users backport this into specific 2.x branches they need and leave it only on trunk / branch-2. That said, I would love to hear others’ thoughts on this, but let’s fork that discussion off from this 2.7.3 thread. Re a fresh 2.8, I have renewed my efforts on 2.8 with a view of cutting an RC in weeks. Not sure if that does or doesn’t help this discussion.

Thanks
+Vinod

> On Apr 5, 2016, at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
> 
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
> 
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
> 
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
> 
> 
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
> 
>> I commented on the JIRA way back (see
>> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
>> saying what I said below. Unfortunately, I haven’t followed the patch along
>> after my initial comment.
>> 
>> This isn’t about any specific release - starting 2.6 we declared support
>> for rolling upgrades and downgrades. Any patch that breaks this should not
>> be in branch-2.
>> 
>> Two options from where I stand
>> (1) For folks who worked on the patch: Is there a way to make (a) the
>> upgrade-downgrade seamless for people who don’t care about this (b) and
>> have explicit documentation for people who care to switch this behavior on
>> and are willing to risk not having downgrades. If this means a new
>> configuration property, so be it. It’s a necessary evil.
>> (2) Just let specific users backport this into specific 2.x branches they
>> need and leave it only on trunk.
>> 
>> Unless this behavior stops breaking rolling upgrades/downgrades, I think
>> we should just revert it from branch-2 and definitely 2.7.3 as it stands
>> today.
>> 
>> +Vinod
>> 
>> 
>>> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
>>> 
>>> A few thoughts:
>>> 
>>> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
>>> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
>>> for data nodes depending on your setup.
>>> 
>>> 2. We have already deployed this patch internally so, with my Twitter hat
>>> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
>>> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
>>> releases on a large production cluster that has a diverse set of block
>> ids
>>> without this patch, especially if your data nodes have a large number of
>>> disks or you are using federation. To be clear though: this highly
>> depends
>>> on your setup and at a minimum you should verify that this regression
>> will
>>> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
>> a
>>> performance regression that gets worse over time. When you see it
>> happening
>>> on a live cluster, it is one of the harder issues to identify a root
>> cause
>>> and debug. I do understand that this is currently only affecting a
>> smaller
>>> number of users, but I also think this number has potential to increase
>> as
>>> time goes on. Maybe we can issue a warning in the release notes for
>> future
>>> 2.7.x and 2.6.x releases?
>>> 
>>> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
>> to
>>> this proposal on this thread) would be to cut a 2.8 release off of the
>>> 2.7.3 release with the new layout. What people currently think of as 2.8
>>> would then become 2.9. This would give customers a stable release that
>> they
>>> could deploy with the new layout and would not break upgrade and
>> downgrade
>>> expectations.
>>> 
>>> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>> 
>>>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
>> would
>>>> patch the release to revert HDFS-8791 before pushing it out to
>> production.
>>>> For what it's worth.
>>>> 
>>>> 
>>>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>>>> backported the parallel DN upgrade improvement (HDFS-8578) to
>> branch-2.6.
>>>>> HDFS-8578 is a very important related fix since otherwise upgrade will
>> be
>>>>> very slow.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>>>> maintenance release. I've only seen it crop up on a handful of our
>>>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>>>> to
>>>>> be
>>>>>> more affected are also the most able to patch this change in
>>>> themselves.
>>>>>> 
>>>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>>>> (in
>>>>>> my experience) vast majority of users.
>>>>>> 
>>>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
>> but
>>>>>> will let him elaborate.
>>>>>> 
>>>>>> Best,
>>>>>> Andrew
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>>>> wrote:
>>>>>> 
>>>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>>>> HDFS-8791 means that folks will have cluster downtime in order to
>> back
>>>>>>> things out. If that happens to any substantial number of downstream
>>>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>>>> maintenance releases. That's a pretty steep cost.
>>>>>>> 
>>>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>>>> unreasonable burden.
>>>>>>> 
>>>>>>> I agree that this fix is important, I just think we should either cut
>>>>>>> a version of 2.8 that includes it or find a way to do it that gives
>> an
>>>>>>> operational path for rolling downgrade.
>>>>>>> 
>>>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>>>> wrote:
>>>>>>>> Thanks for bringing up this topic, Sean.
>>>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>>>> HDFS-8791
>>>>>>> haven't been committed in so that's why we didn't discuss this
>>>> earlier.
>>>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>>>> Blocker bug that fixing a significant performance regression before
>>>> but
>>>>> not
>>>>>>> a normal performance improvement. And I believe HDFS community
>> already
>>>>> did
>>>>>>> their best with careful and patient to deliver the fix and other
>>>> related
>>>>>>> patches (like upgrade fix in HDFS-8578). Take an example of
>> HDFS-8578,
>>>>> you
>>>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>>>> not to
>>>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
>> RM
>>>>> for
>>>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>>>> into
>>>>>>> 2.6.5 now.
>>>>>>>> Just my 2 cents.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Junping
>>>>>>>> 
>>>>>>>> ________________________________________
>>>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> Subject: Re: 2.7.3 release plan
>>>>>>>> 
>>>>>>>> A layout change in a maintenance release sounds very risky. I saw
>>>> some
>>>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>>>> to
>>>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>>>> thought
>>>>>>>> we did RMs per release rather than per branch? No one claiming to
>>>> be a
>>>>>>>> release manager ever spoke up AFAICT.
>>>>>>>> 
>>>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>>>> release as mentioned in the ticket?
>>>>>>>> 
>>>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>>>> Thank you Vinod!
>>>>>>>>> 
>>>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>>>> 
>>>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>>>> is impossible. We can rollback instead.
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Akira
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>>> (which
>>>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>>>> 
>>>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>>>> towards a
>>>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>>>> bug-fixes
>>>>>>> and *no*
>>>>>>>>>> features / improvements.
>>>>>>>>>> 
>>>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>>> blocker
>>>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>>>> blockers
>>>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>>>> critical
>>>>>>> tickets
>>>>>>>>>> outstanding as of now.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> +Vinod
>>>>>>>>>> 
>>>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> busbey
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> busbey
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>>  - Andy
>>>> 
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>> (via Tom White)
>>>> 
>> 
>> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks Chris.

+1 for reverting form 2.7. This is the least we should do. Can you help doing the needful?

I personally am not completely sold on a release with *only* the layout changes. Like I was saying before, we can let specific users backport this into specific 2.x branches they need and leave it only on trunk / branch-2. That said, I would love to hear others’ thoughts on this, but let’s fork that discussion off from this 2.7.3 thread. Re a fresh 2.8, I have renewed my efforts on 2.8 with a view of cutting an RC in weeks. Not sure if that does or doesn’t help this discussion.

Thanks
+Vinod

> On Apr 5, 2016, at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
> 
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
> 
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
> 
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
> 
> 
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
> 
>> I commented on the JIRA way back (see
>> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
>> saying what I said below. Unfortunately, I haven’t followed the patch along
>> after my initial comment.
>> 
>> This isn’t about any specific release - starting 2.6 we declared support
>> for rolling upgrades and downgrades. Any patch that breaks this should not
>> be in branch-2.
>> 
>> Two options from where I stand
>> (1) For folks who worked on the patch: Is there a way to make (a) the
>> upgrade-downgrade seamless for people who don’t care about this (b) and
>> have explicit documentation for people who care to switch this behavior on
>> and are willing to risk not having downgrades. If this means a new
>> configuration property, so be it. It’s a necessary evil.
>> (2) Just let specific users backport this into specific 2.x branches they
>> need and leave it only on trunk.
>> 
>> Unless this behavior stops breaking rolling upgrades/downgrades, I think
>> we should just revert it from branch-2 and definitely 2.7.3 as it stands
>> today.
>> 
>> +Vinod
>> 
>> 
>>> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
>>> 
>>> A few thoughts:
>>> 
>>> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
>>> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
>>> for data nodes depending on your setup.
>>> 
>>> 2. We have already deployed this patch internally so, with my Twitter hat
>>> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
>>> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
>>> releases on a large production cluster that has a diverse set of block
>> ids
>>> without this patch, especially if your data nodes have a large number of
>>> disks or you are using federation. To be clear though: this highly
>> depends
>>> on your setup and at a minimum you should verify that this regression
>> will
>>> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
>> a
>>> performance regression that gets worse over time. When you see it
>> happening
>>> on a live cluster, it is one of the harder issues to identify a root
>> cause
>>> and debug. I do understand that this is currently only affecting a
>> smaller
>>> number of users, but I also think this number has potential to increase
>> as
>>> time goes on. Maybe we can issue a warning in the release notes for
>> future
>>> 2.7.x and 2.6.x releases?
>>> 
>>> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
>> to
>>> this proposal on this thread) would be to cut a 2.8 release off of the
>>> 2.7.3 release with the new layout. What people currently think of as 2.8
>>> would then become 2.9. This would give customers a stable release that
>> they
>>> could deploy with the new layout and would not break upgrade and
>> downgrade
>>> expectations.
>>> 
>>> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>> 
>>>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
>> would
>>>> patch the release to revert HDFS-8791 before pushing it out to
>> production.
>>>> For what it's worth.
>>>> 
>>>> 
>>>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>>>> backported the parallel DN upgrade improvement (HDFS-8578) to
>> branch-2.6.
>>>>> HDFS-8578 is a very important related fix since otherwise upgrade will
>> be
>>>>> very slow.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>>>> maintenance release. I've only seen it crop up on a handful of our
>>>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>>>> to
>>>>> be
>>>>>> more affected are also the most able to patch this change in
>>>> themselves.
>>>>>> 
>>>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>>>> (in
>>>>>> my experience) vast majority of users.
>>>>>> 
>>>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
>> but
>>>>>> will let him elaborate.
>>>>>> 
>>>>>> Best,
>>>>>> Andrew
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>>>> wrote:
>>>>>> 
>>>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>>>> HDFS-8791 means that folks will have cluster downtime in order to
>> back
>>>>>>> things out. If that happens to any substantial number of downstream
>>>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>>>> maintenance releases. That's a pretty steep cost.
>>>>>>> 
>>>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>>>> unreasonable burden.
>>>>>>> 
>>>>>>> I agree that this fix is important, I just think we should either cut
>>>>>>> a version of 2.8 that includes it or find a way to do it that gives
>> an
>>>>>>> operational path for rolling downgrade.
>>>>>>> 
>>>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>>>> wrote:
>>>>>>>> Thanks for bringing up this topic, Sean.
>>>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>>>> HDFS-8791
>>>>>>> haven't been committed in so that's why we didn't discuss this
>>>> earlier.
>>>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>>>> Blocker bug that fixing a significant performance regression before
>>>> but
>>>>> not
>>>>>>> a normal performance improvement. And I believe HDFS community
>> already
>>>>> did
>>>>>>> their best with careful and patient to deliver the fix and other
>>>> related
>>>>>>> patches (like upgrade fix in HDFS-8578). Take an example of
>> HDFS-8578,
>>>>> you
>>>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>>>> not to
>>>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
>> RM
>>>>> for
>>>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>>>> into
>>>>>>> 2.6.5 now.
>>>>>>>> Just my 2 cents.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Junping
>>>>>>>> 
>>>>>>>> ________________________________________
>>>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> Subject: Re: 2.7.3 release plan
>>>>>>>> 
>>>>>>>> A layout change in a maintenance release sounds very risky. I saw
>>>> some
>>>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>>>> to
>>>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>>>> thought
>>>>>>>> we did RMs per release rather than per branch? No one claiming to
>>>> be a
>>>>>>>> release manager ever spoke up AFAICT.
>>>>>>>> 
>>>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>>>> release as mentioned in the ticket?
>>>>>>>> 
>>>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>>>> Thank you Vinod!
>>>>>>>>> 
>>>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>>>> 
>>>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>>>> is impossible. We can rollback instead.
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Akira
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>>> (which
>>>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>>>> 
>>>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>>>> towards a
>>>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>>>> bug-fixes
>>>>>>> and *no*
>>>>>>>>>> features / improvements.
>>>>>>>>>> 
>>>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>>> blocker
>>>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>>>> blockers
>>>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>>>> critical
>>>>>>> tickets
>>>>>>>>>> outstanding as of now.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> +Vinod
>>>>>>>>>> 
>>>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> busbey
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> busbey
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>>  - Andy
>>>> 
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>> (via Tom White)
>>>> 
>> 
>> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks Chris.

+1 for reverting form 2.7. This is the least we should do. Can you help doing the needful?

I personally am not completely sold on a release with *only* the layout changes. Like I was saying before, we can let specific users backport this into specific 2.x branches they need and leave it only on trunk / branch-2. That said, I would love to hear others’ thoughts on this, but let’s fork that discussion off from this 2.7.3 thread. Re a fresh 2.8, I have renewed my efforts on 2.8 with a view of cutting an RC in weeks. Not sure if that does or doesn’t help this discussion.

Thanks
+Vinod

> On Apr 5, 2016, at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
> 
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
> 
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
> 
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
> 
> 
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
> 
>> I commented on the JIRA way back (see
>> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
>> saying what I said below. Unfortunately, I haven’t followed the patch along
>> after my initial comment.
>> 
>> This isn’t about any specific release - starting 2.6 we declared support
>> for rolling upgrades and downgrades. Any patch that breaks this should not
>> be in branch-2.
>> 
>> Two options from where I stand
>> (1) For folks who worked on the patch: Is there a way to make (a) the
>> upgrade-downgrade seamless for people who don’t care about this (b) and
>> have explicit documentation for people who care to switch this behavior on
>> and are willing to risk not having downgrades. If this means a new
>> configuration property, so be it. It’s a necessary evil.
>> (2) Just let specific users backport this into specific 2.x branches they
>> need and leave it only on trunk.
>> 
>> Unless this behavior stops breaking rolling upgrades/downgrades, I think
>> we should just revert it from branch-2 and definitely 2.7.3 as it stands
>> today.
>> 
>> +Vinod
>> 
>> 
>>> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
>>> 
>>> A few thoughts:
>>> 
>>> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
>>> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
>>> for data nodes depending on your setup.
>>> 
>>> 2. We have already deployed this patch internally so, with my Twitter hat
>>> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
>>> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
>>> releases on a large production cluster that has a diverse set of block
>> ids
>>> without this patch, especially if your data nodes have a large number of
>>> disks or you are using federation. To be clear though: this highly
>> depends
>>> on your setup and at a minimum you should verify that this regression
>> will
>>> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
>> a
>>> performance regression that gets worse over time. When you see it
>> happening
>>> on a live cluster, it is one of the harder issues to identify a root
>> cause
>>> and debug. I do understand that this is currently only affecting a
>> smaller
>>> number of users, but I also think this number has potential to increase
>> as
>>> time goes on. Maybe we can issue a warning in the release notes for
>> future
>>> 2.7.x and 2.6.x releases?
>>> 
>>> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
>> to
>>> this proposal on this thread) would be to cut a 2.8 release off of the
>>> 2.7.3 release with the new layout. What people currently think of as 2.8
>>> would then become 2.9. This would give customers a stable release that
>> they
>>> could deploy with the new layout and would not break upgrade and
>> downgrade
>>> expectations.
>>> 
>>> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>> 
>>>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
>> would
>>>> patch the release to revert HDFS-8791 before pushing it out to
>> production.
>>>> For what it's worth.
>>>> 
>>>> 
>>>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>>>> wrote:
>>>> 
>>>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>>>> backported the parallel DN upgrade improvement (HDFS-8578) to
>> branch-2.6.
>>>>> HDFS-8578 is a very important related fix since otherwise upgrade will
>> be
>>>>> very slow.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
>> andrew.wang@cloudera.com>
>>>>> wrote:
>>>>> 
>>>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>>>> maintenance release. I've only seen it crop up on a handful of our
>>>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>>>> to
>>>>> be
>>>>>> more affected are also the most able to patch this change in
>>>> themselves.
>>>>>> 
>>>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>>>> (in
>>>>>> my experience) vast majority of users.
>>>>>> 
>>>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
>> but
>>>>>> will let him elaborate.
>>>>>> 
>>>>>> Best,
>>>>>> Andrew
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>>>> wrote:
>>>>>> 
>>>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>>>> HDFS-8791 means that folks will have cluster downtime in order to
>> back
>>>>>>> things out. If that happens to any substantial number of downstream
>>>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>>>> maintenance releases. That's a pretty steep cost.
>>>>>>> 
>>>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>>>> unreasonable burden.
>>>>>>> 
>>>>>>> I agree that this fix is important, I just think we should either cut
>>>>>>> a version of 2.8 that includes it or find a way to do it that gives
>> an
>>>>>>> operational path for rolling downgrade.
>>>>>>> 
>>>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>>>> wrote:
>>>>>>>> Thanks for bringing up this topic, Sean.
>>>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>>>> HDFS-8791
>>>>>>> haven't been committed in so that's why we didn't discuss this
>>>> earlier.
>>>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>>>> Blocker bug that fixing a significant performance regression before
>>>> but
>>>>> not
>>>>>>> a normal performance improvement. And I believe HDFS community
>> already
>>>>> did
>>>>>>> their best with careful and patient to deliver the fix and other
>>>> related
>>>>>>> patches (like upgrade fix in HDFS-8578). Take an example of
>> HDFS-8578,
>>>>> you
>>>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>>>> not to
>>>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
>> RM
>>>>> for
>>>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>>>> into
>>>>>>> 2.6.5 now.
>>>>>>>> Just my 2 cents.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Junping
>>>>>>>> 
>>>>>>>> ________________________________________
>>>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> Subject: Re: 2.7.3 release plan
>>>>>>>> 
>>>>>>>> A layout change in a maintenance release sounds very risky. I saw
>>>> some
>>>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>>>> to
>>>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>>>> thought
>>>>>>>> we did RMs per release rather than per branch? No one claiming to
>>>> be a
>>>>>>>> release manager ever spoke up AFAICT.
>>>>>>>> 
>>>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>>>> release as mentioned in the ticket?
>>>>>>>> 
>>>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>>>> Thank you Vinod!
>>>>>>>>> 
>>>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>>>> 
>>>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>>>> is impossible. We can rollback instead.
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Akira
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>>> (which
>>>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>>>> 
>>>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>>>> towards a
>>>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>>>> bug-fixes
>>>>>>> and *no*
>>>>>>>>>> features / improvements.
>>>>>>>>>> 
>>>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>>> blocker
>>>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>>>> blockers
>>>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>>>> critical
>>>>>>> tickets
>>>>>>>>>> outstanding as of now.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> +Vinod
>>>>>>>>>> 
>>>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> busbey
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> busbey
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>>  - Andy
>>>> 
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>> (via Tom White)
>>>> 
>> 
>> 

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

I'm +1 for ctrezzo's proposal, happy to do the revert from branch-2.7 if
this is acceptable to Vinod.

There's some additional discussion on the HDFS-8791 JIRA for those who are
only following this email thread.

Best,
Andrew

On Tue, Apr 5, 2016 at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:

> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
>
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
>
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
>
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
>
>
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> > I commented on the JIRA way back (see
> >
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666
> ),
> > saying what I said below. Unfortunately, I haven’t followed the patch
> along
> > after my initial comment.
> >
> > This isn’t about any specific release - starting 2.6 we declared support
> > for rolling upgrades and downgrades. Any patch that breaks this should
> not
> > be in branch-2.
> >
> > Two options from where I stand
> >  (1) For folks who worked on the patch: Is there a way to make (a) the
> > upgrade-downgrade seamless for people who don’t care about this (b) and
> > have explicit documentation for people who care to switch this behavior
> on
> > and are willing to risk not having downgrades. If this means a new
> > configuration property, so be it. It’s a necessary evil.
> >  (2) Just let specific users backport this into specific 2.x branches
> they
> > need and leave it only on trunk.
> >
> > Unless this behavior stops breaking rolling upgrades/downgrades, I think
> > we should just revert it from branch-2 and definitely 2.7.3 as it stands
> > today.
> >
> > +Vinod
> >
> >
> > > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> > >
> > > A few thoughts:
> > >
> > > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > > prerequisite for HDFS-8791. Without that patch, upgrades can be very
> slow
> > > for data nodes depending on your setup.
> > >
> > > 2. We have already deployed this patch internally so, with my Twitter
> hat
> > > on, I would be perfectly happy as long as it makes it into trunk and
> 2.8.
> > > That being said, I would be hesitant to deploy the current 2.7.x or
> 2.6.x
> > > releases on a large production cluster that has a diverse set of block
> > ids
> > > without this patch, especially if your data nodes have a large number
> of
> > > disks or you are using federation. To be clear though: this highly
> > depends
> > > on your setup and at a minimum you should verify that this regression
> > will
> > > not affect you. The current block-id based layout in 2.6.x and 2.7.2
> has
> > a
> > > performance regression that gets worse over time. When you see it
> > happening
> > > on a live cluster, it is one of the harder issues to identify a root
> > cause
> > > and debug. I do understand that this is currently only affecting a
> > smaller
> > > number of users, but I also think this number has potential to increase
> > as
> > > time goes on. Maybe we can issue a warning in the release notes for
> > future
> > > 2.7.x and 2.6.x releases?
> > >
> > > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> > to
> > > this proposal on this thread) would be to cut a 2.8 release off of the
> > > 2.7.3 release with the new layout. What people currently think of as
> 2.8
> > > would then become 2.9. This would give customers a stable release that
> > they
> > > could deploy with the new layout and would not break upgrade and
> > downgrade
> > > expectations.
> > >
> > > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> > would
> > >> patch the release to revert HDFS-8791 before pushing it out to
> > production.
> > >> For what it's worth.
> > >>
> > >>
> > >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > >> wrote:
> > >>
> > >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> > branch-2.6.
> > >>> HDFS-8578 is a very important related fix since otherwise upgrade
> will
> > be
> > >>> very slow.
> > >>>
> > >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > >>>> maintenance release. I've only seen it crop up on a handful of our
> > >>>> customer's clusters, and large users like Twitter and Yahoo that
> seem
> > >> to
> > >>> be
> > >>>> more affected are also the most able to patch this change in
> > >> themselves.
> > >>>>
> > >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> > >>>> breaking upgrade and downgrade expectations when it doesn't affect
> the
> > >>> (in
> > >>>> my experience) vast majority of users.
> > >>>>
> > >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> > but
> > >>>> will let him elaborate.
> > >>>>
> > >>>> Best,
> > >>>> Andrew
> > >>>>
> > >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > >>> wrote:
> > >>>>
> > >>>>> As of 2 days ago, there were already 135 jiras associated with
> 2.7.3,
> > >>>>> if *any* of them end up introducing a regression the inclusion of
> > >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> > back
> > >>>>> things out. If that happens to any substantial number of downstream
> > >>>>> folks, or any particularly vocal downstream folks, then it is very
> > >>>>> likely we'll lose the remaining trust of operators for rolling out
> > >>>>> maintenance releases. That's a pretty steep cost.
> > >>>>>
> > >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having
> to
> > >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >>>>> unreasonable burden.
> > >>>>>
> > >>>>> I agree that this fix is important, I just think we should either
> cut
> > >>>>> a version of 2.8 that includes it or find a way to do it that gives
> > an
> > >>>>> operational path for rolling downgrade.
> > >>>>>
> > >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > >>> wrote:
> > >>>>>> Thanks for bringing up this topic, Sean.
> > >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> > >>> HDFS-8791
> > >>>>> haven't been committed in so that's why we didn't discuss this
> > >> earlier.
> > >>>>>> I remember in JIRA discussion, we treated this layout change as a
> > >>>>> Blocker bug that fixing a significant performance regression before
> > >> but
> > >>> not
> > >>>>> a normal performance improvement. And I believe HDFS community
> > already
> > >>> did
> > >>>>> their best with careful and patient to deliver the fix and other
> > >> related
> > >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> > HDFS-8578,
> > >>> you
> > >>>>> can see 30+ rounds patch review back and forth by senior
> committers,
> > >>> not to
> > >>>>> mention the outstanding performance test data in HDFS-8791.
> > >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> > RM
> > >>> for
> > >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> > >> into
> > >>>>> 2.6.5 now.
> > >>>>>> Just my 2 cents.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Junping
> > >>>>>>
> > >>>>>> ________________________________________
> > >>>>>> From: Sean Busbey <bu...@cloudera.com>
> > >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> > >>>>>> To: hdfs-dev@hadoop.apache.org
> > >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >>>>> mapreduce-dev@hadoop.apache.org
> > >>>>>> Subject: Re: 2.7.3 release plan
> > >>>>>>
> > >>>>>> A layout change in a maintenance release sounds very risky. I saw
> > >> some
> > >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> > >> to
> > >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> > >> thought
> > >>>>>> we did RMs per release rather than per branch? No one claiming to
> > >> be a
> > >>>>>> release manager ever spoke up AFAICT.
> > >>>>>>
> > >>>>>> Should this change be included? Should it go into a special 2.8
> > >>>>>> release as mentioned in the ticket?
> > >>>>>>
> > >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> > >>>>>>> Thank you Vinod!
> > >>>>>>>
> > >>>>>>> FYI: 2.7.3 will be a bit special release.
> > >>>>>>>
> > >>>>>>> HDFS-8791 bumped up the datanode layout version,
> > >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >>>>>>> is impossible. We can rollback instead.
> > >>>>>>>
> > >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> > >>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Akira
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >>>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go
> out
> > >>>>> (which
> > >>>>>>>> did go out mid February). Got a little busy since.
> > >>>>>>>>
> > >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> > >> towards a
> > >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> > >> bug-fixes
> > >>>>> and *no*
> > >>>>>>>> features / improvements.
> > >>>>>>>>
> > >>>>>>>> I hope to cut an RC in a week - giving enough time for
> outstanding
> > >>>>> blocker
> > >>>>>>>> / critical issues. Will start moving out any tickets that are
> not
> > >>>>> blockers
> > >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> > >> critical
> > >>>>> tickets
> > >>>>>>>> outstanding as of now.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> +Vinod
> > >>>>>>>>
> > >>>>>>>> [1] 2.7.3 release blockers:
> > >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> busbey
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> busbey
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
> >
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

I'm +1 for ctrezzo's proposal, happy to do the revert from branch-2.7 if
this is acceptable to Vinod.

There's some additional discussion on the HDFS-8791 JIRA for those who are
only following this email thread.

Best,
Andrew

On Tue, Apr 5, 2016 at 2:03 PM, Chris Trezzo <ct...@gmail.com> wrote:

> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
>
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
>
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
>
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
>
>
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> > I commented on the JIRA way back (see
> >
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666
> ),
> > saying what I said below. Unfortunately, I haven’t followed the patch
> along
> > after my initial comment.
> >
> > This isn’t about any specific release - starting 2.6 we declared support
> > for rolling upgrades and downgrades. Any patch that breaks this should
> not
> > be in branch-2.
> >
> > Two options from where I stand
> >  (1) For folks who worked on the patch: Is there a way to make (a) the
> > upgrade-downgrade seamless for people who don’t care about this (b) and
> > have explicit documentation for people who care to switch this behavior
> on
> > and are willing to risk not having downgrades. If this means a new
> > configuration property, so be it. It’s a necessary evil.
> >  (2) Just let specific users backport this into specific 2.x branches
> they
> > need and leave it only on trunk.
> >
> > Unless this behavior stops breaking rolling upgrades/downgrades, I think
> > we should just revert it from branch-2 and definitely 2.7.3 as it stands
> > today.
> >
> > +Vinod
> >
> >
> > > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> > >
> > > A few thoughts:
> > >
> > > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > > prerequisite for HDFS-8791. Without that patch, upgrades can be very
> slow
> > > for data nodes depending on your setup.
> > >
> > > 2. We have already deployed this patch internally so, with my Twitter
> hat
> > > on, I would be perfectly happy as long as it makes it into trunk and
> 2.8.
> > > That being said, I would be hesitant to deploy the current 2.7.x or
> 2.6.x
> > > releases on a large production cluster that has a diverse set of block
> > ids
> > > without this patch, especially if your data nodes have a large number
> of
> > > disks or you are using federation. To be clear though: this highly
> > depends
> > > on your setup and at a minimum you should verify that this regression
> > will
> > > not affect you. The current block-id based layout in 2.6.x and 2.7.2
> has
> > a
> > > performance regression that gets worse over time. When you see it
> > happening
> > > on a live cluster, it is one of the harder issues to identify a root
> > cause
> > > and debug. I do understand that this is currently only affecting a
> > smaller
> > > number of users, but I also think this number has potential to increase
> > as
> > > time goes on. Maybe we can issue a warning in the release notes for
> > future
> > > 2.7.x and 2.6.x releases?
> > >
> > > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> > to
> > > this proposal on this thread) would be to cut a 2.8 release off of the
> > > 2.7.3 release with the new layout. What people currently think of as
> 2.8
> > > would then become 2.9. This would give customers a stable release that
> > they
> > > could deploy with the new layout and would not break upgrade and
> > downgrade
> > > expectations.
> > >
> > > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> > would
> > >> patch the release to revert HDFS-8791 before pushing it out to
> > production.
> > >> For what it's worth.
> > >>
> > >>
> > >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > >> wrote:
> > >>
> > >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> > branch-2.6.
> > >>> HDFS-8578 is a very important related fix since otherwise upgrade
> will
> > be
> > >>> very slow.
> > >>>
> > >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > >>>> maintenance release. I've only seen it crop up on a handful of our
> > >>>> customer's clusters, and large users like Twitter and Yahoo that
> seem
> > >> to
> > >>> be
> > >>>> more affected are also the most able to patch this change in
> > >> themselves.
> > >>>>
> > >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> > >>>> breaking upgrade and downgrade expectations when it doesn't affect
> the
> > >>> (in
> > >>>> my experience) vast majority of users.
> > >>>>
> > >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> > but
> > >>>> will let him elaborate.
> > >>>>
> > >>>> Best,
> > >>>> Andrew
> > >>>>
> > >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > >>> wrote:
> > >>>>
> > >>>>> As of 2 days ago, there were already 135 jiras associated with
> 2.7.3,
> > >>>>> if *any* of them end up introducing a regression the inclusion of
> > >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> > back
> > >>>>> things out. If that happens to any substantial number of downstream
> > >>>>> folks, or any particularly vocal downstream folks, then it is very
> > >>>>> likely we'll lose the remaining trust of operators for rolling out
> > >>>>> maintenance releases. That's a pretty steep cost.
> > >>>>>
> > >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having
> to
> > >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >>>>> unreasonable burden.
> > >>>>>
> > >>>>> I agree that this fix is important, I just think we should either
> cut
> > >>>>> a version of 2.8 that includes it or find a way to do it that gives
> > an
> > >>>>> operational path for rolling downgrade.
> > >>>>>
> > >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > >>> wrote:
> > >>>>>> Thanks for bringing up this topic, Sean.
> > >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> > >>> HDFS-8791
> > >>>>> haven't been committed in so that's why we didn't discuss this
> > >> earlier.
> > >>>>>> I remember in JIRA discussion, we treated this layout change as a
> > >>>>> Blocker bug that fixing a significant performance regression before
> > >> but
> > >>> not
> > >>>>> a normal performance improvement. And I believe HDFS community
> > already
> > >>> did
> > >>>>> their best with careful and patient to deliver the fix and other
> > >> related
> > >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> > HDFS-8578,
> > >>> you
> > >>>>> can see 30+ rounds patch review back and forth by senior
> committers,
> > >>> not to
> > >>>>> mention the outstanding performance test data in HDFS-8791.
> > >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> > RM
> > >>> for
> > >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> > >> into
> > >>>>> 2.6.5 now.
> > >>>>>> Just my 2 cents.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Junping
> > >>>>>>
> > >>>>>> ________________________________________
> > >>>>>> From: Sean Busbey <bu...@cloudera.com>
> > >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> > >>>>>> To: hdfs-dev@hadoop.apache.org
> > >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >>>>> mapreduce-dev@hadoop.apache.org
> > >>>>>> Subject: Re: 2.7.3 release plan
> > >>>>>>
> > >>>>>> A layout change in a maintenance release sounds very risky. I saw
> > >> some
> > >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> > >> to
> > >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> > >> thought
> > >>>>>> we did RMs per release rather than per branch? No one claiming to
> > >> be a
> > >>>>>> release manager ever spoke up AFAICT.
> > >>>>>>
> > >>>>>> Should this change be included? Should it go into a special 2.8
> > >>>>>> release as mentioned in the ticket?
> > >>>>>>
> > >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> > >>>>>>> Thank you Vinod!
> > >>>>>>>
> > >>>>>>> FYI: 2.7.3 will be a bit special release.
> > >>>>>>>
> > >>>>>>> HDFS-8791 bumped up the datanode layout version,
> > >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >>>>>>> is impossible. We can rollback instead.
> > >>>>>>>
> > >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> > >>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Akira
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >>>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go
> out
> > >>>>> (which
> > >>>>>>>> did go out mid February). Got a little busy since.
> > >>>>>>>>
> > >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> > >> towards a
> > >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> > >> bug-fixes
> > >>>>> and *no*
> > >>>>>>>> features / improvements.
> > >>>>>>>>
> > >>>>>>>> I hope to cut an RC in a week - giving enough time for
> outstanding
> > >>>>> blocker
> > >>>>>>>> / critical issues. Will start moving out any tickets that are
> not
> > >>>>> blockers
> > >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> > >> critical
> > >>>>> tickets
> > >>>>>>>> outstanding as of now.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> +Vinod
> > >>>>>>>>
> > >>>>>>>> [1] 2.7.3 release blockers:
> > >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> busbey
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> busbey
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>   - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
> >
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

In light of the additional conversation on HDFS-8791, I would like to
re-propose the following:

1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
layout change currently does not support downgrades which breaks our
upgrade/downgrade policies for dot releases.

2. Cut a 2.8 release off of the 2.7.3 release with the addition of
HDFS-8791. This would give customers a stable release that they could
deploy with the new layout. As discussed on the jira, this is still in line
with user expectation for minor releases as we have done layout changes in
a number of 2.x minor releases already. The current 2.8 would become 2.9
and continue its current release schedule.

What does everyone think? If unsupported downgrades between minor releases
is still not agreeable, then as stated by Vinod, we would need to either
add support for downgrades with dn layout changes or revert the layout
change from branch-2. If we are OK with the layout change in a minor
release, but think that the issue does not affect enough customers to
warrant a separate release, we could simply leave it in branch-2 and let it
be released with the current 2.8.


On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> I commented on the JIRA way back (see
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
> saying what I said below. Unfortunately, I haven’t followed the patch along
> after my initial comment.
>
> This isn’t about any specific release - starting 2.6 we declared support
> for rolling upgrades and downgrades. Any patch that breaks this should not
> be in branch-2.
>
> Two options from where I stand
>  (1) For folks who worked on the patch: Is there a way to make (a) the
> upgrade-downgrade seamless for people who don’t care about this (b) and
> have explicit documentation for people who care to switch this behavior on
> and are willing to risk not having downgrades. If this means a new
> configuration property, so be it. It’s a necessary evil.
>  (2) Just let specific users backport this into specific 2.x branches they
> need and leave it only on trunk.
>
> Unless this behavior stops breaking rolling upgrades/downgrades, I think
> we should just revert it from branch-2 and definitely 2.7.3 as it stands
> today.
>
> +Vinod
>
>
> > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > A few thoughts:
> >
> > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> > for data nodes depending on your setup.
> >
> > 2. We have already deployed this patch internally so, with my Twitter hat
> > on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> > That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> > releases on a large production cluster that has a diverse set of block
> ids
> > without this patch, especially if your data nodes have a large number of
> > disks or you are using federation. To be clear though: this highly
> depends
> > on your setup and at a minimum you should verify that this regression
> will
> > not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
> a
> > performance regression that gets worse over time. When you see it
> happening
> > on a live cluster, it is one of the harder issues to identify a root
> cause
> > and debug. I do understand that this is currently only affecting a
> smaller
> > number of users, but I also think this number has potential to increase
> as
> > time goes on. Maybe we can issue a warning in the release notes for
> future
> > 2.7.x and 2.6.x releases?
> >
> > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> to
> > this proposal on this thread) would be to cut a 2.8 release off of the
> > 2.7.3 release with the new layout. What people currently think of as 2.8
> > would then become 2.9. This would give customers a stable release that
> they
> > could deploy with the new layout and would not break upgrade and
> downgrade
> > expectations.
> >
> > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> would
> >> patch the release to revert HDFS-8791 before pushing it out to
> production.
> >> For what it's worth.
> >>
> >>
> >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> >> wrote:
> >>
> >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> branch-2.6.
> >>> HDFS-8578 is a very important related fix since otherwise upgrade will
> be
> >>> very slow.
> >>>
> >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>> wrote:
> >>>
> >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> >>>> maintenance release. I've only seen it crop up on a handful of our
> >>>> customer's clusters, and large users like Twitter and Yahoo that seem
> >> to
> >>> be
> >>>> more affected are also the most able to patch this change in
> >> themselves.
> >>>>
> >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> >>>> breaking upgrade and downgrade expectations when it doesn't affect the
> >>> (in
> >>>> my experience) vast majority of users.
> >>>>
> >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> but
> >>>> will let him elaborate.
> >>>>
> >>>> Best,
> >>>> Andrew
> >>>>
> >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> >>> wrote:
> >>>>
> >>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >>>>> if *any* of them end up introducing a regression the inclusion of
> >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> back
> >>>>> things out. If that happens to any substantial number of downstream
> >>>>> folks, or any particularly vocal downstream folks, then it is very
> >>>>> likely we'll lose the remaining trust of operators for rolling out
> >>>>> maintenance releases. That's a pretty steep cost.
> >>>>>
> >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >>>>> unreasonable burden.
> >>>>>
> >>>>> I agree that this fix is important, I just think we should either cut
> >>>>> a version of 2.8 that includes it or find a way to do it that gives
> an
> >>>>> operational path for rolling downgrade.
> >>>>>
> >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> >>> wrote:
> >>>>>> Thanks for bringing up this topic, Sean.
> >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> >>> HDFS-8791
> >>>>> haven't been committed in so that's why we didn't discuss this
> >> earlier.
> >>>>>> I remember in JIRA discussion, we treated this layout change as a
> >>>>> Blocker bug that fixing a significant performance regression before
> >> but
> >>> not
> >>>>> a normal performance improvement. And I believe HDFS community
> already
> >>> did
> >>>>> their best with careful and patient to deliver the fix and other
> >> related
> >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> HDFS-8578,
> >>> you
> >>>>> can see 30+ rounds patch review back and forth by senior committers,
> >>> not to
> >>>>> mention the outstanding performance test data in HDFS-8791.
> >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> RM
> >>> for
> >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> >> into
> >>>>> 2.6.5 now.
> >>>>>> Just my 2 cents.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Junping
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Sean Busbey <bu...@cloudera.com>
> >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> >>>>>> To: hdfs-dev@hadoop.apache.org
> >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >>>>> mapreduce-dev@hadoop.apache.org
> >>>>>> Subject: Re: 2.7.3 release plan
> >>>>>>
> >>>>>> A layout change in a maintenance release sounds very risky. I saw
> >> some
> >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> >> to
> >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> >> thought
> >>>>>> we did RMs per release rather than per branch? No one claiming to
> >> be a
> >>>>>> release manager ever spoke up AFAICT.
> >>>>>>
> >>>>>> Should this change be included? Should it go into a special 2.8
> >>>>>> release as mentioned in the ticket?
> >>>>>>
> >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> >>>>>>> Thank you Vinod!
> >>>>>>>
> >>>>>>> FYI: 2.7.3 will be a bit special release.
> >>>>>>>
> >>>>>>> HDFS-8791 bumped up the datanode layout version,
> >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >>>>>>> is impossible. We can rollback instead.
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> >>>>>>>
> >>>>>
> >>>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akira
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >>>>> (which
> >>>>>>>> did go out mid February). Got a little busy since.
> >>>>>>>>
> >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> >> towards a
> >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> >> bug-fixes
> >>>>> and *no*
> >>>>>>>> features / improvements.
> >>>>>>>>
> >>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
> >>>>> blocker
> >>>>>>>> / critical issues. Will start moving out any tickets that are not
> >>>>> blockers
> >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> >> critical
> >>>>> tickets
> >>>>>>>> outstanding as of now.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>> [1] 2.7.3 release blockers:
> >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> busbey
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> busbey
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

In light of the additional conversation on HDFS-8791, I would like to
re-propose the following:

1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
layout change currently does not support downgrades which breaks our
upgrade/downgrade policies for dot releases.

2. Cut a 2.8 release off of the 2.7.3 release with the addition of
HDFS-8791. This would give customers a stable release that they could
deploy with the new layout. As discussed on the jira, this is still in line
with user expectation for minor releases as we have done layout changes in
a number of 2.x minor releases already. The current 2.8 would become 2.9
and continue its current release schedule.

What does everyone think? If unsupported downgrades between minor releases
is still not agreeable, then as stated by Vinod, we would need to either
add support for downgrades with dn layout changes or revert the layout
change from branch-2. If we are OK with the layout change in a minor
release, but think that the issue does not affect enough customers to
warrant a separate release, we could simply leave it in branch-2 and let it
be released with the current 2.8.


On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> I commented on the JIRA way back (see
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
> saying what I said below. Unfortunately, I haven’t followed the patch along
> after my initial comment.
>
> This isn’t about any specific release - starting 2.6 we declared support
> for rolling upgrades and downgrades. Any patch that breaks this should not
> be in branch-2.
>
> Two options from where I stand
>  (1) For folks who worked on the patch: Is there a way to make (a) the
> upgrade-downgrade seamless for people who don’t care about this (b) and
> have explicit documentation for people who care to switch this behavior on
> and are willing to risk not having downgrades. If this means a new
> configuration property, so be it. It’s a necessary evil.
>  (2) Just let specific users backport this into specific 2.x branches they
> need and leave it only on trunk.
>
> Unless this behavior stops breaking rolling upgrades/downgrades, I think
> we should just revert it from branch-2 and definitely 2.7.3 as it stands
> today.
>
> +Vinod
>
>
> > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > A few thoughts:
> >
> > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> > for data nodes depending on your setup.
> >
> > 2. We have already deployed this patch internally so, with my Twitter hat
> > on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> > That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> > releases on a large production cluster that has a diverse set of block
> ids
> > without this patch, especially if your data nodes have a large number of
> > disks or you are using federation. To be clear though: this highly
> depends
> > on your setup and at a minimum you should verify that this regression
> will
> > not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
> a
> > performance regression that gets worse over time. When you see it
> happening
> > on a live cluster, it is one of the harder issues to identify a root
> cause
> > and debug. I do understand that this is currently only affecting a
> smaller
> > number of users, but I also think this number has potential to increase
> as
> > time goes on. Maybe we can issue a warning in the release notes for
> future
> > 2.7.x and 2.6.x releases?
> >
> > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> to
> > this proposal on this thread) would be to cut a 2.8 release off of the
> > 2.7.3 release with the new layout. What people currently think of as 2.8
> > would then become 2.9. This would give customers a stable release that
> they
> > could deploy with the new layout and would not break upgrade and
> downgrade
> > expectations.
> >
> > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> would
> >> patch the release to revert HDFS-8791 before pushing it out to
> production.
> >> For what it's worth.
> >>
> >>
> >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> >> wrote:
> >>
> >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> branch-2.6.
> >>> HDFS-8578 is a very important related fix since otherwise upgrade will
> be
> >>> very slow.
> >>>
> >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>> wrote:
> >>>
> >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> >>>> maintenance release. I've only seen it crop up on a handful of our
> >>>> customer's clusters, and large users like Twitter and Yahoo that seem
> >> to
> >>> be
> >>>> more affected are also the most able to patch this change in
> >> themselves.
> >>>>
> >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> >>>> breaking upgrade and downgrade expectations when it doesn't affect the
> >>> (in
> >>>> my experience) vast majority of users.
> >>>>
> >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> but
> >>>> will let him elaborate.
> >>>>
> >>>> Best,
> >>>> Andrew
> >>>>
> >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> >>> wrote:
> >>>>
> >>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >>>>> if *any* of them end up introducing a regression the inclusion of
> >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> back
> >>>>> things out. If that happens to any substantial number of downstream
> >>>>> folks, or any particularly vocal downstream folks, then it is very
> >>>>> likely we'll lose the remaining trust of operators for rolling out
> >>>>> maintenance releases. That's a pretty steep cost.
> >>>>>
> >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >>>>> unreasonable burden.
> >>>>>
> >>>>> I agree that this fix is important, I just think we should either cut
> >>>>> a version of 2.8 that includes it or find a way to do it that gives
> an
> >>>>> operational path for rolling downgrade.
> >>>>>
> >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> >>> wrote:
> >>>>>> Thanks for bringing up this topic, Sean.
> >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> >>> HDFS-8791
> >>>>> haven't been committed in so that's why we didn't discuss this
> >> earlier.
> >>>>>> I remember in JIRA discussion, we treated this layout change as a
> >>>>> Blocker bug that fixing a significant performance regression before
> >> but
> >>> not
> >>>>> a normal performance improvement. And I believe HDFS community
> already
> >>> did
> >>>>> their best with careful and patient to deliver the fix and other
> >> related
> >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> HDFS-8578,
> >>> you
> >>>>> can see 30+ rounds patch review back and forth by senior committers,
> >>> not to
> >>>>> mention the outstanding performance test data in HDFS-8791.
> >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> RM
> >>> for
> >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> >> into
> >>>>> 2.6.5 now.
> >>>>>> Just my 2 cents.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Junping
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Sean Busbey <bu...@cloudera.com>
> >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> >>>>>> To: hdfs-dev@hadoop.apache.org
> >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >>>>> mapreduce-dev@hadoop.apache.org
> >>>>>> Subject: Re: 2.7.3 release plan
> >>>>>>
> >>>>>> A layout change in a maintenance release sounds very risky. I saw
> >> some
> >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> >> to
> >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> >> thought
> >>>>>> we did RMs per release rather than per branch? No one claiming to
> >> be a
> >>>>>> release manager ever spoke up AFAICT.
> >>>>>>
> >>>>>> Should this change be included? Should it go into a special 2.8
> >>>>>> release as mentioned in the ticket?
> >>>>>>
> >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> >>>>>>> Thank you Vinod!
> >>>>>>>
> >>>>>>> FYI: 2.7.3 will be a bit special release.
> >>>>>>>
> >>>>>>> HDFS-8791 bumped up the datanode layout version,
> >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >>>>>>> is impossible. We can rollback instead.
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> >>>>>>>
> >>>>>
> >>>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akira
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >>>>> (which
> >>>>>>>> did go out mid February). Got a little busy since.
> >>>>>>>>
> >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> >> towards a
> >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> >> bug-fixes
> >>>>> and *no*
> >>>>>>>> features / improvements.
> >>>>>>>>
> >>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
> >>>>> blocker
> >>>>>>>> / critical issues. Will start moving out any tickets that are not
> >>>>> blockers
> >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> >> critical
> >>>>> tickets
> >>>>>>>> outstanding as of now.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>> [1] 2.7.3 release blockers:
> >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> busbey
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> busbey
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

In light of the additional conversation on HDFS-8791, I would like to
re-propose the following:

1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
layout change currently does not support downgrades which breaks our
upgrade/downgrade policies for dot releases.

2. Cut a 2.8 release off of the 2.7.3 release with the addition of
HDFS-8791. This would give customers a stable release that they could
deploy with the new layout. As discussed on the jira, this is still in line
with user expectation for minor releases as we have done layout changes in
a number of 2.x minor releases already. The current 2.8 would become 2.9
and continue its current release schedule.

What does everyone think? If unsupported downgrades between minor releases
is still not agreeable, then as stated by Vinod, we would need to either
add support for downgrades with dn layout changes or revert the layout
change from branch-2. If we are OK with the layout change in a minor
release, but think that the issue does not affect enough customers to
warrant a separate release, we could simply leave it in branch-2 and let it
be released with the current 2.8.


On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> I commented on the JIRA way back (see
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
> saying what I said below. Unfortunately, I haven’t followed the patch along
> after my initial comment.
>
> This isn’t about any specific release - starting 2.6 we declared support
> for rolling upgrades and downgrades. Any patch that breaks this should not
> be in branch-2.
>
> Two options from where I stand
>  (1) For folks who worked on the patch: Is there a way to make (a) the
> upgrade-downgrade seamless for people who don’t care about this (b) and
> have explicit documentation for people who care to switch this behavior on
> and are willing to risk not having downgrades. If this means a new
> configuration property, so be it. It’s a necessary evil.
>  (2) Just let specific users backport this into specific 2.x branches they
> need and leave it only on trunk.
>
> Unless this behavior stops breaking rolling upgrades/downgrades, I think
> we should just revert it from branch-2 and definitely 2.7.3 as it stands
> today.
>
> +Vinod
>
>
> > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > A few thoughts:
> >
> > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> > for data nodes depending on your setup.
> >
> > 2. We have already deployed this patch internally so, with my Twitter hat
> > on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> > That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> > releases on a large production cluster that has a diverse set of block
> ids
> > without this patch, especially if your data nodes have a large number of
> > disks or you are using federation. To be clear though: this highly
> depends
> > on your setup and at a minimum you should verify that this regression
> will
> > not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
> a
> > performance regression that gets worse over time. When you see it
> happening
> > on a live cluster, it is one of the harder issues to identify a root
> cause
> > and debug. I do understand that this is currently only affecting a
> smaller
> > number of users, but I also think this number has potential to increase
> as
> > time goes on. Maybe we can issue a warning in the release notes for
> future
> > 2.7.x and 2.6.x releases?
> >
> > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> to
> > this proposal on this thread) would be to cut a 2.8 release off of the
> > 2.7.3 release with the new layout. What people currently think of as 2.8
> > would then become 2.9. This would give customers a stable release that
> they
> > could deploy with the new layout and would not break upgrade and
> downgrade
> > expectations.
> >
> > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> would
> >> patch the release to revert HDFS-8791 before pushing it out to
> production.
> >> For what it's worth.
> >>
> >>
> >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> >> wrote:
> >>
> >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> branch-2.6.
> >>> HDFS-8578 is a very important related fix since otherwise upgrade will
> be
> >>> very slow.
> >>>
> >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>> wrote:
> >>>
> >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> >>>> maintenance release. I've only seen it crop up on a handful of our
> >>>> customer's clusters, and large users like Twitter and Yahoo that seem
> >> to
> >>> be
> >>>> more affected are also the most able to patch this change in
> >> themselves.
> >>>>
> >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> >>>> breaking upgrade and downgrade expectations when it doesn't affect the
> >>> (in
> >>>> my experience) vast majority of users.
> >>>>
> >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> but
> >>>> will let him elaborate.
> >>>>
> >>>> Best,
> >>>> Andrew
> >>>>
> >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> >>> wrote:
> >>>>
> >>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >>>>> if *any* of them end up introducing a regression the inclusion of
> >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> back
> >>>>> things out. If that happens to any substantial number of downstream
> >>>>> folks, or any particularly vocal downstream folks, then it is very
> >>>>> likely we'll lose the remaining trust of operators for rolling out
> >>>>> maintenance releases. That's a pretty steep cost.
> >>>>>
> >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >>>>> unreasonable burden.
> >>>>>
> >>>>> I agree that this fix is important, I just think we should either cut
> >>>>> a version of 2.8 that includes it or find a way to do it that gives
> an
> >>>>> operational path for rolling downgrade.
> >>>>>
> >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> >>> wrote:
> >>>>>> Thanks for bringing up this topic, Sean.
> >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> >>> HDFS-8791
> >>>>> haven't been committed in so that's why we didn't discuss this
> >> earlier.
> >>>>>> I remember in JIRA discussion, we treated this layout change as a
> >>>>> Blocker bug that fixing a significant performance regression before
> >> but
> >>> not
> >>>>> a normal performance improvement. And I believe HDFS community
> already
> >>> did
> >>>>> their best with careful and patient to deliver the fix and other
> >> related
> >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> HDFS-8578,
> >>> you
> >>>>> can see 30+ rounds patch review back and forth by senior committers,
> >>> not to
> >>>>> mention the outstanding performance test data in HDFS-8791.
> >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> RM
> >>> for
> >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> >> into
> >>>>> 2.6.5 now.
> >>>>>> Just my 2 cents.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Junping
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Sean Busbey <bu...@cloudera.com>
> >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> >>>>>> To: hdfs-dev@hadoop.apache.org
> >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >>>>> mapreduce-dev@hadoop.apache.org
> >>>>>> Subject: Re: 2.7.3 release plan
> >>>>>>
> >>>>>> A layout change in a maintenance release sounds very risky. I saw
> >> some
> >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> >> to
> >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> >> thought
> >>>>>> we did RMs per release rather than per branch? No one claiming to
> >> be a
> >>>>>> release manager ever spoke up AFAICT.
> >>>>>>
> >>>>>> Should this change be included? Should it go into a special 2.8
> >>>>>> release as mentioned in the ticket?
> >>>>>>
> >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> >>>>>>> Thank you Vinod!
> >>>>>>>
> >>>>>>> FYI: 2.7.3 will be a bit special release.
> >>>>>>>
> >>>>>>> HDFS-8791 bumped up the datanode layout version,
> >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >>>>>>> is impossible. We can rollback instead.
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> >>>>>>>
> >>>>>
> >>>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akira
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >>>>> (which
> >>>>>>>> did go out mid February). Got a little busy since.
> >>>>>>>>
> >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> >> towards a
> >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> >> bug-fixes
> >>>>> and *no*
> >>>>>>>> features / improvements.
> >>>>>>>>
> >>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
> >>>>> blocker
> >>>>>>>> / critical issues. Will start moving out any tickets that are not
> >>>>> blockers
> >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> >> critical
> >>>>> tickets
> >>>>>>>> outstanding as of now.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>> [1] 2.7.3 release blockers:
> >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> busbey
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> busbey
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

In light of the additional conversation on HDFS-8791, I would like to
re-propose the following:

1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
layout change currently does not support downgrades which breaks our
upgrade/downgrade policies for dot releases.

2. Cut a 2.8 release off of the 2.7.3 release with the addition of
HDFS-8791. This would give customers a stable release that they could
deploy with the new layout. As discussed on the jira, this is still in line
with user expectation for minor releases as we have done layout changes in
a number of 2.x minor releases already. The current 2.8 would become 2.9
and continue its current release schedule.

What does everyone think? If unsupported downgrades between minor releases
is still not agreeable, then as stated by Vinod, we would need to either
add support for downgrades with dn layout changes or revert the layout
change from branch-2. If we are OK with the layout change in a minor
release, but think that the issue does not affect enough customers to
warrant a separate release, we could simply leave it in branch-2 and let it
be released with the current 2.8.


On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:

> I commented on the JIRA way back (see
> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666),
> saying what I said below. Unfortunately, I haven’t followed the patch along
> after my initial comment.
>
> This isn’t about any specific release - starting 2.6 we declared support
> for rolling upgrades and downgrades. Any patch that breaks this should not
> be in branch-2.
>
> Two options from where I stand
>  (1) For folks who worked on the patch: Is there a way to make (a) the
> upgrade-downgrade seamless for people who don’t care about this (b) and
> have explicit documentation for people who care to switch this behavior on
> and are willing to risk not having downgrades. If this means a new
> configuration property, so be it. It’s a necessary evil.
>  (2) Just let specific users backport this into specific 2.x branches they
> need and leave it only on trunk.
>
> Unless this behavior stops breaking rolling upgrades/downgrades, I think
> we should just revert it from branch-2 and definitely 2.7.3 as it stands
> today.
>
> +Vinod
>
>
> > On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> >
> > A few thoughts:
> >
> > 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> > prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> > for data nodes depending on your setup.
> >
> > 2. We have already deployed this patch internally so, with my Twitter hat
> > on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> > That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> > releases on a large production cluster that has a diverse set of block
> ids
> > without this patch, especially if your data nodes have a large number of
> > disks or you are using federation. To be clear though: this highly
> depends
> > on your setup and at a minimum you should verify that this regression
> will
> > not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
> a
> > performance regression that gets worse over time. When you see it
> happening
> > on a live cluster, it is one of the harder issues to identify a root
> cause
> > and debug. I do understand that this is currently only affecting a
> smaller
> > number of users, but I also think this number has potential to increase
> as
> > time goes on. Maybe we can issue a warning in the release notes for
> future
> > 2.7.x and 2.6.x releases?
> >
> > 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> to
> > this proposal on this thread) would be to cut a 2.8 release off of the
> > 2.7.3 release with the new layout. What people currently think of as 2.8
> > would then become 2.9. This would give customers a stable release that
> they
> > could deploy with the new layout and would not break upgrade and
> downgrade
> > expectations.
> >
> > On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> >> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we
> would
> >> patch the release to revert HDFS-8791 before pushing it out to
> production.
> >> For what it's worth.
> >>
> >>
> >> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> >> wrote:
> >>
> >>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> >>> backported the parallel DN upgrade improvement (HDFS-8578) to
> branch-2.6.
> >>> HDFS-8578 is a very important related fix since otherwise upgrade will
> be
> >>> very slow.
> >>>
> >>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> >>> wrote:
> >>>
> >>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> >>>> maintenance release. I've only seen it crop up on a handful of our
> >>>> customer's clusters, and large users like Twitter and Yahoo that seem
> >> to
> >>> be
> >>>> more affected are also the most able to patch this change in
> >> themselves.
> >>>>
> >>>> Layout upgrades are quite disruptive, and I don't think it's worth
> >>>> breaking upgrade and downgrade expectations when it doesn't affect the
> >>> (in
> >>>> my experience) vast majority of users.
> >>>>
> >>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791,
> but
> >>>> will let him elaborate.
> >>>>
> >>>> Best,
> >>>> Andrew
> >>>>
> >>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> >>> wrote:
> >>>>
> >>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >>>>> if *any* of them end up introducing a regression the inclusion of
> >>>>> HDFS-8791 means that folks will have cluster downtime in order to
> back
> >>>>> things out. If that happens to any substantial number of downstream
> >>>>> folks, or any particularly vocal downstream folks, then it is very
> >>>>> likely we'll lose the remaining trust of operators for rolling out
> >>>>> maintenance releases. That's a pretty steep cost.
> >>>>>
> >>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >>>>> unreasonable burden.
> >>>>>
> >>>>> I agree that this fix is important, I just think we should either cut
> >>>>> a version of 2.8 that includes it or find a way to do it that gives
> an
> >>>>> operational path for rolling downgrade.
> >>>>>
> >>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> >>> wrote:
> >>>>>> Thanks for bringing up this topic, Sean.
> >>>>>> When I released our latest Hadoop release 2.6.4, the patch of
> >>> HDFS-8791
> >>>>> haven't been committed in so that's why we didn't discuss this
> >> earlier.
> >>>>>> I remember in JIRA discussion, we treated this layout change as a
> >>>>> Blocker bug that fixing a significant performance regression before
> >> but
> >>> not
> >>>>> a normal performance improvement. And I believe HDFS community
> already
> >>> did
> >>>>> their best with careful and patient to deliver the fix and other
> >> related
> >>>>> patches (like upgrade fix in HDFS-8578). Take an example of
> HDFS-8578,
> >>> you
> >>>>> can see 30+ rounds patch review back and forth by senior committers,
> >>> not to
> >>>>> mention the outstanding performance test data in HDFS-8791.
> >>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
> >>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as
> RM
> >>> for
> >>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
> >> into
> >>>>> 2.6.5 now.
> >>>>>> Just my 2 cents.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Junping
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Sean Busbey <bu...@cloudera.com>
> >>>>>> Sent: Thursday, March 31, 2016 2:57 PM
> >>>>>> To: hdfs-dev@hadoop.apache.org
> >>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >>>>> mapreduce-dev@hadoop.apache.org
> >>>>>> Subject: Re: 2.7.3 release plan
> >>>>>>
> >>>>>> A layout change in a maintenance release sounds very risky. I saw
> >> some
> >>>>>> discussion on the JIRA about those risks, but the consensus seemed
> >> to
> >>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
> >> thought
> >>>>>> we did RMs per release rather than per branch? No one claiming to
> >> be a
> >>>>>> release manager ever spoke up AFAICT.
> >>>>>>
> >>>>>> Should this change be included? Should it go into a special 2.8
> >>>>>> release as mentioned in the ticket?
> >>>>>>
> >>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >>>>>> <aj...@oss.nttdata.co.jp> wrote:
> >>>>>>> Thank you Vinod!
> >>>>>>>
> >>>>>>> FYI: 2.7.3 will be a bit special release.
> >>>>>>>
> >>>>>>> HDFS-8791 bumped up the datanode layout version,
> >>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >>>>>>> is impossible. We can rollback instead.
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
> >>>>>>>
> >>>>>
> >>>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akira
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >>>>> (which
> >>>>>>>> did go out mid February). Got a little busy since.
> >>>>>>>>
> >>>>>>>> Following up the 2.7.2 maintenance release, we should work
> >> towards a
> >>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
> >> bug-fixes
> >>>>> and *no*
> >>>>>>>> features / improvements.
> >>>>>>>>
> >>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
> >>>>> blocker
> >>>>>>>> / critical issues. Will start moving out any tickets that are not
> >>>>> blockers
> >>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
> >> critical
> >>>>> tickets
> >>>>>>>> outstanding as of now.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>> [1] 2.7.3 release blockers:
> >>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> busbey
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> busbey
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>   - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

I commented on the JIRA way back (see https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666), saying what I said below. Unfortunately, I haven’t followed the patch along after my initial comment. 

This isn’t about any specific release - starting 2.6 we declared support for rolling upgrades and downgrades. Any patch that breaks this should not be in branch-2.

Two options from where I stand
 (1) For folks who worked on the patch: Is there a way to make (a) the upgrade-downgrade seamless for people who don’t care about this (b) and have explicit documentation for people who care to switch this behavior on and are willing to risk not having downgrades. If this means a new configuration property, so be it. It’s a necessary evil.
 (2) Just let specific users backport this into specific 2.x branches they need and leave it only on trunk.

Unless this behavior stops breaking rolling upgrades/downgrades, I think we should just revert it from branch-2 and definitely 2.7.3 as it stands today.

+Vinod


> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> A few thoughts:
> 
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
> 
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
> 
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
> 
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
> 
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>> 
>> 
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> 
>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>>> HDFS-8578 is a very important related fix since otherwise upgrade will be
>>> very slow.
>>> 
>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>>> wrote:
>>> 
>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>> maintenance release. I've only seen it crop up on a handful of our
>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>>> be
>>>> more affected are also the most able to patch this change in
>> themselves.
>>>> 
>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>> (in
>>>> my experience) vast majority of users.
>>>> 
>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>>>> will let him elaborate.
>>>> 
>>>> Best,
>>>> Andrew
>>>> 
>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>> 
>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>> HDFS-8791 means that folks will have cluster downtime in order to back
>>>>> things out. If that happens to any substantial number of downstream
>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>> maintenance releases. That's a pretty steep cost.
>>>>> 
>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>> unreasonable burden.
>>>>> 
>>>>> I agree that this fix is important, I just think we should either cut
>>>>> a version of 2.8 that includes it or find a way to do it that gives an
>>>>> operational path for rolling downgrade.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>> wrote:
>>>>>> Thanks for bringing up this topic, Sean.
>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>> HDFS-8791
>>>>> haven't been committed in so that's why we didn't discuss this
>> earlier.
>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>> Blocker bug that fixing a significant performance regression before
>> but
>>> not
>>>>> a normal performance improvement. And I believe HDFS community already
>>> did
>>>>> their best with careful and patient to deliver the fix and other
>> related
>>>>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>>> you
>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>> not to
>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>>> for
>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>>>>> 2.6.5 now.
>>>>>> Just my 2 cents.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: 2.7.3 release plan
>>>>>> 
>>>>>> A layout change in a maintenance release sounds very risky. I saw
>> some
>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>> to
>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>>>>>> we did RMs per release rather than per branch? No one claiming to
>> be a
>>>>>> release manager ever spoke up AFAICT.
>>>>>> 
>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>> release as mentioned in the ticket?
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>> Thank you Vinod!
>>>>>>> 
>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>> 
>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>> is impossible. We can rollback instead.
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>> 
>>>>> 
>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which
>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>>>>> and *no*
>>>>>>>> features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker
>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>> blockers
>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>>>>> tickets
>>>>>>>> outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> busbey
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> busbey
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>> 

Re: 2.7.3 release plan

[Colin McCabe <cm...@apache.org>]

I agree that HDFS-8578 should be a prerequisite for backporting
HDFS-8791.

I think we are overestimating the number of people affected by
HDFS-8791, and underestimating the disruption that would be caused by a
layout version upgrade in a dot release.  As Andrew, Sean, and others in
the thread pointed out, this could reduce people's trust in the
stabilization branches.

The maintenance branches were never intended to live forever. 
Eventually, people should start using newer releases.  A more efficient
DataNode layout is just one more motivation to upgrade.

best,
Colin


On Fri, Apr 1, 2016, at 14:54, Chris Trezzo wrote:
> A few thoughts:
> 
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
> 
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block
> ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly
> depends
> on your setup and at a minimum you should verify that this regression
> will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
> a
> performance regression that gets worse over time. When you see it
> happening
> on a live cluster, it is one of the harder issues to identify a root
> cause
> and debug. I do understand that this is currently only affecting a
> smaller
> number of users, but I also think this number has potential to increase
> as
> time goes on. Maybe we can issue a warning in the release notes for
> future
> 2.7.x and 2.6.x releases?
> 
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded
> to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that
> they
> could deploy with the new layout and would not break upgrade and
> downgrade
> expectations.
> 
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> 
> > As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
> > patch the release to revert HDFS-8791 before pushing it out to production.
> > For what it's worth.
> >
> >
> > On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> > > HDFS-8578 is a very important related fix since otherwise upgrade will be
> > > very slow.
> > >
> > > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> > > wrote:
> > >
> > > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > > > maintenance release. I've only seen it crop up on a handful of our
> > > > customer's clusters, and large users like Twitter and Yahoo that seem
> > to
> > > be
> > > > more affected are also the most able to patch this change in
> > themselves.
> > > >
> > > > Layout upgrades are quite disruptive, and I don't think it's worth
> > > > breaking upgrade and downgrade expectations when it doesn't affect the
> > > (in
> > > > my experience) vast majority of users.
> > > >
> > > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > > > will let him elaborate.
> > > >
> > > > Best,
> > > > Andrew
> > > >
> > > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > > wrote:
> > > >
> > > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> > > >> if *any* of them end up introducing a regression the inclusion of
> > > >> HDFS-8791 means that folks will have cluster downtime in order to back
> > > >> things out. If that happens to any substantial number of downstream
> > > >> folks, or any particularly vocal downstream folks, then it is very
> > > >> likely we'll lose the remaining trust of operators for rolling out
> > > >> maintenance releases. That's a pretty steep cost.
> > > >>
> > > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> > > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > > >> unreasonable burden.
> > > >>
> > > >> I agree that this fix is important, I just think we should either cut
> > > >> a version of 2.8 that includes it or find a way to do it that gives an
> > > >> operational path for rolling downgrade.
> > > >>
> > > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > > wrote:
> > > >> > Thanks for bringing up this topic, Sean.
> > > >> > When I released our latest Hadoop release 2.6.4, the patch of
> > > HDFS-8791
> > > >> haven't been committed in so that's why we didn't discuss this
> > earlier.
> > > >> > I remember in JIRA discussion, we treated this layout change as a
> > > >> Blocker bug that fixing a significant performance regression before
> > but
> > > not
> > > >> a normal performance improvement. And I believe HDFS community already
> > > did
> > > >> their best with careful and patient to deliver the fix and other
> > related
> > > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> > > you
> > > >> can see 30+ rounds patch review back and forth by senior committers,
> > > not to
> > > >> mention the outstanding performance test data in HDFS-8791.
> > > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> > > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> > > for
> > > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
> > into
> > > >> 2.6.5 now.
> > > >> > Just my 2 cents.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Junping
> > > >> >
> > > >> > ________________________________________
> > > >> > From: Sean Busbey <bu...@cloudera.com>
> > > >> > Sent: Thursday, March 31, 2016 2:57 PM
> > > >> > To: hdfs-dev@hadoop.apache.org
> > > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > > >> mapreduce-dev@hadoop.apache.org
> > > >> > Subject: Re: 2.7.3 release plan
> > > >> >
> > > >> > A layout change in a maintenance release sounds very risky. I saw
> > some
> > > >> > discussion on the JIRA about those risks, but the consensus seemed
> > to
> > > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
> > thought
> > > >> > we did RMs per release rather than per branch? No one claiming to
> > be a
> > > >> > release manager ever spoke up AFAICT.
> > > >> >
> > > >> > Should this change be included? Should it go into a special 2.8
> > > >> > release as mentioned in the ticket?
> > > >> >
> > > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > > >> > <aj...@oss.nttdata.co.jp> wrote:
> > > >> >> Thank you Vinod!
> > > >> >>
> > > >> >> FYI: 2.7.3 will be a bit special release.
> > > >> >>
> > > >> >> HDFS-8791 bumped up the datanode layout version,
> > > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > > >> >> is impossible. We can rollback instead.
> > > >> >>
> > > >> >> https://issues.apache.org/jira/browse/HDFS-8791
> > > >> >>
> > > >>
> > >
> > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > > >> >>
> > > >> >> Regards,
> > > >> >> Akira
> > > >> >>
> > > >> >>
> > > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > > >> >>>
> > > >> >>> Hi all,
> > > >> >>>
> > > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> > > >> (which
> > > >> >>> did go out mid February). Got a little busy since.
> > > >> >>>
> > > >> >>> Following up the 2.7.2 maintenance release, we should work
> > towards a
> > > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
> > bug-fixes
> > > >> and *no*
> > > >> >>> features / improvements.
> > > >> >>>
> > > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> > > >> blocker
> > > >> >>> / critical issues. Will start moving out any tickets that are not
> > > >> blockers
> > > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
> > critical
> > > >> tickets
> > > >> >>> outstanding as of now.
> > > >> >>>
> > > >> >>> Thanks,
> > > >> >>> +Vinod
> > > >> >>>
> > > >> >>> [1] 2.7.3 release blockers:
> > > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> > > >> >>>
> > > >> >>
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > busbey
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> busbey
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >

Re: 2.7.3 release plan

[Tsz Wo Sze <sz...@yahoo.com.INVALID>]

> ... I volunteer to help on releasing 2.8.0 as 2.7.3 + HDFS-8578 and HDFS-8791.

HDFS-8578 (parallel upgrade), which does not change layout, should remain in 2.7.3.  Only HDFS-8791 (new layout) is needed to be put in 2.8.0.
Tsz-Wo
 

    On Monday, April 4, 2016 2:51 AM, Haohui Mai <ri...@gmail.com> wrote:
 
 

 +1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

 
  

Re: 2.7.3 release plan

[Tsz Wo Sze <sz...@yahoo.com.INVALID>]

> ... I volunteer to help on releasing 2.8.0 as 2.7.3 + HDFS-8578 and HDFS-8791.

HDFS-8578 (parallel upgrade), which does not change layout, should remain in 2.7.3.  Only HDFS-8791 (new layout) is needed to be put in 2.8.0.
Tsz-Wo
 

    On Monday, April 4, 2016 2:51 AM, Haohui Mai <ri...@gmail.com> wrote:
 
 

 +1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

 
  

Re: 2.7.3 release plan

[Haohui Mai <ri...@gmail.com>]

+1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: 2.7.3 release plan

[Haohui Mai <ri...@gmail.com>]

+1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

I commented on the JIRA way back (see https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666), saying what I said below. Unfortunately, I haven’t followed the patch along after my initial comment. 

This isn’t about any specific release - starting 2.6 we declared support for rolling upgrades and downgrades. Any patch that breaks this should not be in branch-2.

Two options from where I stand
 (1) For folks who worked on the patch: Is there a way to make (a) the upgrade-downgrade seamless for people who don’t care about this (b) and have explicit documentation for people who care to switch this behavior on and are willing to risk not having downgrades. If this means a new configuration property, so be it. It’s a necessary evil.
 (2) Just let specific users backport this into specific 2.x branches they need and leave it only on trunk.

Unless this behavior stops breaking rolling upgrades/downgrades, I think we should just revert it from branch-2 and definitely 2.7.3 as it stands today.

+Vinod


> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> A few thoughts:
> 
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
> 
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
> 
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
> 
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
> 
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>> 
>> 
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> 
>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>>> HDFS-8578 is a very important related fix since otherwise upgrade will be
>>> very slow.
>>> 
>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>>> wrote:
>>> 
>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>> maintenance release. I've only seen it crop up on a handful of our
>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>>> be
>>>> more affected are also the most able to patch this change in
>> themselves.
>>>> 
>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>> (in
>>>> my experience) vast majority of users.
>>>> 
>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>>>> will let him elaborate.
>>>> 
>>>> Best,
>>>> Andrew
>>>> 
>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>> 
>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>> HDFS-8791 means that folks will have cluster downtime in order to back
>>>>> things out. If that happens to any substantial number of downstream
>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>> maintenance releases. That's a pretty steep cost.
>>>>> 
>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>> unreasonable burden.
>>>>> 
>>>>> I agree that this fix is important, I just think we should either cut
>>>>> a version of 2.8 that includes it or find a way to do it that gives an
>>>>> operational path for rolling downgrade.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>> wrote:
>>>>>> Thanks for bringing up this topic, Sean.
>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>> HDFS-8791
>>>>> haven't been committed in so that's why we didn't discuss this
>> earlier.
>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>> Blocker bug that fixing a significant performance regression before
>> but
>>> not
>>>>> a normal performance improvement. And I believe HDFS community already
>>> did
>>>>> their best with careful and patient to deliver the fix and other
>> related
>>>>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>>> you
>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>> not to
>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>>> for
>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>>>>> 2.6.5 now.
>>>>>> Just my 2 cents.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: 2.7.3 release plan
>>>>>> 
>>>>>> A layout change in a maintenance release sounds very risky. I saw
>> some
>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>> to
>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>>>>>> we did RMs per release rather than per branch? No one claiming to
>> be a
>>>>>> release manager ever spoke up AFAICT.
>>>>>> 
>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>> release as mentioned in the ticket?
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>> Thank you Vinod!
>>>>>>> 
>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>> 
>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>> is impossible. We can rollback instead.
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>> 
>>>>> 
>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which
>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>>>>> and *no*
>>>>>>>> features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker
>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>> blockers
>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>>>>> tickets
>>>>>>>> outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> busbey
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> busbey
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

I commented on the JIRA way back (see https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666), saying what I said below. Unfortunately, I haven’t followed the patch along after my initial comment. 

This isn’t about any specific release - starting 2.6 we declared support for rolling upgrades and downgrades. Any patch that breaks this should not be in branch-2.

Two options from where I stand
 (1) For folks who worked on the patch: Is there a way to make (a) the upgrade-downgrade seamless for people who don’t care about this (b) and have explicit documentation for people who care to switch this behavior on and are willing to risk not having downgrades. If this means a new configuration property, so be it. It’s a necessary evil.
 (2) Just let specific users backport this into specific 2.x branches they need and leave it only on trunk.

Unless this behavior stops breaking rolling upgrades/downgrades, I think we should just revert it from branch-2 and definitely 2.7.3 as it stands today.

+Vinod


> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> A few thoughts:
> 
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
> 
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
> 
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
> 
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
> 
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>> 
>> 
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> 
>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>>> HDFS-8578 is a very important related fix since otherwise upgrade will be
>>> very slow.
>>> 
>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>>> wrote:
>>> 
>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>> maintenance release. I've only seen it crop up on a handful of our
>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>>> be
>>>> more affected are also the most able to patch this change in
>> themselves.
>>>> 
>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>> (in
>>>> my experience) vast majority of users.
>>>> 
>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>>>> will let him elaborate.
>>>> 
>>>> Best,
>>>> Andrew
>>>> 
>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>> 
>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>> HDFS-8791 means that folks will have cluster downtime in order to back
>>>>> things out. If that happens to any substantial number of downstream
>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>> maintenance releases. That's a pretty steep cost.
>>>>> 
>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>> unreasonable burden.
>>>>> 
>>>>> I agree that this fix is important, I just think we should either cut
>>>>> a version of 2.8 that includes it or find a way to do it that gives an
>>>>> operational path for rolling downgrade.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>> wrote:
>>>>>> Thanks for bringing up this topic, Sean.
>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>> HDFS-8791
>>>>> haven't been committed in so that's why we didn't discuss this
>> earlier.
>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>> Blocker bug that fixing a significant performance regression before
>> but
>>> not
>>>>> a normal performance improvement. And I believe HDFS community already
>>> did
>>>>> their best with careful and patient to deliver the fix and other
>> related
>>>>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>>> you
>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>> not to
>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>>> for
>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>>>>> 2.6.5 now.
>>>>>> Just my 2 cents.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: 2.7.3 release plan
>>>>>> 
>>>>>> A layout change in a maintenance release sounds very risky. I saw
>> some
>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>> to
>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>>>>>> we did RMs per release rather than per branch? No one claiming to
>> be a
>>>>>> release manager ever spoke up AFAICT.
>>>>>> 
>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>> release as mentioned in the ticket?
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>> Thank you Vinod!
>>>>>>> 
>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>> 
>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>> is impossible. We can rollback instead.
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>> 
>>>>> 
>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which
>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>>>>> and *no*
>>>>>>>> features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker
>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>> blockers
>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>>>>> tickets
>>>>>>>> outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> busbey
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> busbey
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>> 

Re: 2.7.3 release plan

[Haohui Mai <ri...@gmail.com>]

+1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: 2.7.3 release plan

[Haohui Mai <ri...@gmail.com>]

+1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.

~Haohui

On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bu...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> > >> mapreduce-dev@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <aj...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

I commented on the JIRA way back (see https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=15036666&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036666), saying what I said below. Unfortunately, I haven’t followed the patch along after my initial comment. 

This isn’t about any specific release - starting 2.6 we declared support for rolling upgrades and downgrades. Any patch that breaks this should not be in branch-2.

Two options from where I stand
 (1) For folks who worked on the patch: Is there a way to make (a) the upgrade-downgrade seamless for people who don’t care about this (b) and have explicit documentation for people who care to switch this behavior on and are willing to risk not having downgrades. If this means a new configuration property, so be it. It’s a necessary evil.
 (2) Just let specific users backport this into specific 2.x branches they need and leave it only on trunk.

Unless this behavior stops breaking rolling upgrades/downgrades, I think we should just revert it from branch-2 and definitely 2.7.3 as it stands today.

+Vinod


> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ct...@gmail.com> wrote:
> 
> A few thoughts:
> 
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
> 
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
> 
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
> 
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:
> 
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>> 
>> 
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> 
>>> One other thing I wanted to bring up regarding HDFS-8791, we haven't
>>> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>>> HDFS-8578 is a very important related fix since otherwise upgrade will be
>>> very slow.
>>> 
>>> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
>>> wrote:
>>> 
>>>> As I expressed on HDFS-8791, I do not want to include this JIRA in a
>>>> maintenance release. I've only seen it crop up on a handful of our
>>>> customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>>> be
>>>> more affected are also the most able to patch this change in
>> themselves.
>>>> 
>>>> Layout upgrades are quite disruptive, and I don't think it's worth
>>>> breaking upgrade and downgrade expectations when it doesn't affect the
>>> (in
>>>> my experience) vast majority of users.
>>>> 
>>>> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>>>> will let him elaborate.
>>>> 
>>>> Best,
>>>> Andrew
>>>> 
>>>> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>> 
>>>>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>>>>> if *any* of them end up introducing a regression the inclusion of
>>>>> HDFS-8791 means that folks will have cluster downtime in order to back
>>>>> things out. If that happens to any substantial number of downstream
>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>> maintenance releases. That's a pretty steep cost.
>>>>> 
>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>> unreasonable burden.
>>>>> 
>>>>> I agree that this fix is important, I just think we should either cut
>>>>> a version of 2.8 that includes it or find a way to do it that gives an
>>>>> operational path for rolling downgrade.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
>>> wrote:
>>>>>> Thanks for bringing up this topic, Sean.
>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>> HDFS-8791
>>>>> haven't been committed in so that's why we didn't discuss this
>> earlier.
>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>> Blocker bug that fixing a significant performance regression before
>> but
>>> not
>>>>> a normal performance improvement. And I believe HDFS community already
>>> did
>>>>> their best with careful and patient to deliver the fix and other
>> related
>>>>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>>> you
>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>> not to
>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>>> for
>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>>>>> 2.6.5 now.
>>>>>> Just my 2 cents.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> ________________________________________
>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>> To: hdfs-dev@hadoop.apache.org
>>>>>> Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: 2.7.3 release plan
>>>>>> 
>>>>>> A layout change in a maintenance release sounds very risky. I saw
>> some
>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>> to
>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>>>>>> we did RMs per release rather than per branch? No one claiming to
>> be a
>>>>>> release manager ever spoke up AFAICT.
>>>>>> 
>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>> release as mentioned in the ticket?
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>> <aj...@oss.nttdata.co.jp> wrote:
>>>>>>> Thank you Vinod!
>>>>>>> 
>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>> 
>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>> is impossible. We can rollback instead.
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>> 
>>>>> 
>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which
>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>>>>> and *no*
>>>>>>>> features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker
>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>> blockers
>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>>>>> tickets
>>>>>>>> outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> busbey
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> busbey
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>> 

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

A few thoughts:

1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
for data nodes depending on your setup.

2. We have already deployed this patch internally so, with my Twitter hat
on, I would be perfectly happy as long as it makes it into trunk and 2.8.
That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
releases on a large production cluster that has a diverse set of block ids
without this patch, especially if your data nodes have a large number of
disks or you are using federation. To be clear though: this highly depends
on your setup and at a minimum you should verify that this regression will
not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
performance regression that gets worse over time. When you see it happening
on a live cluster, it is one of the harder issues to identify a root cause
and debug. I do understand that this is currently only affecting a smaller
number of users, but I also think this number has potential to increase as
time goes on. Maybe we can issue a warning in the release notes for future
2.7.x and 2.6.x releases?

3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
this proposal on this thread) would be to cut a 2.8 release off of the
2.7.3 release with the new layout. What people currently think of as 2.8
would then become 2.9. This would give customers a stable release that they
could deploy with the new layout and would not break upgrade and downgrade
expectations.

On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:

> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
> patch the release to revert HDFS-8791 before pushing it out to production.
> For what it's worth.
>
>
> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> > HDFS-8578 is a very important related fix since otherwise upgrade will be
> > very slow.
> >
> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > > maintenance release. I've only seen it crop up on a handful of our
> > > customer's clusters, and large users like Twitter and Yahoo that seem
> to
> > be
> > > more affected are also the most able to patch this change in
> themselves.
> > >
> > > Layout upgrades are quite disruptive, and I don't think it's worth
> > > breaking upgrade and downgrade expectations when it doesn't affect the
> > (in
> > > my experience) vast majority of users.
> > >
> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > > will let him elaborate.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> > >> if *any* of them end up introducing a regression the inclusion of
> > >> HDFS-8791 means that folks will have cluster downtime in order to back
> > >> things out. If that happens to any substantial number of downstream
> > >> folks, or any particularly vocal downstream folks, then it is very
> > >> likely we'll lose the remaining trust of operators for rolling out
> > >> maintenance releases. That's a pretty steep cost.
> > >>
> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >> unreasonable burden.
> > >>
> > >> I agree that this fix is important, I just think we should either cut
> > >> a version of 2.8 that includes it or find a way to do it that gives an
> > >> operational path for rolling downgrade.
> > >>
> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > wrote:
> > >> > Thanks for bringing up this topic, Sean.
> > >> > When I released our latest Hadoop release 2.6.4, the patch of
> > HDFS-8791
> > >> haven't been committed in so that's why we didn't discuss this
> earlier.
> > >> > I remember in JIRA discussion, we treated this layout change as a
> > >> Blocker bug that fixing a significant performance regression before
> but
> > not
> > >> a normal performance improvement. And I believe HDFS community already
> > did
> > >> their best with careful and patient to deliver the fix and other
> related
> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> > you
> > >> can see 30+ rounds patch review back and forth by senior committers,
> > not to
> > >> mention the outstanding performance test data in HDFS-8791.
> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> > for
> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
> into
> > >> 2.6.5 now.
> > >> > Just my 2 cents.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Junping
> > >> >
> > >> > ________________________________________
> > >> > From: Sean Busbey <bu...@cloudera.com>
> > >> > Sent: Thursday, March 31, 2016 2:57 PM
> > >> > To: hdfs-dev@hadoop.apache.org
> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >> mapreduce-dev@hadoop.apache.org
> > >> > Subject: Re: 2.7.3 release plan
> > >> >
> > >> > A layout change in a maintenance release sounds very risky. I saw
> some
> > >> > discussion on the JIRA about those risks, but the consensus seemed
> to
> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
> thought
> > >> > we did RMs per release rather than per branch? No one claiming to
> be a
> > >> > release manager ever spoke up AFAICT.
> > >> >
> > >> > Should this change be included? Should it go into a special 2.8
> > >> > release as mentioned in the ticket?
> > >> >
> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >> > <aj...@oss.nttdata.co.jp> wrote:
> > >> >> Thank you Vinod!
> > >> >>
> > >> >> FYI: 2.7.3 will be a bit special release.
> > >> >>
> > >> >> HDFS-8791 bumped up the datanode layout version,
> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >> >> is impossible. We can rollback instead.
> > >> >>
> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
> > >> >>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >> >>
> > >> >> Regards,
> > >> >> Akira
> > >> >>
> > >> >>
> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >> >>>
> > >> >>> Hi all,
> > >> >>>
> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> > >> (which
> > >> >>> did go out mid February). Got a little busy since.
> > >> >>>
> > >> >>> Following up the 2.7.2 maintenance release, we should work
> towards a
> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
> bug-fixes
> > >> and *no*
> > >> >>> features / improvements.
> > >> >>>
> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> > >> blocker
> > >> >>> / critical issues. Will start moving out any tickets that are not
> > >> blockers
> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
> critical
> > >> tickets
> > >> >>> outstanding as of now.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> +Vinod
> > >> >>>
> > >> >>> [1] 2.7.3 release blockers:
> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> > >> >>>
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > busbey
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

A few thoughts:

1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
for data nodes depending on your setup.

2. We have already deployed this patch internally so, with my Twitter hat
on, I would be perfectly happy as long as it makes it into trunk and 2.8.
That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
releases on a large production cluster that has a diverse set of block ids
without this patch, especially if your data nodes have a large number of
disks or you are using federation. To be clear though: this highly depends
on your setup and at a minimum you should verify that this regression will
not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
performance regression that gets worse over time. When you see it happening
on a live cluster, it is one of the harder issues to identify a root cause
and debug. I do understand that this is currently only affecting a smaller
number of users, but I also think this number has potential to increase as
time goes on. Maybe we can issue a warning in the release notes for future
2.7.x and 2.6.x releases?

3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
this proposal on this thread) would be to cut a 2.8 release off of the
2.7.3 release with the new layout. What people currently think of as 2.8
would then become 2.9. This would give customers a stable release that they
could deploy with the new layout and would not break upgrade and downgrade
expectations.

On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:

> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
> patch the release to revert HDFS-8791 before pushing it out to production.
> For what it's worth.
>
>
> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> > HDFS-8578 is a very important related fix since otherwise upgrade will be
> > very slow.
> >
> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > > maintenance release. I've only seen it crop up on a handful of our
> > > customer's clusters, and large users like Twitter and Yahoo that seem
> to
> > be
> > > more affected are also the most able to patch this change in
> themselves.
> > >
> > > Layout upgrades are quite disruptive, and I don't think it's worth
> > > breaking upgrade and downgrade expectations when it doesn't affect the
> > (in
> > > my experience) vast majority of users.
> > >
> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > > will let him elaborate.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> > >> if *any* of them end up introducing a regression the inclusion of
> > >> HDFS-8791 means that folks will have cluster downtime in order to back
> > >> things out. If that happens to any substantial number of downstream
> > >> folks, or any particularly vocal downstream folks, then it is very
> > >> likely we'll lose the remaining trust of operators for rolling out
> > >> maintenance releases. That's a pretty steep cost.
> > >>
> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >> unreasonable burden.
> > >>
> > >> I agree that this fix is important, I just think we should either cut
> > >> a version of 2.8 that includes it or find a way to do it that gives an
> > >> operational path for rolling downgrade.
> > >>
> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > wrote:
> > >> > Thanks for bringing up this topic, Sean.
> > >> > When I released our latest Hadoop release 2.6.4, the patch of
> > HDFS-8791
> > >> haven't been committed in so that's why we didn't discuss this
> earlier.
> > >> > I remember in JIRA discussion, we treated this layout change as a
> > >> Blocker bug that fixing a significant performance regression before
> but
> > not
> > >> a normal performance improvement. And I believe HDFS community already
> > did
> > >> their best with careful and patient to deliver the fix and other
> related
> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> > you
> > >> can see 30+ rounds patch review back and forth by senior committers,
> > not to
> > >> mention the outstanding performance test data in HDFS-8791.
> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> > for
> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
> into
> > >> 2.6.5 now.
> > >> > Just my 2 cents.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Junping
> > >> >
> > >> > ________________________________________
> > >> > From: Sean Busbey <bu...@cloudera.com>
> > >> > Sent: Thursday, March 31, 2016 2:57 PM
> > >> > To: hdfs-dev@hadoop.apache.org
> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >> mapreduce-dev@hadoop.apache.org
> > >> > Subject: Re: 2.7.3 release plan
> > >> >
> > >> > A layout change in a maintenance release sounds very risky. I saw
> some
> > >> > discussion on the JIRA about those risks, but the consensus seemed
> to
> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
> thought
> > >> > we did RMs per release rather than per branch? No one claiming to
> be a
> > >> > release manager ever spoke up AFAICT.
> > >> >
> > >> > Should this change be included? Should it go into a special 2.8
> > >> > release as mentioned in the ticket?
> > >> >
> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >> > <aj...@oss.nttdata.co.jp> wrote:
> > >> >> Thank you Vinod!
> > >> >>
> > >> >> FYI: 2.7.3 will be a bit special release.
> > >> >>
> > >> >> HDFS-8791 bumped up the datanode layout version,
> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >> >> is impossible. We can rollback instead.
> > >> >>
> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
> > >> >>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >> >>
> > >> >> Regards,
> > >> >> Akira
> > >> >>
> > >> >>
> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >> >>>
> > >> >>> Hi all,
> > >> >>>
> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> > >> (which
> > >> >>> did go out mid February). Got a little busy since.
> > >> >>>
> > >> >>> Following up the 2.7.2 maintenance release, we should work
> towards a
> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
> bug-fixes
> > >> and *no*
> > >> >>> features / improvements.
> > >> >>>
> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> > >> blocker
> > >> >>> / critical issues. Will start moving out any tickets that are not
> > >> blockers
> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
> critical
> > >> tickets
> > >> >>> outstanding as of now.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> +Vinod
> > >> >>>
> > >> >>> [1] 2.7.3 release blockers:
> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> > >> >>>
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > busbey
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

A few thoughts:

1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
for data nodes depending on your setup.

2. We have already deployed this patch internally so, with my Twitter hat
on, I would be perfectly happy as long as it makes it into trunk and 2.8.
That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
releases on a large production cluster that has a diverse set of block ids
without this patch, especially if your data nodes have a large number of
disks or you are using federation. To be clear though: this highly depends
on your setup and at a minimum you should verify that this regression will
not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
performance regression that gets worse over time. When you see it happening
on a live cluster, it is one of the harder issues to identify a root cause
and debug. I do understand that this is currently only affecting a smaller
number of users, but I also think this number has potential to increase as
time goes on. Maybe we can issue a warning in the release notes for future
2.7.x and 2.6.x releases?

3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
this proposal on this thread) would be to cut a 2.8 release off of the
2.7.3 release with the new layout. What people currently think of as 2.8
would then become 2.9. This would give customers a stable release that they
could deploy with the new layout and would not break upgrade and downgrade
expectations.

On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:

> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
> patch the release to revert HDFS-8791 before pushing it out to production.
> For what it's worth.
>
>
> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> > HDFS-8578 is a very important related fix since otherwise upgrade will be
> > very slow.
> >
> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > > maintenance release. I've only seen it crop up on a handful of our
> > > customer's clusters, and large users like Twitter and Yahoo that seem
> to
> > be
> > > more affected are also the most able to patch this change in
> themselves.
> > >
> > > Layout upgrades are quite disruptive, and I don't think it's worth
> > > breaking upgrade and downgrade expectations when it doesn't affect the
> > (in
> > > my experience) vast majority of users.
> > >
> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > > will let him elaborate.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> > >> if *any* of them end up introducing a regression the inclusion of
> > >> HDFS-8791 means that folks will have cluster downtime in order to back
> > >> things out. If that happens to any substantial number of downstream
> > >> folks, or any particularly vocal downstream folks, then it is very
> > >> likely we'll lose the remaining trust of operators for rolling out
> > >> maintenance releases. That's a pretty steep cost.
> > >>
> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >> unreasonable burden.
> > >>
> > >> I agree that this fix is important, I just think we should either cut
> > >> a version of 2.8 that includes it or find a way to do it that gives an
> > >> operational path for rolling downgrade.
> > >>
> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > wrote:
> > >> > Thanks for bringing up this topic, Sean.
> > >> > When I released our latest Hadoop release 2.6.4, the patch of
> > HDFS-8791
> > >> haven't been committed in so that's why we didn't discuss this
> earlier.
> > >> > I remember in JIRA discussion, we treated this layout change as a
> > >> Blocker bug that fixing a significant performance regression before
> but
> > not
> > >> a normal performance improvement. And I believe HDFS community already
> > did
> > >> their best with careful and patient to deliver the fix and other
> related
> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> > you
> > >> can see 30+ rounds patch review back and forth by senior committers,
> > not to
> > >> mention the outstanding performance test data in HDFS-8791.
> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> > for
> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
> into
> > >> 2.6.5 now.
> > >> > Just my 2 cents.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Junping
> > >> >
> > >> > ________________________________________
> > >> > From: Sean Busbey <bu...@cloudera.com>
> > >> > Sent: Thursday, March 31, 2016 2:57 PM
> > >> > To: hdfs-dev@hadoop.apache.org
> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >> mapreduce-dev@hadoop.apache.org
> > >> > Subject: Re: 2.7.3 release plan
> > >> >
> > >> > A layout change in a maintenance release sounds very risky. I saw
> some
> > >> > discussion on the JIRA about those risks, but the consensus seemed
> to
> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
> thought
> > >> > we did RMs per release rather than per branch? No one claiming to
> be a
> > >> > release manager ever spoke up AFAICT.
> > >> >
> > >> > Should this change be included? Should it go into a special 2.8
> > >> > release as mentioned in the ticket?
> > >> >
> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >> > <aj...@oss.nttdata.co.jp> wrote:
> > >> >> Thank you Vinod!
> > >> >>
> > >> >> FYI: 2.7.3 will be a bit special release.
> > >> >>
> > >> >> HDFS-8791 bumped up the datanode layout version,
> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >> >> is impossible. We can rollback instead.
> > >> >>
> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
> > >> >>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >> >>
> > >> >> Regards,
> > >> >> Akira
> > >> >>
> > >> >>
> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >> >>>
> > >> >>> Hi all,
> > >> >>>
> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> > >> (which
> > >> >>> did go out mid February). Got a little busy since.
> > >> >>>
> > >> >>> Following up the 2.7.2 maintenance release, we should work
> towards a
> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
> bug-fixes
> > >> and *no*
> > >> >>> features / improvements.
> > >> >>>
> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> > >> blocker
> > >> >>> / critical issues. Will start moving out any tickets that are not
> > >> blockers
> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
> critical
> > >> tickets
> > >> >>> outstanding as of now.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> +Vinod
> > >> >>>
> > >> >>> [1] 2.7.3 release blockers:
> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> > >> >>>
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > busbey
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: 2.7.3 release plan

[Chris Trezzo <ct...@gmail.com>]

A few thoughts:

1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
for data nodes depending on your setup.

2. We have already deployed this patch internally so, with my Twitter hat
on, I would be perfectly happy as long as it makes it into trunk and 2.8.
That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
releases on a large production cluster that has a diverse set of block ids
without this patch, especially if your data nodes have a large number of
disks or you are using federation. To be clear though: this highly depends
on your setup and at a minimum you should verify that this regression will
not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
performance regression that gets worse over time. When you see it happening
on a live cluster, it is one of the harder issues to identify a root cause
and debug. I do understand that this is currently only affecting a smaller
number of users, but I also think this number has potential to increase as
time goes on. Maybe we can issue a warning in the release notes for future
2.7.x and 2.6.x releases?

3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
this proposal on this thread) would be to cut a 2.8 release off of the
2.7.3 release with the new layout. What people currently think of as 2.8
would then become 2.9. This would give customers a stable release that they
could deploy with the new layout and would not break upgrade and downgrade
expectations.

On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <ap...@apache.org> wrote:

> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
> patch the release to revert HDFS-8791 before pushing it out to production.
> For what it's worth.
>
>
> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> > HDFS-8578 is a very important related fix since otherwise upgrade will be
> > very slow.
> >
> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > > maintenance release. I've only seen it crop up on a handful of our
> > > customer's clusters, and large users like Twitter and Yahoo that seem
> to
> > be
> > > more affected are also the most able to patch this change in
> themselves.
> > >
> > > Layout upgrades are quite disruptive, and I don't think it's worth
> > > breaking upgrade and downgrade expectations when it doesn't affect the
> > (in
> > > my experience) vast majority of users.
> > >
> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > > will let him elaborate.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> > >> if *any* of them end up introducing a regression the inclusion of
> > >> HDFS-8791 means that folks will have cluster downtime in order to back
> > >> things out. If that happens to any substantial number of downstream
> > >> folks, or any particularly vocal downstream folks, then it is very
> > >> likely we'll lose the remaining trust of operators for rolling out
> > >> maintenance releases. That's a pretty steep cost.
> > >>
> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> > >> unreasonable burden.
> > >>
> > >> I agree that this fix is important, I just think we should either cut
> > >> a version of 2.8 that includes it or find a way to do it that gives an
> > >> operational path for rolling downgrade.
> > >>
> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> > wrote:
> > >> > Thanks for bringing up this topic, Sean.
> > >> > When I released our latest Hadoop release 2.6.4, the patch of
> > HDFS-8791
> > >> haven't been committed in so that's why we didn't discuss this
> earlier.
> > >> > I remember in JIRA discussion, we treated this layout change as a
> > >> Blocker bug that fixing a significant performance regression before
> but
> > not
> > >> a normal performance improvement. And I believe HDFS community already
> > did
> > >> their best with careful and patient to deliver the fix and other
> related
> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> > you
> > >> can see 30+ rounds patch review back and forth by senior committers,
> > not to
> > >> mention the outstanding performance test data in HDFS-8791.
> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> > for
> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
> into
> > >> 2.6.5 now.
> > >> > Just my 2 cents.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Junping
> > >> >
> > >> > ________________________________________
> > >> > From: Sean Busbey <bu...@cloudera.com>
> > >> > Sent: Thursday, March 31, 2016 2:57 PM
> > >> > To: hdfs-dev@hadoop.apache.org
> > >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> > >> mapreduce-dev@hadoop.apache.org
> > >> > Subject: Re: 2.7.3 release plan
> > >> >
> > >> > A layout change in a maintenance release sounds very risky. I saw
> some
> > >> > discussion on the JIRA about those risks, but the consensus seemed
> to
> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
> thought
> > >> > we did RMs per release rather than per branch? No one claiming to
> be a
> > >> > release manager ever spoke up AFAICT.
> > >> >
> > >> > Should this change be included? Should it go into a special 2.8
> > >> > release as mentioned in the ticket?
> > >> >
> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > >> > <aj...@oss.nttdata.co.jp> wrote:
> > >> >> Thank you Vinod!
> > >> >>
> > >> >> FYI: 2.7.3 will be a bit special release.
> > >> >>
> > >> >> HDFS-8791 bumped up the datanode layout version,
> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> > >> >> is impossible. We can rollback instead.
> > >> >>
> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
> > >> >>
> > >>
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> > >> >>
> > >> >> Regards,
> > >> >> Akira
> > >> >>
> > >> >>
> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> > >> >>>
> > >> >>> Hi all,
> > >> >>>
> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> > >> (which
> > >> >>> did go out mid February). Got a little busy since.
> > >> >>>
> > >> >>> Following up the 2.7.2 maintenance release, we should work
> towards a
> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
> bug-fixes
> > >> and *no*
> > >> >>> features / improvements.
> > >> >>>
> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> > >> blocker
> > >> >>> / critical issues. Will start moving out any tickets that are not
> > >> blockers
> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
> critical
> > >> tickets
> > >> >>> outstanding as of now.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> +Vinod
> > >> >>>
> > >> >>> [1] 2.7.3 release blockers:
> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> > >> >>>
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > busbey
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: 2.7.3 release plan

[Andrew Purtell <ap...@apache.org>]

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
patch the release to revert HDFS-8791 before pushing it out to production.
For what it's worth.


On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
wrote:

> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> HDFS-8578 is a very important related fix since otherwise upgrade will be
> very slow.
>
> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > maintenance release. I've only seen it crop up on a handful of our
> > customer's clusters, and large users like Twitter and Yahoo that seem to
> be
> > more affected are also the most able to patch this change in themselves.
> >
> > Layout upgrades are quite disruptive, and I don't think it's worth
> > breaking upgrade and downgrade expectations when it doesn't affect the
> (in
> > my experience) vast majority of users.
> >
> > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > will let him elaborate.
> >
> > Best,
> > Andrew
> >
> > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >> if *any* of them end up introducing a regression the inclusion of
> >> HDFS-8791 means that folks will have cluster downtime in order to back
> >> things out. If that happens to any substantial number of downstream
> >> folks, or any particularly vocal downstream folks, then it is very
> >> likely we'll lose the remaining trust of operators for rolling out
> >> maintenance releases. That's a pretty steep cost.
> >>
> >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >> unreasonable burden.
> >>
> >> I agree that this fix is important, I just think we should either cut
> >> a version of 2.8 that includes it or find a way to do it that gives an
> >> operational path for rolling downgrade.
> >>
> >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> wrote:
> >> > Thanks for bringing up this topic, Sean.
> >> > When I released our latest Hadoop release 2.6.4, the patch of
> HDFS-8791
> >> haven't been committed in so that's why we didn't discuss this earlier.
> >> > I remember in JIRA discussion, we treated this layout change as a
> >> Blocker bug that fixing a significant performance regression before but
> not
> >> a normal performance improvement. And I believe HDFS community already
> did
> >> their best with careful and patient to deliver the fix and other related
> >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> you
> >> can see 30+ rounds patch review back and forth by senior committers,
> not to
> >> mention the outstanding performance test data in HDFS-8791.
> >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> for
> >> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> >> 2.6.5 now.
> >> > Just my 2 cents.
> >> >
> >> > Thanks,
> >> >
> >> > Junping
> >> >
> >> > ________________________________________
> >> > From: Sean Busbey <bu...@cloudera.com>
> >> > Sent: Thursday, March 31, 2016 2:57 PM
> >> > To: hdfs-dev@hadoop.apache.org
> >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> > Subject: Re: 2.7.3 release plan
> >> >
> >> > A layout change in a maintenance release sounds very risky. I saw some
> >> > discussion on the JIRA about those risks, but the consensus seemed to
> >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> >> > we did RMs per release rather than per branch? No one claiming to be a
> >> > release manager ever spoke up AFAICT.
> >> >
> >> > Should this change be included? Should it go into a special 2.8
> >> > release as mentioned in the ticket?
> >> >
> >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >> > <aj...@oss.nttdata.co.jp> wrote:
> >> >> Thank you Vinod!
> >> >>
> >> >> FYI: 2.7.3 will be a bit special release.
> >> >>
> >> >> HDFS-8791 bumped up the datanode layout version,
> >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> >> is impossible. We can rollback instead.
> >> >>
> >> >> https://issues.apache.org/jira/browse/HDFS-8791
> >> >>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >> >>
> >> >> Regards,
> >> >> Akira
> >> >>
> >> >>
> >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >> >>>
> >> >>> Hi all,
> >> >>>
> >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >> (which
> >> >>> did go out mid February). Got a little busy since.
> >> >>>
> >> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> >> and *no*
> >> >>> features / improvements.
> >> >>>
> >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> >> blocker
> >> >>> / critical issues. Will start moving out any tickets that are not
> >> blockers
> >> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> >> tickets
> >> >>> outstanding as of now.
> >> >>>
> >> >>> Thanks,
> >> >>> +Vinod
> >> >>>
> >> >>> [1] 2.7.3 release blockers:
> >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >> >>>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > busbey
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: 2.7.3 release plan

[Andrew Purtell <ap...@apache.org>]

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
patch the release to revert HDFS-8791 before pushing it out to production.
For what it's worth.


On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
wrote:

> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> HDFS-8578 is a very important related fix since otherwise upgrade will be
> very slow.
>
> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > maintenance release. I've only seen it crop up on a handful of our
> > customer's clusters, and large users like Twitter and Yahoo that seem to
> be
> > more affected are also the most able to patch this change in themselves.
> >
> > Layout upgrades are quite disruptive, and I don't think it's worth
> > breaking upgrade and downgrade expectations when it doesn't affect the
> (in
> > my experience) vast majority of users.
> >
> > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > will let him elaborate.
> >
> > Best,
> > Andrew
> >
> > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >> if *any* of them end up introducing a regression the inclusion of
> >> HDFS-8791 means that folks will have cluster downtime in order to back
> >> things out. If that happens to any substantial number of downstream
> >> folks, or any particularly vocal downstream folks, then it is very
> >> likely we'll lose the remaining trust of operators for rolling out
> >> maintenance releases. That's a pretty steep cost.
> >>
> >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >> unreasonable burden.
> >>
> >> I agree that this fix is important, I just think we should either cut
> >> a version of 2.8 that includes it or find a way to do it that gives an
> >> operational path for rolling downgrade.
> >>
> >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> wrote:
> >> > Thanks for bringing up this topic, Sean.
> >> > When I released our latest Hadoop release 2.6.4, the patch of
> HDFS-8791
> >> haven't been committed in so that's why we didn't discuss this earlier.
> >> > I remember in JIRA discussion, we treated this layout change as a
> >> Blocker bug that fixing a significant performance regression before but
> not
> >> a normal performance improvement. And I believe HDFS community already
> did
> >> their best with careful and patient to deliver the fix and other related
> >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> you
> >> can see 30+ rounds patch review back and forth by senior committers,
> not to
> >> mention the outstanding performance test data in HDFS-8791.
> >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> for
> >> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> >> 2.6.5 now.
> >> > Just my 2 cents.
> >> >
> >> > Thanks,
> >> >
> >> > Junping
> >> >
> >> > ________________________________________
> >> > From: Sean Busbey <bu...@cloudera.com>
> >> > Sent: Thursday, March 31, 2016 2:57 PM
> >> > To: hdfs-dev@hadoop.apache.org
> >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> > Subject: Re: 2.7.3 release plan
> >> >
> >> > A layout change in a maintenance release sounds very risky. I saw some
> >> > discussion on the JIRA about those risks, but the consensus seemed to
> >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> >> > we did RMs per release rather than per branch? No one claiming to be a
> >> > release manager ever spoke up AFAICT.
> >> >
> >> > Should this change be included? Should it go into a special 2.8
> >> > release as mentioned in the ticket?
> >> >
> >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >> > <aj...@oss.nttdata.co.jp> wrote:
> >> >> Thank you Vinod!
> >> >>
> >> >> FYI: 2.7.3 will be a bit special release.
> >> >>
> >> >> HDFS-8791 bumped up the datanode layout version,
> >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> >> is impossible. We can rollback instead.
> >> >>
> >> >> https://issues.apache.org/jira/browse/HDFS-8791
> >> >>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >> >>
> >> >> Regards,
> >> >> Akira
> >> >>
> >> >>
> >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >> >>>
> >> >>> Hi all,
> >> >>>
> >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >> (which
> >> >>> did go out mid February). Got a little busy since.
> >> >>>
> >> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> >> and *no*
> >> >>> features / improvements.
> >> >>>
> >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> >> blocker
> >> >>> / critical issues. Will start moving out any tickets that are not
> >> blockers
> >> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> >> tickets
> >> >>> outstanding as of now.
> >> >>>
> >> >>> Thanks,
> >> >>> +Vinod
> >> >>>
> >> >>> [1] 2.7.3 release blockers:
> >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >> >>>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > busbey
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: 2.7.3 release plan

[Andrew Purtell <ap...@apache.org>]

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
patch the release to revert HDFS-8791 before pushing it out to production.
For what it's worth.


On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
wrote:

> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> HDFS-8578 is a very important related fix since otherwise upgrade will be
> very slow.
>
> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > maintenance release. I've only seen it crop up on a handful of our
> > customer's clusters, and large users like Twitter and Yahoo that seem to
> be
> > more affected are also the most able to patch this change in themselves.
> >
> > Layout upgrades are quite disruptive, and I don't think it's worth
> > breaking upgrade and downgrade expectations when it doesn't affect the
> (in
> > my experience) vast majority of users.
> >
> > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > will let him elaborate.
> >
> > Best,
> > Andrew
> >
> > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >> if *any* of them end up introducing a regression the inclusion of
> >> HDFS-8791 means that folks will have cluster downtime in order to back
> >> things out. If that happens to any substantial number of downstream
> >> folks, or any particularly vocal downstream folks, then it is very
> >> likely we'll lose the remaining trust of operators for rolling out
> >> maintenance releases. That's a pretty steep cost.
> >>
> >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >> unreasonable burden.
> >>
> >> I agree that this fix is important, I just think we should either cut
> >> a version of 2.8 that includes it or find a way to do it that gives an
> >> operational path for rolling downgrade.
> >>
> >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> wrote:
> >> > Thanks for bringing up this topic, Sean.
> >> > When I released our latest Hadoop release 2.6.4, the patch of
> HDFS-8791
> >> haven't been committed in so that's why we didn't discuss this earlier.
> >> > I remember in JIRA discussion, we treated this layout change as a
> >> Blocker bug that fixing a significant performance regression before but
> not
> >> a normal performance improvement. And I believe HDFS community already
> did
> >> their best with careful and patient to deliver the fix and other related
> >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> you
> >> can see 30+ rounds patch review back and forth by senior committers,
> not to
> >> mention the outstanding performance test data in HDFS-8791.
> >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> for
> >> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> >> 2.6.5 now.
> >> > Just my 2 cents.
> >> >
> >> > Thanks,
> >> >
> >> > Junping
> >> >
> >> > ________________________________________
> >> > From: Sean Busbey <bu...@cloudera.com>
> >> > Sent: Thursday, March 31, 2016 2:57 PM
> >> > To: hdfs-dev@hadoop.apache.org
> >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> > Subject: Re: 2.7.3 release plan
> >> >
> >> > A layout change in a maintenance release sounds very risky. I saw some
> >> > discussion on the JIRA about those risks, but the consensus seemed to
> >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> >> > we did RMs per release rather than per branch? No one claiming to be a
> >> > release manager ever spoke up AFAICT.
> >> >
> >> > Should this change be included? Should it go into a special 2.8
> >> > release as mentioned in the ticket?
> >> >
> >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >> > <aj...@oss.nttdata.co.jp> wrote:
> >> >> Thank you Vinod!
> >> >>
> >> >> FYI: 2.7.3 will be a bit special release.
> >> >>
> >> >> HDFS-8791 bumped up the datanode layout version,
> >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> >> is impossible. We can rollback instead.
> >> >>
> >> >> https://issues.apache.org/jira/browse/HDFS-8791
> >> >>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >> >>
> >> >> Regards,
> >> >> Akira
> >> >>
> >> >>
> >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >> >>>
> >> >>> Hi all,
> >> >>>
> >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >> (which
> >> >>> did go out mid February). Got a little busy since.
> >> >>>
> >> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> >> and *no*
> >> >>> features / improvements.
> >> >>>
> >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> >> blocker
> >> >>> / critical issues. Will start moving out any tickets that are not
> >> blockers
> >> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> >> tickets
> >> >>> outstanding as of now.
> >> >>>
> >> >>> Thanks,
> >> >>> +Vinod
> >> >>>
> >> >>> [1] 2.7.3 release blockers:
> >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >> >>>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > busbey
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: 2.7.3 release plan

[Andrew Purtell <ap...@apache.org>]

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
patch the release to revert HDFS-8791 before pushing it out to production.
For what it's worth.


On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <an...@cloudera.com>
wrote:

> One other thing I wanted to bring up regarding HDFS-8791, we haven't
> backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
> HDFS-8578 is a very important related fix since otherwise upgrade will be
> very slow.
>
> On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > As I expressed on HDFS-8791, I do not want to include this JIRA in a
> > maintenance release. I've only seen it crop up on a handful of our
> > customer's clusters, and large users like Twitter and Yahoo that seem to
> be
> > more affected are also the most able to patch this change in themselves.
> >
> > Layout upgrades are quite disruptive, and I don't think it's worth
> > breaking upgrade and downgrade expectations when it doesn't affect the
> (in
> > my experience) vast majority of users.
> >
> > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> > will let him elaborate.
> >
> > Best,
> > Andrew
> >
> > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> >> if *any* of them end up introducing a regression the inclusion of
> >> HDFS-8791 means that folks will have cluster downtime in order to back
> >> things out. If that happens to any substantial number of downstream
> >> folks, or any particularly vocal downstream folks, then it is very
> >> likely we'll lose the remaining trust of operators for rolling out
> >> maintenance releases. That's a pretty steep cost.
> >>
> >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> >> unreasonable burden.
> >>
> >> I agree that this fix is important, I just think we should either cut
> >> a version of 2.8 that includes it or find a way to do it that gives an
> >> operational path for rolling downgrade.
> >>
> >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com>
> wrote:
> >> > Thanks for bringing up this topic, Sean.
> >> > When I released our latest Hadoop release 2.6.4, the patch of
> HDFS-8791
> >> haven't been committed in so that's why we didn't discuss this earlier.
> >> > I remember in JIRA discussion, we treated this layout change as a
> >> Blocker bug that fixing a significant performance regression before but
> not
> >> a normal performance improvement. And I believe HDFS community already
> did
> >> their best with careful and patient to deliver the fix and other related
> >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
> you
> >> can see 30+ rounds patch review back and forth by senior committers,
> not to
> >> mention the outstanding performance test data in HDFS-8791.
> >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
> >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
> for
> >> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> >> 2.6.5 now.
> >> > Just my 2 cents.
> >> >
> >> > Thanks,
> >> >
> >> > Junping
> >> >
> >> > ________________________________________
> >> > From: Sean Busbey <bu...@cloudera.com>
> >> > Sent: Thursday, March 31, 2016 2:57 PM
> >> > To: hdfs-dev@hadoop.apache.org
> >> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> >> mapreduce-dev@hadoop.apache.org
> >> > Subject: Re: 2.7.3 release plan
> >> >
> >> > A layout change in a maintenance release sounds very risky. I saw some
> >> > discussion on the JIRA about those risks, but the consensus seemed to
> >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> >> > we did RMs per release rather than per branch? No one claiming to be a
> >> > release manager ever spoke up AFAICT.
> >> >
> >> > Should this change be included? Should it go into a special 2.8
> >> > release as mentioned in the ticket?
> >> >
> >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> >> > <aj...@oss.nttdata.co.jp> wrote:
> >> >> Thank you Vinod!
> >> >>
> >> >> FYI: 2.7.3 will be a bit special release.
> >> >>
> >> >> HDFS-8791 bumped up the datanode layout version,
> >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> >> is impossible. We can rollback instead.
> >> >>
> >> >> https://issues.apache.org/jira/browse/HDFS-8791
> >> >>
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >> >>
> >> >> Regards,
> >> >> Akira
> >> >>
> >> >>
> >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >> >>>
> >> >>> Hi all,
> >> >>>
> >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >> (which
> >> >>> did go out mid February). Got a little busy since.
> >> >>>
> >> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> >> and *no*
> >> >>> features / improvements.
> >> >>>
> >> >>> I hope to cut an RC in a week - giving enough time for outstanding
> >> blocker
> >> >>> / critical issues. Will start moving out any tickets that are not
> >> blockers
> >> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> >> tickets
> >> >>> outstanding as of now.
> >> >>>
> >> >>> Thanks,
> >> >>> +Vinod
> >> >>>
> >> >>> [1] 2.7.3 release blockers:
> >> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >> >>>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > busbey
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

One other thing I wanted to bring up regarding HDFS-8791, we haven't
backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
HDFS-8578 is a very important related fix since otherwise upgrade will be
very slow.

On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
wrote:

> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> maintenance release. I've only seen it crop up on a handful of our
> customer's clusters, and large users like Twitter and Yahoo that seem to be
> more affected are also the most able to patch this change in themselves.
>
> Layout upgrades are quite disruptive, and I don't think it's worth
> breaking upgrade and downgrade expectations when it doesn't affect the (in
> my experience) vast majority of users.
>
> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> will let him elaborate.
>
> Best,
> Andrew
>
> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> if *any* of them end up introducing a regression the inclusion of
>> HDFS-8791 means that folks will have cluster downtime in order to back
>> things out. If that happens to any substantial number of downstream
>> folks, or any particularly vocal downstream folks, then it is very
>> likely we'll lose the remaining trust of operators for rolling out
>> maintenance releases. That's a pretty steep cost.
>>
>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> unreasonable burden.
>>
>> I agree that this fix is important, I just think we should either cut
>> a version of 2.8 that includes it or find a way to do it that gives an
>> operational path for rolling downgrade.
>>
>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
>> > Thanks for bringing up this topic, Sean.
>> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
>> haven't been committed in so that's why we didn't discuss this earlier.
>> > I remember in JIRA discussion, we treated this layout change as a
>> Blocker bug that fixing a significant performance regression before but not
>> a normal performance improvement. And I believe HDFS community already did
>> their best with careful and patient to deliver the fix and other related
>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
>> can see 30+ rounds patch review back and forth by senior committers, not to
>> mention the outstanding performance test data in HDFS-8791.
>> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM for
>> branch-2.7. In addition, I didn't see any blocker issue to bring it into
>> 2.6.5 now.
>> > Just my 2 cents.
>> >
>> > Thanks,
>> >
>> > Junping
>> >
>> > ________________________________________
>> > From: Sean Busbey <bu...@cloudera.com>
>> > Sent: Thursday, March 31, 2016 2:57 PM
>> > To: hdfs-dev@hadoop.apache.org
>> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> > Subject: Re: 2.7.3 release plan
>> >
>> > A layout change in a maintenance release sounds very risky. I saw some
>> > discussion on the JIRA about those risks, but the consensus seemed to
>> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
>> > we did RMs per release rather than per branch? No one claiming to be a
>> > release manager ever spoke up AFAICT.
>> >
>> > Should this change be included? Should it go into a special 2.8
>> > release as mentioned in the ticket?
>> >
>> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > <aj...@oss.nttdata.co.jp> wrote:
>> >> Thank you Vinod!
>> >>
>> >> FYI: 2.7.3 will be a bit special release.
>> >>
>> >> HDFS-8791 bumped up the datanode layout version,
>> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> >> is impossible. We can rollback instead.
>> >>
>> >> https://issues.apache.org/jira/browse/HDFS-8791
>> >>
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> >>
>> >> Regards,
>> >> Akira
>> >>
>> >>
>> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> (which
>> >>> did go out mid February). Got a little busy since.
>> >>>
>> >>> Following up the 2.7.2 maintenance release, we should work towards a
>> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
>> and *no*
>> >>> features / improvements.
>> >>>
>> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> blocker
>> >>> / critical issues. Will start moving out any tickets that are not
>> blockers
>> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
>> tickets
>> >>> outstanding as of now.
>> >>>
>> >>> Thanks,
>> >>> +Vinod
>> >>>
>> >>> [1] 2.7.3 release blockers:
>> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > busbey
>>
>>
>>
>> --
>> busbey
>>
>
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

One other thing I wanted to bring up regarding HDFS-8791, we haven't
backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
HDFS-8578 is a very important related fix since otherwise upgrade will be
very slow.

On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
wrote:

> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> maintenance release. I've only seen it crop up on a handful of our
> customer's clusters, and large users like Twitter and Yahoo that seem to be
> more affected are also the most able to patch this change in themselves.
>
> Layout upgrades are quite disruptive, and I don't think it's worth
> breaking upgrade and downgrade expectations when it doesn't affect the (in
> my experience) vast majority of users.
>
> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> will let him elaborate.
>
> Best,
> Andrew
>
> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> if *any* of them end up introducing a regression the inclusion of
>> HDFS-8791 means that folks will have cluster downtime in order to back
>> things out. If that happens to any substantial number of downstream
>> folks, or any particularly vocal downstream folks, then it is very
>> likely we'll lose the remaining trust of operators for rolling out
>> maintenance releases. That's a pretty steep cost.
>>
>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> unreasonable burden.
>>
>> I agree that this fix is important, I just think we should either cut
>> a version of 2.8 that includes it or find a way to do it that gives an
>> operational path for rolling downgrade.
>>
>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
>> > Thanks for bringing up this topic, Sean.
>> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
>> haven't been committed in so that's why we didn't discuss this earlier.
>> > I remember in JIRA discussion, we treated this layout change as a
>> Blocker bug that fixing a significant performance regression before but not
>> a normal performance improvement. And I believe HDFS community already did
>> their best with careful and patient to deliver the fix and other related
>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
>> can see 30+ rounds patch review back and forth by senior committers, not to
>> mention the outstanding performance test data in HDFS-8791.
>> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM for
>> branch-2.7. In addition, I didn't see any blocker issue to bring it into
>> 2.6.5 now.
>> > Just my 2 cents.
>> >
>> > Thanks,
>> >
>> > Junping
>> >
>> > ________________________________________
>> > From: Sean Busbey <bu...@cloudera.com>
>> > Sent: Thursday, March 31, 2016 2:57 PM
>> > To: hdfs-dev@hadoop.apache.org
>> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> > Subject: Re: 2.7.3 release plan
>> >
>> > A layout change in a maintenance release sounds very risky. I saw some
>> > discussion on the JIRA about those risks, but the consensus seemed to
>> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
>> > we did RMs per release rather than per branch? No one claiming to be a
>> > release manager ever spoke up AFAICT.
>> >
>> > Should this change be included? Should it go into a special 2.8
>> > release as mentioned in the ticket?
>> >
>> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > <aj...@oss.nttdata.co.jp> wrote:
>> >> Thank you Vinod!
>> >>
>> >> FYI: 2.7.3 will be a bit special release.
>> >>
>> >> HDFS-8791 bumped up the datanode layout version,
>> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> >> is impossible. We can rollback instead.
>> >>
>> >> https://issues.apache.org/jira/browse/HDFS-8791
>> >>
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> >>
>> >> Regards,
>> >> Akira
>> >>
>> >>
>> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> (which
>> >>> did go out mid February). Got a little busy since.
>> >>>
>> >>> Following up the 2.7.2 maintenance release, we should work towards a
>> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
>> and *no*
>> >>> features / improvements.
>> >>>
>> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> blocker
>> >>> / critical issues. Will start moving out any tickets that are not
>> blockers
>> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
>> tickets
>> >>> outstanding as of now.
>> >>>
>> >>> Thanks,
>> >>> +Vinod
>> >>>
>> >>> [1] 2.7.3 release blockers:
>> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > busbey
>>
>>
>>
>> --
>> busbey
>>
>
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

One other thing I wanted to bring up regarding HDFS-8791, we haven't
backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
HDFS-8578 is a very important related fix since otherwise upgrade will be
very slow.

On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
wrote:

> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> maintenance release. I've only seen it crop up on a handful of our
> customer's clusters, and large users like Twitter and Yahoo that seem to be
> more affected are also the most able to patch this change in themselves.
>
> Layout upgrades are quite disruptive, and I don't think it's worth
> breaking upgrade and downgrade expectations when it doesn't affect the (in
> my experience) vast majority of users.
>
> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> will let him elaborate.
>
> Best,
> Andrew
>
> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> if *any* of them end up introducing a regression the inclusion of
>> HDFS-8791 means that folks will have cluster downtime in order to back
>> things out. If that happens to any substantial number of downstream
>> folks, or any particularly vocal downstream folks, then it is very
>> likely we'll lose the remaining trust of operators for rolling out
>> maintenance releases. That's a pretty steep cost.
>>
>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> unreasonable burden.
>>
>> I agree that this fix is important, I just think we should either cut
>> a version of 2.8 that includes it or find a way to do it that gives an
>> operational path for rolling downgrade.
>>
>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
>> > Thanks for bringing up this topic, Sean.
>> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
>> haven't been committed in so that's why we didn't discuss this earlier.
>> > I remember in JIRA discussion, we treated this layout change as a
>> Blocker bug that fixing a significant performance regression before but not
>> a normal performance improvement. And I believe HDFS community already did
>> their best with careful and patient to deliver the fix and other related
>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
>> can see 30+ rounds patch review back and forth by senior committers, not to
>> mention the outstanding performance test data in HDFS-8791.
>> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM for
>> branch-2.7. In addition, I didn't see any blocker issue to bring it into
>> 2.6.5 now.
>> > Just my 2 cents.
>> >
>> > Thanks,
>> >
>> > Junping
>> >
>> > ________________________________________
>> > From: Sean Busbey <bu...@cloudera.com>
>> > Sent: Thursday, March 31, 2016 2:57 PM
>> > To: hdfs-dev@hadoop.apache.org
>> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> > Subject: Re: 2.7.3 release plan
>> >
>> > A layout change in a maintenance release sounds very risky. I saw some
>> > discussion on the JIRA about those risks, but the consensus seemed to
>> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
>> > we did RMs per release rather than per branch? No one claiming to be a
>> > release manager ever spoke up AFAICT.
>> >
>> > Should this change be included? Should it go into a special 2.8
>> > release as mentioned in the ticket?
>> >
>> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > <aj...@oss.nttdata.co.jp> wrote:
>> >> Thank you Vinod!
>> >>
>> >> FYI: 2.7.3 will be a bit special release.
>> >>
>> >> HDFS-8791 bumped up the datanode layout version,
>> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> >> is impossible. We can rollback instead.
>> >>
>> >> https://issues.apache.org/jira/browse/HDFS-8791
>> >>
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> >>
>> >> Regards,
>> >> Akira
>> >>
>> >>
>> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> (which
>> >>> did go out mid February). Got a little busy since.
>> >>>
>> >>> Following up the 2.7.2 maintenance release, we should work towards a
>> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
>> and *no*
>> >>> features / improvements.
>> >>>
>> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> blocker
>> >>> / critical issues. Will start moving out any tickets that are not
>> blockers
>> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
>> tickets
>> >>> outstanding as of now.
>> >>>
>> >>> Thanks,
>> >>> +Vinod
>> >>>
>> >>> [1] 2.7.3 release blockers:
>> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > busbey
>>
>>
>>
>> --
>> busbey
>>
>
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

One other thing I wanted to bring up regarding HDFS-8791, we haven't
backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
HDFS-8578 is a very important related fix since otherwise upgrade will be
very slow.

On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <an...@cloudera.com>
wrote:

> As I expressed on HDFS-8791, I do not want to include this JIRA in a
> maintenance release. I've only seen it crop up on a handful of our
> customer's clusters, and large users like Twitter and Yahoo that seem to be
> more affected are also the most able to patch this change in themselves.
>
> Layout upgrades are quite disruptive, and I don't think it's worth
> breaking upgrade and downgrade expectations when it doesn't affect the (in
> my experience) vast majority of users.
>
> Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
> will let him elaborate.
>
> Best,
> Andrew
>
> On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> if *any* of them end up introducing a regression the inclusion of
>> HDFS-8791 means that folks will have cluster downtime in order to back
>> things out. If that happens to any substantial number of downstream
>> folks, or any particularly vocal downstream folks, then it is very
>> likely we'll lose the remaining trust of operators for rolling out
>> maintenance releases. That's a pretty steep cost.
>>
>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> unreasonable burden.
>>
>> I agree that this fix is important, I just think we should either cut
>> a version of 2.8 that includes it or find a way to do it that gives an
>> operational path for rolling downgrade.
>>
>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
>> > Thanks for bringing up this topic, Sean.
>> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
>> haven't been committed in so that's why we didn't discuss this earlier.
>> > I remember in JIRA discussion, we treated this layout change as a
>> Blocker bug that fixing a significant performance regression before but not
>> a normal performance improvement. And I believe HDFS community already did
>> their best with careful and patient to deliver the fix and other related
>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
>> can see 30+ rounds patch review back and forth by senior committers, not to
>> mention the outstanding performance test data in HDFS-8791.
>> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM for
>> branch-2.7. In addition, I didn't see any blocker issue to bring it into
>> 2.6.5 now.
>> > Just my 2 cents.
>> >
>> > Thanks,
>> >
>> > Junping
>> >
>> > ________________________________________
>> > From: Sean Busbey <bu...@cloudera.com>
>> > Sent: Thursday, March 31, 2016 2:57 PM
>> > To: hdfs-dev@hadoop.apache.org
>> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
>> mapreduce-dev@hadoop.apache.org
>> > Subject: Re: 2.7.3 release plan
>> >
>> > A layout change in a maintenance release sounds very risky. I saw some
>> > discussion on the JIRA about those risks, but the consensus seemed to
>> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
>> > we did RMs per release rather than per branch? No one claiming to be a
>> > release manager ever spoke up AFAICT.
>> >
>> > Should this change be included? Should it go into a special 2.8
>> > release as mentioned in the ticket?
>> >
>> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > <aj...@oss.nttdata.co.jp> wrote:
>> >> Thank you Vinod!
>> >>
>> >> FYI: 2.7.3 will be a bit special release.
>> >>
>> >> HDFS-8791 bumped up the datanode layout version,
>> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> >> is impossible. We can rollback instead.
>> >>
>> >> https://issues.apache.org/jira/browse/HDFS-8791
>> >>
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> >>
>> >> Regards,
>> >> Akira
>> >>
>> >>
>> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> (which
>> >>> did go out mid February). Got a little busy since.
>> >>>
>> >>> Following up the 2.7.2 maintenance release, we should work towards a
>> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
>> and *no*
>> >>> features / improvements.
>> >>>
>> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> blocker
>> >>> / critical issues. Will start moving out any tickets that are not
>> blockers
>> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
>> tickets
>> >>> outstanding as of now.
>> >>>
>> >>> Thanks,
>> >>> +Vinod
>> >>>
>> >>> [1] 2.7.3 release blockers:
>> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > busbey
>>
>>
>>
>> --
>> busbey
>>
>
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

As I expressed on HDFS-8791, I do not want to include this JIRA in a
maintenance release. I've only seen it crop up on a handful of our
customer's clusters, and large users like Twitter and Yahoo that seem to be
more affected are also the most able to patch this change in themselves.

Layout upgrades are quite disruptive, and I don't think it's worth breaking
upgrade and downgrade expectations when it doesn't affect the (in my
experience) vast majority of users.

Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
will let him elaborate.

Best,
Andrew

On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:

> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> if *any* of them end up introducing a regression the inclusion of
> HDFS-8791 means that folks will have cluster downtime in order to back
> things out. If that happens to any substantial number of downstream
> folks, or any particularly vocal downstream folks, then it is very
> likely we'll lose the remaining trust of operators for rolling out
> maintenance releases. That's a pretty steep cost.
>
> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> unreasonable burden.
>
> I agree that this fix is important, I just think we should either cut
> a version of 2.8 that includes it or find a way to do it that gives an
> operational path for rolling downgrade.
>
> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> > Thanks for bringing up this topic, Sean.
> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
> haven't been committed in so that's why we didn't discuss this earlier.
> > I remember in JIRA discussion, we treated this layout change as a
> Blocker bug that fixing a significant performance regression before but not
> a normal performance improvement. And I believe HDFS community already did
> their best with careful and patient to deliver the fix and other related
> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
> can see 30+ rounds patch review back and forth by senior committers, not to
> mention the outstanding performance test data in HDFS-8791.
> > I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3.
> However, that needs Vinod's final confirmation who serves as RM for
> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> 2.6.5 now.
> > Just my 2 cents.
> >
> > Thanks,
> >
> > Junping
> >
> > ________________________________________
> > From: Sean Busbey <bu...@cloudera.com>
> > Sent: Thursday, March 31, 2016 2:57 PM
> > To: hdfs-dev@hadoop.apache.org
> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> > Subject: Re: 2.7.3 release plan
> >
> > A layout change in a maintenance release sounds very risky. I saw some
> > discussion on the JIRA about those risks, but the consensus seemed to
> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> > we did RMs per release rather than per branch? No one claiming to be a
> > release manager ever spoke up AFAICT.
> >
> > Should this change be included? Should it go into a special 2.8
> > release as mentioned in the ticket?
> >
> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > <aj...@oss.nttdata.co.jp> wrote:
> >> Thank you Vinod!
> >>
> >> FYI: 2.7.3 will be a bit special release.
> >>
> >> HDFS-8791 bumped up the datanode layout version,
> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> is impossible. We can rollback instead.
> >>
> >> https://issues.apache.org/jira/browse/HDFS-8791
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which
> >>> did go out mid February). Got a little busy since.
> >>>
> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> and *no*
> >>> features / improvements.
> >>>
> >>> I hope to cut an RC in a week - giving enough time for outstanding
> blocker
> >>> / critical issues. Will start moving out any tickets that are not
> blockers
> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> tickets
> >>> outstanding as of now.
> >>>
> >>> Thanks,
> >>> +Vinod
> >>>
> >>> [1] 2.7.3 release blockers:
> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>
> >>
> >
> >
> >
> > --
> > busbey
>
>
>
> --
> busbey
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

As I expressed on HDFS-8791, I do not want to include this JIRA in a
maintenance release. I've only seen it crop up on a handful of our
customer's clusters, and large users like Twitter and Yahoo that seem to be
more affected are also the most able to patch this change in themselves.

Layout upgrades are quite disruptive, and I don't think it's worth breaking
upgrade and downgrade expectations when it doesn't affect the (in my
experience) vast majority of users.

Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
will let him elaborate.

Best,
Andrew

On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:

> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> if *any* of them end up introducing a regression the inclusion of
> HDFS-8791 means that folks will have cluster downtime in order to back
> things out. If that happens to any substantial number of downstream
> folks, or any particularly vocal downstream folks, then it is very
> likely we'll lose the remaining trust of operators for rolling out
> maintenance releases. That's a pretty steep cost.
>
> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> unreasonable burden.
>
> I agree that this fix is important, I just think we should either cut
> a version of 2.8 that includes it or find a way to do it that gives an
> operational path for rolling downgrade.
>
> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> > Thanks for bringing up this topic, Sean.
> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
> haven't been committed in so that's why we didn't discuss this earlier.
> > I remember in JIRA discussion, we treated this layout change as a
> Blocker bug that fixing a significant performance regression before but not
> a normal performance improvement. And I believe HDFS community already did
> their best with careful and patient to deliver the fix and other related
> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
> can see 30+ rounds patch review back and forth by senior committers, not to
> mention the outstanding performance test data in HDFS-8791.
> > I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3.
> However, that needs Vinod's final confirmation who serves as RM for
> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> 2.6.5 now.
> > Just my 2 cents.
> >
> > Thanks,
> >
> > Junping
> >
> > ________________________________________
> > From: Sean Busbey <bu...@cloudera.com>
> > Sent: Thursday, March 31, 2016 2:57 PM
> > To: hdfs-dev@hadoop.apache.org
> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> > Subject: Re: 2.7.3 release plan
> >
> > A layout change in a maintenance release sounds very risky. I saw some
> > discussion on the JIRA about those risks, but the consensus seemed to
> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> > we did RMs per release rather than per branch? No one claiming to be a
> > release manager ever spoke up AFAICT.
> >
> > Should this change be included? Should it go into a special 2.8
> > release as mentioned in the ticket?
> >
> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > <aj...@oss.nttdata.co.jp> wrote:
> >> Thank you Vinod!
> >>
> >> FYI: 2.7.3 will be a bit special release.
> >>
> >> HDFS-8791 bumped up the datanode layout version,
> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> is impossible. We can rollback instead.
> >>
> >> https://issues.apache.org/jira/browse/HDFS-8791
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which
> >>> did go out mid February). Got a little busy since.
> >>>
> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> and *no*
> >>> features / improvements.
> >>>
> >>> I hope to cut an RC in a week - giving enough time for outstanding
> blocker
> >>> / critical issues. Will start moving out any tickets that are not
> blockers
> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> tickets
> >>> outstanding as of now.
> >>>
> >>> Thanks,
> >>> +Vinod
> >>>
> >>> [1] 2.7.3 release blockers:
> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>
> >>
> >
> >
> >
> > --
> > busbey
>
>
>
> --
> busbey
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

As I expressed on HDFS-8791, I do not want to include this JIRA in a
maintenance release. I've only seen it crop up on a handful of our
customer's clusters, and large users like Twitter and Yahoo that seem to be
more affected are also the most able to patch this change in themselves.

Layout upgrades are quite disruptive, and I don't think it's worth breaking
upgrade and downgrade expectations when it doesn't affect the (in my
experience) vast majority of users.

Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
will let him elaborate.

Best,
Andrew

On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:

> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> if *any* of them end up introducing a regression the inclusion of
> HDFS-8791 means that folks will have cluster downtime in order to back
> things out. If that happens to any substantial number of downstream
> folks, or any particularly vocal downstream folks, then it is very
> likely we'll lose the remaining trust of operators for rolling out
> maintenance releases. That's a pretty steep cost.
>
> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> unreasonable burden.
>
> I agree that this fix is important, I just think we should either cut
> a version of 2.8 that includes it or find a way to do it that gives an
> operational path for rolling downgrade.
>
> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> > Thanks for bringing up this topic, Sean.
> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
> haven't been committed in so that's why we didn't discuss this earlier.
> > I remember in JIRA discussion, we treated this layout change as a
> Blocker bug that fixing a significant performance regression before but not
> a normal performance improvement. And I believe HDFS community already did
> their best with careful and patient to deliver the fix and other related
> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
> can see 30+ rounds patch review back and forth by senior committers, not to
> mention the outstanding performance test data in HDFS-8791.
> > I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3.
> However, that needs Vinod's final confirmation who serves as RM for
> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> 2.6.5 now.
> > Just my 2 cents.
> >
> > Thanks,
> >
> > Junping
> >
> > ________________________________________
> > From: Sean Busbey <bu...@cloudera.com>
> > Sent: Thursday, March 31, 2016 2:57 PM
> > To: hdfs-dev@hadoop.apache.org
> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> > Subject: Re: 2.7.3 release plan
> >
> > A layout change in a maintenance release sounds very risky. I saw some
> > discussion on the JIRA about those risks, but the consensus seemed to
> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> > we did RMs per release rather than per branch? No one claiming to be a
> > release manager ever spoke up AFAICT.
> >
> > Should this change be included? Should it go into a special 2.8
> > release as mentioned in the ticket?
> >
> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > <aj...@oss.nttdata.co.jp> wrote:
> >> Thank you Vinod!
> >>
> >> FYI: 2.7.3 will be a bit special release.
> >>
> >> HDFS-8791 bumped up the datanode layout version,
> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> is impossible. We can rollback instead.
> >>
> >> https://issues.apache.org/jira/browse/HDFS-8791
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which
> >>> did go out mid February). Got a little busy since.
> >>>
> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> and *no*
> >>> features / improvements.
> >>>
> >>> I hope to cut an RC in a week - giving enough time for outstanding
> blocker
> >>> / critical issues. Will start moving out any tickets that are not
> blockers
> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> tickets
> >>> outstanding as of now.
> >>>
> >>> Thanks,
> >>> +Vinod
> >>>
> >>> [1] 2.7.3 release blockers:
> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>
> >>
> >
> >
> >
> > --
> > busbey
>
>
>
> --
> busbey
>

Re: 2.7.3 release plan

[Andrew Wang <an...@cloudera.com>]

As I expressed on HDFS-8791, I do not want to include this JIRA in a
maintenance release. I've only seen it crop up on a handful of our
customer's clusters, and large users like Twitter and Yahoo that seem to be
more affected are also the most able to patch this change in themselves.

Layout upgrades are quite disruptive, and I don't think it's worth breaking
upgrade and downgrade expectations when it doesn't affect the (in my
experience) vast majority of users.

Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
will let him elaborate.

Best,
Andrew

On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bu...@cloudera.com> wrote:

> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
> if *any* of them end up introducing a regression the inclusion of
> HDFS-8791 means that folks will have cluster downtime in order to back
> things out. If that happens to any substantial number of downstream
> folks, or any particularly vocal downstream folks, then it is very
> likely we'll lose the remaining trust of operators for rolling out
> maintenance releases. That's a pretty steep cost.
>
> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
> unreasonable burden.
>
> I agree that this fix is important, I just think we should either cut
> a version of 2.8 that includes it or find a way to do it that gives an
> operational path for rolling downgrade.
>
> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> > Thanks for bringing up this topic, Sean.
> > When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791
> haven't been committed in so that's why we didn't discuss this earlier.
> > I remember in JIRA discussion, we treated this layout change as a
> Blocker bug that fixing a significant performance regression before but not
> a normal performance improvement. And I believe HDFS community already did
> their best with careful and patient to deliver the fix and other related
> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you
> can see 30+ rounds patch review back and forth by senior committers, not to
> mention the outstanding performance test data in HDFS-8791.
> > I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3.
> However, that needs Vinod's final confirmation who serves as RM for
> branch-2.7. In addition, I didn't see any blocker issue to bring it into
> 2.6.5 now.
> > Just my 2 cents.
> >
> > Thanks,
> >
> > Junping
> >
> > ________________________________________
> > From: Sean Busbey <bu...@cloudera.com>
> > Sent: Thursday, March 31, 2016 2:57 PM
> > To: hdfs-dev@hadoop.apache.org
> > Cc: Hadoop Common; yarn-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> > Subject: Re: 2.7.3 release plan
> >
> > A layout change in a maintenance release sounds very risky. I saw some
> > discussion on the JIRA about those risks, but the consensus seemed to
> > be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> > we did RMs per release rather than per branch? No one claiming to be a
> > release manager ever spoke up AFAICT.
> >
> > Should this change be included? Should it go into a special 2.8
> > release as mentioned in the ticket?
> >
> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> > <aj...@oss.nttdata.co.jp> wrote:
> >> Thank you Vinod!
> >>
> >> FYI: 2.7.3 will be a bit special release.
> >>
> >> HDFS-8791 bumped up the datanode layout version,
> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> >> is impossible. We can rollback instead.
> >>
> >> https://issues.apache.org/jira/browse/HDFS-8791
> >>
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which
> >>> did go out mid February). Got a little busy since.
> >>>
> >>> Following up the 2.7.2 maintenance release, we should work towards a
> >>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes
> and *no*
> >>> features / improvements.
> >>>
> >>> I hope to cut an RC in a week - giving enough time for outstanding
> blocker
> >>> / critical issues. Will start moving out any tickets that are not
> blockers
> >>> and/or won’t fit the timeline - there are 3 blockers and 15 critical
> tickets
> >>> outstanding as of now.
> >>>
> >>> Thanks,
> >>> +Vinod
> >>>
> >>> [1] 2.7.3 release blockers:
> >>> https://issues.apache.org/jira/issues/?filter=12335343
> >>>
> >>
> >
> >
> >
> > --
> > busbey
>
>
>
> --
> busbey
>

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Sean Busbey <bu...@cloudera.com>
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> <aj...@oss.nttdata.co.jp> wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Sean Busbey <bu...@cloudera.com>
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> <aj...@oss.nttdata.co.jp> wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Sean Busbey <bu...@cloudera.com>
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> <aj...@oss.nttdata.co.jp> wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <jd...@hortonworks.com> wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Sean Busbey <bu...@cloudera.com>
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> <aj...@oss.nttdata.co.jp> wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: 2.7.3 release plan

[Junping Du <jd...@hortonworks.com>]

Thanks for bringing up this topic, Sean. 
When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
Just my 2 cents.

Thanks,

Junping

________________________________________
From: Sean Busbey <bu...@cloudera.com>
Sent: Thursday, March 31, 2016 2:57 PM
To: hdfs-dev@hadoop.apache.org
Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



--
busbey

Re: 2.7.3 release plan

[Junping Du <jd...@hortonworks.com>]

Thanks for bringing up this topic, Sean. 
When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
Just my 2 cents.

Thanks,

Junping

________________________________________
From: Sean Busbey <bu...@cloudera.com>
Sent: Thursday, March 31, 2016 2:57 PM
To: hdfs-dev@hadoop.apache.org
Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



--
busbey

Re: 2.7.3 release plan

[Junping Du <jd...@hortonworks.com>]

Thanks for bringing up this topic, Sean. 
When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
Just my 2 cents.

Thanks,

Junping

________________________________________
From: Sean Busbey <bu...@cloudera.com>
Sent: Thursday, March 31, 2016 2:57 PM
To: hdfs-dev@hadoop.apache.org
Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



--
busbey

Re: 2.7.3 release plan

[Junping Du <jd...@hortonworks.com>]

Thanks for bringing up this topic, Sean. 
When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 haven't been committed in so that's why we didn't discuss this earlier.
I remember in JIRA discussion, we treated this layout change as a Blocker bug that fixing a significant performance regression before but not a normal performance improvement. And I believe HDFS community already did their best with careful and patient to deliver the fix and other related patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ rounds patch review back and forth by senior committers, not to mention the outstanding performance test data in HDFS-8791.
I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. However, that needs Vinod's final confirmation who serves as RM for branch-2.7. In addition, I didn't see any blocker issue to bring it into 2.6.5 now.
Just my 2 cents.

Thanks,

Junping

________________________________________
From: Sean Busbey <bu...@cloudera.com>
Sent: Thursday, March 31, 2016 2:57 PM
To: hdfs-dev@hadoop.apache.org
Cc: Hadoop Common; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



--
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: 2.7.3 release plan

[payam rastogi <pa...@outlook.com>]

Please unsubscribe me from this group.

________________________________________
From: Akira AJISAKA <aj...@oss.nttdata.co.jp>
Sent: Thursday, March 31, 2016 12:15 PM
To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: 2.7.3 release plan

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey <bu...@cloudera.com>]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
<aj...@oss.nttdata.co.jp> wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: 2.7.3 release plan

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

The nexus issue persists to date. I have been constantly getting “Remote end closed due to a SSH handshake” issue. Tried to upload the artifacts in piece, but that was too slow, spanned across days.

Finally found a “ DretryFailedDeploymentCount=10” option to mvn-deploy.

Sending out the vote now.

Thanks
+Vinod

> On Jul 21, 2016, at 11:53 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Started on the RC two days ago, but running into a nexus issue - not able to push the jars up stream for two days - the remote end is dropping the connection consistently.
> 
> Trying to figure out workarounds.
> 
> Thanks
> +Vinod
> 
>> On Jul 13, 2016, at 4:09 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> 
>> HADOOP-12893 is done after 4 months, thanks to great work from a bunch of folks besides Xiao Chen.
>> 
>> Creating the RC now.
>> 
>> Thanks
>> +Vinod
>> 
>>> On Jun 14, 2016, at 7:28 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> Release branch 2.7.3 is created. I also updated branch-2.7 to point to 2.7.4-SNAPSHOT now.
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>>> On Jun 14, 2016, at 3:54 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> HADOOP-12893 is finally close to completion after >  3months thanks to efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.
>>>> 
>>>> I’m creating the release branch and kickstarting the release activities.
>>>> 
>>>> Thanks
>>>> +Vinod
>>>> 
>>>>> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>>> 
>>>>> I am just waiting on HADOOP-12893.
>>>>> 
>>>>> HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.
>>>>> 
>>>>> Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.
>>>>> 
>>>>> Thanks
>>>>> +Vinod
>>>>> 
>>>>>> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
>>>>>> 
>>>>>> Curious on the status of 2.7.3....
>>>>>> 
>>>>>> It seems that we still have two outstanding critical/blocker JIRAs:
>>>>>> 
>>>>>> 1. [image: Bug] HADOOP-12893
>>>>>> Verify LICENSE.txt and NOTICE.txt
>>>>>> <https://issues.apache.org/jira/browse/HADOOP-12893>
>>>>>> 2. [image: Sub-task] HADOOP-13154
>>>>>> <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>>>>>> printAmazonServiceException/printAmazonClientException appear copy & paste
>>>>>> of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
>>>>>> 
>>>>>> 
>>>>>> But 45-ish when we include Majors as well.
>>>>>> 
>>>>>> I know there are a number of critical issues with fixes that need to go out.
>>>>>> 
>>>>>> What is the plan?
>>>>>> 
>>>>>> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>>>>>>> wrote:
>>>>>> 
>>>>>>> Others and I committed a few, I pushed out a few.
>>>>>>> 
>>>>>>> Down to just three now!
>>>>>>> 
>>>>>>> +Vinod
>>>>>>> 
>>>>>>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Down to only 10 blocker / critical tickets (
>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>>>>>>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>>> (which did go out mid February). Got a little busy since.
>>>>>>>>> 
>>>>>>>>> Following up the 2.7.2 maintenance release, we should work towards a
>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>>>>>>> *no* features / improvements.
>>>>>>>>> 
>>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>>> blocker / critical issues. Will start moving out any tickets that are not
>>>>>>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>>>>>>> critical tickets outstanding as of now.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> +Vinod
>>>>>>>>> 
>>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343>
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Started on the RC two days ago, but running into a nexus issue - not able to push the jars up stream for two days - the remote end is dropping the connection consistently.

Trying to figure out workarounds.

Thanks
+Vinod

> On Jul 13, 2016, at 4:09 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> HADOOP-12893 is done after 4 months, thanks to great work from a bunch of folks besides Xiao Chen.
> 
> Creating the RC now.
> 
> Thanks
> +Vinod
> 
>> On Jun 14, 2016, at 7:28 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> 
>> Release branch 2.7.3 is created. I also updated branch-2.7 to point to 2.7.4-SNAPSHOT now.
>> 
>> Thanks
>> +Vinod
>> 
>>> On Jun 14, 2016, at 3:54 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> HADOOP-12893 is finally close to completion after >  3months thanks to efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.
>>> 
>>> I’m creating the release branch and kickstarting the release activities.
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>>> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>> 
>>>> I am just waiting on HADOOP-12893.
>>>> 
>>>> HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.
>>>> 
>>>> Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.
>>>> 
>>>> Thanks
>>>> +Vinod
>>>> 
>>>>> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
>>>>> 
>>>>> Curious on the status of 2.7.3....
>>>>> 
>>>>> It seems that we still have two outstanding critical/blocker JIRAs:
>>>>> 
>>>>> 1. [image: Bug] HADOOP-12893
>>>>> Verify LICENSE.txt and NOTICE.txt
>>>>> <https://issues.apache.org/jira/browse/HADOOP-12893>
>>>>> 2. [image: Sub-task] HADOOP-13154
>>>>> <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>>>>> printAmazonServiceException/printAmazonClientException appear copy & paste
>>>>> of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
>>>>> 
>>>>> 
>>>>> But 45-ish when we include Majors as well.
>>>>> 
>>>>> I know there are a number of critical issues with fixes that need to go out.
>>>>> 
>>>>> What is the plan?
>>>>> 
>>>>> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>>>>>> wrote:
>>>>> 
>>>>>> Others and I committed a few, I pushed out a few.
>>>>>> 
>>>>>> Down to just three now!
>>>>>> 
>>>>>> +Vinod
>>>>>> 
>>>>>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Down to only 10 blocker / critical tickets (
>>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>>>>>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>>> (which did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work towards a
>>>>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>>>>>> *no* features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>>> blocker / critical issues. Will start moving out any tickets that are not
>>>>>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>>>>>> critical tickets outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>>> https://issues.apache.org/jira/issues/?filter=12335343>
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>> 
>>>> 
>>> 
>> 
>> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

HADOOP-12893 is done after 4 months, thanks to great work from a bunch of folks besides Xiao Chen.

Creating the RC now.

Thanks
+Vinod

> On Jun 14, 2016, at 7:28 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Release branch 2.7.3 is created. I also updated branch-2.7 to point to 2.7.4-SNAPSHOT now.
> 
> Thanks
> +Vinod
> 
>> On Jun 14, 2016, at 3:54 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> 
>> HADOOP-12893 is finally close to completion after >  3months thanks to efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.
>> 
>> I’m creating the release branch and kickstarting the release activities.
>> 
>> Thanks
>> +Vinod
>> 
>>> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>> 
>>> I am just waiting on HADOOP-12893.
>>> 
>>> HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.
>>> 
>>> Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>>> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
>>>> 
>>>> Curious on the status of 2.7.3....
>>>> 
>>>> It seems that we still have two outstanding critical/blocker JIRAs:
>>>> 
>>>> 1. [image: Bug] HADOOP-12893
>>>> Verify LICENSE.txt and NOTICE.txt
>>>> <https://issues.apache.org/jira/browse/HADOOP-12893>
>>>> 2. [image: Sub-task] HADOOP-13154
>>>> <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>>>> printAmazonServiceException/printAmazonClientException appear copy & paste
>>>> of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
>>>> 
>>>> 
>>>> But 45-ish when we include Majors as well.
>>>> 
>>>> I know there are a number of critical issues with fixes that need to go out.
>>>> 
>>>> What is the plan?
>>>> 
>>>> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>>>>> wrote:
>>>> 
>>>>> Others and I committed a few, I pushed out a few.
>>>>> 
>>>>> Down to just three now!
>>>>> 
>>>>> +Vinod
>>>>> 
>>>>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>>>>> wrote:
>>>>>> 
>>>>>> Down to only 10 blocker / critical tickets (
>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>>>>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which did go out mid February). Got a little busy since.
>>>>>>> 
>>>>>>> Following up the 2.7.2 maintenance release, we should work towards a
>>>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>>>>> *no* features / improvements.
>>>>>>> 
>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker / critical issues. Will start moving out any tickets that are not
>>>>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>>>>> critical tickets outstanding as of now.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> +Vinod
>>>>>>> 
>>>>>>> [1] 2.7.3 release blockers:
>>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>>> https://issues.apache.org/jira/issues/?filter=12335343>
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> 
>>> 
>> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Release branch 2.7.3 is created. I also updated branch-2.7 to point to 2.7.4-SNAPSHOT now.

Thanks
+Vinod

> On Jun 14, 2016, at 3:54 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> HADOOP-12893 is finally close to completion after >  3months thanks to efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.
> 
> I’m creating the release branch and kickstarting the release activities.
> 
> Thanks
> +Vinod
> 
>> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>> 
>> I am just waiting on HADOOP-12893.
>> 
>> HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.
>> 
>> Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.
>> 
>> Thanks
>> +Vinod
>> 
>>> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
>>> 
>>> Curious on the status of 2.7.3....
>>> 
>>> It seems that we still have two outstanding critical/blocker JIRAs:
>>> 
>>> 1. [image: Bug] HADOOP-12893
>>> Verify LICENSE.txt and NOTICE.txt
>>> <https://issues.apache.org/jira/browse/HADOOP-12893>
>>> 2. [image: Sub-task] HADOOP-13154
>>> <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>>> printAmazonServiceException/printAmazonClientException appear copy & paste
>>> of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
>>> 
>>> 
>>> But 45-ish when we include Majors as well.
>>> 
>>> I know there are a number of critical issues with fixes that need to go out.
>>> 
>>> What is the plan?
>>> 
>>> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>>>> wrote:
>>> 
>>>> Others and I committed a few, I pushed out a few.
>>>> 
>>>> Down to just three now!
>>>> 
>>>> +Vinod
>>>> 
>>>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>>>> wrote:
>>>>> 
>>>>> Down to only 10 blocker / critical tickets (
>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>>>> 
>>>>> Thanks
>>>>> +Vinod
>>>>> 
>>>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>>>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>> (which did go out mid February). Got a little busy since.
>>>>>> 
>>>>>> Following up the 2.7.2 maintenance release, we should work towards a
>>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>>>> *no* features / improvements.
>>>>>> 
>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>> blocker / critical issues. Will start moving out any tickets that are not
>>>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>>>> critical tickets outstanding as of now.
>>>>>> 
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>> 
>>>>>> [1] 2.7.3 release blockers:
>>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>>> https://issues.apache.org/jira/issues/?filter=12335343>
>>>>> 
>>>> 
>>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>> 
>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

HADOOP-12893 is finally close to completion after >  3months thanks to efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.

I’m creating the release branch and kickstarting the release activities.

Thanks
+Vinod

> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> I am just waiting on HADOOP-12893.
> 
> HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.
> 
> Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.
> 
> Thanks
> +Vinod
> 
>> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
>> 
>> Curious on the status of 2.7.3....
>> 
>> It seems that we still have two outstanding critical/blocker JIRAs:
>> 
>>  1. [image: Bug] HADOOP-12893
>>  Verify LICENSE.txt and NOTICE.txt
>>  <https://issues.apache.org/jira/browse/HADOOP-12893>
>>  2. [image: Sub-task] HADOOP-13154
>>  <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>>  printAmazonServiceException/printAmazonClientException appear copy & paste
>>  of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
>> 
>> 
>> But 45-ish when we include Majors as well.
>> 
>> I know there are a number of critical issues with fixes that need to go out.
>> 
>> What is the plan?
>> 
>> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>>> wrote:
>> 
>>> Others and I committed a few, I pushed out a few.
>>> 
>>> Down to just three now!
>>> 
>>> +Vinod
>>> 
>>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>>> wrote:
>>>> 
>>>> Down to only 10 blocker / critical tickets (
>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>>> 
>>>> Thanks
>>>> +Vinod
>>>> 
>>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>> (which did go out mid February). Got a little busy since.
>>>>> 
>>>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>>> *no* features / improvements.
>>>>> 
>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>> blocker / critical issues. Will start moving out any tickets that are not
>>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>>> critical tickets outstanding as of now.
>>>>> 
>>>>> Thanks,
>>>>> +Vinod
>>>>> 
>>>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343 <
>>> https://issues.apache.org/jira/issues/?filter=12335343>
>>>> 
>>> 
>>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[larry mccay <lm...@apache.org>]

Great!
Let me know if I can help in any way.

On Mon, May 16, 2016 at 8:39 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> I am just waiting on HADOOP-12893.
>
> HADOOP-13154 just got created in the last one day, will have to see if it
> really should block the release.
>
> Major tickets are usually taken on a time basis: if they get in by the
> proposed timelines, we get them in. Otherwise, we move them over.
>
> Thanks
> +Vinod
>
> > On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
> >
> > Curious on the status of 2.7.3....
> >
> > It seems that we still have two outstanding critical/blocker JIRAs:
> >
> >   1. [image: Bug] HADOOP-12893
> >   Verify LICENSE.txt and NOTICE.txt
> >   <https://issues.apache.org/jira/browse/HADOOP-12893>
> >   2. [image: Sub-task] HADOOP-13154
> >   <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
> >   printAmazonServiceException/printAmazonClientException appear copy &
> paste
> >   of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
> >
> >
> > But 45-ish when we include Majors as well.
> >
> > I know there are a number of critical issues with fixes that need to go
> out.
> >
> > What is the plan?
> >
> > On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> >> wrote:
> >
> >> Others and I committed a few, I pushed out a few.
> >>
> >> Down to just three now!
> >>
> >> +Vinod
> >>
> >>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> >> wrote:
> >>>
> >>> Down to only 10 blocker / critical tickets (
> >> https://issues.apache.org/jira/issues/?filter=12335343 <
> >> https://issues.apache.org/jira/issues/?filter=12335343>) now!
> >>>
> >>> Thanks
> >>> +Vinod
> >>>
> >>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
> >> vinodkv@apache.org <ma...@apache.org>> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> >> (which did go out mid February). Got a little busy since.
> >>>>
> >>>> Following up the 2.7.2 maintenance release, we should work towards a
> >> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
> >> *no* features / improvements.
> >>>>
> >>>> I hope to cut an RC in a week - giving enough time for outstanding
> >> blocker / critical issues. Will start moving out any tickets that are
> not
> >> blockers and/or won’t fit the timeline - there are 3 blockers and 15
> >> critical tickets outstanding as of now.
> >>>>
> >>>> Thanks,
> >>>> +Vinod
> >>>>
> >>>> [1] 2.7.3 release blockers:
> >> https://issues.apache.org/jira/issues/?filter=12335343 <
> >> https://issues.apache.org/jira/issues/?filter=12335343>
> >>>
> >>
> >>
>
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

I am just waiting on HADOOP-12893.

HADOOP-13154 just got created in the last one day, will have to see if it really should block the release.

Major tickets are usually taken on a time basis: if they get in by the proposed timelines, we get them in. Otherwise, we move them over.

Thanks
+Vinod

> On May 16, 2016, at 5:20 PM, larry mccay <lm...@apache.org> wrote:
> 
> Curious on the status of 2.7.3....
> 
> It seems that we still have two outstanding critical/blocker JIRAs:
> 
>   1. [image: Bug] HADOOP-12893
>   Verify LICENSE.txt and NOTICE.txt
>   <https://issues.apache.org/jira/browse/HADOOP-12893>
>   2. [image: Sub-task] HADOOP-13154
>   <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
>   printAmazonServiceException/printAmazonClientException appear copy & paste
>   of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>
> 
> 
> But 45-ish when we include Majors as well.
> 
> I know there are a number of critical issues with fixes that need to go out.
> 
> What is the plan?
> 
> On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
>> wrote:
> 
>> Others and I committed a few, I pushed out a few.
>> 
>> Down to just three now!
>> 
>> +Vinod
>> 
>>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
>> wrote:
>>> 
>>> Down to only 10 blocker / critical tickets (
>> https://issues.apache.org/jira/issues/?filter=12335343 <
>> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
>> vinodkv@apache.org <ma...@apache.org>> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> (which did go out mid February). Got a little busy since.
>>>> 
>>>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
>> *no* features / improvements.
>>>> 
>>>> I hope to cut an RC in a week - giving enough time for outstanding
>> blocker / critical issues. Will start moving out any tickets that are not
>> blockers and/or won’t fit the timeline - there are 3 blockers and 15
>> critical tickets outstanding as of now.
>>>> 
>>>> Thanks,
>>>> +Vinod
>>>> 
>>>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343 <
>> https://issues.apache.org/jira/issues/?filter=12335343>
>>> 
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: 2.7.3 release plan

[larry mccay <lm...@apache.org>]

Curious on the status of 2.7.3....

It seems that we still have two outstanding critical/blocker JIRAs:

   1. [image: Bug] HADOOP-12893
   Verify LICENSE.txt and NOTICE.txt
   <https://issues.apache.org/jira/browse/HADOOP-12893>
   2. [image: Sub-task] HADOOP-13154
   <https://issues.apache.org/jira/browse/HADOOP-13154>S3AFileSystem
   printAmazonServiceException/printAmazonClientException appear copy & paste
   of AWS examples <https://issues.apache.org/jira/browse/HADOOP-13154>


But 45-ish when we include Majors as well.

I know there are a number of critical issues with fixes that need to go out.

What is the plan?

On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> Others and I committed a few, I pushed out a few.
>
> Down to just three now!
>
> +Vinod
>
> > On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
> >
> > Down to only 10 blocker / critical tickets (
> https://issues.apache.org/jira/issues/?filter=12335343 <
> https://issues.apache.org/jira/issues/?filter=12335343>) now!
> >
> > Thanks
> > +Vinod
> >
> >> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org <ma...@apache.org>> wrote:
> >>
> >> Hi all,
> >>
> >> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which did go out mid February). Got a little busy since.
> >>
> >> Following up the 2.7.2 maintenance release, we should work towards a
> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
> *no* features / improvements.
> >>
> >> I hope to cut an RC in a week - giving enough time for outstanding
> blocker / critical issues. Will start moving out any tickets that are not
> blockers and/or won’t fit the timeline - there are 3 blockers and 15
> critical tickets outstanding as of now.
> >>
> >> Thanks,
> >> +Vinod
> >>
> >> [1] 2.7.3 release blockers:
> https://issues.apache.org/jira/issues/?filter=12335343 <
> https://issues.apache.org/jira/issues/?filter=12335343>
> >
>
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Others and I committed a few, I pushed out a few.

Down to just three now!

+Vinod

> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!
> 
> Thanks
> +Vinod
> 
>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>> 
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>> 
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>> 
>> Thanks,
>> +Vinod
>> 
>> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>
> 

Re: 2.7.3 release plan

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a better version, it did caught an upgrade bug while in branch-2.7.
Kihwal


      From: Vinod Kumar Vavilapalli <vi...@apache.org>
 To: yarn-dev@hadoop.apache.org 
Cc: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
 Sent: Wednesday, April 6, 2016 5:00 PM
 Subject: Re: 2.7.3 release plan
   
Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343


  

Re: 2.7.3 release plan

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a better version, it did caught an upgrade bug while in branch-2.7.
Kihwal


      From: Vinod Kumar Vavilapalli <vi...@apache.org>
 To: yarn-dev@hadoop.apache.org 
Cc: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
 Sent: Wednesday, April 6, 2016 5:00 PM
 Subject: Re: 2.7.3 release plan
   
Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343


  

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Others and I committed a few, I pushed out a few.

Down to just three now!

+Vinod

> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!
> 
> Thanks
> +Vinod
> 
>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>> 
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>> 
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>> 
>> Thanks,
>> +Vinod
>> 
>> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Others and I committed a few, I pushed out a few.

Down to just three now!

+Vinod

> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!
> 
> Thanks
> +Vinod
> 
>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>> 
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>> 
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>> 
>> Thanks,
>> +Vinod
>> 
>> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Others and I committed a few, I pushed out a few.

Down to just three now!

+Vinod

> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!
> 
> Thanks
> +Vinod
> 
>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>> 
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>> 
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>> 
>> Thanks,
>> +Vinod
>> 
>> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>
> 

Re: 2.7.3 release plan

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a better version, it did caught an upgrade bug while in branch-2.7.
Kihwal


      From: Vinod Kumar Vavilapalli <vi...@apache.org>
 To: yarn-dev@hadoop.apache.org 
Cc: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
 Sent: Wednesday, April 6, 2016 5:00 PM
 Subject: Re: 2.7.3 release plan
   
Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343


  

Re: 2.7.3 release plan

[Kihwal Lee <ki...@yahoo-inc.com.INVALID>]

Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a better version, it did caught an upgrade bug while in branch-2.7.
Kihwal


      From: Vinod Kumar Vavilapalli <vi...@apache.org>
 To: yarn-dev@hadoop.apache.org 
Cc: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
 Sent: Wednesday, April 6, 2016 5:00 PM
 Subject: Re: 2.7.3 release plan
   
Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343


  

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343

Re: 2.7.3 release plan

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343

Re: 2.7.3 release plan

[Akira AJISAKA <aj...@oss.nttdata.co.jp>]

Thank you Vinod!

FYI: 2.7.3 will be a bit special release.

HDFS-8791 bumped up the datanode layout version,
so rolling downgrade from 2.7.3 to 2.7.[0-2]
is impossible. We can rollback instead.

https://issues.apache.org/jira/browse/HDFS-8791
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html

Regards,
Akira

On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
> Hi all,
>
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
>
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
>
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
>
> Thanks,
> +Vinod
>
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343
>

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Down to only 10 blocker / critical tickets (https://issues.apache.org/jira/issues/?filter=12335343 <https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no* features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / critical issues. Will start moving out any tickets that are not blockers and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: https://issues.apache.org/jira/issues/?filter=12335343