Re: Streamlining the Hadoop release process

[Konstantin Boudnik <co...@apache.org>]

> 4000 node cluster. I don't know if BigTop can help with that, but any type
> of stress testing is desirable.  The other HDFS features like snapshots

An interesting data point: Roman just made some rough estimation of the size
of the execution that we are running as a part of usual BigTop testing.
There's about 400 different apps that are testing pretty much all levels of
Hadoop and downstream stack. 

I'd say this is most extensive test coverage of the Hadoop stack to-date. Anywhere.
I am pretty sure Bigtop can help somewhat: that's always a question of people
willing to contribute decent and valuable implementations of test scenarios.

Cos


On Tue, Apr 30, 2013 at 05:39PM, Robert Evans wrote:
> We are running nightly tests on branch-2 and branch-0.23.  These include
> unit and integration tests and branch-2 seems fairly good.  We would like
> to move off of 0.23 to 2.0.  But, there are relatively few things in 2.0
> that really push us in that direction over what we already have in 0.23.
> The major ones are HDFS HA and RM recovery as steps towards rolling
> upgrades.  For 2.0.5 to be beta for me really requires that the APIs are
> locked down, for both wire and binary compatibility.  This fits with what
> Arun has stated as his intentions and also with the compatibility
> discussion that has been going on. All of this is great but also requires
> a strong commitment from the entire community to really make this happen.
> I wish I could have participated more in those discussions but with
> everything else that has been on my plate I have not even had enough time
> to read them in a timely fashion.  For us to adopt it widely we need 2.0.X
> to be stable at scale.  We have worked out most of the issues on YARN,
> there may be a few more because of the deltas between 0.23 and 2.0 but
> that should be relatively small.  The big question is with HDFS.  The base
> HA work seems to be fairly stable with the fuzz testing and everything
> else that has happened, but I just don't know how it is going to handle a
> 4000 node cluster. I don't know if BigTop can help with that, but any type
> of stress testing is desirable.  The other HDFS features like snapshots
> just seem to add more risk for something that we do not see using in the
> short term.  Not that it is a bad feature, we just don't need it yet.
> 
> --Bobby
> 
> On 4/29/13 11:34 PM, "Roman Shaposhnik" <rv...@apache.org> wrote:
> 
> >On Mon, Apr 29, 2013 at 11:53 AM, Robert Evans <ev...@yahoo-inc.com>
> >wrote:
> >> This depends on how quickly Yahoo will be off of branch-0.23 and if
> >>Yahoo
> >> is the only one maintaining it then it will not receive any bug fixes.
> >> Does that make branch-0.23 any less stable, no.  Does it make it less
> >> desirable for people to move to? Yes.
> >
> >Do you think there's anything that we can do as a community to help with
> >that?
> >Or on a flip side -- do you think there's any chance for Yahoo! team
> >to formalize
> >the kind of workflows that the company cares about in Bigtop integration
> >or
> >component unit tests?
> >
> >Personally, I'm mostly nervous about the fact that I'm yet to hear any of
> >the
> >DevOps from major Hadoop using companies to chime in on the 2.0.5-beta
> >discussion with their take on what 'beta' means to them.
> >
> >Thanks,
> >Roman.
>