You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Konstantin Boudnik <co...@apache.org> on 2013/03/06 19:10:12 UTC

[DISCUSS] Setting up long term support mindset

I want to throw in the following discussion topic.

We are clearly having integration issues when putting together
bleeding edge Hadoop based stacks. I don't want to argue about the
roots of the phenomena not about how it can be fixed. The purpose of
this discussion is different. 

There is an ongoing discussion happening at
  http://s.apache.org/hadoop-stablization

and http://is.gd/HX5JA2

I think this is the time for community to start doing something and
improving the situation, helping our users in the passing. I am sure
the effort outlined below would beneficial for individuals as well as
commercial vendors alike.

Now, think Ubuntu - arguably the _most_ successful community driven
Linux distribution in the world. There are two trains of the releases
happening all the time: STS and LTS. The former have about 6 months of
support in the form of updates and is used as a technology preview for
the next LTS release. The latter is good for 3 years and normally is a
way more stable then STS releases.

Currently, BigTop is focusing on the latest (and apparently not
greatest) of its component releases. While sometimes it gives certain
guarantees that A of X.y version will work with B of W.v version, it
isn't much of reconciliation to the users and vendors who simply want
to have a stable reference implementation of a Hadoop stack that has
been defined and supported by the community, with clear path for the
updates and testing put out in the open.

Because of the nature of the differences between Hadoop 1 and Hadoop 2
we can even try to run two different LTS for both kinds of stack.

I believe there's not much more to say about it, except that this is,
in my opinion, a good way to establish our project as de-facto go-to
place for community driven Hadoop based stacks and a focal point for
the integration in the ASF storage and analytics projects.

Let's the discussion begin.
  Cos


Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Mar 07, 2013 at 07:25AM, Arun C Murthy wrote:
> As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will
> help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys
> be willing to help?

As has been agreed on the same thread, Bigtop is indeed willing to help with
the effort. This thread is slightly different: we are trying to find a model
that will help to bring users to Hadoop2 line. Right now, none of the
production HBase deployment I am aware of using Hadoop2. That's a telling
story. I want to see this project as a place where people can come and working
stacks guaranteed. With support from the community.

So, let's not go into the discussion about specific bugs on this thread,
please.

Cos

Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Mar 07, 2013 at 07:25AM, Arun C Murthy wrote:
> As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will
> help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys
> be willing to help?

As has been agreed on the same thread, Bigtop is indeed willing to help with
the effort. This thread is slightly different: we are trying to find a model
that will help to bring users to Hadoop2 line. Right now, none of the
production HBase deployment I am aware of using Hadoop2. That's a telling
story. I want to see this project as a place where people can come and working
stacks guaranteed. With support from the community.

So, let's not go into the discussion about specific bugs on this thread,
please.

Cos

Re: [DISCUSS] Setting up long term support mindset

Posted by Arun C Murthy <ac...@hortonworks.com>.
On Mar 6, 2013, at 1:27 PM, Roman Shaposhnik wrote:

>  #2 Do we have a reasonable conviction that the beta
>       release of Hadoop 2.X is withing reach?

FTR: A stated goal is to get to a hadoop-2.0.5-beta within the next couple of months. I, and my team, have been spending an inordinate amount of time vetting and solidifying YARN APIs and protocols for long-term sustenance:
https://issues.apache.org/jira/browse/YARN-386

The same effort is undergoing for HDFS (wire) protocols.

To be clear: do not confuse API work with stability of YARN itself. YARN, the system, and MapReduce atop, are very stable and has very, very large-scale deployments already. The above effort is an endeavor to ensure long-term maintainability of the APIs.

>  #3 How do we influence Hadoop community to help us
>       produce the first ever LTS of Bigtop?

As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys be willing to help?

thanks,
Arun

Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
There's only one goal _here_: providing stable collection of data processing
software. Stabilization of Hadoop 2.0 is discussed elsewhere, so let's
not mix two together nor bring the discussion of Hadoop PMC' role to this
list.

Perhaps there's a misunderstanding of the intent of LTS: it is an attempt to
provide users with a less turbulent stack, that isn't a subject of unexpected
breakage, because "Hadoop 2.0 is young". I would describe it as "LST won't
update underlying Hadoop bits just because new Hadoop release is out, unless
there's a guarantee that the rest of the stack works with it".

It is an attempt to step away from fragile model of "cutting edge releases
based on the _latest and not so greatest_ component releases". While Hadoop
might have its sweet time maturing and BigTop would certainly be helping along
the way, there are need to be a stable community supported stack environment
for users to download/install/work with. The content of it is apparently up to
the community at large to decide.

Cos

On Sat, Mar 16, 2013 at 10:06AM, Eric Baldeschwieler wrote:
> There are two goals here & I'm not sure how they can be met with a single release.
> - Stabilizing Apache Hadoop 2.0
> - Providing a known stable collection of hadoop components for hadoop users.  
> 
> I support both goals, but I'm not sure how you'd base them on the same release...
> 
> Currently my recommendation for anyone looking for the most stable release
> of hadoop is to use the 1.0 line.  That may change in 2013, but that is not
> yet proven.  I encourage folks to use 2.0, but with eyes open that they may
> cut their knuckles.
> 
> Hadoop 2.0 is young.  That's why it is tagged alpha.  APIs are still
> changing for good reasons, which seems like more than a labeling problem to
> me.  I certainly support the goal of a LTS on 2, but until the Hadoop PMC
> declares Hadoop 2.0 GA, my read is that anyone declaring that they are going
> to be LTS on 2.0 is betting that they understand Hadoop's stability better
> than the Hadoop PMC.
> 
> But maybe I don't understand the intent of the LTS label?
> 
> E14
> 
> On Mar 7, 2013, at 10:57 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > Andrew,
> > I think the influence part is about how to make LTS accepted and not how to
> > make LTS happened, right?
> > 
> > With respect to Roman's points:
> > 
> > 1. Agree
> > 
> > 2. a lot of people are keep using Hadoop-1 because of the perception that
> > Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
> > hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
> > including a reasonably stable Hadoop. Beta is a good aim, I think
> > 
> > 3. can be solved by having an RM for a Hadoop release combined with an RM for
> > BigTop release, who can coordinate the cross-effort.
> > 
> > 4. The only way is to help these components be more comfortable with the base
> > of the stack. 2-3 above are addressing it, I believe.
> > 
> > 5. I tend to agree with Andrew - it might be quite hard to do.
> > 
> > Cos
> > 
> > On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> >> A lot of behavior among loosely coupled projects is emergent. So in that
> >> sense some of what bigtop can produce is defined by something beyond direct
> >> control. What you can do is try to influence the processes at work. So,
> >> outreach to each component project. Also, constructive pressure. Publicize
> >> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> >> components if there is an integration issue and an unresponsive community.
> >> This would apply constructive pressure on the unresponsive community
> >> commensurate with the influence of bigtop.
> >> 
> >> To grow the influence of bigtop, engaging vendors might be useful. For
> >> those pursuing an open strategy or open core strategy then commoditizing
> >> and amortizing the costs of baseline packaging and integration concerns
> >> makes sense. Everybody wins because more bandwidth is available to focus on
> >> differentiators. Such vendors typically employ committers for community
> >> based development. If some of these vendors can publicly get behind bigtop
> >> and invest in its credibility, then as an emergent consequence there will
> >> be more community engagement on integration issues under its umbrella.
> >> Everyone will win.
> >> 
> >> On some of the specific points, my humble opinion:
> >> 
> >> 2. Hadoop 2 is the only long term viable option even if some parts may be
> >> still unstable in terms of API.
> >> 
> >> 3. This seems a case of "build it and they will come". Iterate on a BOM
> >> that (mostly) works. Publicize it. For integration blockers, track the
> >> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> >> respective dev lists with gentle reminders, but not too frequently.
> >> 
> >> 4. See above.
> >> 
> >> 5. You can't.
> >> 
> >> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> >> 
> >>> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> >>> wrote:
> >>>> I believe there's not much more to say about it, except that this is,
> >>>> in my opinion, a good way to establish our project as de-facto go-to
> >>>> place for community driven Hadoop based stacks and a focal point for
> >>>> the integration in the ASF storage and analytics projects.
> >>> 
> >>> I like this idea very much! A couple of things that I'd love to hear other
> >>> chime in on:
> >>>  #1 I think it is too late to change the focus of Bigtop 0.6.0
> >>>  #2 Do we have a reasonable conviction that the beta
> >>>       release of Hadoop 2.X is withing reach?
> >>>  #3 How do we influence Hadoop community to help us
> >>>       produce the first ever LTS of Bigtop?
> >>>  #4 How do we get the downstream communities (pig, oozie, etc)
> >>>       on-board so that we can all work towards this common goal?
> >>>  #5 Suppose we do all the work in all of the downstream components,
> >>>       how can we at least make sure that there will be patch
> >>>       releases incorporating all the changes we've done?
> >>> 
> >>> Now, Bigtop (well, me personally at least ;-)) would be more than
> >>> willing to help on all of these with automation, testing, etc. But
> >>> we *have* to get all of the communities involved on-board with this.
> >>> 
> >>> Thanks,
> >>> Roman.
> >>> 
> >> 
> >> 
> >> -- 
> >> Best regards,
> >> 
> >>   - Andy
> >> 
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> 

Re: [DISCUSS] Setting up long term support mindset

Posted by Eric Baldeschwieler <er...@hortonworks.com>.
There are two goals here & I'm not sure how they can be met with a single release.
- Stabilizing Apache Hadoop 2.0
- Providing a known stable collection of hadoop components for hadoop users.  

I support both goals, but I'm not sure how you'd base them on the same release...

Currently my recommendation for anyone looking for the most stable release of hadoop is to use the 1.0 line.  That may change in 2013, but that is not yet proven.  I encourage folks to use 2.0, but with eyes open that they may cut their knuckles.

Hadoop 2.0 is young.  That's why it is tagged alpha.  APIs are still changing for good reasons, which seems like more than a labeling problem to me.  I certainly support the goal of a LTS on 2, but until the Hadoop PMC declares Hadoop 2.0 GA, my read is that anyone declaring that they are going to be LTS on 2.0 is betting that they understand Hadoop's stability better than the Hadoop PMC.

But maybe I don't understand the intent of the LTS label?

E14

On Mar 7, 2013, at 10:57 AM, Konstantin Boudnik <co...@apache.org> wrote:

> Andrew,
> I think the influence part is about how to make LTS accepted and not how to
> make LTS happened, right?
> 
> With respect to Roman's points:
> 
> 1. Agree
> 
> 2. a lot of people are keep using Hadoop-1 because of the perception that
> Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
> hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
> including a reasonably stable Hadoop. Beta is a good aim, I think
> 
> 3. can be solved by having an RM for a Hadoop release combined with an RM for
> BigTop release, who can coordinate the cross-effort.
> 
> 4. The only way is to help these components be more comfortable with the base
> of the stack. 2-3 above are addressing it, I believe.
> 
> 5. I tend to agree with Andrew - it might be quite hard to do.
> 
> Cos
> 
> On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
>> A lot of behavior among loosely coupled projects is emergent. So in that
>> sense some of what bigtop can produce is defined by something beyond direct
>> control. What you can do is try to influence the processes at work. So,
>> outreach to each component project. Also, constructive pressure. Publicize
>> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
>> components if there is an integration issue and an unresponsive community.
>> This would apply constructive pressure on the unresponsive community
>> commensurate with the influence of bigtop.
>> 
>> To grow the influence of bigtop, engaging vendors might be useful. For
>> those pursuing an open strategy or open core strategy then commoditizing
>> and amortizing the costs of baseline packaging and integration concerns
>> makes sense. Everybody wins because more bandwidth is available to focus on
>> differentiators. Such vendors typically employ committers for community
>> based development. If some of these vendors can publicly get behind bigtop
>> and invest in its credibility, then as an emergent consequence there will
>> be more community engagement on integration issues under its umbrella.
>> Everyone will win.
>> 
>> On some of the specific points, my humble opinion:
>> 
>> 2. Hadoop 2 is the only long term viable option even if some parts may be
>> still unstable in terms of API.
>> 
>> 3. This seems a case of "build it and they will come". Iterate on a BOM
>> that (mostly) works. Publicize it. For integration blockers, track the
>> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
>> respective dev lists with gentle reminders, but not too frequently.
>> 
>> 4. See above.
>> 
>> 5. You can't.
>> 
>> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
>> 
>>> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
>>> wrote:
>>>> I believe there's not much more to say about it, except that this is,
>>>> in my opinion, a good way to establish our project as de-facto go-to
>>>> place for community driven Hadoop based stacks and a focal point for
>>>> the integration in the ASF storage and analytics projects.
>>> 
>>> I like this idea very much! A couple of things that I'd love to hear other
>>> chime in on:
>>>  #1 I think it is too late to change the focus of Bigtop 0.6.0
>>>  #2 Do we have a reasonable conviction that the beta
>>>       release of Hadoop 2.X is withing reach?
>>>  #3 How do we influence Hadoop community to help us
>>>       produce the first ever LTS of Bigtop?
>>>  #4 How do we get the downstream communities (pig, oozie, etc)
>>>       on-board so that we can all work towards this common goal?
>>>  #5 Suppose we do all the work in all of the downstream components,
>>>       how can we at least make sure that there will be patch
>>>       releases incorporating all the changes we've done?
>>> 
>>> Now, Bigtop (well, me personally at least ;-)) would be more than
>>> willing to help on all of these with automation, testing, etc. But
>>> we *have* to get all of the communities involved on-board with this.
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> -- 
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)


Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
Andrew,
I think the influence part is about how to make LTS accepted and not how to
make LTS happened, right?

With respect to Roman's points:

1. Agree

2. a lot of people are keep using Hadoop-1 because of the perception that
Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
including a reasonably stable Hadoop. Beta is a good aim, I think

3. can be solved by having an RM for a Hadoop release combined with an RM for
BigTop release, who can coordinate the cross-effort.

4. The only way is to help these components be more comfortable with the base
of the stack. 2-3 above are addressing it, I believe.

5. I tend to agree with Andrew - it might be quite hard to do.

Cos

On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> A lot of behavior among loosely coupled projects is emergent. So in that
> sense some of what bigtop can produce is defined by something beyond direct
> control. What you can do is try to influence the processes at work. So,
> outreach to each component project. Also, constructive pressure. Publicize
> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> components if there is an integration issue and an unresponsive community.
> This would apply constructive pressure on the unresponsive community
> commensurate with the influence of bigtop.
> 
> To grow the influence of bigtop, engaging vendors might be useful. For
> those pursuing an open strategy or open core strategy then commoditizing
> and amortizing the costs of baseline packaging and integration concerns
> makes sense. Everybody wins because more bandwidth is available to focus on
> differentiators. Such vendors typically employ committers for community
> based development. If some of these vendors can publicly get behind bigtop
> and invest in its credibility, then as an emergent consequence there will
> be more community engagement on integration issues under its umbrella.
> Everyone will win.
> 
> On some of the specific points, my humble opinion:
> 
> 2. Hadoop 2 is the only long term viable option even if some parts may be
> still unstable in terms of API.
> 
> 3. This seems a case of "build it and they will come". Iterate on a BOM
> that (mostly) works. Publicize it. For integration blockers, track the
> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> respective dev lists with gentle reminders, but not too frequently.
> 
> 4. See above.
> 
> 5. You can't.
> 
> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> 
> > On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> > wrote:
> > > I believe there's not much more to say about it, except that this is,
> > > in my opinion, a good way to establish our project as de-facto go-to
> > > place for community driven Hadoop based stacks and a focal point for
> > > the integration in the ASF storage and analytics projects.
> >
> > I like this idea very much! A couple of things that I'd love to hear other
> > chime in on:
> >   #1 I think it is too late to change the focus of Bigtop 0.6.0
> >   #2 Do we have a reasonable conviction that the beta
> >        release of Hadoop 2.X is withing reach?
> >   #3 How do we influence Hadoop community to help us
> >        produce the first ever LTS of Bigtop?
> >   #4 How do we get the downstream communities (pig, oozie, etc)
> >        on-board so that we can all work towards this common goal?
> >   #5 Suppose we do all the work in all of the downstream components,
> >        how can we at least make sure that there will be patch
> >        releases incorporating all the changes we've done?
> >
> > Now, Bigtop (well, me personally at least ;-)) would be more than
> > willing to help on all of these with automation, testing, etc. But
> > we *have* to get all of the communities involved on-board with this.
> >
> > Thanks,
> > Roman.
> >
> 
> 
> -- 
> Best regards,
> 
>    - Andy
> 
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
Andrew,
I think the influence part is about how to make LTS accepted and not how to
make LTS happened, right?

With respect to Roman's points:

1. Agree

2. a lot of people are keep using Hadoop-1 because of the perception that
Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
including a reasonably stable Hadoop. Beta is a good aim, I think

3. can be solved by having an RM for a Hadoop release combined with an RM for
BigTop release, who can coordinate the cross-effort.

4. The only way is to help these components be more comfortable with the base
of the stack. 2-3 above are addressing it, I believe.

5. I tend to agree with Andrew - it might be quite hard to do.

Cos

On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> A lot of behavior among loosely coupled projects is emergent. So in that
> sense some of what bigtop can produce is defined by something beyond direct
> control. What you can do is try to influence the processes at work. So,
> outreach to each component project. Also, constructive pressure. Publicize
> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> components if there is an integration issue and an unresponsive community.
> This would apply constructive pressure on the unresponsive community
> commensurate with the influence of bigtop.
> 
> To grow the influence of bigtop, engaging vendors might be useful. For
> those pursuing an open strategy or open core strategy then commoditizing
> and amortizing the costs of baseline packaging and integration concerns
> makes sense. Everybody wins because more bandwidth is available to focus on
> differentiators. Such vendors typically employ committers for community
> based development. If some of these vendors can publicly get behind bigtop
> and invest in its credibility, then as an emergent consequence there will
> be more community engagement on integration issues under its umbrella.
> Everyone will win.
> 
> On some of the specific points, my humble opinion:
> 
> 2. Hadoop 2 is the only long term viable option even if some parts may be
> still unstable in terms of API.
> 
> 3. This seems a case of "build it and they will come". Iterate on a BOM
> that (mostly) works. Publicize it. For integration blockers, track the
> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> respective dev lists with gentle reminders, but not too frequently.
> 
> 4. See above.
> 
> 5. You can't.
> 
> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> 
> > On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> > wrote:
> > > I believe there's not much more to say about it, except that this is,
> > > in my opinion, a good way to establish our project as de-facto go-to
> > > place for community driven Hadoop based stacks and a focal point for
> > > the integration in the ASF storage and analytics projects.
> >
> > I like this idea very much! A couple of things that I'd love to hear other
> > chime in on:
> >   #1 I think it is too late to change the focus of Bigtop 0.6.0
> >   #2 Do we have a reasonable conviction that the beta
> >        release of Hadoop 2.X is withing reach?
> >   #3 How do we influence Hadoop community to help us
> >        produce the first ever LTS of Bigtop?
> >   #4 How do we get the downstream communities (pig, oozie, etc)
> >        on-board so that we can all work towards this common goal?
> >   #5 Suppose we do all the work in all of the downstream components,
> >        how can we at least make sure that there will be patch
> >        releases incorporating all the changes we've done?
> >
> > Now, Bigtop (well, me personally at least ;-)) would be more than
> > willing to help on all of these with automation, testing, etc. But
> > we *have* to get all of the communities involved on-board with this.
> >
> > Thanks,
> > Roman.
> >
> 
> 
> -- 
> Best regards,
> 
>    - Andy
> 
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: [DISCUSS] Setting up long term support mindset

Posted by Andrew Purtell <ap...@apache.org>.
A lot of behavior among loosely coupled projects is emergent. So in that
sense some of what bigtop can produce is defined by something beyond direct
control. What you can do is try to influence the processes at work. So,
outreach to each component project. Also, constructive pressure. Publicize
bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
components if there is an integration issue and an unresponsive community.
This would apply constructive pressure on the unresponsive community
commensurate with the influence of bigtop.

To grow the influence of bigtop, engaging vendors might be useful. For
those pursuing an open strategy or open core strategy then commoditizing
and amortizing the costs of baseline packaging and integration concerns
makes sense. Everybody wins because more bandwidth is available to focus on
differentiators. Such vendors typically employ committers for community
based development. If some of these vendors can publicly get behind bigtop
and invest in its credibility, then as an emergent consequence there will
be more community engagement on integration issues under its umbrella.
Everyone will win.

On some of the specific points, my humble opinion:

2. Hadoop 2 is the only long term viable option even if some parts may be
still unstable in terms of API.

3. This seems a case of "build it and they will come". Iterate on a BOM
that (mostly) works. Publicize it. For integration blockers, track the
respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
respective dev lists with gentle reminders, but not too frequently.

4. See above.

5. You can't.

On Thursday, March 7, 2013, Roman Shaposhnik wrote:

> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> wrote:
> > I believe there's not much more to say about it, except that this is,
> > in my opinion, a good way to establish our project as de-facto go-to
> > place for community driven Hadoop based stacks and a focal point for
> > the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
>   #1 I think it is too late to change the focus of Bigtop 0.6.0
>   #2 Do we have a reasonable conviction that the beta
>        release of Hadoop 2.X is withing reach?
>   #3 How do we influence Hadoop community to help us
>        produce the first ever LTS of Bigtop?
>   #4 How do we get the downstream communities (pig, oozie, etc)
>        on-board so that we can all work towards this common goal?
>   #5 Suppose we do all the work in all of the downstream components,
>        how can we at least make sure that there will be patch
>        releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Setting up long term support mindset

Posted by Bruno Mahé <bm...@apache.org>.
On 03/06/2013 01:27 PM, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
>> I believe there's not much more to say about it, except that this is,
>> in my opinion, a good way to establish our project as de-facto go-to
>> place for community driven Hadoop based stacks and a focal point for
>> the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
>    #1 I think it is too late to change the focus of Bigtop 0.6.0
>    #2 Do we have a reasonable conviction that the beta
>         release of Hadoop 2.X is withing reach?
>    #3 How do we influence Hadoop community to help us
>         produce the first ever LTS of Bigtop?
>    #4 How do we get the downstream communities (pig, oozie, etc)
>         on-board so that we can all work towards this common goal?
>    #5 Suppose we do all the work in all of the downstream components,
>         how can we at least make sure that there will be patch
>         releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>

Hi,

It seems like there are at least two different topics meshed together:
1/ Offering a stable version of Apache Bigtop
2/ How can Apache Bigtop help raise the quality of downstream projects


Regarding 1/ I was under the impression this is what the branch 0.3.X is 
all about. There was some interest, but apparently not enough to get 
releases out. So I am all for it but I am not sure building the 
consensus for a LTS stamp will be enough if we don't get enough 
interest/people/time to have stable releases out in the first place.

As of 2/, isn't it just a matter to put in action what we talked about 
numerous time: having builds based of trunk/rc branches of downstream 
projects? But there again, this is very time consuming and we do not 
have that many volunteers/time. This would also not be an answer to all 
the issues, but that would cover quite a few.
We would not be able to guarantee that issues would be fixed, but these 
communities would have some incentive to fix them anyway. And if they 
don't fix it, nothing forces us to update these projects.

Having downstream communities on board would also be very nice, but I am 
not sure it would be a requirement.


Thanks,
Bruno

Re: [DISCUSS] Setting up long term support mindset

Posted by Bruno Mahé <bm...@apache.org>.
On 03/06/2013 01:27 PM, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
>> I believe there's not much more to say about it, except that this is,
>> in my opinion, a good way to establish our project as de-facto go-to
>> place for community driven Hadoop based stacks and a focal point for
>> the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
>    #1 I think it is too late to change the focus of Bigtop 0.6.0
>    #2 Do we have a reasonable conviction that the beta
>         release of Hadoop 2.X is withing reach?
>    #3 How do we influence Hadoop community to help us
>         produce the first ever LTS of Bigtop?
>    #4 How do we get the downstream communities (pig, oozie, etc)
>         on-board so that we can all work towards this common goal?
>    #5 Suppose we do all the work in all of the downstream components,
>         how can we at least make sure that there will be patch
>         releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>

Hi,

It seems like there are at least two different topics meshed together:
1/ Offering a stable version of Apache Bigtop
2/ How can Apache Bigtop help raise the quality of downstream projects


Regarding 1/ I was under the impression this is what the branch 0.3.X is 
all about. There was some interest, but apparently not enough to get 
releases out. So I am all for it but I am not sure building the 
consensus for a LTS stamp will be enough if we don't get enough 
interest/people/time to have stable releases out in the first place.

As of 2/, isn't it just a matter to put in action what we talked about 
numerous time: having builds based of trunk/rc branches of downstream 
projects? But there again, this is very time consuming and we do not 
have that many volunteers/time. This would also not be an answer to all 
the issues, but that would cover quite a few.
We would not be able to guarantee that issues would be fixed, but these 
communities would have some incentive to fix them anyway. And if they 
don't fix it, nothing forces us to update these projects.

Having downstream communities on board would also be very nice, but I am 
not sure it would be a requirement.


Thanks,
Bruno

Re: [DISCUSS] Setting up long term support mindset

Posted by Arun C Murthy <ac...@hortonworks.com>.
On Mar 6, 2013, at 1:27 PM, Roman Shaposhnik wrote:

>  #2 Do we have a reasonable conviction that the beta
>       release of Hadoop 2.X is withing reach?

FTR: A stated goal is to get to a hadoop-2.0.5-beta within the next couple of months. I, and my team, have been spending an inordinate amount of time vetting and solidifying YARN APIs and protocols for long-term sustenance:
https://issues.apache.org/jira/browse/YARN-386

The same effort is undergoing for HDFS (wire) protocols.

To be clear: do not confuse API work with stability of YARN itself. YARN, the system, and MapReduce atop, are very stable and has very, very large-scale deployments already. The above effort is an endeavor to ensure long-term maintainability of the APIs.

>  #3 How do we influence Hadoop community to help us
>       produce the first ever LTS of Bigtop?

As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys be willing to help?

thanks,
Arun

Re: [DISCUSS] Setting up long term support mindset

Posted by Andrew Purtell <ap...@apache.org>.
A lot of behavior among loosely coupled projects is emergent. So in that
sense some of what bigtop can produce is defined by something beyond direct
control. What you can do is try to influence the processes at work. So,
outreach to each component project. Also, constructive pressure. Publicize
bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
components if there is an integration issue and an unresponsive community.
This would apply constructive pressure on the unresponsive community
commensurate with the influence of bigtop.

To grow the influence of bigtop, engaging vendors might be useful. For
those pursuing an open strategy or open core strategy then commoditizing
and amortizing the costs of baseline packaging and integration concerns
makes sense. Everybody wins because more bandwidth is available to focus on
differentiators. Such vendors typically employ committers for community
based development. If some of these vendors can publicly get behind bigtop
and invest in its credibility, then as an emergent consequence there will
be more community engagement on integration issues under its umbrella.
Everyone will win.

On some of the specific points, my humble opinion:

2. Hadoop 2 is the only long term viable option even if some parts may be
still unstable in terms of API.

3. This seems a case of "build it and they will come". Iterate on a BOM
that (mostly) works. Publicize it. For integration blockers, track the
respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
respective dev lists with gentle reminders, but not too frequently.

4. See above.

5. You can't.

On Thursday, March 7, 2013, Roman Shaposhnik wrote:

> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> wrote:
> > I believe there's not much more to say about it, except that this is,
> > in my opinion, a good way to establish our project as de-facto go-to
> > place for community driven Hadoop based stacks and a focal point for
> > the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
>   #1 I think it is too late to change the focus of Bigtop 0.6.0
>   #2 Do we have a reasonable conviction that the beta
>        release of Hadoop 2.X is withing reach?
>   #3 How do we influence Hadoop community to help us
>        produce the first ever LTS of Bigtop?
>   #4 How do we get the downstream communities (pig, oozie, etc)
>        on-board so that we can all work towards this common goal?
>   #5 Suppose we do all the work in all of the downstream components,
>        how can we at least make sure that there will be patch
>        releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSS] Setting up long term support mindset

Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
> I believe there's not much more to say about it, except that this is,
> in my opinion, a good way to establish our project as de-facto go-to
> place for community driven Hadoop based stacks and a focal point for
> the integration in the ASF storage and analytics projects.

I like this idea very much! A couple of things that I'd love to hear other
chime in on:
  #1 I think it is too late to change the focus of Bigtop 0.6.0
  #2 Do we have a reasonable conviction that the beta
       release of Hadoop 2.X is withing reach?
  #3 How do we influence Hadoop community to help us
       produce the first ever LTS of Bigtop?
  #4 How do we get the downstream communities (pig, oozie, etc)
       on-board so that we can all work towards this common goal?
  #5 Suppose we do all the work in all of the downstream components,
       how can we at least make sure that there will be patch
       releases incorporating all the changes we've done?

Now, Bigtop (well, me personally at least ;-)) would be more than
willing to help on all of these with automation, testing, etc. But
we *have* to get all of the communities involved on-board with this.

Thanks,
Roman.

Re: [DISCUSS] Setting up long term support mindset

Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
> I believe there's not much more to say about it, except that this is,
> in my opinion, a good way to establish our project as de-facto go-to
> place for community driven Hadoop based stacks and a focal point for
> the integration in the ASF storage and analytics projects.

I like this idea very much! A couple of things that I'd love to hear other
chime in on:
  #1 I think it is too late to change the focus of Bigtop 0.6.0
  #2 Do we have a reasonable conviction that the beta
       release of Hadoop 2.X is withing reach?
  #3 How do we influence Hadoop community to help us
       produce the first ever LTS of Bigtop?
  #4 How do we get the downstream communities (pig, oozie, etc)
       on-board so that we can all work towards this common goal?
  #5 Suppose we do all the work in all of the downstream components,
       how can we at least make sure that there will be patch
       releases incorporating all the changes we've done?

Now, Bigtop (well, me personally at least ;-)) would be more than
willing to help on all of these with automation, testing, etc. But
we *have* to get all of the communities involved on-board with this.

Thanks,
Roman.

Re: [DISCUSS] Setting up long term support mindset

Posted by Roman Shaposhnik <rv...@apache.org>.
On Sun, Mar 17, 2013 at 2:30 PM, Konstantin Boudnik <co...@apache.org> wrote:
> The ideal outcome would be a sort of consensus on or a complete rejection of
> this whole LTS idea. As I said - I am not sure if it should be 0.3 line.

My reading of the thread seems to indicate that most of the Bigtop
folks support the concept of LTS. Of course, another huge issue is
whether downstream component think it would be worth their
time to help with a a coherent stack around Hadoop 2.X. We
don't really have much control over that (and as I pointed out elsewhere:
Hadoop 2.X history of alphas has created a bit of a barrier there for us).

I'm not quite sure how to address that, but I'd more than welcome
any proposal/thoughts.

> P.S. On a completely separate note: it would be good to finish 0.3.1 after all
> that long hibernation. I have re-started vote on 0.3.1 BOM and will update the
> jiras tomorrow to reflect it. As usual - everyone is very welcome to jump on
> the fix the tickets: there are shouldn't be many of them; but let's not turn
> this thread into 0.3.1 release discussion.

That sounds like a really good idea, provided that folks are willing to spend
cycles on it. Personally, I really think having one "last" release
around Hadoop 1.X
would be extremely nice, but I'm not sure how much I'd be able to help
with that effort :-(

Thanks,
Roman.

Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
The ideal outcome would be a sort of consensus on or a complete rejection of
this whole LTS idea. As I said - I am not sure if it should be 0.3 line.

Cos

P.S. On a completely separate note: it would be good to finish 0.3.1 after all
that long hibernation. I have re-started vote on 0.3.1 BOM and will update the
jiras tomorrow to reflect it. As usual - everyone is very welcome to jump on
the fix the tickets: there are shouldn't be many of them; but let's not turn
this thread into 0.3.1 release discussion.

On Sun, Mar 17, 2013 at 01:35PM, Bruno Mahé wrote:
> On 03/17/2013 11:22 AM, Konstantin Boudnik wrote:
> >Indeed. Nothing but an effort is preventing us from doing.
> >
> >While I am not convinced that LTS has to be based on Hadoop 1.x, considering
> >the growing interest in Hadoop 2-alpha line, keeping 0.3 alive is definitely
> >something worth doing.
> >
> >Cos
> >
> >On Sat, Mar 16, 2013 at 09:46PM, Bruno Mahé wrote:
> >>On 03/06/2013 10:10 AM, Konstantin Boudnik wrote:
> >>>I want to throw in the following discussion topic.
> >>>
> >>>We are clearly having integration issues when putting together
> >>>bleeding edge Hadoop based stacks. I don't want to argue about the
> >>>roots of the phenomena not about how it can be fixed. The purpose of
> >>>this discussion is different.
> >>>
> >>>There is an ongoing discussion happening at
> >>>   http://s.apache.org/hadoop-stablization
> >>>
> >>>and http://is.gd/HX5JA2
> >>>
> >>>I think this is the time for community to start doing something and
> >>>improving the situation, helping our users in the passing. I am sure
> >>>the effort outlined below would beneficial for individuals as well as
> >>>commercial vendors alike.
> >>>
> >>>Now, think Ubuntu - arguably the _most_ successful community driven
> >>>Linux distribution in the world. There are two trains of the releases
> >>>happening all the time: STS and LTS. The former have about 6 months of
> >>>support in the form of updates and is used as a technology preview for
> >>>the next LTS release. The latter is good for 3 years and normally is a
> >>>way more stable then STS releases.
> >>>
> >>>Currently, BigTop is focusing on the latest (and apparently not
> >>>greatest) of its component releases. While sometimes it gives certain
> >>>guarantees that A of X.y version will work with B of W.v version, it
> >>>isn't much of reconciliation to the users and vendors who simply want
> >>>to have a stable reference implementation of a Hadoop stack that has
> >>>been defined and supported by the community, with clear path for the
> >>>updates and testing put out in the open.
> >>>
> >>>Because of the nature of the differences between Hadoop 1 and Hadoop 2
> >>>we can even try to run two different LTS for both kinds of stack.
> >>>
> >>>I believe there's not much more to say about it, except that this is,
> >>>in my opinion, a good way to establish our project as de-facto go-to
> >>>place for community driven Hadoop based stacks and a focal point for
> >>>the integration in the ASF storage and analytics projects.
> >>>
> >>>Let's the discussion begin.
> >>>   Cos
> >>>
> >>
> >>
> >>Hi,
> >>
> >>What was preventing anyone from offering such stack until now?
> >>I was under the impression this is what the branch 0.3.X is all
> >>about. There was some interest, but apparently not enough to get
> >>releases out. So I am all for it but I am not sure building the
> >>consensus for a LTS stamp will be enough if we don't get enough
> >>interest/people/time to have stable releases out in the first place.
> >>
> >>Why not:
> >>* Backport applicable bugfixes from trunk into a stable branch of
> >>your choice (let say 0.3.X)
> >>* Update minor version (supposedly bugfix only) of the downstream
> >>components in the stable branch of your choice (let say 0.3.X)
> >>* Add more tests to bigtop if necessary
> >>* Run the tests
> >>* Start a vote on a release
> >>Rinse and repeat.
> >>
> >>Note that I agree with you, having some LTS releases would be great.
> >>But beside doing the work, I fail to see what is preventing us from
> >>doing so right now.
> >>
> >>Thanks,
> >>Bruno
> >>
> 
> 
> So what would the ideal outcome of this email be?
> Do you need more help in some areas?
> What can I do to help you crank the 0.3.X release train?
> 
> Thanks,
> Bruno

Re: [DISCUSS] Setting up long term support mindset

Posted by Bruno Mahé <bm...@apache.org>.
On 03/17/2013 11:22 AM, Konstantin Boudnik wrote:
> Indeed. Nothing but an effort is preventing us from doing.
>
> While I am not convinced that LTS has to be based on Hadoop 1.x, considering
> the growing interest in Hadoop 2-alpha line, keeping 0.3 alive is definitely
> something worth doing.
>
> Cos
>
> On Sat, Mar 16, 2013 at 09:46PM, Bruno Mahé wrote:
>> On 03/06/2013 10:10 AM, Konstantin Boudnik wrote:
>>> I want to throw in the following discussion topic.
>>>
>>> We are clearly having integration issues when putting together
>>> bleeding edge Hadoop based stacks. I don't want to argue about the
>>> roots of the phenomena not about how it can be fixed. The purpose of
>>> this discussion is different.
>>>
>>> There is an ongoing discussion happening at
>>>    http://s.apache.org/hadoop-stablization
>>>
>>> and http://is.gd/HX5JA2
>>>
>>> I think this is the time for community to start doing something and
>>> improving the situation, helping our users in the passing. I am sure
>>> the effort outlined below would beneficial for individuals as well as
>>> commercial vendors alike.
>>>
>>> Now, think Ubuntu - arguably the _most_ successful community driven
>>> Linux distribution in the world. There are two trains of the releases
>>> happening all the time: STS and LTS. The former have about 6 months of
>>> support in the form of updates and is used as a technology preview for
>>> the next LTS release. The latter is good for 3 years and normally is a
>>> way more stable then STS releases.
>>>
>>> Currently, BigTop is focusing on the latest (and apparently not
>>> greatest) of its component releases. While sometimes it gives certain
>>> guarantees that A of X.y version will work with B of W.v version, it
>>> isn't much of reconciliation to the users and vendors who simply want
>>> to have a stable reference implementation of a Hadoop stack that has
>>> been defined and supported by the community, with clear path for the
>>> updates and testing put out in the open.
>>>
>>> Because of the nature of the differences between Hadoop 1 and Hadoop 2
>>> we can even try to run two different LTS for both kinds of stack.
>>>
>>> I believe there's not much more to say about it, except that this is,
>>> in my opinion, a good way to establish our project as de-facto go-to
>>> place for community driven Hadoop based stacks and a focal point for
>>> the integration in the ASF storage and analytics projects.
>>>
>>> Let's the discussion begin.
>>>    Cos
>>>
>>
>>
>> Hi,
>>
>> What was preventing anyone from offering such stack until now?
>> I was under the impression this is what the branch 0.3.X is all
>> about. There was some interest, but apparently not enough to get
>> releases out. So I am all for it but I am not sure building the
>> consensus for a LTS stamp will be enough if we don't get enough
>> interest/people/time to have stable releases out in the first place.
>>
>> Why not:
>> * Backport applicable bugfixes from trunk into a stable branch of
>> your choice (let say 0.3.X)
>> * Update minor version (supposedly bugfix only) of the downstream
>> components in the stable branch of your choice (let say 0.3.X)
>> * Add more tests to bigtop if necessary
>> * Run the tests
>> * Start a vote on a release
>> Rinse and repeat.
>>
>> Note that I agree with you, having some LTS releases would be great.
>> But beside doing the work, I fail to see what is preventing us from
>> doing so right now.
>>
>> Thanks,
>> Bruno
>>


So what would the ideal outcome of this email be?
Do you need more help in some areas?
What can I do to help you crank the 0.3.X release train?

Thanks,
Bruno

Re: [DISCUSS] Setting up long term support mindset

Posted by Konstantin Boudnik <co...@apache.org>.
Indeed. Nothing but an effort is preventing us from doing. 

While I am not convinced that LTS has to be based on Hadoop 1.x, considering
the growing interest in Hadoop 2-alpha line, keeping 0.3 alive is definitely
something worth doing. 

Cos

On Sat, Mar 16, 2013 at 09:46PM, Bruno Mahé wrote:
> On 03/06/2013 10:10 AM, Konstantin Boudnik wrote:
> >I want to throw in the following discussion topic.
> >
> >We are clearly having integration issues when putting together
> >bleeding edge Hadoop based stacks. I don't want to argue about the
> >roots of the phenomena not about how it can be fixed. The purpose of
> >this discussion is different.
> >
> >There is an ongoing discussion happening at
> >   http://s.apache.org/hadoop-stablization
> >
> >and http://is.gd/HX5JA2
> >
> >I think this is the time for community to start doing something and
> >improving the situation, helping our users in the passing. I am sure
> >the effort outlined below would beneficial for individuals as well as
> >commercial vendors alike.
> >
> >Now, think Ubuntu - arguably the _most_ successful community driven
> >Linux distribution in the world. There are two trains of the releases
> >happening all the time: STS and LTS. The former have about 6 months of
> >support in the form of updates and is used as a technology preview for
> >the next LTS release. The latter is good for 3 years and normally is a
> >way more stable then STS releases.
> >
> >Currently, BigTop is focusing on the latest (and apparently not
> >greatest) of its component releases. While sometimes it gives certain
> >guarantees that A of X.y version will work with B of W.v version, it
> >isn't much of reconciliation to the users and vendors who simply want
> >to have a stable reference implementation of a Hadoop stack that has
> >been defined and supported by the community, with clear path for the
> >updates and testing put out in the open.
> >
> >Because of the nature of the differences between Hadoop 1 and Hadoop 2
> >we can even try to run two different LTS for both kinds of stack.
> >
> >I believe there's not much more to say about it, except that this is,
> >in my opinion, a good way to establish our project as de-facto go-to
> >place for community driven Hadoop based stacks and a focal point for
> >the integration in the ASF storage and analytics projects.
> >
> >Let's the discussion begin.
> >   Cos
> >
> 
> 
> Hi,
> 
> What was preventing anyone from offering such stack until now?
> I was under the impression this is what the branch 0.3.X is all
> about. There was some interest, but apparently not enough to get
> releases out. So I am all for it but I am not sure building the
> consensus for a LTS stamp will be enough if we don't get enough
> interest/people/time to have stable releases out in the first place.
> 
> Why not:
> * Backport applicable bugfixes from trunk into a stable branch of
> your choice (let say 0.3.X)
> * Update minor version (supposedly bugfix only) of the downstream
> components in the stable branch of your choice (let say 0.3.X)
> * Add more tests to bigtop if necessary
> * Run the tests
> * Start a vote on a release
> Rinse and repeat.
> 
> Note that I agree with you, having some LTS releases would be great.
> But beside doing the work, I fail to see what is preventing us from
> doing so right now.
> 
> Thanks,
> Bruno
> 

Re: [DISCUSS] Setting up long term support mindset

Posted by Bruno Mahé <bm...@apache.org>.
On 03/06/2013 10:10 AM, Konstantin Boudnik wrote:
> I want to throw in the following discussion topic.
>
> We are clearly having integration issues when putting together
> bleeding edge Hadoop based stacks. I don't want to argue about the
> roots of the phenomena not about how it can be fixed. The purpose of
> this discussion is different.
>
> There is an ongoing discussion happening at
>    http://s.apache.org/hadoop-stablization
>
> and http://is.gd/HX5JA2
>
> I think this is the time for community to start doing something and
> improving the situation, helping our users in the passing. I am sure
> the effort outlined below would beneficial for individuals as well as
> commercial vendors alike.
>
> Now, think Ubuntu - arguably the _most_ successful community driven
> Linux distribution in the world. There are two trains of the releases
> happening all the time: STS and LTS. The former have about 6 months of
> support in the form of updates and is used as a technology preview for
> the next LTS release. The latter is good for 3 years and normally is a
> way more stable then STS releases.
>
> Currently, BigTop is focusing on the latest (and apparently not
> greatest) of its component releases. While sometimes it gives certain
> guarantees that A of X.y version will work with B of W.v version, it
> isn't much of reconciliation to the users and vendors who simply want
> to have a stable reference implementation of a Hadoop stack that has
> been defined and supported by the community, with clear path for the
> updates and testing put out in the open.
>
> Because of the nature of the differences between Hadoop 1 and Hadoop 2
> we can even try to run two different LTS for both kinds of stack.
>
> I believe there's not much more to say about it, except that this is,
> in my opinion, a good way to establish our project as de-facto go-to
> place for community driven Hadoop based stacks and a focal point for
> the integration in the ASF storage and analytics projects.
>
> Let's the discussion begin.
>    Cos
>


Hi,

What was preventing anyone from offering such stack until now?
I was under the impression this is what the branch 0.3.X is all about. 
There was some interest, but apparently not enough to get releases out. 
So I am all for it but I am not sure building the consensus for a LTS 
stamp will be enough if we don't get enough interest/people/time to have 
stable releases out in the first place.

Why not:
* Backport applicable bugfixes from trunk into a stable branch of your 
choice (let say 0.3.X)
* Update minor version (supposedly bugfix only) of the downstream 
components in the stable branch of your choice (let say 0.3.X)
* Add more tests to bigtop if necessary
* Run the tests
* Start a vote on a release
Rinse and repeat.

Note that I agree with you, having some LTS releases would be great. But 
beside doing the work, I fail to see what is preventing us from doing so 
right now.

Thanks,
Bruno