You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Roman Shaposhnik <rv...@apache.org> on 2013/03/06 22:27:00 UTC
Re: [DISCUSS] Setting up long term support mindset
On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
> I believe there's not much more to say about it, except that this is,
> in my opinion, a good way to establish our project as de-facto go-to
> place for community driven Hadoop based stacks and a focal point for
> the integration in the ASF storage and analytics projects.
I like this idea very much! A couple of things that I'd love to hear other
chime in on:
#1 I think it is too late to change the focus of Bigtop 0.6.0
#2 Do we have a reasonable conviction that the beta
release of Hadoop 2.X is withing reach?
#3 How do we influence Hadoop community to help us
produce the first ever LTS of Bigtop?
#4 How do we get the downstream communities (pig, oozie, etc)
on-board so that we can all work towards this common goal?
#5 Suppose we do all the work in all of the downstream components,
how can we at least make sure that there will be patch
releases incorporating all the changes we've done?
Now, Bigtop (well, me personally at least ;-)) would be more than
willing to help on all of these with automation, testing, etc. But
we *have* to get all of the communities involved on-board with this.
Thanks,
Roman.
Re: [DISCUSS] Setting up long term support mindset
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Mar 07, 2013 at 07:25AM, Arun C Murthy wrote:
> As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will
> help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys
> be willing to help?
As has been agreed on the same thread, Bigtop is indeed willing to help with
the effort. This thread is slightly different: we are trying to find a model
that will help to bring users to Hadoop2 line. Right now, none of the
production HBase deployment I am aware of using Hadoop2. That's a telling
story. I want to see this project as a place where people can come and working
stacks guaranteed. With support from the community.
So, let's not go into the discussion about specific bugs on this thread,
please.
Cos
Re: [DISCUSS] Setting up long term support mindset
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Mar 07, 2013 at 07:25AM, Arun C Murthy wrote:
> As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will
> help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys
> be willing to help?
As has been agreed on the same thread, Bigtop is indeed willing to help with
the effort. This thread is slightly different: we are trying to find a model
that will help to bring users to Hadoop2 line. Right now, none of the
production HBase deployment I am aware of using Hadoop2. That's a telling
story. I want to see this project as a place where people can come and working
stacks guaranteed. With support from the community.
So, let's not go into the discussion about specific bugs on this thread,
please.
Cos
Re: [DISCUSS] Setting up long term support mindset
Posted by Arun C Murthy <ac...@hortonworks.com>.
On Mar 6, 2013, at 1:27 PM, Roman Shaposhnik wrote:
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
FTR: A stated goal is to get to a hadoop-2.0.5-beta within the next couple of months. I, and my team, have been spending an inordinate amount of time vetting and solidifying YARN APIs and protocols for long-term sustenance:
https://issues.apache.org/jira/browse/YARN-386
The same effort is undergoing for HDFS (wire) protocols.
To be clear: do not confuse API work with stability of YARN itself. YARN, the system, and MapReduce atop, are very stable and has very, very large-scale deployments already. The above effort is an endeavor to ensure long-term maintainability of the APIs.
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys be willing to help?
thanks,
Arun
Re: [DISCUSS] Setting up long term support mindset
Posted by Konstantin Boudnik <co...@apache.org>.
There's only one goal _here_: providing stable collection of data processing
software. Stabilization of Hadoop 2.0 is discussed elsewhere, so let's
not mix two together nor bring the discussion of Hadoop PMC' role to this
list.
Perhaps there's a misunderstanding of the intent of LTS: it is an attempt to
provide users with a less turbulent stack, that isn't a subject of unexpected
breakage, because "Hadoop 2.0 is young". I would describe it as "LST won't
update underlying Hadoop bits just because new Hadoop release is out, unless
there's a guarantee that the rest of the stack works with it".
It is an attempt to step away from fragile model of "cutting edge releases
based on the _latest and not so greatest_ component releases". While Hadoop
might have its sweet time maturing and BigTop would certainly be helping along
the way, there are need to be a stable community supported stack environment
for users to download/install/work with. The content of it is apparently up to
the community at large to decide.
Cos
On Sat, Mar 16, 2013 at 10:06AM, Eric Baldeschwieler wrote:
> There are two goals here & I'm not sure how they can be met with a single release.
> - Stabilizing Apache Hadoop 2.0
> - Providing a known stable collection of hadoop components for hadoop users.
>
> I support both goals, but I'm not sure how you'd base them on the same release...
>
> Currently my recommendation for anyone looking for the most stable release
> of hadoop is to use the 1.0 line. That may change in 2013, but that is not
> yet proven. I encourage folks to use 2.0, but with eyes open that they may
> cut their knuckles.
>
> Hadoop 2.0 is young. That's why it is tagged alpha. APIs are still
> changing for good reasons, which seems like more than a labeling problem to
> me. I certainly support the goal of a LTS on 2, but until the Hadoop PMC
> declares Hadoop 2.0 GA, my read is that anyone declaring that they are going
> to be LTS on 2.0 is betting that they understand Hadoop's stability better
> than the Hadoop PMC.
>
> But maybe I don't understand the intent of the LTS label?
>
> E14
>
> On Mar 7, 2013, at 10:57 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Andrew,
> > I think the influence part is about how to make LTS accepted and not how to
> > make LTS happened, right?
> >
> > With respect to Roman's points:
> >
> > 1. Agree
> >
> > 2. a lot of people are keep using Hadoop-1 because of the perception that
> > Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
> > hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
> > including a reasonably stable Hadoop. Beta is a good aim, I think
> >
> > 3. can be solved by having an RM for a Hadoop release combined with an RM for
> > BigTop release, who can coordinate the cross-effort.
> >
> > 4. The only way is to help these components be more comfortable with the base
> > of the stack. 2-3 above are addressing it, I believe.
> >
> > 5. I tend to agree with Andrew - it might be quite hard to do.
> >
> > Cos
> >
> > On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> >> A lot of behavior among loosely coupled projects is emergent. So in that
> >> sense some of what bigtop can produce is defined by something beyond direct
> >> control. What you can do is try to influence the processes at work. So,
> >> outreach to each component project. Also, constructive pressure. Publicize
> >> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> >> components if there is an integration issue and an unresponsive community.
> >> This would apply constructive pressure on the unresponsive community
> >> commensurate with the influence of bigtop.
> >>
> >> To grow the influence of bigtop, engaging vendors might be useful. For
> >> those pursuing an open strategy or open core strategy then commoditizing
> >> and amortizing the costs of baseline packaging and integration concerns
> >> makes sense. Everybody wins because more bandwidth is available to focus on
> >> differentiators. Such vendors typically employ committers for community
> >> based development. If some of these vendors can publicly get behind bigtop
> >> and invest in its credibility, then as an emergent consequence there will
> >> be more community engagement on integration issues under its umbrella.
> >> Everyone will win.
> >>
> >> On some of the specific points, my humble opinion:
> >>
> >> 2. Hadoop 2 is the only long term viable option even if some parts may be
> >> still unstable in terms of API.
> >>
> >> 3. This seems a case of "build it and they will come". Iterate on a BOM
> >> that (mostly) works. Publicize it. For integration blockers, track the
> >> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> >> respective dev lists with gentle reminders, but not too frequently.
> >>
> >> 4. See above.
> >>
> >> 5. You can't.
> >>
> >> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> >>
> >>> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> >>> wrote:
> >>>> I believe there's not much more to say about it, except that this is,
> >>>> in my opinion, a good way to establish our project as de-facto go-to
> >>>> place for community driven Hadoop based stacks and a focal point for
> >>>> the integration in the ASF storage and analytics projects.
> >>>
> >>> I like this idea very much! A couple of things that I'd love to hear other
> >>> chime in on:
> >>> #1 I think it is too late to change the focus of Bigtop 0.6.0
> >>> #2 Do we have a reasonable conviction that the beta
> >>> release of Hadoop 2.X is withing reach?
> >>> #3 How do we influence Hadoop community to help us
> >>> produce the first ever LTS of Bigtop?
> >>> #4 How do we get the downstream communities (pig, oozie, etc)
> >>> on-board so that we can all work towards this common goal?
> >>> #5 Suppose we do all the work in all of the downstream components,
> >>> how can we at least make sure that there will be patch
> >>> releases incorporating all the changes we've done?
> >>>
> >>> Now, Bigtop (well, me personally at least ;-)) would be more than
> >>> willing to help on all of these with automation, testing, etc. But
> >>> we *have* to get all of the communities involved on-board with this.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
>
Re: [DISCUSS] Setting up long term support mindset
Posted by Eric Baldeschwieler <er...@hortonworks.com>.
There are two goals here & I'm not sure how they can be met with a single release.
- Stabilizing Apache Hadoop 2.0
- Providing a known stable collection of hadoop components for hadoop users.
I support both goals, but I'm not sure how you'd base them on the same release...
Currently my recommendation for anyone looking for the most stable release of hadoop is to use the 1.0 line. That may change in 2013, but that is not yet proven. I encourage folks to use 2.0, but with eyes open that they may cut their knuckles.
Hadoop 2.0 is young. That's why it is tagged alpha. APIs are still changing for good reasons, which seems like more than a labeling problem to me. I certainly support the goal of a LTS on 2, but until the Hadoop PMC declares Hadoop 2.0 GA, my read is that anyone declaring that they are going to be LTS on 2.0 is betting that they understand Hadoop's stability better than the Hadoop PMC.
But maybe I don't understand the intent of the LTS label?
E14
On Mar 7, 2013, at 10:57 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Andrew,
> I think the influence part is about how to make LTS accepted and not how to
> make LTS happened, right?
>
> With respect to Roman's points:
>
> 1. Agree
>
> 2. a lot of people are keep using Hadoop-1 because of the perception that
> Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
> hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
> including a reasonably stable Hadoop. Beta is a good aim, I think
>
> 3. can be solved by having an RM for a Hadoop release combined with an RM for
> BigTop release, who can coordinate the cross-effort.
>
> 4. The only way is to help these components be more comfortable with the base
> of the stack. 2-3 above are addressing it, I believe.
>
> 5. I tend to agree with Andrew - it might be quite hard to do.
>
> Cos
>
> On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
>> A lot of behavior among loosely coupled projects is emergent. So in that
>> sense some of what bigtop can produce is defined by something beyond direct
>> control. What you can do is try to influence the processes at work. So,
>> outreach to each component project. Also, constructive pressure. Publicize
>> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
>> components if there is an integration issue and an unresponsive community.
>> This would apply constructive pressure on the unresponsive community
>> commensurate with the influence of bigtop.
>>
>> To grow the influence of bigtop, engaging vendors might be useful. For
>> those pursuing an open strategy or open core strategy then commoditizing
>> and amortizing the costs of baseline packaging and integration concerns
>> makes sense. Everybody wins because more bandwidth is available to focus on
>> differentiators. Such vendors typically employ committers for community
>> based development. If some of these vendors can publicly get behind bigtop
>> and invest in its credibility, then as an emergent consequence there will
>> be more community engagement on integration issues under its umbrella.
>> Everyone will win.
>>
>> On some of the specific points, my humble opinion:
>>
>> 2. Hadoop 2 is the only long term viable option even if some parts may be
>> still unstable in terms of API.
>>
>> 3. This seems a case of "build it and they will come". Iterate on a BOM
>> that (mostly) works. Publicize it. For integration blockers, track the
>> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
>> respective dev lists with gentle reminders, but not too frequently.
>>
>> 4. See above.
>>
>> 5. You can't.
>>
>> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
>>
>>> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
>>> wrote:
>>>> I believe there's not much more to say about it, except that this is,
>>>> in my opinion, a good way to establish our project as de-facto go-to
>>>> place for community driven Hadoop based stacks and a focal point for
>>>> the integration in the ASF storage and analytics projects.
>>>
>>> I like this idea very much! A couple of things that I'd love to hear other
>>> chime in on:
>>> #1 I think it is too late to change the focus of Bigtop 0.6.0
>>> #2 Do we have a reasonable conviction that the beta
>>> release of Hadoop 2.X is withing reach?
>>> #3 How do we influence Hadoop community to help us
>>> produce the first ever LTS of Bigtop?
>>> #4 How do we get the downstream communities (pig, oozie, etc)
>>> on-board so that we can all work towards this common goal?
>>> #5 Suppose we do all the work in all of the downstream components,
>>> how can we at least make sure that there will be patch
>>> releases incorporating all the changes we've done?
>>>
>>> Now, Bigtop (well, me personally at least ;-)) would be more than
>>> willing to help on all of these with automation, testing, etc. But
>>> we *have* to get all of the communities involved on-board with this.
>>>
>>> Thanks,
>>> Roman.
>>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
Re: [DISCUSS] Setting up long term support mindset
Posted by Konstantin Boudnik <co...@apache.org>.
Andrew,
I think the influence part is about how to make LTS accepted and not how to
make LTS happened, right?
With respect to Roman's points:
1. Agree
2. a lot of people are keep using Hadoop-1 because of the perception that
Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
including a reasonably stable Hadoop. Beta is a good aim, I think
3. can be solved by having an RM for a Hadoop release combined with an RM for
BigTop release, who can coordinate the cross-effort.
4. The only way is to help these components be more comfortable with the base
of the stack. 2-3 above are addressing it, I believe.
5. I tend to agree with Andrew - it might be quite hard to do.
Cos
On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> A lot of behavior among loosely coupled projects is emergent. So in that
> sense some of what bigtop can produce is defined by something beyond direct
> control. What you can do is try to influence the processes at work. So,
> outreach to each component project. Also, constructive pressure. Publicize
> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> components if there is an integration issue and an unresponsive community.
> This would apply constructive pressure on the unresponsive community
> commensurate with the influence of bigtop.
>
> To grow the influence of bigtop, engaging vendors might be useful. For
> those pursuing an open strategy or open core strategy then commoditizing
> and amortizing the costs of baseline packaging and integration concerns
> makes sense. Everybody wins because more bandwidth is available to focus on
> differentiators. Such vendors typically employ committers for community
> based development. If some of these vendors can publicly get behind bigtop
> and invest in its credibility, then as an emergent consequence there will
> be more community engagement on integration issues under its umbrella.
> Everyone will win.
>
> On some of the specific points, my humble opinion:
>
> 2. Hadoop 2 is the only long term viable option even if some parts may be
> still unstable in terms of API.
>
> 3. This seems a case of "build it and they will come". Iterate on a BOM
> that (mostly) works. Publicize it. For integration blockers, track the
> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> respective dev lists with gentle reminders, but not too frequently.
>
> 4. See above.
>
> 5. You can't.
>
> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
>
> > On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> > wrote:
> > > I believe there's not much more to say about it, except that this is,
> > > in my opinion, a good way to establish our project as de-facto go-to
> > > place for community driven Hadoop based stacks and a focal point for
> > > the integration in the ASF storage and analytics projects.
> >
> > I like this idea very much! A couple of things that I'd love to hear other
> > chime in on:
> > #1 I think it is too late to change the focus of Bigtop 0.6.0
> > #2 Do we have a reasonable conviction that the beta
> > release of Hadoop 2.X is withing reach?
> > #3 How do we influence Hadoop community to help us
> > produce the first ever LTS of Bigtop?
> > #4 How do we get the downstream communities (pig, oozie, etc)
> > on-board so that we can all work towards this common goal?
> > #5 Suppose we do all the work in all of the downstream components,
> > how can we at least make sure that there will be patch
> > releases incorporating all the changes we've done?
> >
> > Now, Bigtop (well, me personally at least ;-)) would be more than
> > willing to help on all of these with automation, testing, etc. But
> > we *have* to get all of the communities involved on-board with this.
> >
> > Thanks,
> > Roman.
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: [DISCUSS] Setting up long term support mindset
Posted by Konstantin Boudnik <co...@apache.org>.
Andrew,
I think the influence part is about how to make LTS accepted and not how to
make LTS happened, right?
With respect to Roman's points:
1. Agree
2. a lot of people are keep using Hadoop-1 because of the perception that
Hadoop-2 is alpha. And the "alpha" tag helps a lot to keep this impression
hardened, IMO. So, it is quite critical to produce a stable Bigtop stack,
including a reasonably stable Hadoop. Beta is a good aim, I think
3. can be solved by having an RM for a Hadoop release combined with an RM for
BigTop release, who can coordinate the cross-effort.
4. The only way is to help these components be more comfortable with the base
of the stack. 2-3 above are addressing it, I believe.
5. I tend to agree with Andrew - it might be quite hard to do.
Cos
On Thu, Mar 07, 2013 at 12:38PM, Andrew Purtell wrote:
> A lot of behavior among loosely coupled projects is emergent. So in that
> sense some of what bigtop can produce is defined by something beyond direct
> control. What you can do is try to influence the processes at work. So,
> outreach to each component project. Also, constructive pressure. Publicize
> bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
> components if there is an integration issue and an unresponsive community.
> This would apply constructive pressure on the unresponsive community
> commensurate with the influence of bigtop.
>
> To grow the influence of bigtop, engaging vendors might be useful. For
> those pursuing an open strategy or open core strategy then commoditizing
> and amortizing the costs of baseline packaging and integration concerns
> makes sense. Everybody wins because more bandwidth is available to focus on
> differentiators. Such vendors typically employ committers for community
> based development. If some of these vendors can publicly get behind bigtop
> and invest in its credibility, then as an emergent consequence there will
> be more community engagement on integration issues under its umbrella.
> Everyone will win.
>
> On some of the specific points, my humble opinion:
>
> 2. Hadoop 2 is the only long term viable option even if some parts may be
> still unstable in terms of API.
>
> 3. This seems a case of "build it and they will come". Iterate on a BOM
> that (mostly) works. Publicize it. For integration blockers, track the
> respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
> respective dev lists with gentle reminders, but not too frequently.
>
> 4. See above.
>
> 5. You can't.
>
> On Thursday, March 7, 2013, Roman Shaposhnik wrote:
>
> > On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> > wrote:
> > > I believe there's not much more to say about it, except that this is,
> > > in my opinion, a good way to establish our project as de-facto go-to
> > > place for community driven Hadoop based stacks and a focal point for
> > > the integration in the ASF storage and analytics projects.
> >
> > I like this idea very much! A couple of things that I'd love to hear other
> > chime in on:
> > #1 I think it is too late to change the focus of Bigtop 0.6.0
> > #2 Do we have a reasonable conviction that the beta
> > release of Hadoop 2.X is withing reach?
> > #3 How do we influence Hadoop community to help us
> > produce the first ever LTS of Bigtop?
> > #4 How do we get the downstream communities (pig, oozie, etc)
> > on-board so that we can all work towards this common goal?
> > #5 Suppose we do all the work in all of the downstream components,
> > how can we at least make sure that there will be patch
> > releases incorporating all the changes we've done?
> >
> > Now, Bigtop (well, me personally at least ;-)) would be more than
> > willing to help on all of these with automation, testing, etc. But
> > we *have* to get all of the communities involved on-board with this.
> >
> > Thanks,
> > Roman.
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: [DISCUSS] Setting up long term support mindset
Posted by Andrew Purtell <ap...@apache.org>.
A lot of behavior among loosely coupled projects is emergent. So in that
sense some of what bigtop can produce is defined by something beyond direct
control. What you can do is try to influence the processes at work. So,
outreach to each component project. Also, constructive pressure. Publicize
bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
components if there is an integration issue and an unresponsive community.
This would apply constructive pressure on the unresponsive community
commensurate with the influence of bigtop.
To grow the influence of bigtop, engaging vendors might be useful. For
those pursuing an open strategy or open core strategy then commoditizing
and amortizing the costs of baseline packaging and integration concerns
makes sense. Everybody wins because more bandwidth is available to focus on
differentiators. Such vendors typically employ committers for community
based development. If some of these vendors can publicly get behind bigtop
and invest in its credibility, then as an emergent consequence there will
be more community engagement on integration issues under its umbrella.
Everyone will win.
On some of the specific points, my humble opinion:
2. Hadoop 2 is the only long term viable option even if some parts may be
still unstable in terms of API.
3. This seems a case of "build it and they will come". Iterate on a BOM
that (mostly) works. Publicize it. For integration blockers, track the
respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
respective dev lists with gentle reminders, but not too frequently.
4. See above.
5. You can't.
On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> wrote:
> > I believe there's not much more to say about it, except that this is,
> > in my opinion, a good way to establish our project as de-facto go-to
> > place for community driven Hadoop based stacks and a focal point for
> > the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
> #1 I think it is too late to change the focus of Bigtop 0.6.0
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
> #4 How do we get the downstream communities (pig, oozie, etc)
> on-board so that we can all work towards this common goal?
> #5 Suppose we do all the work in all of the downstream components,
> how can we at least make sure that there will be patch
> releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSS] Setting up long term support mindset
Posted by Bruno Mahé <bm...@apache.org>.
On 03/06/2013 01:27 PM, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
>> I believe there's not much more to say about it, except that this is,
>> in my opinion, a good way to establish our project as de-facto go-to
>> place for community driven Hadoop based stacks and a focal point for
>> the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
> #1 I think it is too late to change the focus of Bigtop 0.6.0
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
> #4 How do we get the downstream communities (pig, oozie, etc)
> on-board so that we can all work towards this common goal?
> #5 Suppose we do all the work in all of the downstream components,
> how can we at least make sure that there will be patch
> releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>
Hi,
It seems like there are at least two different topics meshed together:
1/ Offering a stable version of Apache Bigtop
2/ How can Apache Bigtop help raise the quality of downstream projects
Regarding 1/ I was under the impression this is what the branch 0.3.X is
all about. There was some interest, but apparently not enough to get
releases out. So I am all for it but I am not sure building the
consensus for a LTS stamp will be enough if we don't get enough
interest/people/time to have stable releases out in the first place.
As of 2/, isn't it just a matter to put in action what we talked about
numerous time: having builds based of trunk/rc branches of downstream
projects? But there again, this is very time consuming and we do not
have that many volunteers/time. This would also not be an answer to all
the issues, but that would cover quite a few.
We would not be able to guarantee that issues would be fixed, but these
communities would have some incentive to fix them anyway. And if they
don't fix it, nothing forces us to update these projects.
Having downstream communities on board would also be very nice, but I am
not sure it would be a requirement.
Thanks,
Bruno
Re: [DISCUSS] Setting up long term support mindset
Posted by Bruno Mahé <bm...@apache.org>.
On 03/06/2013 01:27 PM, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <co...@apache.org> wrote:
>> I believe there's not much more to say about it, except that this is,
>> in my opinion, a good way to establish our project as de-facto go-to
>> place for community driven Hadoop based stacks and a focal point for
>> the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
> #1 I think it is too late to change the focus of Bigtop 0.6.0
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
> #4 How do we get the downstream communities (pig, oozie, etc)
> on-board so that we can all work towards this common goal?
> #5 Suppose we do all the work in all of the downstream components,
> how can we at least make sure that there will be patch
> releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>
Hi,
It seems like there are at least two different topics meshed together:
1/ Offering a stable version of Apache Bigtop
2/ How can Apache Bigtop help raise the quality of downstream projects
Regarding 1/ I was under the impression this is what the branch 0.3.X is
all about. There was some interest, but apparently not enough to get
releases out. So I am all for it but I am not sure building the
consensus for a LTS stamp will be enough if we don't get enough
interest/people/time to have stable releases out in the first place.
As of 2/, isn't it just a matter to put in action what we talked about
numerous time: having builds based of trunk/rc branches of downstream
projects? But there again, this is very time consuming and we do not
have that many volunteers/time. This would also not be an answer to all
the issues, but that would cover quite a few.
We would not be able to guarantee that issues would be fixed, but these
communities would have some incentive to fix them anyway. And if they
don't fix it, nothing forces us to update these projects.
Having downstream communities on board would also be very nice, but I am
not sure it would be a requirement.
Thanks,
Bruno
Re: [DISCUSS] Setting up long term support mindset
Posted by Arun C Murthy <ac...@hortonworks.com>.
On Mar 6, 2013, at 1:27 PM, Roman Shaposhnik wrote:
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
FTR: A stated goal is to get to a hadoop-2.0.5-beta within the next couple of months. I, and my team, have been spending an inordinate amount of time vetting and solidifying YARN APIs and protocols for long-term sustenance:
https://issues.apache.org/jira/browse/YARN-386
The same effort is undergoing for HDFS (wire) protocols.
To be clear: do not confuse API work with stability of YARN itself. YARN, the system, and MapReduce atop, are very stable and has very, very large-scale deployments already. The above effort is an endeavor to ensure long-term maintainability of the APIs.
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
As I've said in the thread over at Hadoop (http://s.apache.org/4DR), it will help if Bigtop is willing to step forward and vet Hadoop RCs. Would you guys be willing to help?
thanks,
Arun
Re: [DISCUSS] Setting up long term support mindset
Posted by Andrew Purtell <ap...@apache.org>.
A lot of behavior among loosely coupled projects is emergent. So in that
sense some of what bigtop can produce is defined by something beyond direct
control. What you can do is try to influence the processes at work. So,
outreach to each component project. Also, constructive pressure. Publicize
bigtop as a vetted stable open packaging of Hadoop. Don't upgrade
components if there is an integration issue and an unresponsive community.
This would apply constructive pressure on the unresponsive community
commensurate with the influence of bigtop.
To grow the influence of bigtop, engaging vendors might be useful. For
those pursuing an open strategy or open core strategy then commoditizing
and amortizing the costs of baseline packaging and integration concerns
makes sense. Everybody wins because more bandwidth is available to focus on
differentiators. Such vendors typically employ committers for community
based development. If some of these vendors can publicly get behind bigtop
and invest in its credibility, then as an emergent consequence there will
be more community engagement on integration issues under its umbrella.
Everyone will win.
On some of the specific points, my humble opinion:
2. Hadoop 2 is the only long term viable option even if some parts may be
still unstable in terms of API.
3. This seems a case of "build it and they will come". Iterate on a BOM
that (mostly) works. Publicize it. For integration blockers, track the
respective JIRAs on the bigtop wiki. Be polite. Be proactive. Mail the
respective dev lists with gentle reminders, but not too frequently.
4. See above.
5. You can't.
On Thursday, March 7, 2013, Roman Shaposhnik wrote:
> On Wed, Mar 6, 2013 at 10:10 AM, Konstantin Boudnik <cos@apache.org<javascript:;>>
> wrote:
> > I believe there's not much more to say about it, except that this is,
> > in my opinion, a good way to establish our project as de-facto go-to
> > place for community driven Hadoop based stacks and a focal point for
> > the integration in the ASF storage and analytics projects.
>
> I like this idea very much! A couple of things that I'd love to hear other
> chime in on:
> #1 I think it is too late to change the focus of Bigtop 0.6.0
> #2 Do we have a reasonable conviction that the beta
> release of Hadoop 2.X is withing reach?
> #3 How do we influence Hadoop community to help us
> produce the first ever LTS of Bigtop?
> #4 How do we get the downstream communities (pig, oozie, etc)
> on-board so that we can all work towards this common goal?
> #5 Suppose we do all the work in all of the downstream components,
> how can we at least make sure that there will be patch
> releases incorporating all the changes we've done?
>
> Now, Bigtop (well, me personally at least ;-)) would be more than
> willing to help on all of these with automation, testing, etc. But
> we *have* to get all of the communities involved on-board with this.
>
> Thanks,
> Roman.
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)