You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Eric Sammer <es...@cloudera.com> on 2011/05/07 08:14:01 UTC
Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
I understand Eli's concerns that putting stuff in there that hasn't gone
into trunk yet is danger. However, as the team makes no guarantees of 100%
compatibility between releases, I don't think it's critical. It's just
something that needs to be addressed -which can be done after this release
has shipped.
I was under the impression that the community has been extremely strict
about compatibility between minor version bumps in the past. I though there
were specific guarantees and that was one of the reasons certain behaviors
have persisted so long.
Does this mean API changes can be made in minor releases and it can be made
backward compatible in future releases? That seems very, very counter to
various conversations that have happened in the past. I'm of the mind that
we should continue to promise what we've always promised and if that's
changing, let's make with the refactoring party!
Can some PMC'ers clarify this one for me?
TIA.
Sammer
-Steve
Re: Release compatibility was Re: [VOTE] Release candidate
0.20.203.0-rc1
Posted by Milind Bhandarkar <mb...@linkedin.com>.
Thanks Kos. Archived mailing lists come in handy. Many thanks to Apache to
have http://mail-archives.apache.org/mod_mbox/hadoop-general/.
- milind
--
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167
On 5/6/11 11:57 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
>Wow! Great compilation, Milind! Very nice to have the sequence of events
>handy.
>
>Thanks,
> Cos
>
>On Fri, May 6, 2011 at 23:55, Milind Bhandarkar
><mb...@linkedin.com> wrote:
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>>
>>>On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>>
>>>I understand Eli's concerns that putting stuff in there that hasn't gone
>>>into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>>compatibility between releases, I don't think it's critical. It's just
>>>something that needs to be addressed -which can be done after this
>>>release
>>>has shipped.
>>>
>>>
>>>I was under the impression that the community has been extremely strict
>>>about compatibility between minor version bumps in the past. I though
>>>there
>>>were specific guarantees and that was one of the reasons certain
>>>behaviors
>>>have persisted so long.
>>>
>>>Does this mean API changes can be made in minor releases and it can be
>>>made
>>>backward compatible in future releases? That seems very, very counter to
>>>various conversations that have happened in the past. I'm of the mind
>>>that
>>>we should continue to promise what we've always promised and if that's
>>>changing, let's make with the refactoring party!
>>>
>>>Can some PMC'ers clarify this one for me?
>>>
>>>TIA.
>>>Sammer
>>>
>>>
>>>
>>>-Steve
>>
>>
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Konstantin Boudnik <co...@apache.org>.
Wow! Great compilation, Milind! Very nice to have the sequence of events handy.
Thanks,
Cos
On Fri, May 6, 2011 at 23:55, Milind Bhandarkar
<mb...@linkedin.com> wrote:
> [I am not on PMC, but seeing that PMC may be busy with other issues, I
> will try to answer your questions.]
>
> Eric,
>
> I think the thread
> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
> questions. Here is the timeline as I see it:
>
> 1. Arun proposes to create a release from the security patchset. Says Doug
> has proposed this earlier
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
> earlier by Doug and did not get far due to concerns about the effect this
> would have on development on trunk.") (August 24, 2010)
>
> 2. Lots of +1s, between August 24 to August 30 2010. One particular
> comment is from Tom White: "I think it would be good to have a shared 0.20
> Apache security branch.
> Since security isn't in 0.21, and the 0.22 release is a some way off
> as you mention, this would be useful for folks who want the security
> features sooner (and want to use an Apache release)."
>
> 3. Arun volunteers to create a release (August 30, 2010)
>
> 4. Doug reminds Arun. (October 15, 2010)
>
> 5. Arun apologizes for not creating a branch because he was busy, because
> he had a baby. (January 11, 2011)
>
> 6. Lots of discussion about what to call it (the release, not the baby,
> although I had a good laugh at Patrick Angeles's email: "You're gonna call
> your kid 20.100?" ;-).
>
> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
> something like 20.100 to show that it's a big jump? Anything else?" Jan
> 12, 2011
>
> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
> 12, 2011.
>
> So, as you can see, even if this release is called 0.20.x, the community
> agreed that these are valuable patches to have, and despite backward
> incompatibility, still have them in minor release.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>
>>On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>
>>I understand Eli's concerns that putting stuff in there that hasn't gone
>>into trunk yet is danger. However, as the team makes no guarantees of 100%
>>compatibility between releases, I don't think it's critical. It's just
>>something that needs to be addressed -which can be done after this release
>>has shipped.
>>
>>
>>I was under the impression that the community has been extremely strict
>>about compatibility between minor version bumps in the past. I though
>>there
>>were specific guarantees and that was one of the reasons certain behaviors
>>have persisted so long.
>>
>>Does this mean API changes can be made in minor releases and it can be
>>made
>>backward compatible in future releases? That seems very, very counter to
>>various conversations that have happened in the past. I'm of the mind that
>>we should continue to promise what we've always promised and if that's
>>changing, let's make with the refactoring party!
>>
>>Can some PMC'ers clarify this one for me?
>>
>>TIA.
>>Sammer
>>
>>
>>
>>-Steve
>
>
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Eli Collins <el...@cloudera.com>.
On Sat, May 7, 2011 at 6:36 PM, Ian Holsman <ha...@holsman.net> wrote:
>
> On May 8, 2011, at 9:50 AM, Eric Sammer wrote:
>
>> do we permit
>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>> something we've allowed just for the 203 release?
>
> good question.
> do we allow incompatible (smallish) features to be added to a 20.x release.
> hoping that they will eventually be put into trunk at a later stage.
> and if we need a process or something around it, or will just act on good faith that it will occur.
We do allow it, as the 203 release shows. I don't think we need an
official process, our existing practice of filing a jira with the
appropriate fix version is sufficient. And this seems to be happening
already for all changes to future 20x releases.
Compatibility is just one reason it's important that features be
developed on trunk first. Security and other enhancements were
developed on 20 and forward ported to trunk. Almost two years later
the forward porting is still not complete - if we released from trunk
today, it would not support security. Ditto for append. Some people
who work on HBase consider the code in branch-20-append to be more
reliable than the code on trunk. This is the primary reason why people
are currently on 20.x releases.
People are of course free to contribute to Apache on whatever branch
they please, but I think as a development community we need to try to
make sure a future release off trunk is not a regression against
previous releases. This won't happen unless we invest in trunk. I
support a release from branch-20-security, I just wanted to see the
work go into trunk first. Ie I think the staging matters. (and I agree
with Ray wrt the version scheme but that's independent).
Fortunately, I don't think we're far off. The vast majority of
security work is in trunk (thanks to all those who did the porting),
there's probably only 50 to 100 bugs/enhancements in
branch-20-security not yet in trunk, and the append code (or rather
sync) to support HBase just needs some tests and debugging.
I agree with Eric that is is the desire to put improvements into users
hands more quickly that drives orgs to produce releases of Hadoop
outside Apache. I think the best way to address this is to get solid
releases coming off trunk on a regular basis again. Hence the push to
close out 22 and work on getting trunk in shape. Also, as a
development community, we're at our best when collaborating on trunk.
Thanks,
Eli
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Ian Holsman <ha...@holsman.net>.
On May 8, 2011, at 9:50 AM, Eric Sammer wrote:
> do we permit
> backward incompatible changes between 0.22.0 and 0.22.1 or is this
> something we've allowed just for the 203 release?
good question.
do we allow incompatible (smallish) features to be added to a 20.x release.
hoping that they will eventually be put into trunk at a later stage.
and if we need a process or something around it, or will just act on good faith that it will occur.
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On May 10, 2011, at 10:49 PM, Aaron T. Myers wrote:
> On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <dd...@yahoo-inc.com>
> wrote:
>
> .....
>
> By far the most significant incompatibility that I've seen from a user
> perspective is that setting hadoop.job.ugi no longer has any effect.
> Granted, this interface wasn't used by a large percentage of users,
> but
> those that were using it have no other alternative that is as
> flexible as
> this was. There was discussion about this incompatibility last
> September on
> the mailing lists[1]. The conclusion there was that supporting
> backward
> compatibility for this interface was too difficult, semantically and
> technically, to warrant support. This incompatibility is present
> whether or
> not Kerberos support is enabled on the cluster.
That I why we added the interface audience and stability classification.
From the very beginning (see my early proposals on HADOOP-5073) the
security UGI
interfaces were targeted to be marked as limited private.
We completed the interface annotation in release 21, so some users did
not realize that they should not
be using such interfaces.
sanjay
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <dd...@yahoo-inc.com> wrote:
> I can't think of any major user-facing incompatibilities other than the
> users having to run a 'kinit' when they are working with a secure Hadoop
> cluster (of course the admins need to do more work in order to set up a
> secure cluster).
By far the most significant incompatibility that I've seen from a user
perspective is that setting hadoop.job.ugi no longer has any effect.
Granted, this interface wasn't used by a large percentage of users, but
those that were using it have no other alternative that is as flexible as
this was. There was discussion about this incompatibility last September on
the mailing lists[1]. The conclusion there was that supporting backward
compatibility for this interface was too difficult, semantically and
technically, to warrant support. This incompatibility is present whether or
not Kerberos support is enabled on the cluster.
I totally agree that the pain of upgrading to 0.20 security was felt
substantially more by admins/operators than by users.
--
Aaron T. Myers
Software Engineer, Cloudera
[1]
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <dd...@yahoo-inc.com> wrote:
> Just so that everyone is on the same page w.r.t the compatibility between
> 20.2 & 20.203 (don't think this is documented anywhere yet)..
>
> The aim of the team working on Hadoop Security at Yahoo! was to make Hadoop
> *secure*, and with *minimal* disruption to existing apps. I can't think of
> any major user-facing incompatibilities other than the users having to run a
> 'kinit' when they are working with a secure Hadoop cluster (of course the
> admins need to do more work in order to set up a secure cluster). Also,
> security can switched off, and all the other enhancements (job limits, etc.)
> are still available.. As per users/Operations/Solutions at Yahoo!,
> 20.security was one of the smoothest upgrades ever.
>
And I think you guys did a commendable job with this, given the scope of the
project! :)
But there were certainly plenty of bugs introduced along the way that
affected both secure and non-secure, and even now the security-able branches
don't function on any non-Sun JVM.
Again, I think for this particular case, most of the developers agreed on
the risk/reward trade-off, so I didn't want to start a discussion about
security being a good or bad decision to backport on to 0.20.
But, I'd love to know what our framework is for making such decisions in the
future, if we plan to maintain branches with feature backports as part of
Apache. (eg what scope of change requires what type of vote and when)
-Todd
>
> On 5/10/11 2:28 PM, "Todd Lipcon" <to...@cloudera.com> wrote:
>
> On Tue, May 10, 2011 at 12:41 PM, Scott Carey <scott@richrelevance.com
> >wrote:
>
> >
> > As an observer, this is a very important observation. Sure, the default
> > is that dot releases are bugfix-onl. But exceptions to these rules are
> > sometimes required and often beneficial to the health of the project.
> > Performance enhancements, minor features, and other items are sometimes
> > very low risk and the barrier to getting them to users earlier should be
> > lower.
> >
>
> I agree whole-heartedly.
>
>
> > These issues are the sort of things that get into non-Apache releases
> > quickly and drive the community away from the Apache release. Its been
> > well proven through those vehicles that back-porting minor features and
> > improvements from trunk to an old release can be done safely.
>
>
> However, one shouldn't understate the difficulty of agreeing on the
> risk-reward tradeoff here. While risk is mostly technical, reward may vary
> widely based on the userbase or organization.
>
> For example, everyone would agree that security was a very risky feature to
> add to 20, with known backward compatibilities and a lot of fallout. For
> some people (both CDH and YDH), the security features were an absolute
> necessity on a tight timeline, so the risk-reward decision was clear --
> I've
> heard from many users, though, that they saw none of the reward from
> security and wished they hadn't had to endure the resulting changes and
> bugs
> within the 0.20 series.
>
> Another example is the 0.20-append patch series, which is indispensable for
> the HBase community but seen as overly risky by those who do not use HBase.
>
> So, while I'm in favor of "sustaining" release series like 0.20-security in
> theory, I also think we need a clear inclusion criteria for such branches.
> As I said in a previous email, the criteria used to be "low risk compatible
> bug fixes only" with a vote process for any exceptions. 0.20-security is
> obviously entirely different, but as yet remains undefined (it's way more
> than just "security").
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Release compatibility was Re: [VOTE] Release candidate
0.20.203.0-rc1
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Just so that everyone is on the same page w.r.t the compatibility between 20.2 & 20.203 (don't think this is documented anywhere yet)..
The aim of the team working on Hadoop Security at Yahoo! was to make Hadoop *secure*, and with *minimal* disruption to existing apps. I can't think of any major user-facing incompatibilities other than the users having to run a 'kinit' when they are working with a secure Hadoop cluster (of course the admins need to do more work in order to set up a secure cluster). Also, security can switched off, and all the other enhancements (job limits, etc.) are still available.. As per users/Operations/Solutions at Yahoo!, 20.security was one of the smoothest upgrades ever.
On 5/10/11 2:28 PM, "Todd Lipcon" <to...@cloudera.com> wrote:
On Tue, May 10, 2011 at 12:41 PM, Scott Carey <sc...@richrelevance.com>wrote:
>
> As an observer, this is a very important observation. Sure, the default
> is that dot releases are bugfix-onl. But exceptions to these rules are
> sometimes required and often beneficial to the health of the project.
> Performance enhancements, minor features, and other items are sometimes
> very low risk and the barrier to getting them to users earlier should be
> lower.
>
I agree whole-heartedly.
> These issues are the sort of things that get into non-Apache releases
> quickly and drive the community away from the Apache release. Its been
> well proven through those vehicles that back-porting minor features and
> improvements from trunk to an old release can be done safely.
However, one shouldn't understate the difficulty of agreeing on the
risk-reward tradeoff here. While risk is mostly technical, reward may vary
widely based on the userbase or organization.
For example, everyone would agree that security was a very risky feature to
add to 20, with known backward compatibilities and a lot of fallout. For
some people (both CDH and YDH), the security features were an absolute
necessity on a tight timeline, so the risk-reward decision was clear -- I've
heard from many users, though, that they saw none of the reward from
security and wished they hadn't had to endure the resulting changes and bugs
within the 0.20 series.
Another example is the 0.20-append patch series, which is indispensable for
the HBase community but seen as overly risky by those who do not use HBase.
So, while I'm in favor of "sustaining" release series like 0.20-security in
theory, I also think we need a clear inclusion criteria for such branches.
As I said in a previous email, the criteria used to be "low risk compatible
bug fixes only" with a vote process for any exceptions. 0.20-security is
obviously entirely different, but as yet remains undefined (it's way more
than just "security").
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, May 10, 2011 at 12:41 PM, Scott Carey <sc...@richrelevance.com>wrote:
>
> As an observer, this is a very important observation. Sure, the default
> is that dot releases are bugfix-onl. But exceptions to these rules are
> sometimes required and often beneficial to the health of the project.
> Performance enhancements, minor features, and other items are sometimes
> very low risk and the barrier to getting them to users earlier should be
> lower.
>
I agree whole-heartedly.
> These issues are the sort of things that get into non-Apache releases
> quickly and drive the community away from the Apache release. Its been
> well proven through those vehicles that back-porting minor features and
> improvements from trunk to an old release can be done safely.
However, one shouldn't understate the difficulty of agreeing on the
risk-reward tradeoff here. While risk is mostly technical, reward may vary
widely based on the userbase or organization.
For example, everyone would agree that security was a very risky feature to
add to 20, with known backward compatibilities and a lot of fallout. For
some people (both CDH and YDH), the security features were an absolute
necessity on a tight timeline, so the risk-reward decision was clear -- I've
heard from many users, though, that they saw none of the reward from
security and wished they hadn't had to endure the resulting changes and bugs
within the 0.20 series.
Another example is the 0.20-append patch series, which is indispensable for
the HBase community but seen as overly risky by those who do not use HBase.
So, while I'm in favor of "sustaining" release series like 0.20-security in
theory, I also think we need a clear inclusion criteria for such branches.
As I said in a previous email, the criteria used to be "low risk compatible
bug fixes only" with a vote process for any exceptions. 0.20-security is
obviously entirely different, but as yet remains undefined (it's way more
than just "security").
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Release compatibility was Re: [VOTE] Release candidate
0.20.203.0-rc1
Posted by Scott Carey <sc...@richrelevance.com>.
On 5/8/11 11:10 AM, "Eric Baldeschwieler" <er...@yahoo-inc.com> wrote:
>I'd agree with this too. [same disclaimer as milind, not on PMC]
>
>In general one would not expect to see an incompatible change added in a
>dot release (0.24.1 0.24.2). I'd expect anything like that to require
>community discussion and support.
>
>As milind summarized, we seem to have support for the addition of
>security to 20. The existing mechanism of the required release vote will
>confirm or deny that.
>
>I think it is important that compatible enhancements to hadoop are
>allowed into dot releases. This is something that we've discussed but
>never finalized in the community. It is the desire to put improvements
>into users hands more quickly that the next major release that drives
>orgs to produce private releases of hadoop. In general, I think it is
>fair that such changes go into trunk first. Exceptions to that also need
>discussion and support IMO.
As an observer, this is a very important observation. Sure, the default
is that dot releases are bugfix-onl. But exceptions to these rules are
sometimes required and often beneficial to the health of the project.
Performance enhancements, minor features, and other items are sometimes
very low risk and the barrier to getting them to users earlier should be
lower.
These issues are the sort of things that get into non-Apache releases
quickly and drive the community away from the Apache release. Its been
well proven through those vehicles that back-porting minor features and
improvements from trunk to an old release can be done safely.
>
>I think the key to making progress is discussion and the idea that
>majority support, not consensus is what is needed to make exceptions to
>our process. Process is useful, it reduces friction. Process without
>exception is stifling.
Absolutely -- for a subset of process exceptions, a lazy majority would be
much more useful than consensus. Others are much more dangerous
(backwards compatibility breakage)
>
>On May 7, 2011, at 10:52 PM, Milind Bhandarkar wrote:
>
>> [Mentioning again: I am not on the PMC, and this email contains
>> non-binding opinions based on my reading the general@hadoop.apache.org
>> emails.]
>>
>> It is my understanding that, from the beginning, the 0.20+security was
>> always treated as an exception to the normal (I.e. Pre-0.20) release
>> process. (This has been confirmed by the mailing list threads, in which
>> many of those who are objecting to this release now - stating that it
>>has
>> violated norms - have consented, actually argued for, breaking the
>>norms.)
>>
>> For whatever I have read on this mailing list before the vote for this
>> release, it looked like most of the community agreed that what Yahoo!
>>Had
>> produced on their own branch, outside of Apache trunk, was important
>> contribution, and a release based on that would be a good idea, and
>>that a
>> one-time release should proceed. (After all, whichever organization the
>> contributors belong to, many seem to indicate that they feel ashamed not
>> having an Apache release in more than a year.)
>>
>> From many emails on this thread, it has been clear to me, that it is a
>>one
>> time concession given for parting ways from the normal process, and I
>>hope
>> everyone understands that this is supposed to make Apache Hadoop
>>releases
>> relevant once again.
>>
>> So, to cut it short, the 0.20.203 backward incompatibilities etc have no
>> bearing on the "normal" process, in which no backward incompatibilities
>> should be allowed in minor releases. To answer your specific question, I
>> have no reason to believe that 0.22.1 could be backward incompatible
>>with
>> 0.22.0.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/7/11 4:50 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>>
>>> Milind:
>>>
>>> Thanks for the pointer. I remember this thread. I guess my question
>>> was unrelated to the specific release and more about the general mode
>>> of development under normal release circumstances (ie. do we permit
>>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>>> something we've allowed just for the 203 release?).
>>>
>>> I think it's important to be clear about what the MO is so end users
>>> can plan upgrades appropriately.
>>>
>>> Thanks!
>>> Sammer
>>>
>>> On May 6, 2011, at 11:52 PM, Milind Bhandarkar
>>><mb...@linkedin.com>
>>> wrote:
>>>
>>>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>>>> will try to answer your questions.]
>>>>
>>>> Eric,
>>>>
>>>> I think the thread
>>>>
>>>>
>>>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3
>>>>C1
>>>> 8C
>>>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>>>> questions. Here is the timeline as I see it:
>>>>
>>>> 1. Arun proposes to create a release from the security patchset. Says
>>>> Doug
>>>> has proposed this earlier
>>>>
>>>>
>>>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3
>>>>C4
>>>> BD
>>>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>>>> earlier by Doug and did not get far due to concerns about the effect
>>>> this
>>>> would have on development on trunk.") (August 24, 2010)
>>>>
>>>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>>>> comment is from Tom White: "I think it would be good to have a shared
>>>> 0.20
>>>> Apache security branch.
>>>> Since security isn't in 0.21, and the 0.22 release is a some way off
>>>> as you mention, this would be useful for folks who want the security
>>>> features sooner (and want to use an Apache release)."
>>>>
>>>> 3. Arun volunteers to create a release (August 30, 2010)
>>>>
>>>> 4. Doug reminds Arun. (October 15, 2010)
>>>>
>>>> 5. Arun apologizes for not creating a branch because he was busy,
>>>> because
>>>> he had a baby. (January 11, 2011)
>>>>
>>>> 6. Lots of discussion about what to call it (the release, not the
>>>>baby,
>>>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>>> call
>>>> your kid 20.100?" ;-).
>>>>
>>>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>>> about
>>>> something like 20.100 to show that it's a big jump? Anything else?"
>>>>Jan
>>>> 12, 2011
>>>>
>>>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on
>>>>Jan
>>>> 12, 2011.
>>>>
>>>> So, as you can see, even if this release is called 0.20.x, the
>>>>community
>>>> agreed that these are valuable patches to have, and despite backward
>>>> incompatibility, still have them in minor release.
>>>>
>>>> - milind
>>>>
>>>> --
>>>> Milind Bhandarkar
>>>> mbhandarkar@linkedin.com
>>>> +1-650-776-3167
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>>>>
>>>>> On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>>>>
>>>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>>> gone
>>>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>>> 100%
>>>>> compatibility between releases, I don't think it's critical. It's
>>>>>just
>>>>> something that needs to be addressed -which can be done after this
>>>>> release
>>>>> has shipped.
>>>>>
>>>>>
>>>>> I was under the impression that the community has been extremely
>>>>>strict
>>>>> about compatibility between minor version bumps in the past. I though
>>>>> there
>>>>> were specific guarantees and that was one of the reasons certain
>>>>> behaviors
>>>>> have persisted so long.
>>>>>
>>>>> Does this mean API changes can be made in minor releases and it can
>>>>>be
>>>>> made
>>>>> backward compatible in future releases? That seems very, very counter
>>>>> to
>>>>> various conversations that have happened in the past. I'm of the mind
>>>>> that
>>>>> we should continue to promise what we've always promised and if
>>>>>that's
>>>>> changing, let's make with the refactoring party!
>>>>>
>>>>> Can some PMC'ers clarify this one for me?
>>>>>
>>>>> TIA.
>>>>> Sammer
>>>>>
>>>>>
>>>>>
>>>>> -Steve
>>>>
>>
>
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
I'd agree with this too. [same disclaimer as milind, not on PMC]
In general one would not expect to see an incompatible change added in a dot release (0.24.1 0.24.2). I'd expect anything like that to require community discussion and support.
As milind summarized, we seem to have support for the addition of security to 20. The existing mechanism of the required release vote will confirm or deny that.
I think it is important that compatible enhancements to hadoop are allowed into dot releases. This is something that we've discussed but never finalized in the community. It is the desire to put improvements into users hands more quickly that the next major release that drives orgs to produce private releases of hadoop. In general, I think it is fair that such changes go into trunk first. Exceptions to that also need discussion and support IMO.
I think the key to making progress is discussion and the idea that majority support, not consensus is what is needed to make exceptions to our process. Process is useful, it reduces friction. Process without exception is stifling.
On May 7, 2011, at 10:52 PM, Milind Bhandarkar wrote:
> [Mentioning again: I am not on the PMC, and this email contains
> non-binding opinions based on my reading the general@hadoop.apache.org
> emails.]
>
> It is my understanding that, from the beginning, the 0.20+security was
> always treated as an exception to the normal (I.e. Pre-0.20) release
> process. (This has been confirmed by the mailing list threads, in which
> many of those who are objecting to this release now - stating that it has
> violated norms - have consented, actually argued for, breaking the norms.)
>
> For whatever I have read on this mailing list before the vote for this
> release, it looked like most of the community agreed that what Yahoo! Had
> produced on their own branch, outside of Apache trunk, was important
> contribution, and a release based on that would be a good idea, and that a
> one-time release should proceed. (After all, whichever organization the
> contributors belong to, many seem to indicate that they feel ashamed not
> having an Apache release in more than a year.)
>
> From many emails on this thread, it has been clear to me, that it is a one
> time concession given for parting ways from the normal process, and I hope
> everyone understands that this is supposed to make Apache Hadoop releases
> relevant once again.
>
> So, to cut it short, the 0.20.203 backward incompatibilities etc have no
> bearing on the "normal" process, in which no backward incompatibilities
> should be allowed in minor releases. To answer your specific question, I
> have no reason to believe that 0.22.1 could be backward incompatible with
> 0.22.0.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/7/11 4:50 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>
>> Milind:
>>
>> Thanks for the pointer. I remember this thread. I guess my question
>> was unrelated to the specific release and more about the general mode
>> of development under normal release circumstances (ie. do we permit
>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>> something we've allowed just for the 203 release?).
>>
>> I think it's important to be clear about what the MO is so end users
>> can plan upgrades appropriately.
>>
>> Thanks!
>> Sammer
>>
>> On May 6, 2011, at 11:52 PM, Milind Bhandarkar <mb...@linkedin.com>
>> wrote:
>>
>>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>>> will try to answer your questions.]
>>>
>>> Eric,
>>>
>>> I think the thread
>>>
>>> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>> 8C
>>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>>> questions. Here is the timeline as I see it:
>>>
>>> 1. Arun proposes to create a release from the security patchset. Says
>>> Doug
>>> has proposed this earlier
>>>
>>> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>> BD
>>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>>> earlier by Doug and did not get far due to concerns about the effect
>>> this
>>> would have on development on trunk.") (August 24, 2010)
>>>
>>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>>> comment is from Tom White: "I think it would be good to have a shared
>>> 0.20
>>> Apache security branch.
>>> Since security isn't in 0.21, and the 0.22 release is a some way off
>>> as you mention, this would be useful for folks who want the security
>>> features sooner (and want to use an Apache release)."
>>>
>>> 3. Arun volunteers to create a release (August 30, 2010)
>>>
>>> 4. Doug reminds Arun. (October 15, 2010)
>>>
>>> 5. Arun apologizes for not creating a branch because he was busy,
>>> because
>>> he had a baby. (January 11, 2011)
>>>
>>> 6. Lots of discussion about what to call it (the release, not the baby,
>>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>> call
>>> your kid 20.100?" ;-).
>>>
>>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>> about
>>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>>> 12, 2011
>>>
>>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
>>> 12, 2011.
>>>
>>> So, as you can see, even if this release is called 0.20.x, the community
>>> agreed that these are valuable patches to have, and despite backward
>>> incompatibility, still have them in minor release.
>>>
>>> - milind
>>>
>>> --
>>> Milind Bhandarkar
>>> mbhandarkar@linkedin.com
>>> +1-650-776-3167
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>>>
>>>> On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>>>
>>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>> gone
>>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>> 100%
>>>> compatibility between releases, I don't think it's critical. It's just
>>>> something that needs to be addressed -which can be done after this
>>>> release
>>>> has shipped.
>>>>
>>>>
>>>> I was under the impression that the community has been extremely strict
>>>> about compatibility between minor version bumps in the past. I though
>>>> there
>>>> were specific guarantees and that was one of the reasons certain
>>>> behaviors
>>>> have persisted so long.
>>>>
>>>> Does this mean API changes can be made in minor releases and it can be
>>>> made
>>>> backward compatible in future releases? That seems very, very counter
>>>> to
>>>> various conversations that have happened in the past. I'm of the mind
>>>> that
>>>> we should continue to promise what we've always promised and if that's
>>>> changing, let's make with the refactoring party!
>>>>
>>>> Can some PMC'ers clarify this one for me?
>>>>
>>>> TIA.
>>>> Sammer
>>>>
>>>>
>>>>
>>>> -Steve
>>>
>
Re: Release compatibility was Re: [VOTE] Release candidate
0.20.203.0-rc1
Posted by Milind Bhandarkar <mb...@linkedin.com>.
[Mentioning again: I am not on the PMC, and this email contains
non-binding opinions based on my reading the general@hadoop.apache.org
emails.]
It is my understanding that, from the beginning, the 0.20+security was
always treated as an exception to the normal (I.e. Pre-0.20) release
process. (This has been confirmed by the mailing list threads, in which
many of those who are objecting to this release now - stating that it has
violated norms - have consented, actually argued for, breaking the norms.)
For whatever I have read on this mailing list before the vote for this
release, it looked like most of the community agreed that what Yahoo! Had
produced on their own branch, outside of Apache trunk, was important
contribution, and a release based on that would be a good idea, and that a
one-time release should proceed. (After all, whichever organization the
contributors belong to, many seem to indicate that they feel ashamed not
having an Apache release in more than a year.)
>From many emails on this thread, it has been clear to me, that it is a one
time concession given for parting ways from the normal process, and I hope
everyone understands that this is supposed to make Apache Hadoop releases
relevant once again.
So, to cut it short, the 0.20.203 backward incompatibilities etc have no
bearing on the "normal" process, in which no backward incompatibilities
should be allowed in minor releases. To answer your specific question, I
have no reason to believe that 0.22.1 could be backward incompatible with
0.22.0.
- milind
--
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167
On 5/7/11 4:50 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>Milind:
>
>Thanks for the pointer. I remember this thread. I guess my question
>was unrelated to the specific release and more about the general mode
>of development under normal release circumstances (ie. do we permit
>backward incompatible changes between 0.22.0 and 0.22.1 or is this
>something we've allowed just for the 203 release?).
>
>I think it's important to be clear about what the MO is so end users
>can plan upgrades appropriately.
>
>Thanks!
>Sammer
>
>On May 6, 2011, at 11:52 PM, Milind Bhandarkar <mb...@linkedin.com>
>wrote:
>
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>>
>>> On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>>
>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>gone
>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>> compatibility between releases, I don't think it's critical. It's just
>>> something that needs to be addressed -which can be done after this
>>>release
>>> has shipped.
>>>
>>>
>>> I was under the impression that the community has been extremely strict
>>> about compatibility between minor version bumps in the past. I though
>>> there
>>> were specific guarantees and that was one of the reasons certain
>>>behaviors
>>> have persisted so long.
>>>
>>> Does this mean API changes can be made in minor releases and it can be
>>> made
>>> backward compatible in future releases? That seems very, very counter
>>>to
>>> various conversations that have happened in the past. I'm of the mind
>>>that
>>> we should continue to promise what we've always promised and if that's
>>> changing, let's make with the refactoring party!
>>>
>>> Can some PMC'ers clarify this one for me?
>>>
>>> TIA.
>>> Sammer
>>>
>>>
>>>
>>> -Steve
>>
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Posted by Eric Sammer <es...@cloudera.com>.
Milind:
Thanks for the pointer. I remember this thread. I guess my question
was unrelated to the specific release and more about the general mode
of development under normal release circumstances (ie. do we permit
backward incompatible changes between 0.22.0 and 0.22.1 or is this
something we've allowed just for the 203 release?).
I think it's important to be clear about what the MO is so end users
can plan upgrades appropriately.
Thanks!
Sammer
On May 6, 2011, at 11:52 PM, Milind Bhandarkar <mb...@linkedin.com> wrote:
> [I am not on PMC, but seeing that PMC may be busy with other issues, I
> will try to answer your questions.]
>
> Eric,
>
> I think the thread
> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
> questions. Here is the timeline as I see it:
>
> 1. Arun proposes to create a release from the security patchset. Says Doug
> has proposed this earlier
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
> earlier by Doug and did not get far due to concerns about the effect this
> would have on development on trunk.") (August 24, 2010)
>
> 2. Lots of +1s, between August 24 to August 30 2010. One particular
> comment is from Tom White: "I think it would be good to have a shared 0.20
> Apache security branch.
> Since security isn't in 0.21, and the 0.22 release is a some way off
> as you mention, this would be useful for folks who want the security
> features sooner (and want to use an Apache release)."
>
> 3. Arun volunteers to create a release (August 30, 2010)
>
> 4. Doug reminds Arun. (October 15, 2010)
>
> 5. Arun apologizes for not creating a branch because he was busy, because
> he had a baby. (January 11, 2011)
>
> 6. Lots of discussion about what to call it (the release, not the baby,
> although I had a good laugh at Patrick Angeles's email: "You're gonna call
> your kid 20.100?" ;-).
>
> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
> something like 20.100 to show that it's a big jump? Anything else?" Jan
> 12, 2011
>
> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
> 12, 2011.
>
> So, as you can see, even if this release is called 0.20.x, the community
> agreed that these are valuable patches to have, and despite backward
> incompatibility, still have them in minor release.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>
>> On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>>
>> I understand Eli's concerns that putting stuff in there that hasn't gone
>> into trunk yet is danger. However, as the team makes no guarantees of 100%
>> compatibility between releases, I don't think it's critical. It's just
>> something that needs to be addressed -which can be done after this release
>> has shipped.
>>
>>
>> I was under the impression that the community has been extremely strict
>> about compatibility between minor version bumps in the past. I though
>> there
>> were specific guarantees and that was one of the reasons certain behaviors
>> have persisted so long.
>>
>> Does this mean API changes can be made in minor releases and it can be
>> made
>> backward compatible in future releases? That seems very, very counter to
>> various conversations that have happened in the past. I'm of the mind that
>> we should continue to promise what we've always promised and if that's
>> changing, let's make with the refactoring party!
>>
>> Can some PMC'ers clarify this one for me?
>>
>> TIA.
>> Sammer
>>
>>
>>
>> -Steve
>
Re: Release compatibility was Re: [VOTE] Release candidate
0.20.203.0-rc1
Posted by Milind Bhandarkar <mb...@linkedin.com>.
[I am not on PMC, but seeing that PMC may be busy with other issues, I
will try to answer your questions.]
Eric,
I think the thread
"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
questions. Here is the timeline as I see it:
1. Arun proposes to create a release from the security patchset. Says Doug
has proposed this earlier
(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
earlier by Doug and did not get far due to concerns about the effect this
would have on development on trunk.") (August 24, 2010)
2. Lots of +1s, between August 24 to August 30 2010. One particular
comment is from Tom White: "I think it would be good to have a shared 0.20
Apache security branch.
Since security isn't in 0.21, and the 0.22 release is a some way off
as you mention, this would be useful for folks who want the security
features sooner (and want to use an Apache release)."
3. Arun volunteers to create a release (August 30, 2010)
4. Doug reminds Arun. (October 15, 2010)
5. Arun apologizes for not creating a branch because he was busy, because
he had a baby. (January 11, 2011)
6. Lots of discussion about what to call it (the release, not the baby,
although I had a good laugh at Patrick Angeles's email: "You're gonna call
your kid 20.100?" ;-).
7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
something like 20.100 to show that it's a big jump? Anything else?" Jan
12, 2011
8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
12, 2011.
So, as you can see, even if this release is called 0.20.x, the community
agreed that these are valuable patches to have, and despite backward
incompatibility, still have them in minor release.
- milind
--
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167
On 5/6/11 11:14 PM, "Eric Sammer" <es...@cloudera.com> wrote:
>On May 6, 2011, at 4:53 AM, Steve Loughran <st...@apache.org> wrote:
>
>I understand Eli's concerns that putting stuff in there that hasn't gone
>into trunk yet is danger. However, as the team makes no guarantees of 100%
>compatibility between releases, I don't think it's critical. It's just
>something that needs to be addressed -which can be done after this release
>has shipped.
>
>
>I was under the impression that the community has been extremely strict
>about compatibility between minor version bumps in the past. I though
>there
>were specific guarantees and that was one of the reasons certain behaviors
>have persisted so long.
>
>Does this mean API changes can be made in minor releases and it can be
>made
>backward compatible in future releases? That seems very, very counter to
>various conversations that have happened in the past. I'm of the mind that
>we should continue to promise what we've always promised and if that's
>changing, let's make with the refactoring party!
>
>Can some PMC'ers clarify this one for me?
>
>TIA.
>Sammer
>
>
>
>-Steve