You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Arun C Murthy <ac...@yahoo-inc.com> on 2011/05/02 22:07:16 UTC

Re: [VOTE] Release candidate 0.20.203.0-rc0

Doug,

On May 2, 2011, at 10:58 AM, Doug Cutting wrote:
> The patch selection process for this branch did not appear to be a
> community process.  A massive patch set was committed en-masse with no
> public discussion before or after about its specific composition.

Lets review:

# You proposed to release off the Yahoo security patchset first in  
April, 2010: http://s.apache.org/5Gv
# I started this discussion again in Jan, 2011: http://s.apache.org/uf
# We went through several iterations:
  - I first committed a jumbo patch upon which some reservations were  
expressed.
  - Owen went ahead and broke them up to commit individual patches to  
incorporate the provided feedback.
# Roy clearly clarified the way forward: http://s.apache.org/tD4  
(which Owen has since incorporatedk by breaking into individual  
patches).

Your current stance given the history, is surprising, to say the  
least... we have already discussed this. It is clear that the  
community (including downstream Apache projects like Pig, Hive and  
HCatalog) will substantially benefit from an Apache release of this  
improved codebase.

thanks,
Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> We will work towards ensuring it is a complete superset of the last
> stable release: 0.20.2.

Great!  Who's 'we'?  Do you want any help with this?

Doug

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On May 2, 2011, at 2:49 PM, Ian Holsman wrote:

>
> On May 3, 2011, at 7:33 AM, Arun C Murthy wrote:
>
>>
>> This patchset started from 0.20.1 has is complete superset of 0.20.1.
>>
>> We will work towards ensuring it is a complete superset of the last  
>> stable release: 0.20.2.
>
> so are you intending to make it a superset for 203? or for a future  
> release?

For 0.20.203, it behooves us to incorporate feedback for that.

Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Ian Holsman <ha...@holsman.net>.
On May 3, 2011, at 7:33 AM, Arun C Murthy wrote:

> 
> This patchset started from 0.20.1 has is complete superset of 0.20.1.
> 
> We will work towards ensuring it is a complete superset of the last stable release: 0.20.2.

so are you intending to make it a superset for 203? or for a future release?

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> On May 2, 2011, at 2:21 PM, Doug Cutting wrote:
>> I have absolutely no objection in principle to an Apache 0.20 release
>> including security.  I object to the fact that this patchset started
>> from an arbitrary point and unilaterally applied a large set of patches
>> that are not well correlated with Jira, trunk or other 0.20 branches.
> 
> Completely untrue.

'Completely'?  Really?  Not a true bit in there?  Wow!

> This patchset started from 0.20.1 has is complete superset of 0.20.1.

0.20.1 isn't a branch, it's a tag.  The 0.20 branch includes many
post-0.20.1 patches that are not in this candidate.  Releases in a
series normally share a branch.

> I asked if you were willing to withdraw your objection if the technical
> concerns are satisfied. I think you answered my question - you will not
> withdraw your objection even if it's a technical issue.

That is not what I said.  If this release does not get enough votes then
perhaps another 0.20.203 release candidate will be proposed.  Its
process and contents will be different and I will judge it on the basis
of those when I vote.

Doug

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/03/2011 06:01 PM, Arun C Murthy wrote:
> On May 3, 2011, at 5:17 PM, "Doug Cutting" <cu...@apache.org> wrote:
> 
>> On 05/02/2011 02:33 PM, Arun C Murthy wrote:
>>> Are you simply asking for someone to go through the 450 odd jiras and
>>> set 'fix-for' fields?
>>
>> Every other release we've made is well-correlated with Jira.  It should
>> not be difficult to achieve that for this one.  We could write a script
>> to take all 450 bug IDs from the change log and use Jira's command-line
>> tool to set the "fix-for" to be this 0.20+security release.  Would you
>> like help with that?
>>
> 
> Yes please, that would be great. Thanks!

Please find below a script that will add a fix-version to issues.

Doug

#!/bin/bash

# reads bug ids from standard input
# and adds the fixVersion named on command line

if [ $# -eq 0 ]
then
  echo "Usage: $0 bugid"
  exit 1
fi

fix=$1
echo Setting fix version to $fix.

server=https://issues.apache.org/jira
jira=./jira-cli-2.0.0/jira.sh

set -e

echo -n "Jira username: "
read user
echo -n "Jira password: "
stty -echo
read password
stty echo

while read issue
do
    # first read the old fix versions
    old=`$jira -a getFieldValue --server $server \
         --password $password --user $user \
         --issue $issue --field fixVersions | \
	 tail -n 1 | sed 's/([0-9]*)//g' | sed s/\'//g`

    # now update, adding new value
    # jira will ignore if this value is already present
    $jira -a updateIssue --server $server \
     --password $password --user $user \
     --issue $issue --fixVersions "${old},${fix}"
done

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Arun C Murthy <ar...@yahoo-inc.com>.
On May 3, 2011, at 5:17 PM, "Doug Cutting" <cu...@apache.org> wrote:

> On 05/02/2011 02:33 PM, Arun C Murthy wrote:
>> Are you simply asking for someone to go through the 450 odd jiras and
>> set 'fix-for' fields?
> 
> Every other release we've made is well-correlated with Jira.  It should
> not be difficult to achieve that for this one.  We could write a script
> to take all 450 bug IDs from the change log and use Jira's command-line
> tool to set the "fix-for" to be this 0.20+security release.  Would you
> like help with that?
> 

Yes please, that would be great. Thanks!

Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> Are you simply asking for someone to go through the 450 odd jiras and
> set 'fix-for' fields?

Every other release we've made is well-correlated with Jira.  It should
not be difficult to achieve that for this one.  We could write a script
to take all 450 bug IDs from the change log and use Jira's command-line
tool to set the "fix-for" to be this 0.20+security release.  Would you
like help with that?

Doug

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On May 2, 2011, at 2:21 PM, Doug Cutting wrote:

> On 05/02/2011 02:05 PM, Arun C Murthy wrote:
>> As I noted before you were the first one to propose this release off
>> Yahoo security patch-set in April, 2010:
>> http://s.apache.org/5Gv
>>
>> What has changed since? Clearly, the same situation exists today.
>
> I have absolutely no objection in principle to an Apache 0.20 release
> including security.  I object to the fact that this patchset started
> from an arbitrary point and unilaterally applied a large set of  
> patches
> that are not well correlated with Jira, trunk or other 0.20 branches.

Completely untrue.

This patchset started from 0.20.1 has is complete superset of 0.20.1.

We will work towards ensuring it is a complete superset of the last  
stable release: 0.20.2.

>
>> Also, please note that of the ~450 commits in the branch, only 30 odd
>> jiras are yet to be committed to trunk:
>> http://s.apache.org/7Pe. So it's incorrect to state 'lack of  
>> community
>> involvement'.
>
> This should be easily discoverable from Jira: issues should use the
> "fix-for" field to indicate which branches they've been merged to.   
> This
> standard practice has not been observed for over 400 patches  
> included in
> this release candidate.
>

This seems like parliamentary stalling procedures... sure they don't  
have 'fix-for' fields but they've been verified to be true from  
external committers:

http://s.apache.org/yX

Are you simply asking for someone to go through the 450 odd jiras and  
set 'fix-for' fields?

>> Assuming the technical inconsistencies are sorted out, are you  
>> willing
>> to withdraw you objection?
>
> These are not just technical concerns.  How I vote on any future  
> release
> candidate will in part depend on how the community is involved in its
> production.
>

I understand they aren't technical concerns.

I asked if you were willing to withdraw your objection if the  
technical concerns are satisfied. I think you answered my question -  
you will not withdraw your objection even if it's a technical issue.

thanks,
Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 03:05 PM, Andrew Purtell wrote:
> What strikes me, as an observer to this discussion, is that here
> "community" does not seem equated with Yahoo by implication. Perhaps
> I misread. Nevertheless, Yahoo retains a good percentage of active
> Core developers with standing as both committers and high scale
> users, and these people produced the contribution that is branch
> 0.20.203, and therefore by definition "the community" was entirely
> involved in its production.

Whether or not a subset of contributors acts as the community depends on
whether others outside that subset have a reasonable opportunity to
become involved.  Until this release vote was called it wasn't entirely
clear to all what was happening with these branches.  Wider community
involvement is now starting as folks work to rationalize this 450-issue
patch with respect to past and future releases, Jira, etc.

> Yahoo should be commended for advancing the state of branch 0.20 with
> an obvious commitment to donating the results to Apache. As a
> community we are lucky to have a strong contributor. Their security
> enhancements allow us and many others the option of strong
> authentication and user isolation for multitenant deployments.

+1

> A commercial vendor's product already incorporates Yahoo's donated
> security enhancements. It would be regrettable if nontechnical
> factors ultimately prevents Apache from incorporating the value of
> these contributions into an official release.

+1

Cheers,

Doug

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Eli Collins <el...@cloudera.com>.
On Mon, May 2, 2011 at 3:15 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
>
> On May 2, 2011, at 3:05 PM, Andrew Purtell wrote:
>
>
>> Some technical concerns seem reasonable. Regarding that:
>>
>>> From: Stack <st...@duboce.net>
>>> How hard would it be to get the patches Tom lists below into
>>> branch-0.20-security-203?  I'd think it'd be an easier
>>> sell if it were a superset of all in 0.20, especially since it
>>> bears its name.
>>
>> This suggestion makes a lot of sense. In addition, filing JIRAs for and
>> posting the diffs of the remaining differences could help the process as
>> well, and would be good faith actions of an active contributor.
>>
>
> Agreed, I'm starting the effort to ensure the differences from 0.20.2 are
> resolved.
>
> From my msg on this thread to common-dev@:
>
>> # Remaining for 0.20.203
>>  * HADOOP-5611
>>  * HADOOP-5612
>>  * HADOOP-5623
>>  * HDFS-596
>>  * HDFS-723
>>  * HDFS-732
>>  * HDFS-579
>>  * MAPREDUCE-1070
>>  * HADOOP-6315
>>  * MAPREDUCE-1163
>
> Suresh has kindly agreed to help me, appreciate help from others -
> particularly on the 0.20.3 changes.
>

I'd like to help.  Perhaps let's coordinate in a separate thread?

Thanks,
Eli

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On May 2, 2011, at 3:05 PM, Andrew Purtell wrote:


> Some technical concerns seem reasonable. Regarding that:
>
>> From: Stack <st...@duboce.net>
>> How hard would it be to get the patches Tom lists below into
>> branch-0.20-security-203?  I'd think it'd be an easier
>> sell if it were a superset of all in 0.20, especially since it
>> bears its name.
>
> This suggestion makes a lot of sense. In addition, filing JIRAs for  
> and posting the diffs of the remaining differences could help the  
> process as well, and would be good faith actions of an active  
> contributor.
>

Agreed, I'm starting the effort to ensure the differences from 0.20.2  
are resolved.

 From my msg on this thread to common-dev@:

> # Remaining for 0.20.203
>  * HADOOP-5611
>  * HADOOP-5612
>  * HADOOP-5623
>  * HDFS-596
>  * HDFS-723
>  * HDFS-732
>  * HDFS-579
>  * MAPREDUCE-1070
>  * HADOOP-6315
>  * MAPREDUCE-1163

Suresh has kindly agreed to help me, appreciate help from others -  
particularly on the 0.20.3 changes.

thanks,
Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Andrew Purtell <ap...@apache.org>.
I would like to make one minor but important clarification:

From:

> It would be regrettable if
> nontechnical factors ultimately prevents Apache from
> incorporating the value of these contributions into an
> official release.

To:

It would be regrettable if nontechnical factors ultimately prevents Apache from  incorporating the value of these contributions into an official release OF 0.20. There are some not yet ready to take the leap to 0.22; who do not consider it proven.  

So in this regard I do not wish to minimize concerns about distracting from the success of 0.22 or later releases. 

> From: Andrew Purtell <ap...@apache.org>
> Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
> To: general@hadoop.apache.org
> Date: Monday, May 2, 2011, 3:05 PM

> Most points in this thread are valid,
> having to do with the process of how the contribution was
> assembled; and specific technical aspects of it, e.g. JIRAs
> missing from branch 0.20.203 relative to branch 0.20.
> However,
> 
> > > From: Doug Cutting <cu...@apache.org>
> > > Assuming the technical inconsistencies are sorted
> > > out, are you willing to withdraw you objection?
> > 
> > These are not just technical concerns.  How I vote on
> > any future release candidate will in part depend on how
> > the community is involved in its production.
> 
> What strikes me, as an observer to this discussion, is that
> here "community" does not seem equated with Yahoo by
> implication. Perhaps I misread. Nevertheless, Yahoo retains
> a good percentage of active Core developers with standing as
> both committers and high scale users, and these people
> produced the contribution that is branch 0.20.203, and
> therefore by definition "the community" was entirely
> involved in its production.
> 
> Yahoo should be commended for advancing the state of branch
> 0.20 with an obvious commitment to donating the results to
> Apache. As a community we are lucky to have a strong
> contributor. Their security enhancements allow us and many
> others the option of strong authentication and user
> isolation for multitenant deployments. 
> 
> A commercial vendor's product already incorporates Yahoo's
> donated security enhancements. It would be regrettable if
> nontechnical factors ultimately prevents Apache from
> incorporating the value of these contributions into an
> official release.
> 
> Some technical concerns seem reasonable. Regarding that:
> 
> > From: Stack <st...@duboce.net>
> > How hard would it be to get the patches Tom lists
> below into
> > branch-0.20-security-203?  I'd think it'd be an
> easier
> > sell if it were a superset of all in 0.20, especially
> since it
> > bears its name.
> 
> This suggestion makes a lot of sense. In addition, filing
> JIRAs for and posting the diffs of the remaining differences
> could help the process as well, and would be good faith
> actions of an active contributor.
> 
> Best regards,
> 
>     - Andy
> 
> Problems worthy of attack prove their worth by hitting
> back. - Piet Hein (via Tom White)
> 
>

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Andrew Purtell <ap...@apache.org>.
Most points in this thread are valid, having to do with the process of how the contribution was assembled; and specific technical aspects of it, e.g. JIRAs missing from branch 0.20.203 relative to branch 0.20. However,

> From: Doug Cutting <cu...@apache.org>
> > Assuming the technical inconsistencies are sorted out,
> > are you willing to withdraw you objection?
> 
> These are not just technical concerns.  How I vote on any future
> release candidate will in part depend on how the community is
> involved in its production.

What strikes me, as an observer to this discussion, is that here "community" does not seem equated with Yahoo by implication. Perhaps I misread. Nevertheless, Yahoo retains a good percentage of active Core developers with standing as both committers and high scale users, and these people produced the contribution that is branch 0.20.203, and therefore by definition "the community" was entirely involved in its production.

Yahoo should be commended for advancing the state of branch 0.20 with an obvious commitment to donating the results to Apache. As a community we are lucky to have a strong contributor. Their security enhancements allow us and many others the option of strong authentication and user isolation for multitenant deployments. 

A commercial vendor's product already incorporates Yahoo's donated security enhancements. It would be regrettable if nontechnical factors ultimately prevents Apache from incorporating the value of these contributions into an official release.

Some technical concerns seem reasonable. Regarding that:

> From: Stack <st...@duboce.net>
> How hard would it be to get the patches Tom lists below into
> branch-0.20-security-203?  I'd think it'd be an easier
> sell if it were a superset of all in 0.20, especially since it
> bears its name.

This suggestion makes a lot of sense. In addition, filing JIRAs for and posting the diffs of the remaining differences could help the process as well, and would be good faith actions of an active contributor.

Best regards,

    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 02:05 PM, Arun C Murthy wrote:
> As I noted before you were the first one to propose this release off
> Yahoo security patch-set in April, 2010:
> http://s.apache.org/5Gv
> 
> What has changed since? Clearly, the same situation exists today.

I have absolutely no objection in principle to an Apache 0.20 release
including security.  I object to the fact that this patchset started
from an arbitrary point and unilaterally applied a large set of patches
that are not well correlated with Jira, trunk or other 0.20 branches.

> Also, please note that of the ~450 commits in the branch, only 30 odd
> jiras are yet to be committed to trunk:
> http://s.apache.org/7Pe. So it's incorrect to state 'lack of community
> involvement'.

This should be easily discoverable from Jira: issues should use the
"fix-for" field to indicate which branches they've been merged to.  This
standard practice has not been observed for over 400 patches included in
this release candidate.

> Assuming the technical inconsistencies are sorted out, are you willing
> to withdraw you objection?

These are not just technical concerns.  How I vote on any future release
candidate will in part depend on how the community is involved in its
production.

Thanks,

Doug

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Doug,

On May 2, 2011, at 1:40 PM, Doug Cutting wrote:
>
> Also note that, on the common-dev thread, Eli & Tom have both noted a
> number of inconsistencies between this set of patches and trunk, 0.22
> and even prior 0.20 branches and releases.  In addition to the lack of
> community involvement in patch selection, these issues concern me.
>
> I cannot in good conscience vote for this release as a community  
> product.

As I noted before you were the first one to propose this release off  
Yahoo security patch-set in April, 2010:
http://s.apache.org/5Gv

What has changed since? Clearly, the same situation exists today.

Also, please note that of the ~450 commits in the branch, only 30 odd  
jiras are yet to be committed to trunk:
http://s.apache.org/7Pe. So it's incorrect to state 'lack of community  
involvement'.

Assuming the technical inconsistencies are sorted out, are you willing  
to withdraw you objection?

thanks,
Arun

Re: [VOTE] Release candidate 0.20.203.0-rc0

Posted by Doug Cutting <cu...@apache.org>.
On 05/02/2011 01:07 PM, Arun C Murthy wrote:
> # Roy clearly clarified the way forward: http://s.apache.org/tD4 (which
> Owen has since incorporated by breaking into individual patches).

Roy suggested a three ways forward and possible outcomes:

Roy Fielding wrote:
>  a) break the changes down into a sequence of patches, create jira
>     issues for each one (or append to the existing issue), and then
>     provide the group with a list of the issue links so that people
>     can quickly +1 each one.  When it seems worthwhile to you, create
>     a branch off of some prior Apache release point in svn and commit
>     each patch to it until the branch is identical to (or, in your own
>     opinion, better than) the source code that you have tested locally.
>     Then RM a tarball and start a release vote.  Since all of this is
>     being done in jira and svn, others can help you do all but the
>     first part (breaking down the big patch).
> 
> or
> 
>  b) create a branch off of some prior Apache release point in svn
>     and replay the internal Y! commits on that branch until the branch
>     source code is identical to what you have tested locally.  Then
>     RM a tarball based on that branch and start a release vote.
>     Since the history is now in svn, others could do the RM bit if
>     you don't have time.
> 
> or
> 
>  c) create a branch off of some prior Apache release point in svn
>     and apply one big ugly patch to it.  Then RM a tarball based
>     on that branch and ask for a release vote.
> 
> You will note that none of the above requires a discussion on this
> list prior to the release vote, though (a) would likely result in
> more +1s than (b), and (b) would likely receive more +1s than (c).
> Regardless, the release vote is a lazy majority decision.
>
>  [ ... ]
>
> When the release vote happens, encourage folks to test and +1
> the release.  If it passes, woohoo!  If not, then listen to the
> reasons given by the other PMC members and see if you can make
> enough changes to the release to get those extra +1s.

I believe that Owen chose (b).  We're now at the release vote and I am a
PMC member giving reasons for my vote.

Also note that, on the common-dev thread, Eli & Tom have both noted a
number of inconsistencies between this set of patches and trunk, 0.22
and even prior 0.20 branches and releases.  In addition to the lack of
community involvement in patch selection, these issues concern me.

I cannot in good conscience vote for this release as a community product.

Doug