Community Process for 0.20.205 Sustaining Release

[Matt Foley <mf...@hortonworks.com>]

Dear all,
0.20.204.0 will hopefully finish the release process soon, and it is time to
start talking about 205.  As Owen mentioned,
I would like to volunteer as the Release Manager for 0.20.2xx, if that is
acceptable to the community.  I would also like
to suggest some process changes.

By continuing to provide sustaining releases of 0.20-security, the community
helps production users until later versions (v22
and v23) reach stability.  At the same time, we certainly do not wish
0.20-security to be viewed as a "trunk"; it is important
that all patches go in trunk first, and only patches of manageable risk and
high value to production users, should go into
the sustaining releases.

The goal of the following is to assure adequate community participation in
the release process, rather than just a bunch of
Jiras followed by last-minute debate in a Vote thread :-)

It seems that the default assumption has been that any patch committed to
0.20-security would go into the next sustaining
release.  In theory, contributors and committers have the opportunity to
comment in the Jiras if they disagreed.  However,
in practice most people don't follow Jiras for patches other than trunk, and
the number of such patches can add up.

So, to start discussion for the 205 release, I propose the following:

1. Let us look at the list of patches applied to 0.20-security so far, since
204 (see attached document).

2. Contributors and production users who want the various patches should
speak up for them, providing information such as
    - Jira number and title
    - Value to production users
    - Risk and Testability

3. If there are other patches desired in 205 that have not yet been
committed to 0.20-security, potential contributors should
also speak up, providing the same information, and opening new Jiras if
necessary.

Hopefully over the coming week or so we can share all the inputs, and people
will have time to review them and comment.

After the 205 release, I hope we can discuss potential content for 206 in
advance.  Perhaps we can also circulate some
improvements to the Sustaining Release process currently documented in the
wiki.

Thoughts and feedback welcome,
thanks,
--Matt

Re: Community Process for 0.20.205 Sustaining Release

["Roy T. Fielding" <fi...@gbiv.com>]

On Aug 23, 2011, at 5:17 PM, Matt Foley wrote:

> Dear all,
> 0.20.204.0 will hopefully finish the release process soon, and it is time to start talking about 205.  As Owen mentioned, 
> I would like to volunteer as the Release Manager for 0.20.2xx, if that is acceptable to the community.  I would also like 
> to suggest some process changes.
> 
> By continuing to provide sustaining releases of 0.20-security, the community helps production users until later versions (v22 
> and v23) reach stability.  At the same time, we certainly do not wish 0.20-security to be viewed as a "trunk"; it is important 
> that all patches go in trunk first, and only patches of manageable risk and high value to production users, should go into 
> the sustaining releases.
> 
> The goal of the following is to assure adequate community participation in the release process, rather than just a bunch of 
> Jiras followed by last-minute debate in a Vote thread :-) 

Note that the mechanism for encouraging community participation is
very much in the hands of the RM.  So, just do it.  For example,
the file you provided is much like the original STATUS file
that I added to httpd development a long time ago, though I
strongly suggest you commit it to subversion instead of expecting
people to track it in email.  For that matter, I thought Jira has
the ability to track multiple releases already -- why not use the
tracking system directly?

I do feel a need to remind folks: releases are NOT a consensus
decision, but the specific changes allowed within a given release
can be vetoed with a valid technical reason that may be specific
to that release branch (e.g., compatibility concerns are almost
always scoped by version branch).  You should be asking folks to
make any negative opinions known as well, since that is critical
to pruning the tree down to the set of patches that can actually
be placed in front of the group to vote on as a release candidate.

....Roy

Re: Community Process for 0.20.205 Sustaining Release

[Mi...@emc.com]

>
>
>
>It still has to be asked whether the changes meet the criterion of
>manageable risk.  That definitely should be evaluated
>after we see the proposed patches.

Matt,

As a release manager, it is up to you to decide what the criterion for
inclusion is. And that was what I was asking.

- milind

Re: Community Process for 0.20.205 Sustaining Release

[Matt Foley <mf...@hortonworks.com>]

Hi Milind,
Thanks for your response.  If you'll forgive me for reiterating statements
that you've probably heard before, I'll try to
concisely state why the community might consider accepting changes for HBase
compatibility into 0.20.2xx.

It is my understanding that HBase works correctly in trunk (soon to be
branched as v23), with both security and append.
Therefore, making changes in 0.20.2xx to enable HBase to work there, would
not constitute an innovation or new feature
that isn't already in trunk.  Nor is it a new feature relative to 0.20,
since HBase currently works with (and is recommended for
installation with) 0.20-append, which was branched from 0.20.2, and
developed quite a while back.

There are production users of HBase that want to stay with 0.20, since later
releases with security have not yet reached
stability, but also want the security features of 0.20.2xx.  To meet their
needs, it seems reasonable to consider joining
HBase-compatibility features like those of 0.20-append with the
0.20-security branch.  That doesn't seem to be using
0.20-security as a "trunk", although the semantics of that word can of
course be argued.  And it seems to meet the
criterion of high value to production users.

It still has to be asked whether the changes meet the criterion of
manageable risk.  That definitely should be evaluated
after we see the proposed patches.

Does that address your questions?

Regarding the definition of "manageable risk and high value to production
users", I think the community has to decide
that on a case-by-case basis.  Certainly the production users are capable
of expressing their opinions for what they want.
Developers should (and no doubt will) also do so, in the Jiras and in email
threads.

But the most important suggestion in my previous email is that we provide a
forum for such discussions PRE-release,
"up front", rather than only during the release vote.

Best,
--Matt

On Wed, Aug 24, 2011 at 12:05 AM, <Mi...@emc.com> wrote:

> > At the same time, we certainly do not wish 0.20-security to be viewed as
> a "trunk"; it is important
> that all patches go in trunk first, and only patches of manageable risk and
> high value to production users, should go into
> the sustaining releases.
>
> Matt,
>
> With all due respect, I have heard from "several of your associates", about
> features for making hbase work with the 0.20.2xx. That sounds to me that
> 0.20-security to be trunk.
>
> Can you clarify how that is going to work ?
>
> Basically, what are your criteria for "manageable risk and high value to
> production users" ?
>
> In particular, I would like to know why the insistence on 0.20.2xx being
> the default branch to check-in these "innovations", instead of trunk.
>
>
>  *   Milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or present,
> the author might be affiliated with.)
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Chris Douglas <cd...@apache.org>]

There is some confusion, here.

Nigel is the RM for the current 0.22 branch. If he abandons it then
that branch is done, but nobody has a lock on the next release. Any
committer can manage a branch, package its contents, and propose an
artifact. If it receives a majority of votes from the PMC, then it
becomes 0.22. Whether it's derived from the current 0.22 branch is up
to that RM.

Anyone keen to see a particular release must do the work to effect it.
We don't assign work by voting.

On Tue, Sep 6, 2011 at 12:08 PM,  <Mi...@emc.com> wrote:
> AKAIK, the bylaws do not mention anything about "abandoning release". If
> past is any indication, even if releases 0.19.*, and 0.21.* were not
> adopted by large hadoop installations, these releases were cut, voted on,
> and approved. Not "abandoned".

To your point, a branch targeted as 0.21 was abandoned, despite months
of backporting changes from trunk. A branch containing subsequent work
was released instead. Whether the current 0.23 branch is released as
0.22 will depend exclusively on their respective effort and support.

Though you're right: this probably should be written into the bylaws,
even if any contrary model is illusory in practice. -C

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Allen Wittenauer <aw...@apache.org>]

On Sep 7, 2011, at 9:15 AM, Arun C Murthy wrote:
> Using real data helps - from Apache Jira, here are the statistics for work on trunk/hadoop-0.23 in Q3 of 2011 (i.e. last 2 months alone):
> 
> Hadoop Common - 224 resolved* jiras
> Hadoop HDFS - 153 resolved* jiras
> Hadoop MapReduce - 161* resolved jiras
> 
> Total of 538 jiras resolved. I'm sure LOC is much more impressive, but be that as it may.

	I wonder about the usefulness of those stats though.  Many of those jiras are bug fixes to other jiras committed in the same time frame and also committed to other branches.  Also, I've noticed an absolutely explosion of sub-tasks where one big patch is actually done at commit time.

> So, I'd say it's pretty clear that there is significant interest, involvement & investment from the wider community. That makes me believe that Apache Hadoop is moving along the right path i.e. forward.

	Part of the problem that the Hadoop community has is a PMC and committer group that leans heavily towards a few organizations.  Any movement that those organizations do offsets what the rest of the community may or may not want to do.  The reality is that if anyone who isn't HortonWorks or Cloudera wants to do a release, it is likely doomed at PMC vote time.

	[I'll hold off on commenting on the effectiveness of our current PMC member roster.  That's a different discussion altogether.]  

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Arun C Murthy <ac...@hortonworks.com>]

Konstantin,

On Sep 7, 2011, at 2:04 AM, Konstantin Shvachko wrote:


>  I think continuing with 0.20
> a-three-years-old-technology is not the best place to invest
> resources. In the past you advocated for 0.21 and 0.22, both now
> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
> to see this backward/forward porting going on forever, diverging the
> Apache Hadoop project from natural evolutionary process.

Using real data helps - from Apache Jira, here are the statistics for work on trunk/hadoop-0.23 in Q3 of 2011 (i.e. last 2 months alone):

Hadoop Common - 224 resolved* jiras
Hadoop HDFS - 153 resolved* jiras
Hadoop MapReduce - 161* resolved jiras

Total of 538 jiras resolved. I'm sure LOC is much more impressive, but be that as it may.

OTOH, I've haven't run reports (can't figure), but a cursory glance of CHANGES.txt shows hadoop-0.20.2xx has < 50, while 0.22 has ~10.

So, I'd say it's pretty clear that there is significant interest, involvement & investment from the wider community. That makes me believe that Apache Hadoop is moving along the right path i.e. forward.

thanks,
Arun

* I'm aware that 80% of stats are only 80% right *smile* - these are 'resolved' jiras. It's fair to assume vast majority of the jiras were actually 'fixed'.

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Steve Loughran <st...@apache.org>]

The other reason for a 0.22 release is Apache Bigtop only plans to 
release full stacks of released ASF code, and a lot of the things in the 
ASF Hadoop ecosystem do need the 0.21+ APIs (MRUnit, flume). A 0.22 
release is something that bigtop could get behind.

-steve

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

On Fri, Sep 09, 2011 at 12:21PM, Chris Douglas wrote:
> On Fri, Sep 9, 2011 at 11:08 AM, Eli Collins <el...@cloudera.com> wrote:
> > The release manager - not the developers - are responsible for and
> > have the final say as to what patches get merge to their branch. ═If
> > the RM wants all this work they need to either corral the developers
> > to do the merging or do the merging themselves. In short, it's their
> > responsibility to get people to invest in the branch.
> 
> This.
> 
> The rest of this thread is pointless. -C

Then I musta've been imagining things ;\

Cos

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Shvachko <sh...@gmail.com>]

> I benchmarked 22 v/s 20.xxx there was >5x difference and that was more than a year ago.

Arun do you mind sharing the benchmarks that you ran?

Thanks,
--Konstatnin

On Fri, Sep 9, 2011 at 11:26 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Joep,
>
> On Sep 9, 2011, at 10:41 AM, Rottinghuis, Joep wrote:
>
>
> No one is going to block you from doing any work you want.
>
> All that is required is to have the work in trunk and subsequent branches i.e 0.23 (as applicable) .
>
> The problem is that 0.22 hasn't seen major movement for almost a year since it was branched and there is no incentive for lots of people to contribute - plus there is MRv2 which is completely different beast (see the discussion that Eli pointed to).
>
> None of this is to say you shouldn't contribute or use 0.22, you move the project as you wish by your contributions.
>
>>> MR1 is being maintained on 20x. In fact 20x is the only MR1 code that supports security and disk failure handling. The MR1 code in 22 is a regression in some significant aspects (features, performance, bugs) from the latest stable MR1 (204).
>> ...
>>
>> Eli, aside from the disk failure handling which is a new feature in 205 (not present in earlier 20x releases), could you please elaborate on which other significant aspects 22 would regress from 20x?
>>
>
> I've talked to Konstantin about this.
>
> There is tonnes of performance work missing, including scaling work on JobTracker, CapacityScheduler etc. There is work to add a ton of limits (counters, tasks, etc. etc.). Then there is operability work such as JobHistory, handling logs etc. The last time I benchmarked 22 v/s 20.xxx there was >5x difference and that was more than a year ago. Arguably some of the operability work won't matter for small clusters, but you are welcome to make your own decisions.
>
> It's unfortunate we have landed here, but 22 branched almost a year ago and hence none of this work was ported there since a branch implies critical bugs was supposed to go in. That plus the problems with scaling MR1 which led to investment in MR1 is where we are. As a result, there is no enthusiasm to contribute to MR1 from vast majority of devs given that we've decided we won't support it.
>
>
>>> My understanding of the way Apache operates is that you can't do things like "declare blocks on upgrade paths". People can try to release updates to 21 or 22 (or some new tree). Ie the decisions are made implicitly by where people invest cycles.
>>
>> If a group of committers say that they'll commit to trunk, to 0.23 and to 0.20x, but not to 0.22, then in effect that is like to "declare a block on upgrade path" isn't it? The more such commits that go in to other branches, but not the ones in between essentially is a declaration of a block, because of the very regression argument you make.
>>
>> I agree that nobody can be made to contribute to something they don't want, but does that result into a split?
>> In other words, if a significant bug fix or feature goes into trunk and into 22, can developers then simply say: "I'm not interested to put this into 23, you do this yourself if you want?". Will that be tolerated or vetoed?
>>
>
> Again, no one is going to veto anything. As Eli said decisions are make by people's code.
>
> Arun
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Arun C Murthy <ac...@hortonworks.com>]

Joep,

On Sep 9, 2011, at 10:41 AM, Rottinghuis, Joep wrote:


No one is going to block you from doing any work you want. 

All that is required is to have the work in trunk and subsequent branches i.e 0.23 (as applicable) .

The problem is that 0.22 hasn't seen major movement for almost a year since it was branched and there is no incentive for lots of people to contribute - plus there is MRv2 which is completely different beast (see the discussion that Eli pointed to).

None of this is to say you shouldn't contribute or use 0.22, you move the project as you wish by your contributions.

>> MR1 is being maintained on 20x. In fact 20x is the only MR1 code that supports security and disk failure handling. The MR1 code in 22 is a regression in some significant aspects (features, performance, bugs) from the latest stable MR1 (204).
> ...
> 
> Eli, aside from the disk failure handling which is a new feature in 205 (not present in earlier 20x releases), could you please elaborate on which other significant aspects 22 would regress from 20x? 
> 

I've talked to Konstantin about this.

There is tonnes of performance work missing, including scaling work on JobTracker, CapacityScheduler etc. There is work to add a ton of limits (counters, tasks, etc. etc.). Then there is operability work such as JobHistory, handling logs etc. The last time I benchmarked 22 v/s 20.xxx there was >5x difference and that was more than a year ago. Arguably some of the operability work won't matter for small clusters, but you are welcome to make your own decisions.

It's unfortunate we have landed here, but 22 branched almost a year ago and hence none of this work was ported there since a branch implies critical bugs was supposed to go in. That plus the problems with scaling MR1 which led to investment in MR1 is where we are. As a result, there is no enthusiasm to contribute to MR1 from vast majority of devs given that we've decided we won't support it. 


>> My understanding of the way Apache operates is that you can't do things like "declare blocks on upgrade paths". People can try to release updates to 21 or 22 (or some new tree). Ie the decisions are made implicitly by where people invest cycles.
> 
> If a group of committers say that they'll commit to trunk, to 0.23 and to 0.20x, but not to 0.22, then in effect that is like to "declare a block on upgrade path" isn't it? The more such commits that go in to other branches, but not the ones in between essentially is a declaration of a block, because of the very regression argument you make.
> 
> I agree that nobody can be made to contribute to something they don't want, but does that result into a split?
> In other words, if a significant bug fix or feature goes into trunk and into 22, can developers then simply say: "I'm not interested to put this into 23, you do this yourself if you want?". Will that be tolerated or vetoed?
> 

Again, no one is going to veto anything. As Eli said decisions are make by people's code.

Arun

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Chris Douglas <cd...@apache.org>]

On Fri, Sep 9, 2011 at 11:08 AM, Eli Collins <el...@cloudera.com> wrote:
> The release manager - not the developers - are responsible for and
> have the final say as to what patches get merge to their branch.  If
> the RM wants all this work they need to either corral the developers
> to do the merging or do the merging themselves. In short, it's their
> responsibility to get people to invest in the branch.

This.

The rest of this thread is pointless. -C

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

On Fri, Sep 09, 2011 at 06:17PM, Milind.Bhandarkar@emc.com wrote:
> Never mind. It was an oblique reference to client not writing to a file
> for a long time, so hdfs recovering the lease. (RM=client, release=file,
> recovery=transferring RM role. :-)

Very clever ;) I don't think there's a competition. There's room for more, I
am sure!

Cos

> - milind
> 
> On 9/9/11 2:58 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> 
> >On Fri, Sep 09, 2011 at 05:55PM, Milind.Bhandarkar@emc.com wrote:
> >> But did he do the lease recovery (for more info: HDFS-265) ? I haven
> >>seen
> >> the initiation of lease recovery by Konst, but haven't seen acks.
> >
> >I am not sure I follow you, Milind. HDFS_265 has been in since 0.21. Are
> >you
> >referring to some new development that I might've missed?
> >
> >Cos
> >
> >> - milind 
> >> 
> >> On 9/9/11 2:13 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> >> 
> >> >KOnstantin has stepped forward a couple of days ago ;)
> >> >
> >> >On Fri, Sep 09, 2011 at 05:08PM, Milind.Bhandarkar@emc.com wrote:
> >> >> Has anyone seen the RM for 0.22 lately ?
> >> >> 
> >> >> - milind
> >> >> 
> >> >> On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:
> >> >> 
> >> >> >On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
> >> >> >> The release manager - not the developers - are responsible for and
> >> >> >> have the final say as to what patches get merge to their branch.
> >> >> >
> >> >> >That is simply false.  At no time is any individual responsible for
> >> >> >any part of Apache subversion, no matter how obscure the branch.
> >> >> >No RM has the final say on anything other than their own work.
> >> >> >That is, they can choose not to produce a release candidate.
> >> >> >Furthermore, at no time whatsoever does any person "own" the job
> >> >> >of being RM -- there can be five active RMs on a single branch,
> >> >> >each producing release candidates based on what is in subversion
> >> >> >at the particular time that they decide to tag and build.
> >> >> >
> >> >> >> If the RM wants all this work they need to either corral the
> >> >>developers
> >> >> >> to do the merging or do the merging themselves. In short, it's
> >>their
> >> >> >> responsibility to get people to invest in the branch.
> >> >> >
> >> >> >That is true of anyone, on or off the PMC -- not just the RM.
> >> >> >
> >> >> >....Roy
> >> >> >
> >> >> >
> >> >> 
> >> >
> >> 
> >
> 

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 2:58 PM, Konstantin Boudnik <co...@apache.org> wrote:
> On Fri, Sep 09, 2011 at 05:55PM, Milind.Bhandarkar@emc.com wrote:
>> But did he do the lease recovery (for more info: HDFS-265) ? I haven seen
>> the initiation of lease recovery by Konst, but haven't seen acks.
>
> I am not sure I follow you, Milind. HDFS_265 has been in since 0.21. Are you
> referring to some new development that I might've missed?
>

Milind is saying that Konst volunteered but Nigel hasn't taken him up on it yet.

According to Roy any committer can RM so there's no lease to recover.

Thanks,
Eli

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Mi...@emc.com]

Never mind. It was an oblique reference to client not writing to a file
for a long time, so hdfs recovering the lease. (RM=client, release=file,
recovery=transferring RM role. :-)

- milind

On 9/9/11 2:58 PM, "Konstantin Boudnik" <co...@apache.org> wrote:

>On Fri, Sep 09, 2011 at 05:55PM, Milind.Bhandarkar@emc.com wrote:
>> But did he do the lease recovery (for more info: HDFS-265) ? I haven
>>seen
>> the initiation of lease recovery by Konst, but haven't seen acks.
>
>I am not sure I follow you, Milind. HDFS_265 has been in since 0.21. Are
>you
>referring to some new development that I might've missed?
>
>Cos
>
>> - milind 
>> 
>> On 9/9/11 2:13 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
>> 
>> >KOnstantin has stepped forward a couple of days ago ;)
>> >
>> >On Fri, Sep 09, 2011 at 05:08PM, Milind.Bhandarkar@emc.com wrote:
>> >> Has anyone seen the RM for 0.22 lately ?
>> >> 
>> >> - milind
>> >> 
>> >> On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:
>> >> 
>> >> >On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>> >> >> The release manager - not the developers - are responsible for and
>> >> >> have the final say as to what patches get merge to their branch.
>> >> >
>> >> >That is simply false.  At no time is any individual responsible for
>> >> >any part of Apache subversion, no matter how obscure the branch.
>> >> >No RM has the final say on anything other than their own work.
>> >> >That is, they can choose not to produce a release candidate.
>> >> >Furthermore, at no time whatsoever does any person "own" the job
>> >> >of being RM -- there can be five active RMs on a single branch,
>> >> >each producing release candidates based on what is in subversion
>> >> >at the particular time that they decide to tag and build.
>> >> >
>> >> >> If the RM wants all this work they need to either corral the
>> >>developers
>> >> >> to do the merging or do the merging themselves. In short, it's
>>their
>> >> >> responsibility to get people to invest in the branch.
>> >> >
>> >> >That is true of anyone, on or off the PMC -- not just the RM.
>> >> >
>> >> >....Roy
>> >> >
>> >> >
>> >> 
>> >
>> 
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

On Fri, Sep 09, 2011 at 05:55PM, Milind.Bhandarkar@emc.com wrote:
> But did he do the lease recovery (for more info: HDFS-265) ? I haven seen
> the initiation of lease recovery by Konst, but haven't seen acks.

I am not sure I follow you, Milind. HDFS_265 has been in since 0.21. Are you
referring to some new development that I might've missed?

Cos

> - milind 
> 
> On 9/9/11 2:13 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> 
> >KOnstantin has stepped forward a couple of days ago ;)
> >
> >On Fri, Sep 09, 2011 at 05:08PM, Milind.Bhandarkar@emc.com wrote:
> >> Has anyone seen the RM for 0.22 lately ?
> >> 
> >> - milind
> >> 
> >> On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:
> >> 
> >> >On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
> >> >> The release manager - not the developers - are responsible for and
> >> >> have the final say as to what patches get merge to their branch.
> >> >
> >> >That is simply false.  At no time is any individual responsible for
> >> >any part of Apache subversion, no matter how obscure the branch.
> >> >No RM has the final say on anything other than their own work.
> >> >That is, they can choose not to produce a release candidate.
> >> >Furthermore, at no time whatsoever does any person "own" the job
> >> >of being RM -- there can be five active RMs on a single branch,
> >> >each producing release candidates based on what is in subversion
> >> >at the particular time that they decide to tag and build.
> >> >
> >> >> If the RM wants all this work they need to either corral the
> >>developers
> >> >> to do the merging or do the merging themselves. In short, it's their
> >> >> responsibility to get people to invest in the branch.
> >> >
> >> >That is true of anyone, on or off the PMC -- not just the RM.
> >> >
> >> >....Roy
> >> >
> >> >
> >> 
> >
> 

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Mi...@emc.com]

But did he do the lease recovery (for more info: HDFS-265) ? I haven seen
the initiation of lease recovery by Konst, but haven't seen acks.

- milind 

On 9/9/11 2:13 PM, "Konstantin Boudnik" <co...@apache.org> wrote:

>KOnstantin has stepped forward a couple of days ago ;)
>
>On Fri, Sep 09, 2011 at 05:08PM, Milind.Bhandarkar@emc.com wrote:
>> Has anyone seen the RM for 0.22 lately ?
>> 
>> - milind
>> 
>> On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:
>> 
>> >On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>> >> The release manager - not the developers - are responsible for and
>> >> have the final say as to what patches get merge to their branch.
>> >
>> >That is simply false.  At no time is any individual responsible for
>> >any part of Apache subversion, no matter how obscure the branch.
>> >No RM has the final say on anything other than their own work.
>> >That is, they can choose not to produce a release candidate.
>> >Furthermore, at no time whatsoever does any person "own" the job
>> >of being RM -- there can be five active RMs on a single branch,
>> >each producing release candidates based on what is in subversion
>> >at the particular time that they decide to tag and build.
>> >
>> >> If the RM wants all this work they need to either corral the
>>developers
>> >> to do the merging or do the merging themselves. In short, it's their
>> >> responsibility to get people to invest in the branch.
>> >
>> >That is true of anyone, on or off the PMC -- not just the RM.
>> >
>> >....Roy
>> >
>> >
>> 
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

KOnstantin has stepped forward a couple of days ago ;)

On Fri, Sep 09, 2011 at 05:08PM, Milind.Bhandarkar@emc.com wrote:
> Has anyone seen the RM for 0.22 lately ?
> 
> - milind
> 
> On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:
> 
> >On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
> >> The release manager - not the developers - are responsible for and
> >> have the final say as to what patches get merge to their branch.
> >
> >That is simply false.  At no time is any individual responsible for
> >any part of Apache subversion, no matter how obscure the branch.
> >No RM has the final say on anything other than their own work.
> >That is, they can choose not to produce a release candidate.
> >Furthermore, at no time whatsoever does any person "own" the job
> >of being RM -- there can be five active RMs on a single branch,
> >each producing release candidates based on what is in subversion
> >at the particular time that they decide to tag and build.
> >
> >> If the RM wants all this work they need to either corral the developers
> >> to do the merging or do the merging themselves. In short, it's their
> >> responsibility to get people to invest in the branch.
> >
> >That is true of anyone, on or off the PMC -- not just the RM.
> >
> >....Roy
> >
> >
> 

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 4:42 PM, Chris Douglas <cd...@apache.org> wrote:
> On Fri, Sep 9, 2011 at 2:41 PM, Eli Collins <el...@cloudera.com> wrote:
>> For completeness, Chris proposed a while back [1] that the RM is
>> selected by majority PMC approval. He never proposed a vote to adopt
>> these rules, but if we did, would they be invalid, ie according to
>> other rules established at Apache?  Ie does this project have the
>> ability to establish it's own rules/norms w/o you telling us how our
>> project works?
>>
>> 1. http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E
>
> The intent was to evolve the policies we had at the time to an RM-like
> role. As Tom raised in that thread, much of the intermediate phase is
> unnecessary. In the project's current state, much of that proposal is
> no longer coherent.
>
> For instance, electing an RM is an unnecessary formality. Further,
> enforcing the version compatibility rules across the 0.20.2xx and
> various 0.2x branches is prohibitively expensive. As demonstrated by
> previous experience, making prohibitively expensive rules motivates
> external forks, not coherence. They're useful guidelines for what the
> PMC is likely to approve, but I expect we'll show more flexibility
> than what that proposal outlined.
>
> In practice, we already exercise all the important elements of that
> proposal. Someone creates a branch and intends to release from it,
> others may help them, and if an artifact is produced then the PMC
> votes on whether to release it. Any debate, accusation, and
> recrimination concerning "investing" in a branch is pointless because
> the "result" of the debate is irrelevant. Other than venting some
> spleen, this thread has no functional consequence.
>
> On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
>>> No RM has the final say on anything other than their own work.
>
> That's consistent with the position as forwarded. We're using "the
> RM's branch" as a shorthand for "whatever the RM has elected to
> include." Your comments highlight nuance, not fundamental dissonance.
>
>> I think there's a huge gap between the current understanding of the RM
>> in our community and what you've outlined.
>
> It's vanishingly small, but important. Ownership of a branch isn't
> vested in an RM, neither is it transferrable. If someone wanted to
> commit something to a branch, they aren't required to ask the RM. Now,
> it's *polite*, and I hope that most would give a heads-up for anything
> they were unsure of, but the repository isn't a hierarchy of
> dictatorships. The source tree is lock-free. -C
>

I think the main point is that RMs correspond to releases, not a
release branch. This distinction makes more sense, eg I think it would
be good to have a set of people RM the dot releases off a given branch
instead of an RM for a branch and a series of releases from it (what
it seems like we've been doing).

Thanks,
Eli

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

["Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>]

Chris,

Well put, +1.

Cheers,
Chris

On Sep 9, 2011, at 4:42 PM, Chris Douglas wrote:

> On Fri, Sep 9, 2011 at 2:41 PM, Eli Collins <el...@cloudera.com> wrote:
>> For completeness, Chris proposed a while back [1] that the RM is
>> selected by majority PMC approval. He never proposed a vote to adopt
>> these rules, but if we did, would they be invalid, ie according to
>> other rules established at Apache?  Ie does this project have the
>> ability to establish it's own rules/norms w/o you telling us how our
>> project works?
>> 
>> 1. http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E
> 
> The intent was to evolve the policies we had at the time to an RM-like
> role. As Tom raised in that thread, much of the intermediate phase is
> unnecessary. In the project's current state, much of that proposal is
> no longer coherent.
> 
> For instance, electing an RM is an unnecessary formality. Further,
> enforcing the version compatibility rules across the 0.20.2xx and
> various 0.2x branches is prohibitively expensive. As demonstrated by
> previous experience, making prohibitively expensive rules motivates
> external forks, not coherence. They're useful guidelines for what the
> PMC is likely to approve, but I expect we'll show more flexibility
> than what that proposal outlined.
> 
> In practice, we already exercise all the important elements of that
> proposal. Someone creates a branch and intends to release from it,
> others may help them, and if an artifact is produced then the PMC
> votes on whether to release it. Any debate, accusation, and
> recrimination concerning "investing" in a branch is pointless because
> the "result" of the debate is irrelevant. Other than venting some
> spleen, this thread has no functional consequence.
> 
> On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
>>> No RM has the final say on anything other than their own work.
> 
> That's consistent with the position as forwarded. We're using "the
> RM's branch" as a shorthand for "whatever the RM has elected to
> include." Your comments highlight nuance, not fundamental dissonance.
> 
>> I think there's a huge gap between the current understanding of the RM
>> in our community and what you've outlined.
> 
> It's vanishingly small, but important. Ownership of a branch isn't
> vested in an RM, neither is it transferrable. If someone wanted to
> commit something to a branch, they aren't required to ask the RM. Now,
> it's *polite*, and I hope that most would give a heads-up for anything
> they were unsure of, but the repository isn't a hierarchy of
> dictatorships. The source tree is lock-free. -C


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Chris Douglas <cd...@apache.org>]

On Fri, Sep 9, 2011 at 2:41 PM, Eli Collins <el...@cloudera.com> wrote:
> For completeness, Chris proposed a while back [1] that the RM is
> selected by majority PMC approval. He never proposed a vote to adopt
> these rules, but if we did, would they be invalid, ie according to
> other rules established at Apache?  Ie does this project have the
> ability to establish it's own rules/norms w/o you telling us how our
> project works?
>
> 1. http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E

The intent was to evolve the policies we had at the time to an RM-like
role. As Tom raised in that thread, much of the intermediate phase is
unnecessary. In the project's current state, much of that proposal is
no longer coherent.

For instance, electing an RM is an unnecessary formality. Further,
enforcing the version compatibility rules across the 0.20.2xx and
various 0.2x branches is prohibitively expensive. As demonstrated by
previous experience, making prohibitively expensive rules motivates
external forks, not coherence. They're useful guidelines for what the
PMC is likely to approve, but I expect we'll show more flexibility
than what that proposal outlined.

In practice, we already exercise all the important elements of that
proposal. Someone creates a branch and intends to release from it,
others may help them, and if an artifact is produced then the PMC
votes on whether to release it. Any debate, accusation, and
recrimination concerning "investing" in a branch is pointless because
the "result" of the debate is irrelevant. Other than venting some
spleen, this thread has no functional consequence.

On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
>> No RM has the final say on anything other than their own work.

That's consistent with the position as forwarded. We're using "the
RM's branch" as a shorthand for "whatever the RM has elected to
include." Your comments highlight nuance, not fundamental dissonance.

> I think there's a huge gap between the current understanding of the RM
> in our community and what you've outlined.

It's vanishingly small, but important. Ownership of a branch isn't
vested in an RM, neither is it transferrable. If someone wanted to
commit something to a branch, they aren't required to ask the RM. Now,
it's *polite*, and I hope that most would give a heads-up for anything
they were unsure of, but the repository isn't a hierarchy of
dictatorships. The source tree is lock-free. -C

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

["Roy T. Fielding" <fi...@gbiv.com>]

On Sep 9, 2011, at 2:41 PM, Eli Collins wrote:

> On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
>> On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>>> The release manager - not the developers - are responsible for and
>>> have the final say as to what patches get merge to their branch.
>> 
>> That is simply false.  At no time is any individual responsible for
>> any part of Apache subversion, no matter how obscure the branch.
> 
> This seems to contradict the Apache release policy
> (http://httpd.apache.org/dev/release.html), and the practice adopted
> by this project. Eg "Regarding what makes it into a release, the RM is
> the unquestioned authority.  No one can contest what makes it into the
> release."

I think someone got carried away in their artistic flair.  I wrote the
original httpd guidelines.

> If there is no individual responsible for a release branch then who is
> "the RM" that is the unquestioned authority for a release?
> 
> The page states that "there is no set RM" however in our project
> Nigel, Arun, and Matt volunteered to RM 22, 23 and 20x respectively.

Volunteers are always needed.  That doesn't mean they are the only
volunteers, nor does it mean they have special veto powers.

> Are these people in fact not responsible for their release branches in
> svn, eg any committer is free to merge a patch from trunk into one of
> these branches?

Any committer can commit wherever they have been given permission
to commit by the PMC.  Generally, they do so collaboratively.
I've never encountered a situation in my own projects which developers
were committing at cross-purposes, even when they disagree on content,
though I've seen commit wars elsewhere.  We'd expect the PMC
to step in if they did.

The only thing the RM has authority over is the building of a source
package, based on the contents of our subversion, that can then be
put up for vote.  They can decide what snapshot to tag for a build.
They can decide not to build anything at all.  They can also do all sorts
of organizational support, advocacy, pleading, or whatever in order to
encourage the rest of the project committers to apply changes, vote
for things under issue, etc.

They do not have the right to pick and select whatever variation
of the product they might like to build, short of vetoing (with a
valid reason) any changes that they as a PMC member believe do not
belong on the branch. Likewise, the RM cannot include in the build
any change that has been vetoed by others, and their build cannot
be released if it contains any such changes that have been vetoed
since it was built.  The RM has the right to kill their own build
if they learn something during the release process that they think,
for whatever reason, causes the build to be unreleasable.  But the
RM can't stop anyone else on the PMC from taking the same build
and calling for its release under their own management as RM.

> For completeness, Chris proposed a while back [1] that the RM is
> selected by majority PMC approval. He never proposed a vote to adopt
> these rules, but if we did, would they be invalid, ie according to
> other rules established at Apache?  Ie does this project have the
> ability to establish it's own rules/norms w/o you telling us how our
> project works?
> 
> 1. http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E

The ASF supports collaborative development of open source through
the work of individual volunteers.  If the rules are consistent with
that mission and are applied consistently, then the board is unlikely
to intervene.

As a board member, what I look for is the effect that those rules
have on collaboration.  If I see a bunch of happy developers
collaborating on releases, it really doesn't matter what the rules
are since the rules exist to prevent technical disagreements from
escalating into social conflicts.  If, however, I see no significant
releases, work being done elsewhere, or the project split into
multiple branches that happen to match corporate fiefdoms, then
it is my responsibility to do something to get it back to being
a collaboration.  Sometimes that means we change the rules.

....Roy

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
> On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>> The release manager - not the developers - are responsible for and
>> have the final say as to what patches get merge to their branch.
>
> That is simply false.  At no time is any individual responsible for
> any part of Apache subversion, no matter how obscure the branch.

This seems to contradict the Apache release policy
(http://httpd.apache.org/dev/release.html), and the practice adopted
by this project. Eg "Regarding what makes it into a release, the RM is
the unquestioned authority.  No one can contest what makes it into the
release."

If there is no individual responsible for a release branch then who is
"the RM" that is the unquestioned authority for a release?

The page states that "there is no set RM" however in our project
Nigel, Arun, and Matt volunteered to RM 22, 23 and 20x respectively.
Are these people in fact not responsible for their release branches in
svn, eg any committer is free to merge a patch from trunk into one of
these branches?

For completeness, Chris proposed a while back [1] that the RM is
selected by majority PMC approval. He never proposed a vote to adopt
these rules, but if we did, would they be invalid, ie according to
other rules established at Apache?  Ie does this project have the
ability to establish it's own rules/norms w/o you telling us how our
project works?

1. http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E

> No RM has the final say on anything other than their own work.

The release policy - "Regarding what makes it into a release, the RM
is the unquestioned authority"  - seems to indicate "the RM" has final
say about what makes it into a release. This implies that either the
release is just the RM's work (vs the work of the community) or in
fact the RM is not the unquestioned authority on a release.

I think there's a huge gap between the current understanding of the RM
in our community and what you've outlined.

Thanks,
Eli

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 1:46 PM, Roy T. Fielding <fi...@gbiv.com> wrote:
> On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>> The release manager - not the developers - are responsible for and
>> have the final say as to what patches get merge to their branch.
>
> That is simply false.  At no time is any individual responsible for
> any part of Apache subversion, no matter how obscure the branch.
> No RM has the final say on anything other than their own work.

It seems hard for an RM to be the final authority of what makes it
into a release w/o also being the final authority on what patches get
merged into the branch they're trying to drive a release from.

Thanks,
Eli


> That is, they can choose not to produce a release candidate.
> Furthermore, at no time whatsoever does any person "own" the job
> of being RM -- there can be five active RMs on a single branch,
> each producing release candidates based on what is in subversion
> at the particular time that they decide to tag and build.
>
>> If the RM wants all this work they need to either corral the developers
>> to do the merging or do the merging themselves. In short, it's their
>> responsibility to get people to invest in the branch.
>
> That is true of anyone, on or off the PMC -- not just the RM.
>
> ....Roy
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Mi...@emc.com]

Has anyone seen the RM for 0.22 lately ?

- milind

On 9/9/11 1:46 PM, "Roy T. Fielding" <fi...@gbiv.com> wrote:

>On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
>> The release manager - not the developers - are responsible for and
>> have the final say as to what patches get merge to their branch.
>
>That is simply false.  At no time is any individual responsible for
>any part of Apache subversion, no matter how obscure the branch.
>No RM has the final say on anything other than their own work.
>That is, they can choose not to produce a release candidate.
>Furthermore, at no time whatsoever does any person "own" the job
>of being RM -- there can be five active RMs on a single branch,
>each producing release candidates based on what is in subversion
>at the particular time that they decide to tag and build.
>
>> If the RM wants all this work they need to either corral the developers
>> to do the merging or do the merging themselves. In short, it's their
>> responsibility to get people to invest in the branch.
>
>That is true of anyone, on or off the PMC -- not just the RM.
>
>....Roy
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

["Roy T. Fielding" <fi...@gbiv.com>]

On Sep 9, 2011, at 11:08 AM, Eli Collins wrote:
> The release manager - not the developers - are responsible for and
> have the final say as to what patches get merge to their branch.

That is simply false.  At no time is any individual responsible for
any part of Apache subversion, no matter how obscure the branch.
No RM has the final say on anything other than their own work.
That is, they can choose not to produce a release candidate.
Furthermore, at no time whatsoever does any person "own" the job
of being RM -- there can be five active RMs on a single branch,
each producing release candidates based on what is in subversion
at the particular time that they decide to tag and build.

> If the RM wants all this work they need to either corral the developers
> to do the merging or do the merging themselves. In short, it's their
> responsibility to get people to invest in the branch.

That is true of anyone, on or off the PMC -- not just the RM.

....Roy

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 10:41 AM, Rottinghuis, Joep
<jr...@ebay.com> wrote:
>
>
> -----Original Message-----
> From: Eli Collins [mailto:eli@cloudera.com]
> Sent: Friday, September 09, 2011 10:03 AM
> To: general@hadoop.apache.org
> Subject: Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release
>
> ...
>
>> MR1 is being maintained on 20x. In fact 20x is the only MR1 code that supports security and disk failure handling. The MR1 code in 22 is a regression in some significant aspects (features, performance, bugs) from the latest stable MR1 (204).
> ...
>
> Eli, aside from the disk failure handling which is a new feature in 205 (not present in earlier 20x releases), could you please elaborate on which other significant aspects 22 would regress from 20x?

Check out the MR jiras in the branch-20-security change log. There's a
ton of performance, feature and stability work. 22 doesn't have most
of this.


>
>> My understanding of the way Apache operates is that you can't do things like "declare blocks on upgrade paths". People can try to release updates to 21 or 22 (or some new tree). Ie the decisions are made implicitly by where people invest cycles.
>
> If a group of committers say that they'll commit to trunk, to 0.23 and to 0.20x, but not to 0.22, then in effect that is like to "declare a block on upgrade path" isn't it?

The release manager - not the developers - are responsible for and
have the final say as to what patches get merge to their branch.  If
the RM wants all this work they need to either corral the developers
to do the merging or do the merging themselves. In short, it's their
responsibility to get people to invest in the branch.

Thanks,
Eli

RE: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

["Rottinghuis, Joep" <jr...@ebay.com>]

-----Original Message-----
From: Eli Collins [mailto:eli@cloudera.com] 
Sent: Friday, September 09, 2011 10:03 AM
To: general@hadoop.apache.org
Subject: Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

...

> MR1 is being maintained on 20x. In fact 20x is the only MR1 code that supports security and disk failure handling. The MR1 code in 22 is a regression in some significant aspects (features, performance, bugs) from the latest stable MR1 (204).
...

Eli, aside from the disk failure handling which is a new feature in 205 (not present in earlier 20x releases), could you please elaborate on which other significant aspects 22 would regress from 20x? 

> My understanding of the way Apache operates is that you can't do things like "declare blocks on upgrade paths". People can try to release updates to 21 or 22 (or some new tree). Ie the decisions are made implicitly by where people invest cycles.

If a group of committers say that they'll commit to trunk, to 0.23 and to 0.20x, but not to 0.22, then in effect that is like to "declare a block on upgrade path" isn't it? The more such commits that go in to other branches, but not the ones in between essentially is a declaration of a block, because of the very regression argument you make.

I agree that nobody can be made to contribute to something they don't want, but does that result into a split?
In other words, if a significant bug fix or feature goes into trunk and into 22, can developers then simply say: "I'm not interested to put this into 23, you do this yourself if you want?". Will that be tolerated or vetoed?

Thanks,

Joep

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eli Collins <el...@cloudera.com>]

On Fri, Sep 9, 2011 at 3:40 AM, Steve Loughran <st...@apache.org> wrote:
> On 07/09/11 10:04, Konstantin Shvachko wrote:
>>
>> Eric,
>>
>> It would take the same amount of resources to fix 0.22 as to merge
>> append and security branches aka 0.20.205.
>> Although I understand that Hortonworks needs to support its
>> customer(s) and is eager to bridge the gap in functionality with its
>> competitor(s), I think continuing with 0.20
>> a-three-years-old-technology is not the best place to invest
>> resources. In the past you advocated for 0.21 and 0.22, both now
>> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
>> to see this backward/forward porting going on forever, diverging the
>> Apache Hadoop project from natural evolutionary process.
>>
>> I think 0.22 has all the functionality required to run Hadoop for most
>> production tasks. I see enough momentum and involvement in the
>> community with 0.22 testing. I think there will be enough resources to
>> get it stabilized in near future.
>
> this is interesting.
>
> 1. I've been doing some 0.20.x work and hitting bugs that I know have been
> fixed in trunk a while ago but never backported as they were things that
> weren't critical enough to the people using the 20.x branches (i.e problems
> related to my home network, issues w/ embedding the JARs, etc).
>
> This is why I have to disagree with eric14's "age is irrelevant" claim. The
> APIs show their age, so do other quirks. It's just a known set of quirks
> -like WindowsXP is today.
>
> 2. 0.23 will take a while to stabilise; a big barrier is that projects on
> top of hadoop need to test it. Bigtop can help here, but it will still take
> time.
>
> 3. Where is all maintenance of MR 1.0 code going to go? It can't go in trunk
> or 0.23, as that's on MR2.x. Should all changes to MR1.0 be backported to
> 0.20.x, and new stuff put in there?

MR1 is being maintained on 20x. In fact 20x is the only MR1 code that
supports security and disk failure handling. The MR1 code in 22 is a
regression in some significant aspects (features, performance, bugs)
from the latest stable MR1 (204).

You might find this discussion relevant:

http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201107.mbox/%3CCAPn_vTsdiiqfCB2G0HfsOr3W_4PKoocPcTf2VB93Y3MZrzRczQ@mail.gmail.com%3E

http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201107.mbox/%3CCAPn_vTsdiiqfCB2G0HfsOr3W_4PKoocPcTf2VB93Y3MZrzRczQ@mail.gmail.com%3E

>
> Or are we declaring a complete block on all upgrade paths that don't involve
> a migration to 0.23 or staying on the 0.20.x branch -with only a subset of
> fixes and an aging API- available?

My understanding of the way Apache operates is that you can't do
things like "declare blocks on upgrade paths". People can try to
release updates to 21 or 22 (or some new tree). Ie the decisions are
made implicitly by where people invest cycles.

Thanks,
Eli

>
> -Steve
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Steve Loughran <st...@apache.org>]

On 07/09/11 10:04, Konstantin Shvachko wrote:
> Eric,
>
> It would take the same amount of resources to fix 0.22 as to merge
> append and security branches aka 0.20.205.
> Although I understand that Hortonworks needs to support its
> customer(s) and is eager to bridge the gap in functionality with its
> competitor(s), I think continuing with 0.20
> a-three-years-old-technology is not the best place to invest
> resources. In the past you advocated for 0.21 and 0.22, both now
> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
> to see this backward/forward porting going on forever, diverging the
> Apache Hadoop project from natural evolutionary process.
>
> I think 0.22 has all the functionality required to run Hadoop for most
> production tasks. I see enough momentum and involvement in the
> community with 0.22 testing. I think there will be enough resources to
> get it stabilized in near future.

this is interesting.

1. I've been doing some 0.20.x work and hitting bugs that I know have 
been fixed in trunk a while ago but never backported as they were things 
that weren't critical enough to the people using the 20.x branches (i.e 
problems related to my home network, issues w/ embedding the JARs, etc).

This is why I have to disagree with eric14's "age is irrelevant" claim. 
The APIs show their age, so do other quirks. It's just a known set of 
quirks -like WindowsXP is today.

2. 0.23 will take a while to stabilise; a big barrier is that projects 
on top of hadoop need to test it. Bigtop can help here, but it will 
still take time.

3. Where is all maintenance of MR 1.0 code going to go? It can't go in 
trunk or 0.23, as that's on MR2.x. Should all changes to MR1.0 be 
backported to 0.20.x, and new stuff put in there?

Or are we declaring a complete block on all upgrade paths that don't 
involve a migration to 0.23 or staying on the 0.20.x branch -with only a 
subset of fixes and an aging API- available?

-Steve

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eric Baldeschwieler <er...@hortonworks.com>]

Hi Konstantine,

I've never advocated 21 or 22 since I've never been able to volunteer
help to work on them.

I think it's great if you want to run another branch, do testing etc.
But given that it's neither complete, stable nor the latest majority
supported project I think the expectation should be that the RM /
community behind 22 should take responsibility for backporting
whatever patches they care about from trunk or 23.

(just like 0.20.20*)

Also just like 20 the age is irrelevant, what maters is what folks are
volunteering to work on. I support you volunteering to do whatever
interests you. I'm marshaling help for 20 & 23 because these seem like
the best ways to address the needs of the widest community IMO.

Cheers,
---
E14 - typing on glass

On Sep 7, 2011, at 2:04 AM, Konstantin Shvachko <sh...@gmail.com> wrote:

> Eric,
>
> It would take the same amount of resources to fix 0.22 as to merge
> append and security branches aka 0.20.205.
> Although I understand that Hortonworks needs to support its
> customer(s) and is eager to bridge the gap in functionality with its
> competitor(s), I think continuing with 0.20
> a-three-years-old-technology is not the best place to invest
> resources. In the past you advocated for 0.21 and 0.22, both now
> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
> to see this backward/forward porting going on forever, diverging the
> Apache Hadoop project from natural evolutionary process.
>
> I think 0.22 has all the functionality required to run Hadoop for most
> production tasks. I see enough momentum and involvement in the
> community with 0.22 testing. I think there will be enough resources to
> get it stabilized in near future.
>
> Nigel,
>
> your comment can be understood as a request to commit important fixes
> to the 0.22 branch. I agree with that. But if you choose to abandon
> the RM role I will volunteer to take it over.
>
> Thanks,
> --Konstantin
>
> On Tue, Sep 6, 2011 at 4:06 AM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>>
>> What do others think?
>>
>> IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.
>>
>> E14
>>
>> On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
>>
>>> Matt, Others,
>>>
>>> Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?
>>>
>>> If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
>>>
>>> Cheers,
>>> Nige
>>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Steve Loughran <st...@apache.org>]

On 12/09/11 05:52, Nigel Daley wrote:
>> Nigel,
>>
>> your comment can be understood as a request to commit important fixes
>> to the 0.22 branch. I agree with that. But if you choose to abandon
>> the RM role I will volunteer to take it over.
>>
>> Thanks,
>> --Konstantin
>
> [catching up on the week's email]
>
> Konstantin, you are free to carry on this 0.22 work and call a release vote if you wish.
>
> To me it is becoming clearer every week that 0.22 continues to regress in significant ways from 0.20.20x releases which will cause further confusion in our user base.  Thus I will no longer manage a release from the 0.22 branch.  When 0.20.200 was proposed, I thought this would be the only such release and regressions to 0.22 would be manageable.  This is no longer the case now that there have been many 0.20.20x releases with significant changes and committers have not been merging these changes to intervening branches as was once common practice.
>
> At the current time, I believe the most value that we (the developer community) have derived from 0.22 was finding/fixing issues throughout the spring timeframe that also existed on trunk.  Many thanks to those that ran early versions of 0.22, filled issues and provided fixes.  Trunk, and thus 0.23, are better for it.
>

Well, if that's the case I'm going to start pushing some existing and 
new stuff into 0.20.x

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Shvachko <sh...@gmail.com>]

> Many thanks to those that ran early versions of 0.22, filled issues and provided fixes.

This is one of the three reasons I think 0.22 release is important.
1. A lot of community work has been put into the branch. Developers
were developing, committers were committing, and I heard many
enthusiastic people participated in hackathons. It would be such a
waste.
2. We do not have an Apache release supporting HBase. 0.20.205 is
contemplated to be one. But it will not allow mixed workload. Same as
with other append-based Hadoop versions people will have to use
dedicated HBase clusters separate from "general purpose" clusters.
0.22 is a better choice in this respect.
3. Timing. With 0.23 being de facto a rewrite of Hadoop - both of MR
and HDFS - its stabilization may take longer than anticipated. I
believe 0.22 can be released fairly soon.

I was pleased with recent testing of the 0.22 branch. I plan to set up
a 100-node cluster to test the build with other Hadoop components next
week.

I will very much appreciate any help from the community as I don't
have an army to marshal.

Thanks,
--Konstantin

On Sun, Sep 11, 2011 at 9:52 PM, Nigel Daley <nd...@mac.com> wrote:
>> Nigel,
>>
>> your comment can be understood as a request to commit important fixes
>> to the 0.22 branch. I agree with that. But if you choose to abandon
>> the RM role I will volunteer to take it over.
>>
>> Thanks,
>> --Konstantin
>
> [catching up on the week's email]
>
> Konstantin, you are free to carry on this 0.22 work and call a release vote if you wish.
>
> To me it is becoming clearer every week that 0.22 continues to regress in significant ways from 0.20.20x releases which will cause further confusion in our user base.  Thus I will no longer manage a release from the 0.22 branch.  When 0.20.200 was proposed, I thought this would be the only such release and regressions to 0.22 would be manageable.  This is no longer the case now that there have been many 0.20.20x releases with significant changes and committers have not been merging these changes to intervening branches as was once common practice.
>
> At the current time, I believe the most value that we (the developer community) have derived from 0.22 was finding/fixing issues throughout the spring timeframe that also existed on trunk.  Many thanks to those that ran early versions of 0.22, filled issues and provided fixes.  Trunk, and thus 0.23, are better for it.
>
> Cheers,
> Nige
>
>
> On Sep 7, 2011, at 2:04 AM, Konstantin Shvachko wrote:
>
>> Eric,
>>
>> It would take the same amount of resources to fix 0.22 as to merge
>> append and security branches aka 0.20.205.
>> Although I understand that Hortonworks needs to support its
>> customer(s) and is eager to bridge the gap in functionality with its
>> competitor(s), I think continuing with 0.20
>> a-three-years-old-technology is not the best place to invest
>> resources. In the past you advocated for 0.21 and 0.22, both now
>> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
>> to see this backward/forward porting going on forever, diverging the
>> Apache Hadoop project from natural evolutionary process.
>>
>> I think 0.22 has all the functionality required to run Hadoop for most
>> production tasks. I see enough momentum and involvement in the
>> community with 0.22 testing. I think there will be enough resources to
>> get it stabilized in near future.
>>
>> Nigel,
>>
>> your comment can be understood as a request to commit important fixes
>> to the 0.22 branch. I agree with that. But if you choose to abandon
>> the RM role I will volunteer to take it over.
>>
>> Thanks,
>> --Konstantin
>>
>> On Tue, Sep 6, 2011 at 4:06 AM, Eric Baldeschwieler
>> <er...@hortonworks.com> wrote:
>>>
>>> What do others think?
>>>
>>> IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.
>>>
>>> E14
>>>
>>> On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
>>>
>>>> Matt, Others,
>>>>
>>>> Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?
>>>>
>>>> If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
>>>>
>>>> Cheers,
>>>> Nige
>>>
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Nigel Daley <nd...@mac.com>]

> Nigel,
> 
> your comment can be understood as a request to commit important fixes
> to the 0.22 branch. I agree with that. But if you choose to abandon
> the RM role I will volunteer to take it over.
> 
> Thanks,
> --Konstantin

[catching up on the week's email]

Konstantin, you are free to carry on this 0.22 work and call a release vote if you wish.  

To me it is becoming clearer every week that 0.22 continues to regress in significant ways from 0.20.20x releases which will cause further confusion in our user base.  Thus I will no longer manage a release from the 0.22 branch.  When 0.20.200 was proposed, I thought this would be the only such release and regressions to 0.22 would be manageable.  This is no longer the case now that there have been many 0.20.20x releases with significant changes and committers have not been merging these changes to intervening branches as was once common practice.

At the current time, I believe the most value that we (the developer community) have derived from 0.22 was finding/fixing issues throughout the spring timeframe that also existed on trunk.  Many thanks to those that ran early versions of 0.22, filled issues and provided fixes.  Trunk, and thus 0.23, are better for it.

Cheers,
Nige


On Sep 7, 2011, at 2:04 AM, Konstantin Shvachko wrote:

> Eric,
> 
> It would take the same amount of resources to fix 0.22 as to merge
> append and security branches aka 0.20.205.
> Although I understand that Hortonworks needs to support its
> customer(s) and is eager to bridge the gap in functionality with its
> competitor(s), I think continuing with 0.20
> a-three-years-old-technology is not the best place to invest
> resources. In the past you advocated for 0.21 and 0.22, both now
> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
> to see this backward/forward porting going on forever, diverging the
> Apache Hadoop project from natural evolutionary process.
> 
> I think 0.22 has all the functionality required to run Hadoop for most
> production tasks. I see enough momentum and involvement in the
> community with 0.22 testing. I think there will be enough resources to
> get it stabilized in near future.
> 
> Nigel,
> 
> your comment can be understood as a request to commit important fixes
> to the 0.22 branch. I agree with that. But if you choose to abandon
> the RM role I will volunteer to take it over.
> 
> Thanks,
> --Konstantin
> 
> On Tue, Sep 6, 2011 at 4:06 AM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
>> 
>> What do others think?
>> 
>> IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.
>> 
>> E14
>> 
>> On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
>> 
>>> Matt, Others,
>>> 
>>> Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?
>>> 
>>> If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
>>> 
>>> Cheers,
>>> Nige
>> 

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

On Wed, Sep 07, 2011 at 02:04AM, Konstantin Shvachko wrote:
> Eric,
> 
> It would take the same amount of resources to fix 0.22 as to merge
> append and security branches aka 0.20.205.
> Although I understand that Hortonworks needs to support its
> customer(s) and is eager to bridge the gap in functionality with its
> competitor(s), I think continuing with 0.20
> a-three-years-old-technology is not the best place to invest
> resources. In the past you advocated for 0.21 and 0.22, both now
> abandoned by your team(s) in favor of enhancing 0.20. It will be sad
> to see this backward/forward porting going on forever, diverging the
> Apache Hadoop project from natural evolutionary process.
> 
> I think 0.22 has all the functionality required to run Hadoop for most
> production tasks. I see enough momentum and involvement in the
> community with 0.22 testing. I think there will be enough resources to
> get it stabilized in near future.
> 
> Nigel,
> 
> your comment can be understood as a request to commit important fixes
> to the 0.22 branch. I agree with that. But if you choose to abandon
> the RM role I will volunteer to take it over.

I think this is a grand idea! 0.22 is very close to the stable state. On the
other hand there's no indication of how soon the community can expect to get
0.23 out.

I will be happy to offer any help to complete 0.22, Konstantin!

Cos

> Thanks,
> --Konstantin
> 
> On Tue, Sep 6, 2011 at 4:06 AM, Eric Baldeschwieler
> <er...@hortonworks.com> wrote:
> >
> > What do others think?
> >
> > IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.
> >
> > E14
> >
> > On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
> >
> > > Matt, Others,
> > >
> > > Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?
> > >
> > > If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
> > >
> > > Cheers,
> > > Nige
> >

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Konstantin Shvachko <sh...@gmail.com>]

Eric,

It would take the same amount of resources to fix 0.22 as to merge
append and security branches aka 0.20.205.
Although I understand that Hortonworks needs to support its
customer(s) and is eager to bridge the gap in functionality with its
competitor(s), I think continuing with 0.20
a-three-years-old-technology is not the best place to invest
resources. In the past you advocated for 0.21 and 0.22, both now
abandoned by your team(s) in favor of enhancing 0.20. It will be sad
to see this backward/forward porting going on forever, diverging the
Apache Hadoop project from natural evolutionary process.

I think 0.22 has all the functionality required to run Hadoop for most
production tasks. I see enough momentum and involvement in the
community with 0.22 testing. I think there will be enough resources to
get it stabilized in near future.

Nigel,

your comment can be understood as a request to commit important fixes
to the 0.22 branch. I agree with that. But if you choose to abandon
the RM role I will volunteer to take it over.

Thanks,
--Konstantin

On Tue, Sep 6, 2011 at 4:06 AM, Eric Baldeschwieler
<er...@hortonworks.com> wrote:
>
> What do others think?
>
> IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.
>
> E14
>
> On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
>
> > Matt, Others,
> >
> > Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?
> >
> > If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
> >
> > Cheers,
> > Nige
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Brian Bockelman <bb...@cse.unl.edu>]

On Sep 6, 2011, at 2:08 PM, <Mi...@emc.com> wrote:

> I am very interested in hearing what the community thinks about this
> issue. I believe what happens here has long-term consequences.
> 
> AKAIK, the bylaws do not mention anything about "abandoning release". If
> past is any indication, even if releases 0.19.*, and 0.21.* were not
> adopted by large hadoop installations, these releases were cut, voted on,
> and approved. Not "abandoned".
> 
> I am -1 (non-binding) on abandoning 0.22.
> 

I would note that folks will use the latest release if it has features they want.  I know of about 8PB of HDFS deployments that used 0.19.  Bugs were found, bugs were squashed, patches were committed.

It would be nice to not waste the 0.22 effort, even if we call it a "development release" in the end.

Brian

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Mi...@emc.com]

I am very interested in hearing what the community thinks about this
issue. I believe what happens here has long-term consequences.

AKAIK, the bylaws do not mention anything about "abandoning release". If
past is any indication, even if releases 0.19.*, and 0.21.* were not
adopted by large hadoop installations, these releases were cut, voted on,
and approved. Not "abandoned".

I am -1 (non-binding) on abandoning 0.22.

- Milind

---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)



On 9/6/11 4:06 AM, "Eric Baldeschwieler" <er...@hortonworks.com> wrote:

>What do others think?
>
>IMO 22 is not the best place to invest resources. I support nigel's
>suggestion of abandoning it, but people are free to work on what they are
>passionate about.
>
>E14
>
>On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:
>
>> Matt, Others,
>> 
>> Is the expectation that these fixes go into other release branches too
>>(including 0.22, 0.23) if applicable?
>> 
>> If not, my concern is that 0.22 is regressing further from the popular
>>Apache release and I'm inclined to abandon 0.22. Thoughts?
>> 
>> Cheers,
>> Nige
>
>

Re: abandoning 22 - was: Content request for 0.20.205 Sustaining Release

[Eric Baldeschwieler <er...@hortonworks.com>]

What do others think?

IMO 22 is not the best place to invest resources. I support nigel's suggestion of abandoning it, but people are free to work on what they are passionate about.

E14

On Sep 3, 2011, at 11:11 AM, Nigel Daley wrote:

> Matt, Others,
> 
> Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?  
> 
> If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?
> 
> Cheers,
> Nige

Re: Content request for 0.20.205 Sustaining Release

[Nigel Daley <nd...@mac.com>]

Matt, Others,

Is the expectation that these fixes go into other release branches too (including 0.22, 0.23) if applicable?  

If not, my concern is that 0.22 is regressing further from the popular Apache release and I'm inclined to abandon 0.22. Thoughts?

Cheers,
Nige

On Sep 2, 2011, at 6:40 PM, Arun Murthy <ac...@hortonworks.com> wrote:

> Agree, +1.
> 
> It will be good to ensure we get a more modern version of
> FairScheduler for interested folks.
> 
> Arun
> 
> Sent from my iPhone
> 
> On Sep 2, 2011, at 5:55 PM, Matei Zaharia <ma...@eecs.berkeley.edu> wrote:
> 
>> Hi Matt,
>> 
>> I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.
>> 
>> Matei
>> 
>> On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:
>> 
>>> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
>>> 
>>> Arun
>>> 
>>> On Sep 1, 2011, at 11:02 AM, John George wrote:
>>> 
>>>> Hi Matt,
>>>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>>>> and have a patch uploaded, but I think it should go in to 205.
>>>> Thank you for considering.
>>>> 
>>>> Regads,
>>>> John George
>>>> 
>>>> 
>>>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <jr...@ebay.com> wrote:
>>>> 
>>>>> Could you please add MAPREDUCE-2779 to the list as well?
>>>>> This one has a patch as well.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Joep
>>>>> ________________________________________
>>>>> From: Rottinghuis, Joep [jrottinghuis@ebay.com]
>>>>> Sent: Thursday, September 01, 2011 9:04 AM
>>>>> To: general@hadoop.apache.org
>>>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>>>> 
>>>>> Hi Matt,
>>>>> 
>>>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>>>> We applied it to our branch because our users had problems with getting queue
>>>>> information.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Joep
>>>>> ________________________________________
>>>>> From: Matt Foley [mfoley@hortonworks.com]
>>>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>>>> To: general@hadoop.apache.org
>>>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>>>> 
>>>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>>>> easier to review a single list than 10 or more emails.
>>>>> 
>>>>> Does anyone else have requests for items to include in 0.20.205 - either
>>>>> already submitted to 0.20-security, or pending work?
>>>>> 
>>>>> Thanks,
>>>>> --Matt
>>>>> 
>>>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:
>>>>> 
>>>>>> Hi Matt,
>>>>>> 
>>>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>>>> the following items be included as high priority sustaining improvements in
>>>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>>>> this mailing list, this email aggregates contributions from the following
>>>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>>>> John George.
>>>>>> 
>>>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>>>> significant experience working with large scale Hadoop clusters within
>>>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>>>> 
>>>>>> Notes on the items listed below:
>>>>>> 
>>>>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>>>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>>>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>>>>> over the next few days.
>>>>>> *   All of the jiras listed have been fixed in trunk with the following
>>>>>> exceptions:
>>>>>> *   The four jiras listed above which are still being worked
>>>>>> *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>>>> *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>>>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>>>> *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>>>>> of MR279
>>>>>> 
>>>>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>>>>>> Nathan Roberts
>>>>>> 
>>>>>> ====
>>>>>> [list elided, please see archives at http://tinyurl.com/3k537hs ]
>>>> 
>>> 
>> 

Re: Content request for 0.20.205 Sustaining Release

[Arun Murthy <ac...@hortonworks.com>]

Agree, +1.

It will be good to ensure we get a more modern version of
FairScheduler for interested folks.

Arun

Sent from my iPhone

On Sep 2, 2011, at 5:55 PM, Matei Zaharia <ma...@eecs.berkeley.edu> wrote:

> Hi Matt,
>
> I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.
>
> Matei
>
> On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:
>
>> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
>>
>> Arun
>>
>> On Sep 1, 2011, at 11:02 AM, John George wrote:
>>
>>> Hi Matt,
>>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>>> and have a patch uploaded, but I think it should go in to 205.
>>> Thank you for considering.
>>>
>>> Regads,
>>> John George
>>>
>>>
>>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <jr...@ebay.com> wrote:
>>>
>>>> Could you please add MAPREDUCE-2779 to the list as well?
>>>> This one has a patch as well.
>>>>
>>>> Thanks,
>>>>
>>>> Joep
>>>> ________________________________________
>>>> From: Rottinghuis, Joep [jrottinghuis@ebay.com]
>>>> Sent: Thursday, September 01, 2011 9:04 AM
>>>> To: general@hadoop.apache.org
>>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>>>
>>>> Hi Matt,
>>>>
>>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>>> We applied it to our branch because our users had problems with getting queue
>>>> information.
>>>>
>>>> Thanks,
>>>>
>>>> Joep
>>>> ________________________________________
>>>> From: Matt Foley [mfoley@hortonworks.com]
>>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>>> To: general@hadoop.apache.org
>>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>>>
>>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>>> easier to review a single list than 10 or more emails.
>>>>
>>>> Does anyone else have requests for items to include in 0.20.205 - either
>>>> already submitted to 0.20-security, or pending work?
>>>>
>>>> Thanks,
>>>> --Matt
>>>>
>>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:
>>>>
>>>>> Hi Matt,
>>>>>
>>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>>> the following items be included as high priority sustaining improvements in
>>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>>> this mailing list, this email aggregates contributions from the following
>>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>>> John George.
>>>>>
>>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>>> significant experience working with large scale Hadoop clusters within
>>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>>>
>>>>> Notes on the items listed below:
>>>>>
>>>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>>>> over the next few days.
>>>>> *   All of the jiras listed have been fixed in trunk with the following
>>>>> exceptions:
>>>>> *   The four jiras listed above which are still being worked
>>>>> *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>>> *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>>> *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>>>> of MR279
>>>>>
>>>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>>>>> Nathan Roberts
>>>>>
>>>>> ====
>>>>> [list elided, please see archives at http://tinyurl.com/3k537hs ]
>>>
>>
>

Re: Content request for 0.20.205 Sustaining Release

[Matei Zaharia <ma...@eecs.berkeley.edu>]

Hi Matt,

I'd like to propose backporting some of the new fair scheduler features that have been out for almost 2 years now (in the 0.21 branch and trunk) into 0.20.205, especially https://issues.apache.org/jira/browse/MAPREDUCE-551 and https://issues.apache.org/jira/browse/MAPREDUCE-706. I will try to put together a patch for this. It should be a matter of taking the trunk fairscheduler directory and just tweaking anything that might be required to compile on 0.20.205; I don't think there are other major differences.

Matei

On Sep 1, 2011, at 11:48 AM, Arun C Murthy wrote:

> Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.
> 
> Arun
> 
> On Sep 1, 2011, at 11:02 AM, John George wrote:
> 
>> Hi Matt,
>> I would like to bring HADOOP-7602 to your attention. I just filed it today
>> and have a patch uploaded, but I think it should go in to 205.
>> Thank you for considering.
>> 
>> Regads,
>> John George
>> 
>> 
>> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <jr...@ebay.com> wrote:
>> 
>>> Could you please add MAPREDUCE-2779 to the list as well?
>>> This one has a patch as well.
>>> 
>>> Thanks,
>>> 
>>> Joep
>>> ________________________________________
>>> From: Rottinghuis, Joep [jrottinghuis@ebay.com]
>>> Sent: Thursday, September 01, 2011 9:04 AM
>>> To: general@hadoop.apache.org
>>> Subject: RE: Content request for 0.20.205 Sustaining Release
>>> 
>>> Hi Matt,
>>> 
>>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>>> We applied it to our branch because our users had problems with getting queue
>>> information.
>>> 
>>> Thanks,
>>> 
>>> Joep
>>> ________________________________________
>>> From: Matt Foley [mfoley@hortonworks.com]
>>> Sent: Wednesday, August 31, 2011 11:16 AM
>>> To: general@hadoop.apache.org
>>> Subject: Re: Content request for 0.20.205 Sustaining Release
>>> 
>>> Thank you, Nathan.  And thanks for combining these requests, it's much
>>> easier to review a single list than 10 or more emails.
>>> 
>>> Does anyone else have requests for items to include in 0.20.205 - either
>>> already submitted to 0.20-security, or pending work?
>>> 
>>> Thanks,
>>> --Matt
>>> 
>>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:
>>> 
>>>> Hi Matt,
>>>> 
>>>> My colleagues with experience running 0.20.203 (as well as many previous
>>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>>> the following items be included as high priority sustaining improvements in
>>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>>> this mailing list, this email aggregates contributions from the following
>>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>>> John George.
>>>> 
>>>> Recommendations and suggestions for this list of jiras came from folks with
>>>> significant experience working with large scale Hadoop clusters within
>>>> Yahoo! production environments, including Service Engineering teams, Quality
>>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>>> 
>>>> Notes on the items listed below:
>>>> 
>>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>>> over the next few days.
>>>> *   All of the jiras listed have been fixed in trunk with the following
>>>> exceptions:
>>>>  *   The four jiras listed above which are still being worked
>>>>  *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>>  *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>>  *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>>> of MR279
>>>> 
>>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>>>> Nathan Roberts
>>>> 
>>>> ====
>>>> [list elided, please see archives at http://tinyurl.com/3k537hs ] 
>> 
> 

Re: Content request for 0.20.205 Sustaining Release

[Arun C Murthy <ac...@hortonworks.com>]

Thanks for the input guys - I'll look at MAPREDUCE-2601,MAPREDUCE-2779 & HADOOP-7602.

Arun

On Sep 1, 2011, at 11:02 AM, John George wrote:

> Hi Matt,
> I would like to bring HADOOP-7602 to your attention. I just filed it today
> and have a patch uploaded, but I think it should go in to 205.
> Thank you for considering.
> 
> Regads,
> John George
> 
> 
> On 9/1/11 11:08 AM, "Rottinghuis, Joep" <jr...@ebay.com> wrote:
> 
>> Could you please add MAPREDUCE-2779 to the list as well?
>> This one has a patch as well.
>> 
>> Thanks,
>> 
>> Joep
>> ________________________________________
>> From: Rottinghuis, Joep [jrottinghuis@ebay.com]
>> Sent: Thursday, September 01, 2011 9:04 AM
>> To: general@hadoop.apache.org
>> Subject: RE: Content request for 0.20.205 Sustaining Release
>> 
>> Hi Matt,
>> 
>> IMHO MAPREDUCE-2610 should be included in 205 as well.
>> We applied it to our branch because our users had problems with getting queue
>> information.
>> 
>> Thanks,
>> 
>> Joep
>> ________________________________________
>> From: Matt Foley [mfoley@hortonworks.com]
>> Sent: Wednesday, August 31, 2011 11:16 AM
>> To: general@hadoop.apache.org
>> Subject: Re: Content request for 0.20.205 Sustaining Release
>> 
>> Thank you, Nathan.  And thanks for combining these requests, it's much
>> easier to review a single list than 10 or more emails.
>> 
>> Does anyone else have requests for items to include in 0.20.205 - either
>> already submitted to 0.20-security, or pending work?
>> 
>> Thanks,
>> --Matt
>> 
>> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:
>> 
>>> Hi Matt,
>>> 
>>> My colleagues with experience running 0.20.203 (as well as many previous
>>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>>> the following items be included as high priority sustaining improvements in
>>> 0.20.205.  Rather than contributors sending in several separate requests to
>>> this mailing list, this email aggregates contributions from the following
>>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>>> John George.
>>> 
>>> Recommendations and suggestions for this list of jiras came from folks with
>>> significant experience working with large scale Hadoop clusters within
>>> Yahoo! production environments, including Service Engineering teams, Quality
>>> Engineering teams, Solutions Engineering teams, and Development teams.
>>> 
>>> Notes on the items listed below:
>>> 
>>> *   All the Jiras listed with the exception of HADOOP-7510,
>>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>>> over the next few days.
>>> *   All of the jiras listed have been fixed in trunk with the following
>>> exceptions:
>>>   *   The four jiras listed above which are still being worked
>>>   *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>>   *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>>   *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>>> of MR279
>>> 
>>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>>> Nathan Roberts
>>> 
>>> ====
>>> [list elided, please see archives at http://tinyurl.com/3k537hs ] 
> 

Re: Content request for 0.20.205 Sustaining Release

[John George <jo...@yahoo-inc.com>]

Hi Matt,
I would like to bring HADOOP-7602 to your attention. I just filed it today
and have a patch uploaded, but I think it should go in to 205.
Thank you for considering.

Regads,
John George


On 9/1/11 11:08 AM, "Rottinghuis, Joep" <jr...@ebay.com> wrote:

> Could you please add MAPREDUCE-2779 to the list as well?
> This one has a patch as well.
> 
> Thanks,
> 
> Joep
> ________________________________________
> From: Rottinghuis, Joep [jrottinghuis@ebay.com]
> Sent: Thursday, September 01, 2011 9:04 AM
> To: general@hadoop.apache.org
> Subject: RE: Content request for 0.20.205 Sustaining Release
> 
> Hi Matt,
> 
> IMHO MAPREDUCE-2610 should be included in 205 as well.
> We applied it to our branch because our users had problems with getting queue
> information.
> 
> Thanks,
> 
> Joep
> ________________________________________
> From: Matt Foley [mfoley@hortonworks.com]
> Sent: Wednesday, August 31, 2011 11:16 AM
> To: general@hadoop.apache.org
> Subject: Re: Content request for 0.20.205 Sustaining Release
> 
> Thank you, Nathan.  And thanks for combining these requests, it's much
> easier to review a single list than 10 or more emails.
> 
> Does anyone else have requests for items to include in 0.20.205 - either
> already submitted to 0.20-security, or pending work?
> 
> Thanks,
> --Matt
> 
> On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:
> 
>> Hi Matt,
>> 
>> My colleagues with experience running 0.20.203 (as well as many previous
>> releases of Hadoop) in Yahoo!'s production environment are requesting that
>> the following items be included as high priority sustaining improvements in
>> 0.20.205.  Rather than contributors sending in several separate requests to
>> this mailing list, this email aggregates contributions from the following
>> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
>> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
>> John George.
>> 
>> Recommendations and suggestions for this list of jiras came from folks with
>> significant experience working with large scale Hadoop clusters within
>> Yahoo! production environments, including Service Engineering teams, Quality
>> Engineering teams, Solutions Engineering teams, and Development teams.
>> 
>> Notes on the items listed below:
>> 
>>  *   All the Jiras listed with the exception of HADOOP-7510,
>> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
>> 0.20-security. These remaining four jiras are in-progress and should wrap up
>> over the next few days.
>>  *   All of the jiras listed have been fixed in trunk with the following
>> exceptions:
>>    *   The four jiras listed above which are still being worked
>>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
>> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
>> of MR279
>> 
>> Thank you for considering this list of Jiras for inclusion in 0.20.205.
>> Nathan Roberts
>> 
>> ====
>> [list elided, please see archives at http://tinyurl.com/3k537hs ] 

RE: Content request for 0.20.205 Sustaining Release

["Rottinghuis, Joep" <jr...@ebay.com>]

Could you please add MAPREDUCE-2779 to the list as well?
This one has a patch as well.

Thanks,

Joep
________________________________________
From: Rottinghuis, Joep [jrottinghuis@ebay.com]
Sent: Thursday, September 01, 2011 9:04 AM
To: general@hadoop.apache.org
Subject: RE: Content request for 0.20.205 Sustaining Release

Hi Matt,

IMHO MAPREDUCE-2610 should be included in 205 as well.
We applied it to our branch because our users had problems with getting queue information.

Thanks,

Joep
________________________________________
From: Matt Foley [mfoley@hortonworks.com]
Sent: Wednesday, August 31, 2011 11:16 AM
To: general@hadoop.apache.org
Subject: Re: Content request for 0.20.205 Sustaining Release

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ====
> [list elided, please see archives at http://tinyurl.com/3k537hs ]

RE: Content request for 0.20.205 Sustaining Release

["Rottinghuis, Joep" <jr...@ebay.com>]

Hi Matt,

IMHO MAPREDUCE-2610 should be included in 205 as well.
We applied it to our branch because our users had problems with getting queue information.

Thanks,

Joep
________________________________________
From: Matt Foley [mfoley@hortonworks.com]
Sent: Wednesday, August 31, 2011 11:16 AM
To: general@hadoop.apache.org
Subject: Re: Content request for 0.20.205 Sustaining Release

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ====
> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Re: Content request for 0.20.205 Sustaining Release

[Matt Foley <mf...@hortonworks.com>]

Thank you, Nathan.  And thanks for combining these requests, it's much
easier to review a single list than 10 or more emails.

Does anyone else have requests for items to include in 0.20.205 - either
already submitted to 0.20-security, or pending work?

Thanks,
--Matt

On Wed, Aug 31, 2011 at 8:25 AM, Nathan Roberts <nr...@yahoo-inc.com>wrote:

> Hi Matt,
>
> My colleagues with experience running 0.20.203 (as well as many previous
> releases of Hadoop) in Yahoo!'s production environment are requesting that
> the following items be included as high priority sustaining improvements in
> 0.20.205.  Rather than contributors sending in several separate requests to
> this mailing list, this email aggregates contributions from the following
> individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas
> Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne,
> John George.
>
> Recommendations and suggestions for this list of jiras came from folks with
> significant experience working with large scale Hadoop clusters within
> Yahoo! production environments, including Service Engineering teams, Quality
> Engineering teams, Solutions Engineering teams, and Development teams.
>
> Notes on the items listed below:
>
>  *   All the Jiras listed with the exception of HADOOP-7510,
> MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to
> 0.20-security. These remaining four jiras are in-progress and should wrap up
> over the next few days.
>  *   All of the jiras listed have been fixed in trunk with the following
> exceptions:
>    *   The four jiras listed above which are still being worked
>    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
>    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed
> MAPREDUCE-2723 to make sure this is handled correctly in yarn
>    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration
> of MR279
>
> Thank you for considering this list of Jiras for inclusion in 0.20.205.
> Nathan Roberts
>
> ====
> [list elided, please see archives at http://tinyurl.com/3k537hs ]

Re: Community Process for 0.20.205 Sustaining Release

[Konstantin Boudnik <co...@apache.org>]

On Wed, Aug 24, 2011 at 09:32AM, Eric Baldeschwieler wrote:
> Hi Milind,
> 
> I'm really happy to see 0.23 moving along.  This seems like pretty clear
> evidence that 0.20.xxx is not going to be a new trunk.  However, the time
> between cutting a new release and being able to run production applications
> on it can be very long.  0.23 and any alternative such as 0.22 will take a
> long time to get to the level of quality and stability that is in 0.20.xxx
> today.  Trunk is looking very healthy to me.  Having a good sustaining

I can bet money that  someone will call it 'a very bad taste' joke in a next
20 minutes, but does it really look healthy with last successful MR trunk
build happened 2 mo 25 days ago?

--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author, and do
not necessarily represent the views of any company the author might be
affiliated with at the moment of writing.

> program on 0.20.xxx is intended to compliment mainline dev, not replace it.
> Describing fixing flush/sync as innovation seems like a stretch.  The patch
> sets we are considering merging have been in use for more than a year.
> 
> There is a large constituency that would like to see HBASE work well now.
> I'm hoping to see that happen on 0.20.xxx.  I think this is the best place
> to invest effort.  You are welcome to work with the community members
> investing in 0.20.xxx or on an alternative.  There is room in the community
> for both efforts IMO.
> 
> Thanks,
> 
> E14
> 
> 
> On Aug 24, 2011, at 12:05 AM, <Mi...@emc.com> wrote:
> 
> >> At the same time, we certainly do not wish 0.20-security to be viewed as a "trunk"; it is important
> > that all patches go in trunk first, and only patches of manageable risk and high value to production users, should go into
> > the sustaining releases.
> > 
> > Matt,
> > 
> > With all due respect, I have heard from "several of your associates", about features for making hbase work with the 0.20.2xx. That sounds to me that 0.20-security to be trunk.
> > 
> > Can you clarify how that is going to work ?
> > 
> > Basically, what are your criteria for "manageable risk and high value to production users" ?
> > 
> > In particular, I would like to know why the insistence on 0.20.2xx being the default branch to check-in these "innovations", instead of trunk.
> > 
> > 
> > *   Milind
> > 
> > ---
> > Milind Bhandarkar
> > Greenplum Labs, EMC
> > (Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.)
> > 
> 

Re: Community Process for 0.20.205 Sustaining Release

[Steve Loughran <st...@apache.org>]

On 31/08/11 07:37, Eric Baldeschwieler wrote:
> Hi Milind,
>
> I'd expect to see continued sustaining on the 20 branch until 23 is stable.  Continued bug fixing beyond that if there is need / enthusiasm for such fixes would also seem reasonable. This has certainly been the expect behavior of other released software I've contributed to.
>
> If we want to see apache hadoop widely adopted, it is critical that folks not only volunteer new features, but that someone continue to refine working releases.  I see nothing but upside to this.  If others want to only volunteer new code for trunk or want to work on other releases, that's ok by me.
>
> I don't believe this is a zero sum game.  Any investment I put into 0.20 doesn't block you from putting work into 22 or 23.  We're aligned in the goal of getting 23 and future trunk based releases out.  Asking someone not to do sustaining will not cause them to work faster on other releases, it will just move work out of apache. I don't see how that benefits the community.  If the sustaining work is badly done or does not add value, it can be voted down.
>


I think
  -all patches should go into trunk then backport to 0.23, 0.22, 0.20.x 
if it is deemed important, the criteria for those being things to resolve.

My views

-0.20.x: critical security/stability issues, and critical stack 
compatibility issues. Nothing much else.

-0.22 - this isn't discussed, but it's going to be the last pre-MR-279 
MR engine. Shouldn't all patches to the mainstream MR engine go in here 
first?

-0.23 - subject for discussion. Now that it's into stabilise and ship 
it's bug fixes and not features.

Like milind, I worry about time to stabilise for 0.23, but am somewhat 
more optimistic than 10+ months, because if there is takeup from some 
large hadoop users, stability comes from that use

Re: Community Process for 0.20.205 Sustaining Release

[Eric Baldeschwieler <er...@hortonworks.com>]

Hi Milind,

I'd expect to see continued sustaining on the 20 branch until 23 is stable.  Continued bug fixing beyond that if there is need / enthusiasm for such fixes would also seem reasonable. This has certainly been the expect behavior of other released software I've contributed to.

If we want to see apache hadoop widely adopted, it is critical that folks not only volunteer new features, but that someone continue to refine working releases.  I see nothing but upside to this.  If others want to only volunteer new code for trunk or want to work on other releases, that's ok by me.

I don't believe this is a zero sum game.  Any investment I put into 0.20 doesn't block you from putting work into 22 or 23.  We're aligned in the goal of getting 23 and future trunk based releases out.  Asking someone not to do sustaining will not cause them to work faster on other releases, it will just move work out of apache. I don't see how that benefits the community.  If the sustaining work is badly done or does not add value, it can be voted down.

Thanks,

E14



On Aug 25, 2011, at 1:04 PM, <Mi...@emc.com> <Mi...@emc.com> wrote:

> Thanks for the prompt response Eric.
> 
> I have been traveling, and could reply to you immediately.
> 
> How long do you estimate 0.23 (or 0.22) would take to stabilize ? Based on
> the past experience of release 0.19, even if it was not deployed at 1000+
> node-scale, eventually, it did stabilize, and several small installations
> did exist using that release.
> 
> With that in mind, and knowing the typical deployment schedules at a large
> installation, I would expect 0.23 to be stable for production workloads
> next summer. Since that is still 10 months away, do you expect that
> sustaining releases on the 0.20.2xx branch to continue until then ?
> 
> - milind 
> 
> On 8/24/11 9:32 AM, "Eric Baldeschwieler" <er...@hortonworks.com> wrote:
> 
>> Hi Milind,
>> 
>> I'm really happy to see 0.23 moving along.  This seems like pretty clear
>> evidence that 0.20.xxx is not going to be a new trunk.  However, the time
>> between cutting a new release and being able to run production
>> applications on it can be very long.  0.23 and any alternative such as
>> 0.22 will take a long time to get to the level of quality and stability
>> that is in 0.20.xxx today.  Trunk is looking very healthy to me.  Having
>> a good sustaining program on 0.20.xxx is intended to compliment mainline
>> dev, not replace it.  Describing fixing flush/sync as innovation seems
>> like a stretch.  The patch sets we are considering merging have been in
>> use for more than a year.
>> 
>> There is a large constituency that would like to see HBASE work well now.
>> I'm hoping to see that happen on 0.20.xxx.  I think this is the best
>> place to invest effort.  You are welcome to work with the community
>> members investing in 0.20.xxx or on an alternative.  There is room in the
>> community for both efforts IMO.
>> 
>> Thanks,
>> 
>> E14
>> 
>> 
>> On Aug 24, 2011, at 12:05 AM, <Mi...@emc.com> wrote:
>> 
>>>> At the same time, we certainly do not wish 0.20-security to be viewed
>>>> as a "trunk"; it is important
>>> that all patches go in trunk first, and only patches of manageable risk
>>> and high value to production users, should go into
>>> the sustaining releases.
>>> 
>>> Matt,
>>> 
>>> With all due respect, I have heard from "several of your associates",
>>> about features for making hbase work with the 0.20.2xx. That sounds to
>>> me that 0.20-security to be trunk.
>>> 
>>> Can you clarify how that is going to work ?
>>> 
>>> Basically, what are your criteria for "manageable risk and high value
>>> to production users" ?
>>> 
>>> In particular, I would like to know why the insistence on 0.20.2xx
>>> being the default branch to check-in these "innovations", instead of
>>> trunk.
>>> 
>>> 
>>> *   Milind
>>> 
>>> ---
>>> Milind Bhandarkar
>>> Greenplum Labs, EMC
>>> (Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any organization, past or
>>> present, the author might be affiliated with.)
>>> 
>> 
>> 
> 

Re: Community Process for 0.20.205 Sustaining Release

[Mi...@emc.com]

Thanks for the prompt response Eric.

I have been traveling, and could reply to you immediately.

How long do you estimate 0.23 (or 0.22) would take to stabilize ? Based on
the past experience of release 0.19, even if it was not deployed at 1000+
node-scale, eventually, it did stabilize, and several small installations
did exist using that release.

With that in mind, and knowing the typical deployment schedules at a large
installation, I would expect 0.23 to be stable for production workloads
next summer. Since that is still 10 months away, do you expect that
sustaining releases on the 0.20.2xx branch to continue until then ?

- milind 

On 8/24/11 9:32 AM, "Eric Baldeschwieler" <er...@hortonworks.com> wrote:

>Hi Milind,
>
>I'm really happy to see 0.23 moving along.  This seems like pretty clear
>evidence that 0.20.xxx is not going to be a new trunk.  However, the time
>between cutting a new release and being able to run production
>applications on it can be very long.  0.23 and any alternative such as
>0.22 will take a long time to get to the level of quality and stability
>that is in 0.20.xxx today.  Trunk is looking very healthy to me.  Having
>a good sustaining program on 0.20.xxx is intended to compliment mainline
>dev, not replace it.  Describing fixing flush/sync as innovation seems
>like a stretch.  The patch sets we are considering merging have been in
>use for more than a year.
>
>There is a large constituency that would like to see HBASE work well now.
> I'm hoping to see that happen on 0.20.xxx.  I think this is the best
>place to invest effort.  You are welcome to work with the community
>members investing in 0.20.xxx or on an alternative.  There is room in the
>community for both efforts IMO.
>
>Thanks,
>
>E14
>
>
>On Aug 24, 2011, at 12:05 AM, <Mi...@emc.com> wrote:
>
>>> At the same time, we certainly do not wish 0.20-security to be viewed
>>>as a "trunk"; it is important
>> that all patches go in trunk first, and only patches of manageable risk
>>and high value to production users, should go into
>> the sustaining releases.
>> 
>> Matt,
>> 
>> With all due respect, I have heard from "several of your associates",
>>about features for making hbase work with the 0.20.2xx. That sounds to
>>me that 0.20-security to be trunk.
>> 
>> Can you clarify how that is going to work ?
>> 
>> Basically, what are your criteria for "manageable risk and high value
>>to production users" ?
>> 
>> In particular, I would like to know why the insistence on 0.20.2xx
>>being the default branch to check-in these "innovations", instead of
>>trunk.
>> 
>> 
>> *   Milind
>> 
>> ---
>> Milind Bhandarkar
>> Greenplum Labs, EMC
>> (Disclaimer: Opinions expressed in this email are those of the author,
>>and do not necessarily represent the views of any organization, past or
>>present, the author might be affiliated with.)
>> 
>
>

Re: Community Process for 0.20.205 Sustaining Release

[Eric Baldeschwieler <er...@hortonworks.com>]

Hi Milind,

I'm really happy to see 0.23 moving along.  This seems like pretty clear evidence that 0.20.xxx is not going to be a new trunk.  However, the time between cutting a new release and being able to run production applications on it can be very long.  0.23 and any alternative such as 0.22 will take a long time to get to the level of quality and stability that is in 0.20.xxx today.  Trunk is looking very healthy to me.  Having a good sustaining program on 0.20.xxx is intended to compliment mainline dev, not replace it.  Describing fixing flush/sync as innovation seems like a stretch.  The patch sets we are considering merging have been in use for more than a year.

There is a large constituency that would like to see HBASE work well now.  I'm hoping to see that happen on 0.20.xxx.  I think this is the best place to invest effort.  You are welcome to work with the community members investing in 0.20.xxx or on an alternative.  There is room in the community for both efforts IMO.

Thanks,

E14


On Aug 24, 2011, at 12:05 AM, <Mi...@emc.com> wrote:

>> At the same time, we certainly do not wish 0.20-security to be viewed as a "trunk"; it is important
> that all patches go in trunk first, and only patches of manageable risk and high value to production users, should go into
> the sustaining releases.
> 
> Matt,
> 
> With all due respect, I have heard from "several of your associates", about features for making hbase work with the 0.20.2xx. That sounds to me that 0.20-security to be trunk.
> 
> Can you clarify how that is going to work ?
> 
> Basically, what are your criteria for "manageable risk and high value to production users" ?
> 
> In particular, I would like to know why the insistence on 0.20.2xx being the default branch to check-in these "innovations", instead of trunk.
> 
> 
> *   Milind
> 
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.)
> 

Re: Community Process for 0.20.205 Sustaining Release

[Mi...@emc.com]

> At the same time, we certainly do not wish 0.20-security to be viewed as a "trunk"; it is important
that all patches go in trunk first, and only patches of manageable risk and high value to production users, should go into
the sustaining releases.

Matt,

With all due respect, I have heard from "several of your associates", about features for making hbase work with the 0.20.2xx. That sounds to me that 0.20-security to be trunk.

Can you clarify how that is going to work ?

Basically, what are your criteria for "manageable risk and high value to production users" ?

In particular, I would like to know why the insistence on 0.20.2xx being the default branch to check-in these "innovations", instead of trunk.


 *   Milind

---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.)

Content request for 0.20.205 Sustaining Release

[Nathan Roberts <nr...@yahoo-inc.com>]

Hi Matt,

My colleagues with experience running 0.20.203 (as well as many previous releases of Hadoop) in Yahoo!'s production environment are requesting that the following items be included as high priority sustaining improvements in 0.20.205.  Rather than contributors sending in several separate requests to this mailing list, this email aggregates contributions from the following individuals: Daryn Sharp, Jeffrey Naisbitt, Kihwal Lee, Sherry Chen, Thomas Graves, Bharath Mundlapudi, Robert Joseph Evans, Anupam Seth, Eric Payne, John George.

Recommendations and suggestions for this list of jiras came from folks with significant experience working with large scale Hadoop clusters within Yahoo! production environments, including Service Engineering teams, Quality Engineering teams, Solutions Engineering teams, and Development teams.

Notes on the items listed below:

 *   All the Jiras listed with the exception of HADOOP-7510, MAPREDUCE-2764, MAPREDUCE-2915, and HDFS-2257 have been committed to 0.20-security. These remaining four jiras are in-progress and should wrap up over the next few days.
 *   All of the jiras listed have been fixed in trunk with the following exceptions:
    *   The four jiras listed above which are still being worked
    *   MAPREDUCE-2780 - Similar to previous bullet. In progress now.
    *   MAPREDUCE-2324 - Has a strong interaction with MR279 so filed MAPREDUCE-2723 to make sure this is handled correctly in yarn
    *   MAPREDUCE-2729, MAPREDUCE-2621 - Don't make sense after integration of MR279

Thank you for considering this list of Jiras for inclusion in 0.20.205.
Nathan Roberts

====

MAPREDUCE-2489 - Jobsplits with random hostnames can make the queue unusable
Justification: A broken job that is issuing random hostnames to the job tracker can hang up a queue and severely impact the performance of the job tracker.
Risk: Low. Change involves a simple check for obviously malformed hostnames.

MAPREDUCE-2852 - Remove YDH Bug 2854624 from  code comments
Justification: Comment change only
Risk: Low

HADOOP-7472 - RPC client should deal with the IP address changes
Justification: If the IP address of a namenode is changed, all clients must be restarted. This can be very expensive and difficult to execute when many of the clients are not within the cluster-proper. e.g. distcp
Risk: Low. If an address change is suspected, the code now performs an additional lookup and updates the address. Does not affect normal path.

MAPREDUCE-2729 - Reducers are always counted having pending tasks even if they can't be scheduled yet because not enough of their mappers have completed
Justification: reducer slots are not being properly allocated when reducers are waiting on map tasks to finish, causing situations where a queue can be significantly under utilized. In grids where queues are configured with relatively tight constraints, this can result in substantial throughput degradation when this condition arises.
Risk: Medium/Low. No change to the scheduler can be taken lightly so in those terms it's medium. However, the change itself is straighforward and the experts agree it was a bug.

MAPREDUCE-2705 - tasks localized and launched serially by TaskLauncher - causing other tasks to be delayed
Justification: Large localization processes lock up task launcher for potentially very long periods of time. This can result in significant delays for other tasks assigned to the same compute node.
Risk: Low. Localization is performed in a separate thread but overall flow for a particular task remains unchanged.

MAPREDUCE-2651 - Race condition in Linux Task Controller for job log directory creation
Justification: Tasks can fail because of a race to create the job log directory.
Risk: Low. Deals with EEXIST more consistently.

MAPREDUCE-2650 - back-port MAPREDUCE-2238 to 0.20-security
Justification: Permission handling within localization causes races and can leave directories with broken permissions. Adversely affects test reproducibility.
Risk: Low. Fix has been in trunk and 22 for several months.

MAPREDUCE-2621 - TestCapacityScheduler fails with Queue q1 does not exist
Justification: Hudson unit test failures
Risk: Low. Changes just create an explicit association between the QueueManager and JT.

MAPREDUCE-2494 - Make the distributed cache delete entires using LRU priority
Justification: Some regularly recurring jobs require large distributed cache contents. The current scheme deletes these contents when the distributed cache fills up. The penalty for localizing this type of job is a recurring penalty.
Risk: Low/Medium - Currently eviction is all or nothing. This change just orders the eviction and doesn't do it all at once. All of the races dealing with eviction were already dealt with in the code so no additional risk from that standpoint.

MAPREDUCE-2324 - Job should fail if a reduce task can't be scheduled anywhere
Justification: Jobs can get stuck in limbo emitting tons of messages to the logs about not being able to schedule the reduce. It's best to either just attempt the reduce and let it fail, or  put in more sophisticated logic to attempt to fail these jobs before attempting the reduce at all.
Risk: Low. Change now just removes the check which tried to prevent this. So, the job will be attempted and will just fail through the normal course.

MAPREDUCE-2187 - map tasks timeout during sorting
Justification: No progress is reported during merge sort so if this phase is takes too long, the tasks can timeout and fail.
Risk: Low. Adds new progress report point during merge sort.

HDFS-2202 - Changes to balancer bandwidth should not require datanode restart.
Justification: There are times when operations needs to either speed up or slow down the balancer bandwidth. The system should support doing so without restarting the datanodes.
Risk: Low/Medium - If feature is not used, code paths are the same.

HDFS-1836 - Thousand of CLOSE_WAIT socket
Justification: Clients can chew up socket connections by not closing down correctly.
Risk: Low.

HADOOP-7432 - Back-port HADOOP-7110 to 0.20-security
Justification: Fixes build/UT failures due to racey chmod and improve performance by using JNI chmod rather than forking.
Risk: Low. Backport of fix for 22 from Todd Lipconn.

HADOOP-7314 - Add support for throwing UnknownHostException when a host doesn't resolve
Justification: Tied to MAPREDUCE-2489. Same justification.
Risk: Same risk as MAPREDUCE-2489

MAPREDUCE-2764 - Fix renewal of dfs delegation tokens
Justification: Long running jobs like distcp may repeatedly fail to renew delegation tokens even after an intermittent error has been corrected. The repeated failures can overwhelm the job tracker causing the entire grid to have difficulty.
Risk: Medium risk. Requires a low-level change to the tokens to include enough information so that the token can be renewed later.

HDFS-2257 - HftpFilesysystem should implement GetDelegationTokens
Justification: Required for MAPREDUCE-2764
Risk: Medium - See MAPREDUCE-2764

MAPREDUCE-2780 - MAPREDUCE-2764 Standardize the value of token service
Justification: Required for MAPREDUCE-2764
Risk: Low. Creates a setService method rather than having all token producers do this themselves. No change to actual tokens.

HADOOP-7510 - Tokens should use original hostname provided instead of ip
Justification: Need this in order to support namenode changing IP address. Otherwise as soon as next task tries to look something up in token cache using ip, it will fail to find the proper token and then fail to execute.
Risk: Medium risk. Requires a change to the information maintained in the token.

HADOOP-7539 - merge hadoop archive goodness from trunk to 0.20
Justification: HAR support regressed somewhat when merging to 0.20.203. This jira brings HAR support back to what's in trunk.
Risk: Low risk. Doesn't affect mainline HDFS/MAPREDUCE,

HADOOP-6889 - Make RPC to have an option to timeout
Justification: Clients can hang when issuing RPCs to troubled datanodes because there is no RPC timeout. Has been pulled into 0.22, 0.20.append.
Risk: Low. Running in 0.20.appemd, trunk and 22. Fixed 12 months ago.

MAPREDUCE-2915 LinuxTaskController does not work when JniBasedUnixGroupsNetgroupMapping or JniBasedUnixGroupsMapping is enabled
Justification: If one does not use the JNI versions of these methods, the namenode and jobtracker frequently have to fork. Especially in the case of the namenode, this can cause many seconds of unavailability due to the time it takes the linux kernel to copy hundreds of MB of page tables and exec a new process.
Risk: Low. Fix adds a missing argument when launching the linuxtaskcontroller (path to native libraries)