You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Ian Holsman <ia...@holsman.net> on 2010/12/16 19:54:55 UTC
Plans for the 0.22 Release
Hi Guys.
how are we going with the 0.22 release.
--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
We are not afraid of the truth, in fact we plan on taking the truth out for a nice meal while we persuade it to adopt our views
Re: Plans for the 0.22 Release
Posted by Owen O'Malley <om...@apache.org>.
On Mon, Dec 20, 2010 at 12:12 PM, Tom White <to...@gmail.com> wrote:
> However, to get HADOOP-6685 resolved,
> from my point of view the main thing to sort out are the modularity
> concerns that I and others have raised, so that serializations are
> pluggable and don't add potentially incompatible libraries onto the
> user's classpath.
>
The serializers are already completely pluggable. If you want to propose a
plan for reducing the number of jars on the user's task classpath that might
be interesting. Clearly, it would be a separate jira, since it has nothing
to do with the serialization interface.
-- Owen
Re: Plans for the 0.22 Release
Posted by Tom White <to...@gmail.com>.
[Sending this again as my original post didn't make it to the list for
some reason. Can't see it in the moderation queue either.]
I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.
Thanks,
Tom
On Mon, Dec 20, 2010 at 11:39 AM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> My question is
> What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *to
> move *0.22* forward?
> Asking Owen as the author of the patch, Doug and Tom as they vetoed it.
>
> If I am asking the wrong question please let me know.
> I am not proposing a particular solution, just trying to understand
> 1. if there is any
> 2. What is it if there is.
>
> If, say removing dependency for Avro and updating the patch symmetrically
> removing PB dependency is a solution, if there is a solution to SequenceFile
> and byte array issues, then lets do it.
> If not then lets face it.
>
> Thanks,
> --Konstantin
>
> On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <om...@apache.org> wrote:
>
>> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cu...@apache.org> wrote:
>>
>> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>> >
>> >> It could be a zero-option plan - remove dependencies both for Avro and
>> >> ProtocolBuffers out into libraries, similar to schedulers.
>> >>
>> >
>> > I'd be fine with removing Avro from the mapreduce user's classpath. It's
>> > currently an unused option for RPC, and is used server-side on the
>> > jobtracker. It could be removed from the user classpath today without
>> loss
>> > of critical functionality.
>>
>>
>> That would be non-trivial and no one has requested that. You haven't
>> answered Konstantin's question.
>>
>> What do you consider the technical reasons for rejecting the patch as is?
>>
>> -- Owen
>>
>
Re: Plans for the 0.22 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
My question is
What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *to
move *0.22* forward?
Asking Owen as the author of the patch, Doug and Tom as they vetoed it.
If I am asking the wrong question please let me know.
I am not proposing a particular solution, just trying to understand
1. if there is any
2. What is it if there is.
If, say removing dependency for Avro and updating the patch symmetrically
removing PB dependency is a solution, if there is a solution to SequenceFile
and byte array issues, then lets do it.
If not then lets face it.
Thanks,
--Konstantin
On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <om...@apache.org> wrote:
> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cu...@apache.org> wrote:
>
> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> >
> >> It could be a zero-option plan - remove dependencies both for Avro and
> >> ProtocolBuffers out into libraries, similar to schedulers.
> >>
> >
> > I'd be fine with removing Avro from the mapreduce user's classpath. It's
> > currently an unused option for RPC, and is used server-side on the
> > jobtracker. It could be removed from the user classpath today without
> loss
> > of critical functionality.
>
>
> That would be non-trivial and no one has requested that. You haven't
> answered Konstantin's question.
>
> What do you consider the technical reasons for rejecting the patch as is?
>
> -- Owen
>
Re: Plans for the 0.22 Release
Posted by Owen O'Malley <om...@apache.org>.
On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cu...@apache.org> wrote:
> On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>
>> It could be a zero-option plan - remove dependencies both for Avro and
>> ProtocolBuffers out into libraries, similar to schedulers.
>>
>
> I'd be fine with removing Avro from the mapreduce user's classpath. It's
> currently an unused option for RPC, and is used server-side on the
> jobtracker. It could be removed from the user classpath today without loss
> of critical functionality.
That would be non-trivial and no one has requested that. You haven't
answered Konstantin's question.
What do you consider the technical reasons for rejecting the patch as is?
-- Owen
Re: Plans for the 0.22 Release
Posted by Doug Cutting <cu...@apache.org>.
On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.
I'd be fine with removing Avro from the mapreduce user's classpath.
It's currently an unused option for RPC, and is used server-side on the
jobtracker. It could be removed from the user classpath today without
loss of critical functionality.
Doug
Re: Plans for the 0.22 Release
Posted by Tom White <to...@gmail.com>.
On Fri, Dec 17, 2010 at 2:34 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> Owen, Doug, Tom
>
> Could you please formulate and reply to this email separately
> what would be an *ACCEPTABLE *resolution of
> HADOOP-6685 for *YOU *to move *0.22* forward.
> Just trying to get something to work with to get us beyond the stagnation
> point.
>
> It could be "I want this patch in/out as is, final answer". Then we are
> stuck.
> But at least we will know there is no resolution to hope for anymore.
> And we have to find other ways based on that fact.
>
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.
> Or something else.
> Let's see if there is any common ground. If there is
> we can further talk about implementation and in the mean time declare
> the 0.22 freeze contingent on the completion of H-6685.
I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.
Thanks,
Tom
>
> Thanks,
> --Konstantin
>
>
> On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <ha...@holsman.net> wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>> >> Everyone who has discussed this patch has said it isn't critical to
>> hadoop, and It's holding up everything else 0.22 is going to bring.
>> >
>> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22.
>> I'm just not volunteering to spend my time working on it, if it doesn't have
>> the features that I think it needs.
>> >
>> > -- Owen
>> >
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22
>> with 6685?
>>
>>
>
Re: Plans for the 0.22 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
Owen, Doug, Tom
Could you please formulate and reply to this email separately
what would be an *ACCEPTABLE *resolution of
HADOOP-6685 for *YOU *to move *0.22* forward.
Just trying to get something to work with to get us beyond the stagnation
point.
It could be "I want this patch in/out as is, final answer". Then we are
stuck.
But at least we will know there is no resolution to hope for anymore.
And we have to find other ways based on that fact.
It could be a zero-option plan - remove dependencies both for Avro and
ProtocolBuffers out into libraries, similar to schedulers.
Or something else.
Let's see if there is any common ground. If there is
we can further talk about implementation and in the mean time declare
the 0.22 freeze contingent on the completion of H-6685.
Thanks,
--Konstantin
On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <ha...@holsman.net> wrote:
>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
> >> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22.
> I'm just not volunteering to spend my time working on it, if it doesn't have
> the features that I think it needs.
> >
> > -- Owen
> >
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22
> with 6685?
>
>
Re: Plans for the 0.22 Release
Posted by Konstantin Boudnik <co...@apache.org>.
+1
On Tue, Dec 21, 2010 at 11:24, Nigel Daley <nd...@mac.com> wrote:
> I think it's important we get more frequent releases back on track, so I'll volunteer to do it. Its been a while since I was the release manager so I'll need help.
>
> Looks like in Jira we currently have:
> 26 MR blockers: http://tinyurl.com/23s5vzg
> 17 HDFS blockers: http://tinyurl.com/27w82vl
> 6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers. I'll start cleaning up the list and ping folks after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>>
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>>
>>> -- Owen
>>>
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>>
>
>
Re: Plans for the 0.22 Release
Posted by Konstantin Shvachko <sh...@gmail.com>.
This is great! Thanks, Nigel.
I haven't seen the feature freeze for 0.22 formally announced.
May be it worth confirming it now by setting the start day, as of yesterday?
--Konstantin
On Tue, Dec 21, 2010 at 11:24 AM, Nigel Daley <nd...@mac.com> wrote:
> I think it's important we get more frequent releases back on track, so I'll
> volunteer to do it. Its been a while since I was the release manager so
> I'll need help.
>
> Looks like in Jira we currently have:
> 26 MR blockers: http://tinyurl.com/23s5vzg
> 17 HDFS blockers: http://tinyurl.com/27w82vl
> 6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of
> these are truly blockers. I'll start cleaning up the list and ping folks
> after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
> >
> > On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
> >
> >>> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >>
> >> I disagree that it isn't critical to Hadoop, but I'm not holding up
> 0.22. I'm just not volunteering to spend my time working on it, if it
> doesn't have the features that I think it needs.
> >>
> >> -- Owen
> >>
> >
> > That's a fair point.. and this is a volunteer effort.
> > Do we have anybody else who is willing to be the release manager for 22
> with 6685?
> >
>
>
Re: Plans for the 0.22 Release
Posted by Ian Holsman <ha...@holsman.net>.
Thanks Nige.
On Dec 22, 2010, at 6:24 AM, Nigel Daley wrote:
> I think it's important we get more frequent releases back on track, so I'll volunteer to do it. Its been a while since I was the release manager so I'll need help.
>
> Looks like in Jira we currently have:
> 26 MR blockers: http://tinyurl.com/23s5vzg
> 17 HDFS blockers: http://tinyurl.com/27w82vl
> 6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers. I'll start cleaning up the list and ping folks after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>>
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>>
>>> -- Owen
>>>
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>>
>
Re: Plans for the 0.22 Release
Posted by Ken Walker <Ke...@ca.ibm.com>.
Not sure what state this issue is in, would be nice for 0.22
https://issues.apache.org/jira/browse/HADOOP-6941
I don't see it in the lists below. It's a bit of a blocker running on
non-Sun (Oracle) JREs.
Ken (K.N.) 2670 Queensview
Walker Drive
J9 Java Ottawa, Ontario K2B
Class 8K1
Libraries
Lead
Java Canada
Technology
Centre
IBM
Software
Group
Phone: +1-613-356-5021
e-mail: ken_walker@ca.ibm.com
|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Nigel Daley <nd...@mac.com> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|general@hadoop.apache.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|2010/12/21 02:26 PM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: Plans for the 0.22 Release |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
I think it's important we get more frequent releases back on track, so I'll
volunteer to do it. Its been a while since I was the release manager so
I'll need help.
Looks like in Jira we currently have:
26 MR blockers: http://tinyurl.com/23s5vzg
17 HDFS blockers: http://tinyurl.com/27w82vl
6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of
these are truly blockers. I'll start cleaning up the list and ping folks
after the holidays.
Cheers,
Nige
On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
>>> Everyone who has discussed this patch has said it isn't critical to
hadoop, and It's holding up everything else 0.22 is going to bring.
>>
>> I disagree that it isn't critical to Hadoop, but I'm not holding up
0.22. I'm just not volunteering to spend my time working on it, if it
doesn't have the features that I think it needs.
>>
>> -- Owen
>>
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22
with 6685?
>
Re: Plans for the 0.22 Release
Posted by Nigel Daley <nd...@mac.com>.
I think it's important we get more frequent releases back on track, so I'll volunteer to do it. Its been a while since I was the release manager so I'll need help.
Looks like in Jira we currently have:
26 MR blockers: http://tinyurl.com/23s5vzg
17 HDFS blockers: http://tinyurl.com/27w82vl
6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers. I'll start cleaning up the list and ping folks after the holidays.
Cheers,
Nige
On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>>
>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>>
>> -- Owen
>>
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 with 6685?
>
Re: Plans for the 0.22 Release
Posted by Ian Holsman <ha...@holsman.net>.
On Dec 17, 2010, at 4:16 PM, Konstantin Boudnik wrote:
>
>
> You meant to say "...to be the release manager for 22 without 6685?", didn't you?
Yes..
It's been a long day ;-(
>
> Cos
>
Re: Plans for the 0.22 Release
Posted by Konstantin Boudnik <co...@apache.org>.
On Fri, Dec 17, 2010 at 10:10AM, Ian Holsman wrote:
>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
> >> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
> >
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
> >
> > -- Owen
> >
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 with 6685?
You meant to say "...to be the release manager for 22 without 6685?", didn't you?
Cos
Re: Plans for the 0.22 Release
Posted by Ian Holsman <ha...@holsman.net>.
On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>
> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>
> -- Owen
>
That's a fair point.. and this is a volunteer effort.
Do we have anybody else who is willing to be the release manager for 22 with 6685?
Re: Plans for the 0.22 Release
Posted by Owen O'Malley <om...@apache.org>.
On Dec 16, 2010, at 12:26 PM, Ian Holsman wrote:
> While it may be critical, I still don't understand why we couldn't
> release without it, and carry on the discussion.
> Putting so much emphasis and pressure on this issue is unwarranted.
It took months to stabilize 0.21 and it took a lot of Tom's time.
H-6685 is important to me and needs to be addressed to move the
project forward.
> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
I disagree that it isn't critical to Hadoop, but I'm not holding up
0.22. I'm just not volunteering to spend my time working on it, if it
doesn't have the features that I think it needs.
-- Owen
Re: Plans for the 0.22 Release
Posted by Ian Holsman <ha...@holsman.net>.
On Dec 17, 2010, at 6:15 AM, Owen O'Malley wrote:
>
> On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:
>
>> how are we going with the 0.22 release.
>
> Progress has been blocked because HADOOP-6685 is still blocked. I feel that HADOOP-6685 is critical for Hadoop and don't see much point in working on a 0.22 release without it.
While it may be critical, I still don't understand why we couldn't release without it, and carry on the discussion.
Putting so much emphasis and pressure on this issue is unwarranted.
Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
There are now 4 separate companies who have released their own 0.20 version because of the delay on our releasing...
Delaying this release another 2-3 months for a single patch seems a bit extreme to me.
> In particular, HADOOP-6685 removes the need for third party serialization libraries that work around the current limitations of the MapReduce framework, but splinter the user community.
>
> If someone else wants to take over as release manager for 0.22 while we find a solution to make forward progress on HADOOP-6685, that would be fine.
>
> -- Owen
Re: Plans for the 0.22 Release
Posted by Owen O'Malley <om...@apache.org>.
On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:
> how are we going with the 0.22 release.
Progress has been blocked because HADOOP-6685 is still blocked. I feel
that HADOOP-6685 is critical for Hadoop and don't see much point in
working on a 0.22 release without it. In particular, HADOOP-6685
removes the need for third party serialization libraries that work
around the current limitations of the MapReduce framework, but
splinter the user community.
If someone else wants to take over as release manager for 0.22 while
we find a solution to make forward progress on HADOOP-6685, that would
be fine.
-- Owen