You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Konstantin Shvachko <sh...@gmail.com> on 2011/03/29 06:39:59 UTC
VOTE: Committing HADOOP-6949 to 0.22 branch
HADOOP-6949 introduced a very important optimization to the RPC layer. Based
on the benchmarks presented in HDFS-1583 this provides an order of magnitude
improvement of current RPC implementation.
RPC is a common component of Hadoop projects. Many of them should benefit
from this change. But since this is an incompatible change it requires a
vote to be included into a previous branch.
Please vote for inclusion of this change into branch 0.22.
+1 from me.
Thanks,
--Konstantin
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Konstantin Shvachko <sh...@gmail.com>.
Nigel,
The nature of incompatibility is that the RPC version is changing, which
means
VersionedProtocol-s become incompatible all at once. As opposed to say
only DatanodeProtocol or mr.ClientProtocol.
Doug is right because of our strict requirements for protocol compatibility
this
will not affect users upgrading to 0.22.
Matt, thanks for checking and testing.
--Konstantin
On Tue, Mar 29, 2011 at 11:04 AM, Jakob Homan <jg...@gmail.com> wrote:
> +1
> n.b. that the vote lost hdfs and common dev at some point. I've added
> them back.
>
> On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com>
> wrote:
> > +1
> >
> > On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >> +1
> >>
> >> 2011/3/29 Doug Cutting <cu...@apache.org>
> >>
> >>> +1
> >>>
> >>> I don't think this creates an incompatibility. It changes the RPC wire
> >>> format, but we already require that clients and servers run identical
> >>> builds. No application that ran with a prior version of Hadoop would
> be
> >>> broken by this change when it upgrades to this version of Hadoop.
> >>>
> >>> Doug
> >>>
> >>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
> >>> > HADOOP-6949 introduced a very important optimization to the RPC
> layer.
> >>> Based
> >>> > on the benchmarks presented in HDFS-1583 this provides an order of
> >>> magnitude
> >>> > improvement of current RPC implementation.
> >>> > RPC is a common component of Hadoop projects. Many of them should
> benefit
> >>> > from this change. But since this is an incompatible change it
> requires a
> >>> > vote to be included into a previous branch.
> >>> > Please vote for inclusion of this change into branch 0.22.
> >>> >
> >>> > +1 from me.
> >>> >
> >>> > Thanks,
> >>> > --Konstantin
> >>> >
> >>>
> >>
> >
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Konstantin Shvachko <sh...@gmail.com>.
Nigel,
The nature of incompatibility is that the RPC version is changing, which
means
VersionedProtocol-s become incompatible all at once. As opposed to say
only DatanodeProtocol or mr.ClientProtocol.
Doug is right because of our strict requirements for protocol compatibility
this
will not affect users upgrading to 0.22.
Matt, thanks for checking and testing.
--Konstantin
On Tue, Mar 29, 2011 at 11:04 AM, Jakob Homan <jg...@gmail.com> wrote:
> +1
> n.b. that the vote lost hdfs and common dev at some point. I've added
> them back.
>
> On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com>
> wrote:
> > +1
> >
> > On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >> +1
> >>
> >> 2011/3/29 Doug Cutting <cu...@apache.org>
> >>
> >>> +1
> >>>
> >>> I don't think this creates an incompatibility. It changes the RPC wire
> >>> format, but we already require that clients and servers run identical
> >>> builds. No application that ran with a prior version of Hadoop would
> be
> >>> broken by this change when it upgrades to this version of Hadoop.
> >>>
> >>> Doug
> >>>
> >>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
> >>> > HADOOP-6949 introduced a very important optimization to the RPC
> layer.
> >>> Based
> >>> > on the benchmarks presented in HDFS-1583 this provides an order of
> >>> magnitude
> >>> > improvement of current RPC implementation.
> >>> > RPC is a common component of Hadoop projects. Many of them should
> benefit
> >>> > from this change. But since this is an incompatible change it
> requires a
> >>> > vote to be included into a previous branch.
> >>> > Please vote for inclusion of this change into branch 0.22.
> >>> >
> >>> > +1 from me.
> >>> >
> >>> > Thanks,
> >>> > --Konstantin
> >>> >
> >>>
> >>
> >
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Konstantin Shvachko <sh...@gmail.com>.
Nigel,
The nature of incompatibility is that the RPC version is changing, which
means
VersionedProtocol-s become incompatible all at once. As opposed to say
only DatanodeProtocol or mr.ClientProtocol.
Doug is right because of our strict requirements for protocol compatibility
this
will not affect users upgrading to 0.22.
Matt, thanks for checking and testing.
--Konstantin
On Tue, Mar 29, 2011 at 11:04 AM, Jakob Homan <jg...@gmail.com> wrote:
> +1
> n.b. that the vote lost hdfs and common dev at some point. I've added
> them back.
>
> On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com>
> wrote:
> > +1
> >
> > On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >> +1
> >>
> >> 2011/3/29 Doug Cutting <cu...@apache.org>
> >>
> >>> +1
> >>>
> >>> I don't think this creates an incompatibility. It changes the RPC wire
> >>> format, but we already require that clients and servers run identical
> >>> builds. No application that ran with a prior version of Hadoop would
> be
> >>> broken by this change when it upgrades to this version of Hadoop.
> >>>
> >>> Doug
> >>>
> >>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
> >>> > HADOOP-6949 introduced a very important optimization to the RPC
> layer.
> >>> Based
> >>> > on the benchmarks presented in HDFS-1583 this provides an order of
> >>> magnitude
> >>> > improvement of current RPC implementation.
> >>> > RPC is a common component of Hadoop projects. Many of them should
> benefit
> >>> > from this change. But since this is an incompatible change it
> requires a
> >>> > vote to be included into a previous branch.
> >>> > Please vote for inclusion of this change into branch 0.22.
> >>> >
> >>> > +1 from me.
> >>> >
> >>> > Thanks,
> >>> > --Konstantin
> >>> >
> >>>
> >>
> >
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Jakob Homan <jg...@gmail.com>.
+1
n.b. that the vote lost hdfs and common dev at some point. I've added
them back.
On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com> wrote:
> +1
>
> On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com> wrote:
>> +1
>>
>> 2011/3/29 Doug Cutting <cu...@apache.org>
>>
>>> +1
>>>
>>> I don't think this creates an incompatibility. It changes the RPC wire
>>> format, but we already require that clients and servers run identical
>>> builds. No application that ran with a prior version of Hadoop would be
>>> broken by this change when it upgrades to this version of Hadoop.
>>>
>>> Doug
>>>
>>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
>>> > HADOOP-6949 introduced a very important optimization to the RPC layer.
>>> Based
>>> > on the benchmarks presented in HDFS-1583 this provides an order of
>>> magnitude
>>> > improvement of current RPC implementation.
>>> > RPC is a common component of Hadoop projects. Many of them should benefit
>>> > from this change. But since this is an incompatible change it requires a
>>> > vote to be included into a previous branch.
>>> > Please vote for inclusion of this change into branch 0.22.
>>> >
>>> > +1 from me.
>>> >
>>> > Thanks,
>>> > --Konstantin
>>> >
>>>
>>
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Jakob Homan <jg...@gmail.com>.
+1
n.b. that the vote lost hdfs and common dev at some point. I've added
them back.
On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com> wrote:
> +1
>
> On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com> wrote:
>> +1
>>
>> 2011/3/29 Doug Cutting <cu...@apache.org>
>>
>>> +1
>>>
>>> I don't think this creates an incompatibility. It changes the RPC wire
>>> format, but we already require that clients and servers run identical
>>> builds. No application that ran with a prior version of Hadoop would be
>>> broken by this change when it upgrades to this version of Hadoop.
>>>
>>> Doug
>>>
>>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
>>> > HADOOP-6949 introduced a very important optimization to the RPC layer.
>>> Based
>>> > on the benchmarks presented in HDFS-1583 this provides an order of
>>> magnitude
>>> > improvement of current RPC implementation.
>>> > RPC is a common component of Hadoop projects. Many of them should benefit
>>> > from this change. But since this is an incompatible change it requires a
>>> > vote to be included into a previous branch.
>>> > Please vote for inclusion of this change into branch 0.22.
>>> >
>>> > +1 from me.
>>> >
>>> > Thanks,
>>> > --Konstantin
>>> >
>>>
>>
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Jakob Homan <jg...@gmail.com>.
+1
n.b. that the vote lost hdfs and common dev at some point. I've added
them back.
On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya <sa...@gmail.com> wrote:
> +1
>
> On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com> wrote:
>> +1
>>
>> 2011/3/29 Doug Cutting <cu...@apache.org>
>>
>>> +1
>>>
>>> I don't think this creates an incompatibility. It changes the RPC wire
>>> format, but we already require that clients and servers run identical
>>> builds. No application that ran with a prior version of Hadoop would be
>>> broken by this change when it upgrades to this version of Hadoop.
>>>
>>> Doug
>>>
>>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
>>> > HADOOP-6949 introduced a very important optimization to the RPC layer.
>>> Based
>>> > on the benchmarks presented in HDFS-1583 this provides an order of
>>> magnitude
>>> > improvement of current RPC implementation.
>>> > RPC is a common component of Hadoop projects. Many of them should benefit
>>> > from this change. But since this is an incompatible change it requires a
>>> > vote to be included into a previous branch.
>>> > Please vote for inclusion of this change into branch 0.22.
>>> >
>>> > +1 from me.
>>> >
>>> > Thanks,
>>> > --Konstantin
>>> >
>>>
>>
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Amit Sangroya <sa...@gmail.com>.
+1
On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch <ja...@gmail.com> wrote:
> +1
>
> 2011/3/29 Doug Cutting <cu...@apache.org>
>
>> +1
>>
>> I don't think this creates an incompatibility. It changes the RPC wire
>> format, but we already require that clients and servers run identical
>> builds. No application that ran with a prior version of Hadoop would be
>> broken by this change when it upgrades to this version of Hadoop.
>>
>> Doug
>>
>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
>> > HADOOP-6949 introduced a very important optimization to the RPC layer.
>> Based
>> > on the benchmarks presented in HDFS-1583 this provides an order of
>> magnitude
>> > improvement of current RPC implementation.
>> > RPC is a common component of Hadoop projects. Many of them should benefit
>> > from this change. But since this is an incompatible change it requires a
>> > vote to be included into a previous branch.
>> > Please vote for inclusion of this change into branch 0.22.
>> >
>> > +1 from me.
>> >
>> > Thanks,
>> > --Konstantin
>> >
>>
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Stephen Boesch <ja...@gmail.com>.
+1
2011/3/29 Doug Cutting <cu...@apache.org>
> +1
>
> I don't think this creates an incompatibility. It changes the RPC wire
> format, but we already require that clients and servers run identical
> builds. No application that ran with a prior version of Hadoop would be
> broken by this change when it upgrades to this version of Hadoop.
>
> Doug
>
> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
> > HADOOP-6949 introduced a very important optimization to the RPC layer.
> Based
> > on the benchmarks presented in HDFS-1583 this provides an order of
> magnitude
> > improvement of current RPC implementation.
> > RPC is a common component of Hadoop projects. Many of them should benefit
> > from this change. But since this is an incompatible change it requires a
> > vote to be included into a previous branch.
> > Please vote for inclusion of this change into branch 0.22.
> >
> > +1 from me.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Steve Loughran <st...@apache.org>.
+1
-steve
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Doug Cutting <cu...@apache.org>.
+1
I don't think this creates an incompatibility. It changes the RPC wire
format, but we already require that clients and servers run identical
builds. No application that ran with a prior version of Hadoop would be
broken by this change when it upgrades to this version of Hadoop.
Doug
On 03/28/2011 09:39 PM, Konstantin Shvachko wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer. Based
> on the benchmarks presented in HDFS-1583 this provides an order of magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Eli Collins <el...@cloudera.com>.
+1
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer. Based
> on the benchmarks presented in HDFS-1583 this provides an order of magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Eli Collins <el...@cloudera.com>.
+1
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer. Based
> on the benchmarks presented in HDFS-1583 this provides an order of magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by He Chen <ai...@gmail.com>.
good news. +1.
On Tue, Mar 29, 2011 at 10:02 AM, Nigel Daley <nd...@mac.com> wrote:
> [bcc'ing hdfs-dev and mapreduce-dev]
>
> Konstantin, can you outline the effects of this incompatibility?
>
> Thx,
> Nige
>
> On Mar 28, 2011, at 10:19 PM, Dhruba Borthakur wrote:
>
> > This is a very effective optimization, +1 on pulling it to 0.22.
> >
> > -dhruba
> >
> >
> > On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
> > <sh...@gmail.com>wrote:
> >
> >> HADOOP-6949 introduced a very important optimization to the RPC layer.
> >> Based
> >> on the benchmarks presented in HDFS-1583 this provides an order of
> >> magnitude
> >> improvement of current RPC implementation.
> >> RPC is a common component of Hadoop projects. Many of them should
> benefit
> >> from this change. But since this is an incompatible change it requires a
> >> vote to be included into a previous branch.
> >> Please vote for inclusion of this change into branch 0.22.
> >>
> >> +1 from me.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >
> >
> >
> > --
> > Connect to me at http://www.facebook.com/dhruba
>
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Nigel Daley <nd...@mac.com>.
[bcc'ing hdfs-dev and mapreduce-dev]
Konstantin, can you outline the effects of this incompatibility?
Thx,
Nige
On Mar 28, 2011, at 10:19 PM, Dhruba Borthakur wrote:
> This is a very effective optimization, +1 on pulling it to 0.22.
>
> -dhruba
>
>
> On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
> <sh...@gmail.com>wrote:
>
>> HADOOP-6949 introduced a very important optimization to the RPC layer.
>> Based
>> on the benchmarks presented in HDFS-1583 this provides an order of
>> magnitude
>> improvement of current RPC implementation.
>> RPC is a common component of Hadoop projects. Many of them should benefit
>> from this change. But since this is an incompatible change it requires a
>> vote to be included into a previous branch.
>> Please vote for inclusion of this change into branch 0.22.
>>
>> +1 from me.
>>
>> Thanks,
>> --Konstantin
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Nigel Daley <nd...@mac.com>.
[bcc'ing hdfs-dev and mapreduce-dev]
Konstantin, can you outline the effects of this incompatibility?
Thx,
Nige
On Mar 28, 2011, at 10:19 PM, Dhruba Borthakur wrote:
> This is a very effective optimization, +1 on pulling it to 0.22.
>
> -dhruba
>
>
> On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
> <sh...@gmail.com>wrote:
>
>> HADOOP-6949 introduced a very important optimization to the RPC layer.
>> Based
>> on the benchmarks presented in HDFS-1583 this provides an order of
>> magnitude
>> improvement of current RPC implementation.
>> RPC is a common component of Hadoop projects. Many of them should benefit
>> from this change. But since this is an incompatible change it requires a
>> vote to be included into a previous branch.
>> Please vote for inclusion of this change into branch 0.22.
>>
>> +1 from me.
>>
>> Thanks,
>> --Konstantin
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Nigel Daley <nd...@mac.com>.
[bcc'ing hdfs-dev and mapreduce-dev]
Konstantin, can you outline the effects of this incompatibility?
Thx,
Nige
On Mar 28, 2011, at 10:19 PM, Dhruba Borthakur wrote:
> This is a very effective optimization, +1 on pulling it to 0.22.
>
> -dhruba
>
>
> On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
> <sh...@gmail.com>wrote:
>
>> HADOOP-6949 introduced a very important optimization to the RPC layer.
>> Based
>> on the benchmarks presented in HDFS-1583 this provides an order of
>> magnitude
>> improvement of current RPC implementation.
>> RPC is a common component of Hadoop projects. Many of them should benefit
>> from this change. But since this is an incompatible change it requires a
>> vote to be included into a previous branch.
>> Please vote for inclusion of this change into branch 0.22.
>>
>> +1 from me.
>>
>> Thanks,
>> --Konstantin
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Dhruba Borthakur <dh...@gmail.com>.
This is a very effective optimization, +1 on pulling it to 0.22.
-dhruba
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer.
> Based
> on the benchmarks presented in HDFS-1583 this provides an order of
> magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
--
Connect to me at http://www.facebook.com/dhruba
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Dhruba Borthakur <dh...@gmail.com>.
This is a very effective optimization, +1 on pulling it to 0.22.
-dhruba
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer.
> Based
> on the benchmarks presented in HDFS-1583 this provides an order of
> magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
--
Connect to me at http://www.facebook.com/dhruba
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Eli Collins <el...@cloudera.com>.
+1
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com> wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer. Based
> on the benchmarks presented in HDFS-1583 this provides an order of magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
Posted by Dhruba Borthakur <dh...@gmail.com>.
This is a very effective optimization, +1 on pulling it to 0.22.
-dhruba
On Mon, Mar 28, 2011 at 9:39 PM, Konstantin Shvachko
<sh...@gmail.com>wrote:
> HADOOP-6949 introduced a very important optimization to the RPC layer.
> Based
> on the benchmarks presented in HDFS-1583 this provides an order of
> magnitude
> improvement of current RPC implementation.
> RPC is a common component of Hadoop projects. Many of them should benefit
> from this change. But since this is an incompatible change it requires a
> vote to be included into a previous branch.
> Please vote for inclusion of this change into branch 0.22.
>
> +1 from me.
>
> Thanks,
> --Konstantin
>
--
Connect to me at http://www.facebook.com/dhruba