You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Roman Shaposhnik <rv...@apache.org> on 2013/08/09 00:34:35 UTC
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
I'd be huge +1 on a flag day migrating us to 2.5.x.
We can run a set of validation tests in Bigtop on all components
that could be affected by it. Hadoop and HBase are the obvious
suspects. What else in the ecosystem could be affected?
Thanks,
Roman.
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
>> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>>
>> Kihwal
>>
>>
>> ________________________________
>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> Sent: Thursday, August 8, 2013 1:59 AM
>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>>
>>
>> Hi,
>>
>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> For more detail, please see the JIRA ticket:
>> https://issues.apache.org/jira/browse/HADOOP-9346
>>
>> - Tsuyoshi
>>
>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>
>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>>> justify the upgrade.
>>>
>>> Doing the upgrade now, with the first beta, will make things easier for
>>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>>> we do the upgrade later, downstream projects will have to support 2
>>> different versions and they my get in nasty waters due to classpath issues.
>>>
>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>>
>>> Now, to do this it will require a few things:
>>>
>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>> * A follow up email to dev@ aliases indicating developers should install
>>> locally protobuf 2.5.0
>>>
>>> Thanks.
>>>
>>> --
>>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Andrew Purtell <ap...@apache.org>.
On Thu, Aug 8, 2013 at 3:34 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> > In HBase land we've pretty well discovered that we'll need to have the
> > same version of protobuf that the HDFS/Yarn/MR servers are running.
> > That is to say there are issues with ever having 2.4.x and 2.5.x on
> > the same class path.
>
As long as Hadoop and HBase both are built using protoc = protobuf-java =
2.5.0 things seem ok. I've not done full ecosystem tests though. Would be
interesting to upgrade a build box, do global substitutions in Maven and
Ivy specs, and see what happens.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Andrew Purtell <ap...@apache.org>.
On Thu, Aug 8, 2013 at 3:34 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> > In HBase land we've pretty well discovered that we'll need to have the
> > same version of protobuf that the HDFS/Yarn/MR servers are running.
> > That is to say there are issues with ever having 2.4.x and 2.5.x on
> > the same class path.
>
As long as Hadoop and HBase both are built using protoc = protobuf-java =
2.5.0 things seem ok. I've not done full ecosystem tests though. Would be
interesting to upgrade a build box, do global substitutions in Maven and
Ivy specs, and see what happens.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Alejandro
(phone typing)
On Aug 8, 2013, at 15:34, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>>> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
>>> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>>>
>>> Kihwal
>>>
>>>
>>> ________________________________
>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>> Sent: Thursday, August 8, 2013 1:59 AM
>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>>>
>>>
>>> Hi,
>>>
>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>> For more detail, please see the JIRA ticket:
>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>
>>> - Tsuyoshi
>>>
>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>
>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>>>> justify the upgrade.
>>>>
>>>> Doing the upgrade now, with the first beta, will make things easier for
>>>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>>>> we do the upgrade later, downstream projects will have to support 2
>>>> different versions and they my get in nasty waters due to classpath issues.
>>>>
>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>>>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>>>
>>>> Now, to do this it will require a few things:
>>>>
>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>> * A follow up email to dev@ aliases indicating developers should install
>>>> locally protobuf 2.5.0
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Alejandro