You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2013/08/07 18:49:27 UTC
Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
justify the upgrade.
Doing the upgrade now, with the first beta, will make things easier for
downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
we do the upgrade later, downstream projects will have to support 2
different versions and they my get in nasty waters due to classpath issues.
I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
and it works fine (something is broken in trunk in the RPC layer YARN-885).
Now, to do this it will require a few things:
* Make sure protobuf 2.5.0 is available in the jenkins box
* A follow up email to dev@ aliases indicating developers should install
locally protobuf 2.5.0
Thanks.
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Lisen Mu <im...@gmail.com>.
Another good news!
On Thu, Aug 8, 2013 at 8:34 AM, Jacques Nadeau <ja...@apache.org> wrote:
> I guess they saw that Drill wanted it :D
>
> On Wed, Aug 7, 2013 at 9:56 AM, Timothy Chen <tn...@gmail.com> wrote:
> > About time!
> >
> > Tim
> >
> > Sent from my iPhone
> >
> > Begin forwarded message:
> >
> >> From: Alejandro Abdelnur <tu...@cloudera.com>
> >> Date: August 7, 2013, 9:49:27 AM PDT
> >> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >> Subject: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> >> Reply-To: yarn-dev@hadoop.apache.org
> >>
> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>
> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> justify the upgrade.
> >>
> >> Doing the upgrade now, with the first beta, will make things easier for
> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
> If
> >> we do the upgrade later, downstream projects will have to support 2
> >> different versions and they my get in nasty waters due to classpath
> issues.
> >>
> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> branch
> >> and it works fine (something is broken in trunk in the RPC layer
> YARN-885).
> >>
> >> Now, to do this it will require a few things:
> >>
> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> * A follow up email to dev@ aliases indicating developers should
> install
> >> locally protobuf 2.5.0
> >>
> >> Thanks.
> >>
> >> --
> >> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Jacques Nadeau <ja...@apache.org>.
I guess they saw that Drill wanted it :D
On Wed, Aug 7, 2013 at 9:56 AM, Timothy Chen <tn...@gmail.com> wrote:
> About time!
>
> Tim
>
> Sent from my iPhone
>
> Begin forwarded message:
>
>> From: Alejandro Abdelnur <tu...@cloudera.com>
>> Date: August 7, 2013, 9:49:27 AM PDT
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> Subject: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>> Reply-To: yarn-dev@hadoop.apache.org
>>
>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>
>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> justify the upgrade.
>>
>> Doing the upgrade now, with the first beta, will make things easier for
>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>> we do the upgrade later, downstream projects will have to support 2
>> different versions and they my get in nasty waters due to classpath issues.
>>
>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>
>> Now, to do this it will require a few things:
>>
>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> * A follow up email to dev@ aliases indicating developers should install
>> locally protobuf 2.5.0
>>
>> Thanks.
>>
>> --
>> Alejandro
Fwd: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Timothy Chen <tn...@gmail.com>.
About time!
Tim
Sent from my iPhone
Begin forwarded message:
> From: Alejandro Abdelnur <tu...@cloudera.com>
> Date: August 7, 2013, 9:49:27 AM PDT
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Subject: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> Reply-To: yarn-dev@hadoop.apache.org
>
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
shooting to get it i n for 2.1.0.
at moment is in trunk till the nightly finishes. then we'll decide
in the mean time, you can have multiple versions installed in diff dirs and set the right one in the path
thx
Alejandro
(phone typing)
On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com> wrote:
> Ok. After installing protobuf 2.5.0 I can compile trunk.
> But now I cannot compile Hadoop-2 branches. None of them.
> So if I switch between branches I need to reinstall protobuf?
>
> Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> versions?
> I did not get definite impression there is.
> If not it could be a pretty big disruption.
>
> Thanks,
> --Konst
>
>
>
> On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>
>> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> warning if you don't have it).
>>
>> We'd propagate this to the 2 branches once the precommit build is back to
>> normal and see things are OK.
>>
>> Thanks.
>>
>>
>> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
>>> protoc 2.5.0.
>>>
>>> thx
>>>
>>>
>>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>> gkesavan@hortonworks.com> wrote:
>>>
>>>> I can take care of re-installing 2.4 and installing 2.5 in a different
>>>> location. This would fix 2.0 branch builds as well.
>>>> Thoughts?
>>>>
>>>> -Giri
>>>>
>>>>
>>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>
>>>>> Giri,
>>>>>
>>>>> first of all, thanks for installing protoc 2.5.0.
>>>>>
>>>>> I didn't know we were installing them as the only version and not
>>>> driven by
>>>>> env/path settings.
>>>>>
>>>>> Now we have a bit of a problem, precommit builds are broken because of
>>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>>>>
>>>>> We have to options:
>>>>>
>>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> any
>>>>> follow up issues.
>>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>>>> 2.5.0
>>>>> coexisting
>>>>>
>>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> an
>>>>> then merge it to the other branches.
>>>>>
>>>>> We need to sort this out quickly as precommits are not working.
>>>>>
>>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>>>> commit
>>>>> it to trunk.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>>>>> gkesavan@hortonworks.com> wrote:
>>>>>
>>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> default
>>>>> path.
>>>>>> If we still need 2.4 I may have to install it. Let me know
>>>>>>
>>>>>> -Giri
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>
>>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> so
>>>> we
>>>>>> can
>>>>>>> use and env to set it in the jobs?
>>>>>>>
>>>>>>> thx
>>>>>>>
>>>>>>> Alejandro
>>>>>>> (phone typing)
>>>>>>>
>>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>>>> gkesavan@hortonworks.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Giri
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>>>>>>>> gkesavan@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Alejandro,
>>>>>>>>>
>>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>>>>>>>>
>>>>>>>>> -Giri
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> the
>>>>>> hadoop
>>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>>>>> hours
>>>>>> to
>>>>>>>>>> set
>>>>>>>>>> up myself.
>>>>>>>>>>
>>>>>>>>>> Please!!!
>>>>>>>>>>
>>>>>>>>>> thx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>>>>>> tucu@cloudera.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>>>>> hadoop
>>>>>>>>>>> jenkins boxes.
>>>>>>>>>>>
>>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>>>>>>>>>>>
>>>>>>>>>>> Thx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>>>> eclark@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> to
>>>>> have
>>>>>>> the
>>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>>>>> running.
>>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>>>> 2.5.x
>>>>> on
>>>>>>>>>>>> the same class path.
>>>>>>>>>>>>
>>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>>>> classes
>>>>> we
>>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> a
>>>>>> rather
>>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> in
>>>>>> sooner
>>>>>>>>>>>> rather than later would be great. If we could get this into
>>>>> 2.1.0
>>>>>> it
>>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>>>> story
>>>>>> to
>>>>>>>>>>>> users with regards to protobuf version.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>>>> kihwal@yahoo-inc.com
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>>>> Avro.
>>>>> See
>>>>>>>>>>>> HADOOP-9672.
>>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>>>>> issues.
>>>>>>>>>> We've
>>>>>>>>>>>> seen users running into it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kihwal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>>>>> common-dev@hadoop.apache.org>
>>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> hdfs-dev@hadoop.apache.org
>>>>> ;
>>>>> "
>>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>>>>> mapreduce-dev@hadoop.apache.org>
>>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> release,
>>>>>>>>>>>> HADOOP-9845
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>>>> HADOOP-9346.
>>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Tsuyoshi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>>>>>>>>> tucu@cloudera.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>>>>>> benefits
>>>>>>>>>> to
>>>>>>>>>>>>>> justify the upgrade.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> things
>>>>>> easier
>>>>>>>>>> for
>>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> adopting
>>>>>> Hadoop
>>>>>>>>>> 2.
>>>>>>>>>>>> If
>>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>>>>>> support 2
>>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>>>>>> classpath
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>>>>>> 2.1.0-beta
>>>>>>>>>>>> branch
>>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>>>>> layer
>>>>>>>>>>>> YARN-885).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, to do this it will require a few things:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>>>>> should
>>>>>>>>>>>> install
>>>>>>>>>>>>>> locally protobuf 2.5.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alejandro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>>
>> --
>> Alejandro
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
shooting to get it i n for 2.1.0.
at moment is in trunk till the nightly finishes. then we'll decide
in the mean time, you can have multiple versions installed in diff dirs and set the right one in the path
thx
Alejandro
(phone typing)
On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com> wrote:
> Ok. After installing protobuf 2.5.0 I can compile trunk.
> But now I cannot compile Hadoop-2 branches. None of them.
> So if I switch between branches I need to reinstall protobuf?
>
> Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> versions?
> I did not get definite impression there is.
> If not it could be a pretty big disruption.
>
> Thanks,
> --Konst
>
>
>
> On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>
>> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> warning if you don't have it).
>>
>> We'd propagate this to the 2 branches once the precommit build is back to
>> normal and see things are OK.
>>
>> Thanks.
>>
>>
>> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
>>> protoc 2.5.0.
>>>
>>> thx
>>>
>>>
>>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>> gkesavan@hortonworks.com> wrote:
>>>
>>>> I can take care of re-installing 2.4 and installing 2.5 in a different
>>>> location. This would fix 2.0 branch builds as well.
>>>> Thoughts?
>>>>
>>>> -Giri
>>>>
>>>>
>>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>
>>>>> Giri,
>>>>>
>>>>> first of all, thanks for installing protoc 2.5.0.
>>>>>
>>>>> I didn't know we were installing them as the only version and not
>>>> driven by
>>>>> env/path settings.
>>>>>
>>>>> Now we have a bit of a problem, precommit builds are broken because of
>>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>>>>
>>>>> We have to options:
>>>>>
>>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> any
>>>>> follow up issues.
>>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>>>> 2.5.0
>>>>> coexisting
>>>>>
>>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> an
>>>>> then merge it to the other branches.
>>>>>
>>>>> We need to sort this out quickly as precommits are not working.
>>>>>
>>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>>>> commit
>>>>> it to trunk.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>>>>> gkesavan@hortonworks.com> wrote:
>>>>>
>>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> default
>>>>> path.
>>>>>> If we still need 2.4 I may have to install it. Let me know
>>>>>>
>>>>>> -Giri
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>
>>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> so
>>>> we
>>>>>> can
>>>>>>> use and env to set it in the jobs?
>>>>>>>
>>>>>>> thx
>>>>>>>
>>>>>>> Alejandro
>>>>>>> (phone typing)
>>>>>>>
>>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>>>> gkesavan@hortonworks.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Giri
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>>>>>>>> gkesavan@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Alejandro,
>>>>>>>>>
>>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>>>>>>>>
>>>>>>>>> -Giri
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> the
>>>>>> hadoop
>>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>>>>> hours
>>>>>> to
>>>>>>>>>> set
>>>>>>>>>> up myself.
>>>>>>>>>>
>>>>>>>>>> Please!!!
>>>>>>>>>>
>>>>>>>>>> thx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>>>>>> tucu@cloudera.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>>>>> hadoop
>>>>>>>>>>> jenkins boxes.
>>>>>>>>>>>
>>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>>>>>>>>>>>
>>>>>>>>>>> Thx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>>>> eclark@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> to
>>>>> have
>>>>>>> the
>>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>>>>> running.
>>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>>>> 2.5.x
>>>>> on
>>>>>>>>>>>> the same class path.
>>>>>>>>>>>>
>>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>>>> classes
>>>>> we
>>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> a
>>>>>> rather
>>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> in
>>>>>> sooner
>>>>>>>>>>>> rather than later would be great. If we could get this into
>>>>> 2.1.0
>>>>>> it
>>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>>>> story
>>>>>> to
>>>>>>>>>>>> users with regards to protobuf version.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>>>> kihwal@yahoo-inc.com
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>>>> Avro.
>>>>> See
>>>>>>>>>>>> HADOOP-9672.
>>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>>>>> issues.
>>>>>>>>>> We've
>>>>>>>>>>>> seen users running into it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kihwal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>>>>> common-dev@hadoop.apache.org>
>>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> hdfs-dev@hadoop.apache.org
>>>>> ;
>>>>> "
>>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>>>>> mapreduce-dev@hadoop.apache.org>
>>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> release,
>>>>>>>>>>>> HADOOP-9845
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>>>> HADOOP-9346.
>>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Tsuyoshi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>>>>>>>>> tucu@cloudera.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>>>>>> benefits
>>>>>>>>>> to
>>>>>>>>>>>>>> justify the upgrade.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> things
>>>>>> easier
>>>>>>>>>> for
>>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> adopting
>>>>>> Hadoop
>>>>>>>>>> 2.
>>>>>>>>>>>> If
>>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>>>>>> support 2
>>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>>>>>> classpath
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>>>>>> 2.1.0-beta
>>>>>>>>>>>> branch
>>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>>>>> layer
>>>>>>>>>>>> YARN-885).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, to do this it will require a few things:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>>>>> should
>>>>>>>>>>>> install
>>>>>>>>>>>>>> locally protobuf 2.5.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alejandro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>>
>> --
>> Alejandro
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
shooting to get it i n for 2.1.0.
at moment is in trunk till the nightly finishes. then we'll decide
in the mean time, you can have multiple versions installed in diff dirs and set the right one in the path
thx
Alejandro
(phone typing)
On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com> wrote:
> Ok. After installing protobuf 2.5.0 I can compile trunk.
> But now I cannot compile Hadoop-2 branches. None of them.
> So if I switch between branches I need to reinstall protobuf?
>
> Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> versions?
> I did not get definite impression there is.
> If not it could be a pretty big disruption.
>
> Thanks,
> --Konst
>
>
>
> On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>
>> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> warning if you don't have it).
>>
>> We'd propagate this to the 2 branches once the precommit build is back to
>> normal and see things are OK.
>>
>> Thanks.
>>
>>
>> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
>>> protoc 2.5.0.
>>>
>>> thx
>>>
>>>
>>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>> gkesavan@hortonworks.com> wrote:
>>>
>>>> I can take care of re-installing 2.4 and installing 2.5 in a different
>>>> location. This would fix 2.0 branch builds as well.
>>>> Thoughts?
>>>>
>>>> -Giri
>>>>
>>>>
>>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>
>>>>> Giri,
>>>>>
>>>>> first of all, thanks for installing protoc 2.5.0.
>>>>>
>>>>> I didn't know we were installing them as the only version and not
>>>> driven by
>>>>> env/path settings.
>>>>>
>>>>> Now we have a bit of a problem, precommit builds are broken because of
>>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>>>>
>>>>> We have to options:
>>>>>
>>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> any
>>>>> follow up issues.
>>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>>>> 2.5.0
>>>>> coexisting
>>>>>
>>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> an
>>>>> then merge it to the other branches.
>>>>>
>>>>> We need to sort this out quickly as precommits are not working.
>>>>>
>>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>>>> commit
>>>>> it to trunk.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>>>>> gkesavan@hortonworks.com> wrote:
>>>>>
>>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> default
>>>>> path.
>>>>>> If we still need 2.4 I may have to install it. Let me know
>>>>>>
>>>>>> -Giri
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>
>>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> so
>>>> we
>>>>>> can
>>>>>>> use and env to set it in the jobs?
>>>>>>>
>>>>>>> thx
>>>>>>>
>>>>>>> Alejandro
>>>>>>> (phone typing)
>>>>>>>
>>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>>>> gkesavan@hortonworks.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Giri
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>>>>>>>> gkesavan@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Alejandro,
>>>>>>>>>
>>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>>>>>>>>
>>>>>>>>> -Giri
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> the
>>>>>> hadoop
>>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>>>>> hours
>>>>>> to
>>>>>>>>>> set
>>>>>>>>>> up myself.
>>>>>>>>>>
>>>>>>>>>> Please!!!
>>>>>>>>>>
>>>>>>>>>> thx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>>>>>> tucu@cloudera.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>>>>> hadoop
>>>>>>>>>>> jenkins boxes.
>>>>>>>>>>>
>>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>>>>>>>>>>>
>>>>>>>>>>> Thx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>>>> eclark@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> to
>>>>> have
>>>>>>> the
>>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>>>>> running.
>>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>>>> 2.5.x
>>>>> on
>>>>>>>>>>>> the same class path.
>>>>>>>>>>>>
>>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>>>> classes
>>>>> we
>>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> a
>>>>>> rather
>>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> in
>>>>>> sooner
>>>>>>>>>>>> rather than later would be great. If we could get this into
>>>>> 2.1.0
>>>>>> it
>>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>>>> story
>>>>>> to
>>>>>>>>>>>> users with regards to protobuf version.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>>>> kihwal@yahoo-inc.com
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>>>> Avro.
>>>>> See
>>>>>>>>>>>> HADOOP-9672.
>>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>>>>> issues.
>>>>>>>>>> We've
>>>>>>>>>>>> seen users running into it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kihwal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>>>>> common-dev@hadoop.apache.org>
>>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> hdfs-dev@hadoop.apache.org
>>>>> ;
>>>>> "
>>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>>>>> mapreduce-dev@hadoop.apache.org>
>>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> release,
>>>>>>>>>>>> HADOOP-9845
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>>>> HADOOP-9346.
>>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Tsuyoshi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>>>>>>>>> tucu@cloudera.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>>>>>> benefits
>>>>>>>>>> to
>>>>>>>>>>>>>> justify the upgrade.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> things
>>>>>> easier
>>>>>>>>>> for
>>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> adopting
>>>>>> Hadoop
>>>>>>>>>> 2.
>>>>>>>>>>>> If
>>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>>>>>> support 2
>>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>>>>>> classpath
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>>>>>> 2.1.0-beta
>>>>>>>>>>>> branch
>>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>>>>> layer
>>>>>>>>>>>> YARN-885).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, to do this it will require a few things:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>>>>> should
>>>>>>>>>>>> install
>>>>>>>>>>>>>> locally protobuf 2.5.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alejandro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>>
>> --
>> Alejandro
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
shooting to get it i n for 2.1.0.
at moment is in trunk till the nightly finishes. then we'll decide
in the mean time, you can have multiple versions installed in diff dirs and set the right one in the path
thx
Alejandro
(phone typing)
On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com> wrote:
> Ok. After installing protobuf 2.5.0 I can compile trunk.
> But now I cannot compile Hadoop-2 branches. None of them.
> So if I switch between branches I need to reinstall protobuf?
>
> Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> versions?
> I did not get definite impression there is.
> If not it could be a pretty big disruption.
>
> Thanks,
> --Konst
>
>
>
> On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>
>> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> warning if you don't have it).
>>
>> We'd propagate this to the 2 branches once the precommit build is back to
>> normal and see things are OK.
>>
>> Thanks.
>>
>>
>> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
>>> protoc 2.5.0.
>>>
>>> thx
>>>
>>>
>>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>> gkesavan@hortonworks.com> wrote:
>>>
>>>> I can take care of re-installing 2.4 and installing 2.5 in a different
>>>> location. This would fix 2.0 branch builds as well.
>>>> Thoughts?
>>>>
>>>> -Giri
>>>>
>>>>
>>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>
>>>>> Giri,
>>>>>
>>>>> first of all, thanks for installing protoc 2.5.0.
>>>>>
>>>>> I didn't know we were installing them as the only version and not
>>>> driven by
>>>>> env/path settings.
>>>>>
>>>>> Now we have a bit of a problem, precommit builds are broken because of
>>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>>>>
>>>>> We have to options:
>>>>>
>>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> any
>>>>> follow up issues.
>>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>>>> 2.5.0
>>>>> coexisting
>>>>>
>>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> an
>>>>> then merge it to the other branches.
>>>>>
>>>>> We need to sort this out quickly as precommits are not working.
>>>>>
>>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>>>> commit
>>>>> it to trunk.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>>>>> gkesavan@hortonworks.com> wrote:
>>>>>
>>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> default
>>>>> path.
>>>>>> If we still need 2.4 I may have to install it. Let me know
>>>>>>
>>>>>> -Giri
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>
>>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> so
>>>> we
>>>>>> can
>>>>>>> use and env to set it in the jobs?
>>>>>>>
>>>>>>> thx
>>>>>>>
>>>>>>> Alejandro
>>>>>>> (phone typing)
>>>>>>>
>>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>>>> gkesavan@hortonworks.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Giri
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>>>>>>>> gkesavan@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Alejandro,
>>>>>>>>>
>>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>>>>>>>>
>>>>>>>>> -Giri
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>>>>> tucu@cloudera.com
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> the
>>>>>> hadoop
>>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>>>>> hours
>>>>>> to
>>>>>>>>>> set
>>>>>>>>>> up myself.
>>>>>>>>>>
>>>>>>>>>> Please!!!
>>>>>>>>>>
>>>>>>>>>> thx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>>>>>> tucu@cloudera.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>>>>> hadoop
>>>>>>>>>>> jenkins boxes.
>>>>>>>>>>>
>>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>>>>>>>>>>>
>>>>>>>>>>> Thx
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>>>> eclark@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> to
>>>>> have
>>>>>>> the
>>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>>>>> running.
>>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>>>> 2.5.x
>>>>> on
>>>>>>>>>>>> the same class path.
>>>>>>>>>>>>
>>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>>>> classes
>>>>> we
>>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> a
>>>>>> rather
>>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> in
>>>>>> sooner
>>>>>>>>>>>> rather than later would be great. If we could get this into
>>>>> 2.1.0
>>>>>> it
>>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>>>> story
>>>>>> to
>>>>>>>>>>>> users with regards to protobuf version.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>>>> kihwal@yahoo-inc.com
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>>>> Avro.
>>>>> See
>>>>>>>>>>>> HADOOP-9672.
>>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>>>>> issues.
>>>>>>>>>> We've
>>>>>>>>>>>> seen users running into it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kihwal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>>>>> common-dev@hadoop.apache.org>
>>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> hdfs-dev@hadoop.apache.org
>>>>> ;
>>>>> "
>>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>>>>> mapreduce-dev@hadoop.apache.org>
>>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> release,
>>>>>>>>>>>> HADOOP-9845
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>>>> HADOOP-9346.
>>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Tsuyoshi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>>>>>>>>> tucu@cloudera.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>>>>>> benefits
>>>>>>>>>> to
>>>>>>>>>>>>>> justify the upgrade.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> things
>>>>>> easier
>>>>>>>>>> for
>>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> adopting
>>>>>> Hadoop
>>>>>>>>>> 2.
>>>>>>>>>>>> If
>>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>>>>>> support 2
>>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>>>>>> classpath
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>>>>>> 2.1.0-beta
>>>>>>>>>>>> branch
>>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>>>>> layer
>>>>>>>>>>>> YARN-885).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, to do this it will require a few things:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>>>>> should
>>>>>>>>>>>> install
>>>>>>>>>>>>>> locally protobuf 2.5.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Alejandro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alejandro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Alejandro
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>>
>> --
>> Alejandro
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Konstantin Shvachko <sh...@gmail.com>.
Ok. After installing protobuf 2.5.0 I can compile trunk.
But now I cannot compile Hadoop-2 branches. None of them.
So if I switch between branches I need to reinstall protobuf?
Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
versions?
I did not get definite impression there is.
If not it could be a pretty big disruption.
Thanks,
--Konst
On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>
> To build trunk now you need protoc 2.5.0 (the build will fail with a
> warning if you don't have it).
>
> We'd propagate this to the 2 branches once the precommit build is back to
> normal and see things are OK.
>
> Thanks.
>
>
> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> > protoc 2.5.0.
> >
> > thx
> >
> >
> > On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> I can take care of re-installing 2.4 and installing 2.5 in a different
> >> location. This would fix 2.0 branch builds as well.
> >> Thoughts?
> >>
> >> -Giri
> >>
> >>
> >> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >> >wrote:
> >>
> >> > Giri,
> >> >
> >> > first of all, thanks for installing protoc 2.5.0.
> >> >
> >> > I didn't know we were installing them as the only version and not
> >> driven by
> >> > env/path settings.
> >> >
> >> > Now we have a bit of a problem, precommit builds are broken because of
> >> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >> >
> >> > We have to options:
> >> >
> >> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
> any
> >> > follow up issues.
> >> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> >> 2.5.0
> >> > coexisting
> >> >
> >> > My take would be to commit HADOOP-9845 in trunk, iron out any issues
> an
> >> > then merge it to the other branches.
> >> >
> >> > We need to sort this out quickly as precommits are not working.
> >> >
> >> > I'll wait till 3PM today for objections to option #1, if none I'll
> >> commit
> >> > it to trunk.
> >> >
> >> > Thanks.
> >> >
> >> > Alejandro
> >> >
> >> >
> >> >
> >> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >> > gkesavan@hortonworks.com> wrote:
> >> >
> >> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> default
> >> > path.
> >> > > If we still need 2.4 I may have to install it. Let me know
> >> > >
> >> > > -Giri
> >> > >
> >> > >
> >> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >> > > >wrote:
> >> > >
> >> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both
> so
> >> we
> >> > > can
> >> > > > use and env to set it in the jobs?
> >> > > >
> >> > > > thx
> >> > > >
> >> > > > Alejandro
> >> > > > (phone typing)
> >> > > >
> >> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >> gkesavan@hortonworks.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > -Giri
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> >> > > > > gkesavan@hortonworks.com> wrote:
> >> > > > >
> >> > > > >> Alejandro,
> >> > > > >>
> >> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >> > > > >>
> >> > > > >> -Giri
> >> > > > >>
> >> > > > >>
> >> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> >> > tucu@cloudera.com
> >> > > > >wrote:
> >> > > > >>
> >> > > > >>> pinging again, I need help from somebody with sudo access to
> the
> >> > > hadoop
> >> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> >> > hours
> >> > > to
> >> > > > >>> set
> >> > > > >>> up myself.
> >> > > > >>>
> >> > > > >>> Please!!!
> >> > > > >>>
> >> > > > >>> thx
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> >> > > tucu@cloudera.com
> >> > > > >>>> wrote:
> >> > > > >>>
> >> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> >> > hadoop
> >> > > > >>>> jenkins boxes.
> >> > > > >>>>
> >> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> >> > > > >>>>
> >> > > > >>>> Thx
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> >> eclark@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>
> >> > > > >>>>> In HBase land we've pretty well discovered that we'll need
> to
> >> > have
> >> > > > the
> >> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> >> > running.
> >> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> >> 2.5.x
> >> > on
> >> > > > >>>>> the same class path.
> >> > > > >>>>>
> >> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> >> classes
> >> > we
> >> > > > >>>>> could use. With that said HBase is getting pretty close to
> a
> >> > > rather
> >> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this
> in
> >> > > sooner
> >> > > > >>>>> rather than later would be great. If we could get this into
> >> > 2.1.0
> >> > > it
> >> > > > >>>>> would be great as that would allow us to have a pretty easy
> >> story
> >> > > to
> >> > > > >>>>> users with regards to protobuf version.
> >> > > > >>>>>
> >> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> >> kihwal@yahoo-inc.com
> >> > >
> >> > > > >>> wrote:
> >> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
> >> Avro.
> >> > See
> >> > > > >>>>> HADOOP-9672.
> >> > > > >>>>>> The version we are using has memory leak and inefficiency
> >> > issues.
> >> > > > >>> We've
> >> > > > >>>>> seen users running into it.
> >> > > > >>>>>>
> >> > > > >>>>>> Kihwal
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> ________________________________
> >> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> >> > common-dev@hadoop.apache.org>
> >> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
> hdfs-dev@hadoop.apache.org
> >> >;
> >> > "
> >> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> >> > mapreduce-dev@hadoop.apache.org>
> >> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
> release,
> >> > > > >>>>> HADOOP-9845
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> Hi,
> >> > > > >>>>>>
> >> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> >> HADOOP-9346.
> >> > > > >>>>>> For more detail, please see the JIRA ticket:
> >> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >> > > > >>>>>>
> >> > > > >>>>>> - Tsuyoshi
> >> > > > >>>>>>
> >> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >> > > > >>> tucu@cloudera.com>
> >> > > > >>>>> wrote:
> >> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
> release.
> >> > > > >>>>>>>
> >> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> >> > > benefits
> >> > > > >>> to
> >> > > > >>>>>>> justify the upgrade.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Doing the upgrade now, with the first beta, will make
> things
> >> > > easier
> >> > > > >>> for
> >> > > > >>>>>>> downstream projects (like HBase) using protobuf and
> adopting
> >> > > Hadoop
> >> > > > >>> 2.
> >> > > > >>>>> If
> >> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> >> > > support 2
> >> > > > >>>>>>> different versions and they my get in nasty waters due to
> >> > > classpath
> >> > > > >>>>> issues.
> >> > > > >>>>>>>
> >> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> >> > > 2.1.0-beta
> >> > > > >>>>> branch
> >> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> >> > layer
> >> > > > >>>>> YARN-885).
> >> > > > >>>>>>>
> >> > > > >>>>>>> Now, to do this it will require a few things:
> >> > > > >>>>>>>
> >> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> >> > should
> >> > > > >>>>> install
> >> > > > >>>>>>> locally protobuf 2.5.0
> >> > > > >>>>>>>
> >> > > > >>>>>>> Thanks.
> >> > > > >>>>>>>
> >> > > > >>>>>>> --
> >> > > > >>>>>>> Alejandro
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> --
> >> > > > >>>> Alejandro
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>> Alejandro
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Alejandro
> >> >
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Konstantin Shvachko <sh...@gmail.com>.
Ok. After installing protobuf 2.5.0 I can compile trunk.
But now I cannot compile Hadoop-2 branches. None of them.
So if I switch between branches I need to reinstall protobuf?
Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
versions?
I did not get definite impression there is.
If not it could be a pretty big disruption.
Thanks,
--Konst
On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>
> To build trunk now you need protoc 2.5.0 (the build will fail with a
> warning if you don't have it).
>
> We'd propagate this to the 2 branches once the precommit build is back to
> normal and see things are OK.
>
> Thanks.
>
>
> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> > protoc 2.5.0.
> >
> > thx
> >
> >
> > On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> I can take care of re-installing 2.4 and installing 2.5 in a different
> >> location. This would fix 2.0 branch builds as well.
> >> Thoughts?
> >>
> >> -Giri
> >>
> >>
> >> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >> >wrote:
> >>
> >> > Giri,
> >> >
> >> > first of all, thanks for installing protoc 2.5.0.
> >> >
> >> > I didn't know we were installing them as the only version and not
> >> driven by
> >> > env/path settings.
> >> >
> >> > Now we have a bit of a problem, precommit builds are broken because of
> >> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >> >
> >> > We have to options:
> >> >
> >> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
> any
> >> > follow up issues.
> >> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> >> 2.5.0
> >> > coexisting
> >> >
> >> > My take would be to commit HADOOP-9845 in trunk, iron out any issues
> an
> >> > then merge it to the other branches.
> >> >
> >> > We need to sort this out quickly as precommits are not working.
> >> >
> >> > I'll wait till 3PM today for objections to option #1, if none I'll
> >> commit
> >> > it to trunk.
> >> >
> >> > Thanks.
> >> >
> >> > Alejandro
> >> >
> >> >
> >> >
> >> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >> > gkesavan@hortonworks.com> wrote:
> >> >
> >> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> default
> >> > path.
> >> > > If we still need 2.4 I may have to install it. Let me know
> >> > >
> >> > > -Giri
> >> > >
> >> > >
> >> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >> > > >wrote:
> >> > >
> >> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both
> so
> >> we
> >> > > can
> >> > > > use and env to set it in the jobs?
> >> > > >
> >> > > > thx
> >> > > >
> >> > > > Alejandro
> >> > > > (phone typing)
> >> > > >
> >> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >> gkesavan@hortonworks.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > -Giri
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> >> > > > > gkesavan@hortonworks.com> wrote:
> >> > > > >
> >> > > > >> Alejandro,
> >> > > > >>
> >> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >> > > > >>
> >> > > > >> -Giri
> >> > > > >>
> >> > > > >>
> >> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> >> > tucu@cloudera.com
> >> > > > >wrote:
> >> > > > >>
> >> > > > >>> pinging again, I need help from somebody with sudo access to
> the
> >> > > hadoop
> >> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> >> > hours
> >> > > to
> >> > > > >>> set
> >> > > > >>> up myself.
> >> > > > >>>
> >> > > > >>> Please!!!
> >> > > > >>>
> >> > > > >>> thx
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> >> > > tucu@cloudera.com
> >> > > > >>>> wrote:
> >> > > > >>>
> >> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> >> > hadoop
> >> > > > >>>> jenkins boxes.
> >> > > > >>>>
> >> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> >> > > > >>>>
> >> > > > >>>> Thx
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> >> eclark@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>
> >> > > > >>>>> In HBase land we've pretty well discovered that we'll need
> to
> >> > have
> >> > > > the
> >> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> >> > running.
> >> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> >> 2.5.x
> >> > on
> >> > > > >>>>> the same class path.
> >> > > > >>>>>
> >> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> >> classes
> >> > we
> >> > > > >>>>> could use. With that said HBase is getting pretty close to
> a
> >> > > rather
> >> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this
> in
> >> > > sooner
> >> > > > >>>>> rather than later would be great. If we could get this into
> >> > 2.1.0
> >> > > it
> >> > > > >>>>> would be great as that would allow us to have a pretty easy
> >> story
> >> > > to
> >> > > > >>>>> users with regards to protobuf version.
> >> > > > >>>>>
> >> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> >> kihwal@yahoo-inc.com
> >> > >
> >> > > > >>> wrote:
> >> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
> >> Avro.
> >> > See
> >> > > > >>>>> HADOOP-9672.
> >> > > > >>>>>> The version we are using has memory leak and inefficiency
> >> > issues.
> >> > > > >>> We've
> >> > > > >>>>> seen users running into it.
> >> > > > >>>>>>
> >> > > > >>>>>> Kihwal
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> ________________________________
> >> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> >> > common-dev@hadoop.apache.org>
> >> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
> hdfs-dev@hadoop.apache.org
> >> >;
> >> > "
> >> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> >> > mapreduce-dev@hadoop.apache.org>
> >> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
> release,
> >> > > > >>>>> HADOOP-9845
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> Hi,
> >> > > > >>>>>>
> >> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> >> HADOOP-9346.
> >> > > > >>>>>> For more detail, please see the JIRA ticket:
> >> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >> > > > >>>>>>
> >> > > > >>>>>> - Tsuyoshi
> >> > > > >>>>>>
> >> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >> > > > >>> tucu@cloudera.com>
> >> > > > >>>>> wrote:
> >> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
> release.
> >> > > > >>>>>>>
> >> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> >> > > benefits
> >> > > > >>> to
> >> > > > >>>>>>> justify the upgrade.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Doing the upgrade now, with the first beta, will make
> things
> >> > > easier
> >> > > > >>> for
> >> > > > >>>>>>> downstream projects (like HBase) using protobuf and
> adopting
> >> > > Hadoop
> >> > > > >>> 2.
> >> > > > >>>>> If
> >> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> >> > > support 2
> >> > > > >>>>>>> different versions and they my get in nasty waters due to
> >> > > classpath
> >> > > > >>>>> issues.
> >> > > > >>>>>>>
> >> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> >> > > 2.1.0-beta
> >> > > > >>>>> branch
> >> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> >> > layer
> >> > > > >>>>> YARN-885).
> >> > > > >>>>>>>
> >> > > > >>>>>>> Now, to do this it will require a few things:
> >> > > > >>>>>>>
> >> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> >> > should
> >> > > > >>>>> install
> >> > > > >>>>>>> locally protobuf 2.5.0
> >> > > > >>>>>>>
> >> > > > >>>>>>> Thanks.
> >> > > > >>>>>>>
> >> > > > >>>>>>> --
> >> > > > >>>>>>> Alejandro
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> --
> >> > > > >>>> Alejandro
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>> Alejandro
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Alejandro
> >> >
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Konstantin Shvachko <sh...@gmail.com>.
Ok. After installing protobuf 2.5.0 I can compile trunk.
But now I cannot compile Hadoop-2 branches. None of them.
So if I switch between branches I need to reinstall protobuf?
Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
versions?
I did not get definite impression there is.
If not it could be a pretty big disruption.
Thanks,
--Konst
On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>
> To build trunk now you need protoc 2.5.0 (the build will fail with a
> warning if you don't have it).
>
> We'd propagate this to the 2 branches once the precommit build is back to
> normal and see things are OK.
>
> Thanks.
>
>
> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> > protoc 2.5.0.
> >
> > thx
> >
> >
> > On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> I can take care of re-installing 2.4 and installing 2.5 in a different
> >> location. This would fix 2.0 branch builds as well.
> >> Thoughts?
> >>
> >> -Giri
> >>
> >>
> >> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >> >wrote:
> >>
> >> > Giri,
> >> >
> >> > first of all, thanks for installing protoc 2.5.0.
> >> >
> >> > I didn't know we were installing them as the only version and not
> >> driven by
> >> > env/path settings.
> >> >
> >> > Now we have a bit of a problem, precommit builds are broken because of
> >> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >> >
> >> > We have to options:
> >> >
> >> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
> any
> >> > follow up issues.
> >> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> >> 2.5.0
> >> > coexisting
> >> >
> >> > My take would be to commit HADOOP-9845 in trunk, iron out any issues
> an
> >> > then merge it to the other branches.
> >> >
> >> > We need to sort this out quickly as precommits are not working.
> >> >
> >> > I'll wait till 3PM today for objections to option #1, if none I'll
> >> commit
> >> > it to trunk.
> >> >
> >> > Thanks.
> >> >
> >> > Alejandro
> >> >
> >> >
> >> >
> >> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >> > gkesavan@hortonworks.com> wrote:
> >> >
> >> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> default
> >> > path.
> >> > > If we still need 2.4 I may have to install it. Let me know
> >> > >
> >> > > -Giri
> >> > >
> >> > >
> >> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >> > > >wrote:
> >> > >
> >> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both
> so
> >> we
> >> > > can
> >> > > > use and env to set it in the jobs?
> >> > > >
> >> > > > thx
> >> > > >
> >> > > > Alejandro
> >> > > > (phone typing)
> >> > > >
> >> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >> gkesavan@hortonworks.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > -Giri
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> >> > > > > gkesavan@hortonworks.com> wrote:
> >> > > > >
> >> > > > >> Alejandro,
> >> > > > >>
> >> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >> > > > >>
> >> > > > >> -Giri
> >> > > > >>
> >> > > > >>
> >> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> >> > tucu@cloudera.com
> >> > > > >wrote:
> >> > > > >>
> >> > > > >>> pinging again, I need help from somebody with sudo access to
> the
> >> > > hadoop
> >> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> >> > hours
> >> > > to
> >> > > > >>> set
> >> > > > >>> up myself.
> >> > > > >>>
> >> > > > >>> Please!!!
> >> > > > >>>
> >> > > > >>> thx
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> >> > > tucu@cloudera.com
> >> > > > >>>> wrote:
> >> > > > >>>
> >> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> >> > hadoop
> >> > > > >>>> jenkins boxes.
> >> > > > >>>>
> >> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> >> > > > >>>>
> >> > > > >>>> Thx
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> >> eclark@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>
> >> > > > >>>>> In HBase land we've pretty well discovered that we'll need
> to
> >> > have
> >> > > > the
> >> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> >> > running.
> >> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> >> 2.5.x
> >> > on
> >> > > > >>>>> the same class path.
> >> > > > >>>>>
> >> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> >> classes
> >> > we
> >> > > > >>>>> could use. With that said HBase is getting pretty close to
> a
> >> > > rather
> >> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this
> in
> >> > > sooner
> >> > > > >>>>> rather than later would be great. If we could get this into
> >> > 2.1.0
> >> > > it
> >> > > > >>>>> would be great as that would allow us to have a pretty easy
> >> story
> >> > > to
> >> > > > >>>>> users with regards to protobuf version.
> >> > > > >>>>>
> >> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> >> kihwal@yahoo-inc.com
> >> > >
> >> > > > >>> wrote:
> >> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
> >> Avro.
> >> > See
> >> > > > >>>>> HADOOP-9672.
> >> > > > >>>>>> The version we are using has memory leak and inefficiency
> >> > issues.
> >> > > > >>> We've
> >> > > > >>>>> seen users running into it.
> >> > > > >>>>>>
> >> > > > >>>>>> Kihwal
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> ________________________________
> >> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> >> > common-dev@hadoop.apache.org>
> >> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
> hdfs-dev@hadoop.apache.org
> >> >;
> >> > "
> >> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> >> > mapreduce-dev@hadoop.apache.org>
> >> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
> release,
> >> > > > >>>>> HADOOP-9845
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> Hi,
> >> > > > >>>>>>
> >> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> >> HADOOP-9346.
> >> > > > >>>>>> For more detail, please see the JIRA ticket:
> >> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >> > > > >>>>>>
> >> > > > >>>>>> - Tsuyoshi
> >> > > > >>>>>>
> >> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >> > > > >>> tucu@cloudera.com>
> >> > > > >>>>> wrote:
> >> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
> release.
> >> > > > >>>>>>>
> >> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> >> > > benefits
> >> > > > >>> to
> >> > > > >>>>>>> justify the upgrade.
> >> > > > >>>>>>>
> >> > > > >>>>>>> Doing the upgrade now, with the first beta, will make
> things
> >> > > easier
> >> > > > >>> for
> >> > > > >>>>>>> downstream projects (like HBase) using protobuf and
> adopting
> >> > > Hadoop
> >> > > > >>> 2.
> >> > > > >>>>> If
> >> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> >> > > support 2
> >> > > > >>>>>>> different versions and they my get in nasty waters due to
> >> > > classpath
> >> > > > >>>>> issues.
> >> > > > >>>>>>>
> >> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> >> > > 2.1.0-beta
> >> > > > >>>>> branch
> >> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> >> > layer
> >> > > > >>>>> YARN-885).
> >> > > > >>>>>>>
> >> > > > >>>>>>> Now, to do this it will require a few things:
> >> > > > >>>>>>>
> >> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> >> > should
> >> > > > >>>>> install
> >> > > > >>>>>>> locally protobuf 2.5.0
> >> > > > >>>>>>>
> >> > > > >>>>>>> Thanks.
> >> > > > >>>>>>>
> >> > > > >>>>>>> --
> >> > > > >>>>>>> Alejandro
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> --
> >> > > > >>>> Alejandro
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>> Alejandro
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Alejandro
> >> >
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I've just committed HADOOP-9845 to trunk (only trunk at the moment).
To build trunk now you need protoc 2.5.0 (the build will fail with a
warning if you don't have it).
We'd propagate this to the 2 branches once the precommit build is back to
normal and see things are OK.
Thanks.
On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> protoc 2.5.0.
>
> thx
>
>
> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I can take care of re-installing 2.4 and installing 2.5 in a different
>> location. This would fix 2.0 branch builds as well.
>> Thoughts?
>>
>> -Giri
>>
>>
>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Giri,
>> >
>> > first of all, thanks for installing protoc 2.5.0.
>> >
>> > I didn't know we were installing them as the only version and not
>> driven by
>> > env/path settings.
>> >
>> > Now we have a bit of a problem, precommit builds are broken because of
>> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >
>> > We have to options:
>> >
>> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
>> > follow up issues.
>> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> 2.5.0
>> > coexisting
>> >
>> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
>> > then merge it to the other branches.
>> >
>> > We need to sort this out quickly as precommits are not working.
>> >
>> > I'll wait till 3PM today for objections to option #1, if none I'll
>> commit
>> > it to trunk.
>> >
>> > Thanks.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> > gkesavan@hortonworks.com> wrote:
>> >
>> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
>> > path.
>> > > If we still need 2.4 I may have to install it. Let me know
>> > >
>> > > -Giri
>> > >
>> > >
>> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
>> we
>> > > can
>> > > > use and env to set it in the jobs?
>> > > >
>> > > > thx
>> > > >
>> > > > Alejandro
>> > > > (phone typing)
>> > > >
>> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> gkesavan@hortonworks.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> > > > >
>> > > > >
>> > > > >
>> > > > > -Giri
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> > > > > gkesavan@hortonworks.com> wrote:
>> > > > >
>> > > > >> Alejandro,
>> > > > >>
>> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> > > > >>
>> > > > >> -Giri
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> > tucu@cloudera.com
>> > > > >wrote:
>> > > > >>
>> > > > >>> pinging again, I need help from somebody with sudo access to the
>> > > hadoop
>> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
>> > hours
>> > > to
>> > > > >>> set
>> > > > >>> up myself.
>> > > > >>>
>> > > > >>> Please!!!
>> > > > >>>
>> > > > >>> thx
>> > > > >>>
>> > > > >>>
>> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> > > tucu@cloudera.com
>> > > > >>>> wrote:
>> > > > >>>
>> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
>> > hadoop
>> > > > >>>> jenkins boxes.
>> > > > >>>>
>> > > > >>>> Who can help with this? I assume somebody at Y!, right?
>> > > > >>>>
>> > > > >>>> Thx
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> eclark@apache.org>
>> > > > >>> wrote:
>> > > > >>>>
>> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
>> > have
>> > > > the
>> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> > running.
>> > > > >>>>> That is to say there are issues with ever having 2.4.x and
>> 2.5.x
>> > on
>> > > > >>>>> the same class path.
>> > > > >>>>>
>> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
>> classes
>> > we
>> > > > >>>>> could use. With that said HBase is getting pretty close to a
>> > > rather
>> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
>> > > sooner
>> > > > >>>>> rather than later would be great. If we could get this into
>> > 2.1.0
>> > > it
>> > > > >>>>> would be great as that would allow us to have a pretty easy
>> story
>> > > to
>> > > > >>>>> users with regards to protobuf version.
>> > > > >>>>>
>> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> kihwal@yahoo-inc.com
>> > >
>> > > > >>> wrote:
>> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
>> Avro.
>> > See
>> > > > >>>>> HADOOP-9672.
>> > > > >>>>>> The version we are using has memory leak and inefficiency
>> > issues.
>> > > > >>> We've
>> > > > >>>>> seen users running into it.
>> > > > >>>>>>
>> > > > >>>>>> Kihwal
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> ________________________________
>> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
>> > common-dev@hadoop.apache.org>
>> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
>> >;
>> > "
>> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
>> > mapreduce-dev@hadoop.apache.org>
>> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> > > > >>>>> HADOOP-9845
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Hi,
>> > > > >>>>>>
>> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
>> HADOOP-9346.
>> > > > >>>>>> For more detail, please see the JIRA ticket:
>> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> > > > >>>>>>
>> > > > >>>>>> - Tsuyoshi
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> > > > >>> tucu@cloudera.com>
>> > > > >>>>> wrote:
>> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> > > > >>>>>>>
>> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> > > benefits
>> > > > >>> to
>> > > > >>>>>>> justify the upgrade.
>> > > > >>>>>>>
>> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
>> > > easier
>> > > > >>> for
>> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
>> > > Hadoop
>> > > > >>> 2.
>> > > > >>>>> If
>> > > > >>>>>>> we do the upgrade later, downstream projects will have to
>> > > support 2
>> > > > >>>>>>> different versions and they my get in nasty waters due to
>> > > classpath
>> > > > >>>>> issues.
>> > > > >>>>>>>
>> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
>> > > 2.1.0-beta
>> > > > >>>>> branch
>> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
>> > layer
>> > > > >>>>> YARN-885).
>> > > > >>>>>>>
>> > > > >>>>>>> Now, to do this it will require a few things:
>> > > > >>>>>>>
>> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
>> > should
>> > > > >>>>> install
>> > > > >>>>>>> locally protobuf 2.5.0
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Alejandro
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> Alejandro
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Alejandro
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I've just committed HADOOP-9845 to trunk (only trunk at the moment).
To build trunk now you need protoc 2.5.0 (the build will fail with a
warning if you don't have it).
We'd propagate this to the 2 branches once the precommit build is back to
normal and see things are OK.
Thanks.
On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> protoc 2.5.0.
>
> thx
>
>
> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I can take care of re-installing 2.4 and installing 2.5 in a different
>> location. This would fix 2.0 branch builds as well.
>> Thoughts?
>>
>> -Giri
>>
>>
>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Giri,
>> >
>> > first of all, thanks for installing protoc 2.5.0.
>> >
>> > I didn't know we were installing them as the only version and not
>> driven by
>> > env/path settings.
>> >
>> > Now we have a bit of a problem, precommit builds are broken because of
>> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >
>> > We have to options:
>> >
>> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
>> > follow up issues.
>> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> 2.5.0
>> > coexisting
>> >
>> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
>> > then merge it to the other branches.
>> >
>> > We need to sort this out quickly as precommits are not working.
>> >
>> > I'll wait till 3PM today for objections to option #1, if none I'll
>> commit
>> > it to trunk.
>> >
>> > Thanks.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> > gkesavan@hortonworks.com> wrote:
>> >
>> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
>> > path.
>> > > If we still need 2.4 I may have to install it. Let me know
>> > >
>> > > -Giri
>> > >
>> > >
>> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
>> we
>> > > can
>> > > > use and env to set it in the jobs?
>> > > >
>> > > > thx
>> > > >
>> > > > Alejandro
>> > > > (phone typing)
>> > > >
>> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> gkesavan@hortonworks.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> > > > >
>> > > > >
>> > > > >
>> > > > > -Giri
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> > > > > gkesavan@hortonworks.com> wrote:
>> > > > >
>> > > > >> Alejandro,
>> > > > >>
>> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> > > > >>
>> > > > >> -Giri
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> > tucu@cloudera.com
>> > > > >wrote:
>> > > > >>
>> > > > >>> pinging again, I need help from somebody with sudo access to the
>> > > hadoop
>> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
>> > hours
>> > > to
>> > > > >>> set
>> > > > >>> up myself.
>> > > > >>>
>> > > > >>> Please!!!
>> > > > >>>
>> > > > >>> thx
>> > > > >>>
>> > > > >>>
>> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> > > tucu@cloudera.com
>> > > > >>>> wrote:
>> > > > >>>
>> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
>> > hadoop
>> > > > >>>> jenkins boxes.
>> > > > >>>>
>> > > > >>>> Who can help with this? I assume somebody at Y!, right?
>> > > > >>>>
>> > > > >>>> Thx
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> eclark@apache.org>
>> > > > >>> wrote:
>> > > > >>>>
>> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
>> > have
>> > > > the
>> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> > running.
>> > > > >>>>> That is to say there are issues with ever having 2.4.x and
>> 2.5.x
>> > on
>> > > > >>>>> the same class path.
>> > > > >>>>>
>> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
>> classes
>> > we
>> > > > >>>>> could use. With that said HBase is getting pretty close to a
>> > > rather
>> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
>> > > sooner
>> > > > >>>>> rather than later would be great. If we could get this into
>> > 2.1.0
>> > > it
>> > > > >>>>> would be great as that would allow us to have a pretty easy
>> story
>> > > to
>> > > > >>>>> users with regards to protobuf version.
>> > > > >>>>>
>> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> kihwal@yahoo-inc.com
>> > >
>> > > > >>> wrote:
>> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
>> Avro.
>> > See
>> > > > >>>>> HADOOP-9672.
>> > > > >>>>>> The version we are using has memory leak and inefficiency
>> > issues.
>> > > > >>> We've
>> > > > >>>>> seen users running into it.
>> > > > >>>>>>
>> > > > >>>>>> Kihwal
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> ________________________________
>> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
>> > common-dev@hadoop.apache.org>
>> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
>> >;
>> > "
>> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
>> > mapreduce-dev@hadoop.apache.org>
>> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> > > > >>>>> HADOOP-9845
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Hi,
>> > > > >>>>>>
>> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
>> HADOOP-9346.
>> > > > >>>>>> For more detail, please see the JIRA ticket:
>> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> > > > >>>>>>
>> > > > >>>>>> - Tsuyoshi
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> > > > >>> tucu@cloudera.com>
>> > > > >>>>> wrote:
>> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> > > > >>>>>>>
>> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> > > benefits
>> > > > >>> to
>> > > > >>>>>>> justify the upgrade.
>> > > > >>>>>>>
>> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
>> > > easier
>> > > > >>> for
>> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
>> > > Hadoop
>> > > > >>> 2.
>> > > > >>>>> If
>> > > > >>>>>>> we do the upgrade later, downstream projects will have to
>> > > support 2
>> > > > >>>>>>> different versions and they my get in nasty waters due to
>> > > classpath
>> > > > >>>>> issues.
>> > > > >>>>>>>
>> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
>> > > 2.1.0-beta
>> > > > >>>>> branch
>> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
>> > layer
>> > > > >>>>> YARN-885).
>> > > > >>>>>>>
>> > > > >>>>>>> Now, to do this it will require a few things:
>> > > > >>>>>>>
>> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
>> > should
>> > > > >>>>> install
>> > > > >>>>>>> locally protobuf 2.5.0
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Alejandro
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> Alejandro
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Alejandro
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I've just committed HADOOP-9845 to trunk (only trunk at the moment).
To build trunk now you need protoc 2.5.0 (the build will fail with a
warning if you don't have it).
We'd propagate this to the 2 branches once the precommit build is back to
normal and see things are OK.
Thanks.
On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> protoc 2.5.0.
>
> thx
>
>
> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I can take care of re-installing 2.4 and installing 2.5 in a different
>> location. This would fix 2.0 branch builds as well.
>> Thoughts?
>>
>> -Giri
>>
>>
>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Giri,
>> >
>> > first of all, thanks for installing protoc 2.5.0.
>> >
>> > I didn't know we were installing them as the only version and not
>> driven by
>> > env/path settings.
>> >
>> > Now we have a bit of a problem, precommit builds are broken because of
>> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >
>> > We have to options:
>> >
>> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
>> > follow up issues.
>> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> 2.5.0
>> > coexisting
>> >
>> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
>> > then merge it to the other branches.
>> >
>> > We need to sort this out quickly as precommits are not working.
>> >
>> > I'll wait till 3PM today for objections to option #1, if none I'll
>> commit
>> > it to trunk.
>> >
>> > Thanks.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> > gkesavan@hortonworks.com> wrote:
>> >
>> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
>> > path.
>> > > If we still need 2.4 I may have to install it. Let me know
>> > >
>> > > -Giri
>> > >
>> > >
>> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
>> we
>> > > can
>> > > > use and env to set it in the jobs?
>> > > >
>> > > > thx
>> > > >
>> > > > Alejandro
>> > > > (phone typing)
>> > > >
>> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> gkesavan@hortonworks.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> > > > >
>> > > > >
>> > > > >
>> > > > > -Giri
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> > > > > gkesavan@hortonworks.com> wrote:
>> > > > >
>> > > > >> Alejandro,
>> > > > >>
>> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> > > > >>
>> > > > >> -Giri
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> > tucu@cloudera.com
>> > > > >wrote:
>> > > > >>
>> > > > >>> pinging again, I need help from somebody with sudo access to the
>> > > hadoop
>> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
>> > hours
>> > > to
>> > > > >>> set
>> > > > >>> up myself.
>> > > > >>>
>> > > > >>> Please!!!
>> > > > >>>
>> > > > >>> thx
>> > > > >>>
>> > > > >>>
>> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> > > tucu@cloudera.com
>> > > > >>>> wrote:
>> > > > >>>
>> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
>> > hadoop
>> > > > >>>> jenkins boxes.
>> > > > >>>>
>> > > > >>>> Who can help with this? I assume somebody at Y!, right?
>> > > > >>>>
>> > > > >>>> Thx
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> eclark@apache.org>
>> > > > >>> wrote:
>> > > > >>>>
>> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
>> > have
>> > > > the
>> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> > running.
>> > > > >>>>> That is to say there are issues with ever having 2.4.x and
>> 2.5.x
>> > on
>> > > > >>>>> the same class path.
>> > > > >>>>>
>> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
>> classes
>> > we
>> > > > >>>>> could use. With that said HBase is getting pretty close to a
>> > > rather
>> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
>> > > sooner
>> > > > >>>>> rather than later would be great. If we could get this into
>> > 2.1.0
>> > > it
>> > > > >>>>> would be great as that would allow us to have a pretty easy
>> story
>> > > to
>> > > > >>>>> users with regards to protobuf version.
>> > > > >>>>>
>> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> kihwal@yahoo-inc.com
>> > >
>> > > > >>> wrote:
>> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
>> Avro.
>> > See
>> > > > >>>>> HADOOP-9672.
>> > > > >>>>>> The version we are using has memory leak and inefficiency
>> > issues.
>> > > > >>> We've
>> > > > >>>>> seen users running into it.
>> > > > >>>>>>
>> > > > >>>>>> Kihwal
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> ________________________________
>> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
>> > common-dev@hadoop.apache.org>
>> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
>> >;
>> > "
>> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
>> > mapreduce-dev@hadoop.apache.org>
>> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> > > > >>>>> HADOOP-9845
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Hi,
>> > > > >>>>>>
>> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
>> HADOOP-9346.
>> > > > >>>>>> For more detail, please see the JIRA ticket:
>> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> > > > >>>>>>
>> > > > >>>>>> - Tsuyoshi
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> > > > >>> tucu@cloudera.com>
>> > > > >>>>> wrote:
>> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> > > > >>>>>>>
>> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> > > benefits
>> > > > >>> to
>> > > > >>>>>>> justify the upgrade.
>> > > > >>>>>>>
>> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
>> > > easier
>> > > > >>> for
>> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
>> > > Hadoop
>> > > > >>> 2.
>> > > > >>>>> If
>> > > > >>>>>>> we do the upgrade later, downstream projects will have to
>> > > support 2
>> > > > >>>>>>> different versions and they my get in nasty waters due to
>> > > classpath
>> > > > >>>>> issues.
>> > > > >>>>>>>
>> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
>> > > 2.1.0-beta
>> > > > >>>>> branch
>> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
>> > layer
>> > > > >>>>> YARN-885).
>> > > > >>>>>>>
>> > > > >>>>>>> Now, to do this it will require a few things:
>> > > > >>>>>>>
>> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
>> > should
>> > > > >>>>> install
>> > > > >>>>>>> locally protobuf 2.5.0
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Alejandro
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> Alejandro
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Alejandro
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I've just committed HADOOP-9845 to trunk (only trunk at the moment).
To build trunk now you need protoc 2.5.0 (the build will fail with a
warning if you don't have it).
We'd propagate this to the 2 branches once the precommit build is back to
normal and see things are OK.
Thanks.
On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> protoc 2.5.0.
>
> thx
>
>
> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I can take care of re-installing 2.4 and installing 2.5 in a different
>> location. This would fix 2.0 branch builds as well.
>> Thoughts?
>>
>> -Giri
>>
>>
>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Giri,
>> >
>> > first of all, thanks for installing protoc 2.5.0.
>> >
>> > I didn't know we were installing them as the only version and not
>> driven by
>> > env/path settings.
>> >
>> > Now we have a bit of a problem, precommit builds are broken because of
>> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >
>> > We have to options:
>> >
>> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
>> > follow up issues.
>> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> 2.5.0
>> > coexisting
>> >
>> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
>> > then merge it to the other branches.
>> >
>> > We need to sort this out quickly as precommits are not working.
>> >
>> > I'll wait till 3PM today for objections to option #1, if none I'll
>> commit
>> > it to trunk.
>> >
>> > Thanks.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> > gkesavan@hortonworks.com> wrote:
>> >
>> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
>> > path.
>> > > If we still need 2.4 I may have to install it. Let me know
>> > >
>> > > -Giri
>> > >
>> > >
>> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
>> we
>> > > can
>> > > > use and env to set it in the jobs?
>> > > >
>> > > > thx
>> > > >
>> > > > Alejandro
>> > > > (phone typing)
>> > > >
>> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> gkesavan@hortonworks.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> > > > >
>> > > > >
>> > > > >
>> > > > > -Giri
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> > > > > gkesavan@hortonworks.com> wrote:
>> > > > >
>> > > > >> Alejandro,
>> > > > >>
>> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> > > > >>
>> > > > >> -Giri
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> > tucu@cloudera.com
>> > > > >wrote:
>> > > > >>
>> > > > >>> pinging again, I need help from somebody with sudo access to the
>> > > hadoop
>> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
>> > hours
>> > > to
>> > > > >>> set
>> > > > >>> up myself.
>> > > > >>>
>> > > > >>> Please!!!
>> > > > >>>
>> > > > >>> thx
>> > > > >>>
>> > > > >>>
>> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> > > tucu@cloudera.com
>> > > > >>>> wrote:
>> > > > >>>
>> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
>> > hadoop
>> > > > >>>> jenkins boxes.
>> > > > >>>>
>> > > > >>>> Who can help with this? I assume somebody at Y!, right?
>> > > > >>>>
>> > > > >>>> Thx
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> eclark@apache.org>
>> > > > >>> wrote:
>> > > > >>>>
>> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
>> > have
>> > > > the
>> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> > running.
>> > > > >>>>> That is to say there are issues with ever having 2.4.x and
>> 2.5.x
>> > on
>> > > > >>>>> the same class path.
>> > > > >>>>>
>> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
>> classes
>> > we
>> > > > >>>>> could use. With that said HBase is getting pretty close to a
>> > > rather
>> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
>> > > sooner
>> > > > >>>>> rather than later would be great. If we could get this into
>> > 2.1.0
>> > > it
>> > > > >>>>> would be great as that would allow us to have a pretty easy
>> story
>> > > to
>> > > > >>>>> users with regards to protobuf version.
>> > > > >>>>>
>> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> kihwal@yahoo-inc.com
>> > >
>> > > > >>> wrote:
>> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention
>> Avro.
>> > See
>> > > > >>>>> HADOOP-9672.
>> > > > >>>>>> The version we are using has memory leak and inefficiency
>> > issues.
>> > > > >>> We've
>> > > > >>>>> seen users running into it.
>> > > > >>>>>>
>> > > > >>>>>> Kihwal
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> ________________________________
>> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
>> > common-dev@hadoop.apache.org>
>> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
>> >;
>> > "
>> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
>> > mapreduce-dev@hadoop.apache.org>
>> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> > > > >>>>> HADOOP-9845
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Hi,
>> > > > >>>>>>
>> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
>> HADOOP-9346.
>> > > > >>>>>> For more detail, please see the JIRA ticket:
>> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> > > > >>>>>>
>> > > > >>>>>> - Tsuyoshi
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> > > > >>> tucu@cloudera.com>
>> > > > >>>>> wrote:
>> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> > > > >>>>>>>
>> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> > > benefits
>> > > > >>> to
>> > > > >>>>>>> justify the upgrade.
>> > > > >>>>>>>
>> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
>> > > easier
>> > > > >>> for
>> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
>> > > Hadoop
>> > > > >>> 2.
>> > > > >>>>> If
>> > > > >>>>>>> we do the upgrade later, downstream projects will have to
>> > > support 2
>> > > > >>>>>>> different versions and they my get in nasty waters due to
>> > > classpath
>> > > > >>>>> issues.
>> > > > >>>>>>>
>> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
>> > > 2.1.0-beta
>> > > > >>>>> branch
>> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
>> > layer
>> > > > >>>>> YARN-885).
>> > > > >>>>>>>
>> > > > >>>>>>> Now, to do this it will require a few things:
>> > > > >>>>>>>
>> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
>> > should
>> > > > >>>>> install
>> > > > >>>>>>> locally protobuf 2.5.0
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Alejandro
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> Alejandro
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Alejandro
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
protoc 2.5.0.
thx
On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> I can take care of re-installing 2.4 and installing 2.5 in a different
> location. This would fix 2.0 branch builds as well.
> Thoughts?
>
> -Giri
>
>
> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Giri,
> >
> > first of all, thanks for installing protoc 2.5.0.
> >
> > I didn't know we were installing them as the only version and not driven
> by
> > env/path settings.
> >
> > Now we have a bit of a problem, precommit builds are broken because of
> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >
> > We have to options:
> >
> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> > follow up issues.
> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> 2.5.0
> > coexisting
> >
> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> > then merge it to the other branches.
> >
> > We need to sort this out quickly as precommits are not working.
> >
> > I'll wait till 3PM today for objections to option #1, if none I'll
> commit
> > it to trunk.
> >
> > Thanks.
> >
> > Alejandro
> >
> >
> >
> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> > path.
> > > If we still need 2.4 I may have to install it. Let me know
> > >
> > > -Giri
> > >
> > >
> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
> we
> > > can
> > > > use and env to set it in the jobs?
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> gkesavan@hortonworks.com
> > >
> > > > wrote:
> > > >
> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > > >
> > > > >
> > > > >
> > > > > -Giri
> > > > >
> > > > >
> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > > gkesavan@hortonworks.com> wrote:
> > > > >
> > > > >> Alejandro,
> > > > >>
> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > > >>
> > > > >> -Giri
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >>
> > > > >>> pinging again, I need help from somebody with sudo access to the
> > > hadoop
> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> > hours
> > > to
> > > > >>> set
> > > > >>> up myself.
> > > > >>>
> > > > >>> Please!!!
> > > > >>>
> > > > >>> thx
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> > hadoop
> > > > >>>> jenkins boxes.
> > > > >>>>
> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > > >>>>
> > > > >>>> Thx
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> eclark@apache.org>
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
> > have
> > > > the
> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> > running.
> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> 2.5.x
> > on
> > > > >>>>> the same class path.
> > > > >>>>>
> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> classes
> > we
> > > > >>>>> could use. With that said HBase is getting pretty close to a
> > > rather
> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > > sooner
> > > > >>>>> rather than later would be great. If we could get this into
> > 2.1.0
> > > it
> > > > >>>>> would be great as that would allow us to have a pretty easy
> story
> > > to
> > > > >>>>> users with regards to protobuf version.
> > > > >>>>>
> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> kihwal@yahoo-inc.com
> > >
> > > > >>> wrote:
> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> > See
> > > > >>>>> HADOOP-9672.
> > > > >>>>>> The version we are using has memory leak and inefficiency
> > issues.
> > > > >>> We've
> > > > >>>>> seen users running into it.
> > > > >>>>>>
> > > > >>>>>> Kihwal
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> ________________________________
> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> > common-dev@hadoop.apache.org>
> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
> >;
> > "
> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> > mapreduce-dev@hadoop.apache.org>
> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > > >>>>> HADOOP-9845
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Hi,
> > > > >>>>>>
> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> HADOOP-9346.
> > > > >>>>>> For more detail, please see the JIRA ticket:
> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > > >>>>>>
> > > > >>>>>> - Tsuyoshi
> > > > >>>>>>
> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > > >>> tucu@cloudera.com>
> > > > >>>>> wrote:
> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > > >>>>>>>
> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > > benefits
> > > > >>> to
> > > > >>>>>>> justify the upgrade.
> > > > >>>>>>>
> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > > easier
> > > > >>> for
> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > > Hadoop
> > > > >>> 2.
> > > > >>>>> If
> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> > > support 2
> > > > >>>>>>> different versions and they my get in nasty waters due to
> > > classpath
> > > > >>>>> issues.
> > > > >>>>>>>
> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > > 2.1.0-beta
> > > > >>>>> branch
> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> > layer
> > > > >>>>> YARN-885).
> > > > >>>>>>>
> > > > >>>>>>> Now, to do this it will require a few things:
> > > > >>>>>>>
> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> > should
> > > > >>>>> install
> > > > >>>>>>> locally protobuf 2.5.0
> > > > >>>>>>>
> > > > >>>>>>> Thanks.
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Alejandro
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alejandro
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Alejandro
> > > > >>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
protoc 2.5.0.
thx
On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> I can take care of re-installing 2.4 and installing 2.5 in a different
> location. This would fix 2.0 branch builds as well.
> Thoughts?
>
> -Giri
>
>
> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Giri,
> >
> > first of all, thanks for installing protoc 2.5.0.
> >
> > I didn't know we were installing them as the only version and not driven
> by
> > env/path settings.
> >
> > Now we have a bit of a problem, precommit builds are broken because of
> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >
> > We have to options:
> >
> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> > follow up issues.
> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> 2.5.0
> > coexisting
> >
> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> > then merge it to the other branches.
> >
> > We need to sort this out quickly as precommits are not working.
> >
> > I'll wait till 3PM today for objections to option #1, if none I'll
> commit
> > it to trunk.
> >
> > Thanks.
> >
> > Alejandro
> >
> >
> >
> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> > path.
> > > If we still need 2.4 I may have to install it. Let me know
> > >
> > > -Giri
> > >
> > >
> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
> we
> > > can
> > > > use and env to set it in the jobs?
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> gkesavan@hortonworks.com
> > >
> > > > wrote:
> > > >
> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > > >
> > > > >
> > > > >
> > > > > -Giri
> > > > >
> > > > >
> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > > gkesavan@hortonworks.com> wrote:
> > > > >
> > > > >> Alejandro,
> > > > >>
> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > > >>
> > > > >> -Giri
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >>
> > > > >>> pinging again, I need help from somebody with sudo access to the
> > > hadoop
> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> > hours
> > > to
> > > > >>> set
> > > > >>> up myself.
> > > > >>>
> > > > >>> Please!!!
> > > > >>>
> > > > >>> thx
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> > hadoop
> > > > >>>> jenkins boxes.
> > > > >>>>
> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > > >>>>
> > > > >>>> Thx
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> eclark@apache.org>
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
> > have
> > > > the
> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> > running.
> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> 2.5.x
> > on
> > > > >>>>> the same class path.
> > > > >>>>>
> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> classes
> > we
> > > > >>>>> could use. With that said HBase is getting pretty close to a
> > > rather
> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > > sooner
> > > > >>>>> rather than later would be great. If we could get this into
> > 2.1.0
> > > it
> > > > >>>>> would be great as that would allow us to have a pretty easy
> story
> > > to
> > > > >>>>> users with regards to protobuf version.
> > > > >>>>>
> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> kihwal@yahoo-inc.com
> > >
> > > > >>> wrote:
> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> > See
> > > > >>>>> HADOOP-9672.
> > > > >>>>>> The version we are using has memory leak and inefficiency
> > issues.
> > > > >>> We've
> > > > >>>>> seen users running into it.
> > > > >>>>>>
> > > > >>>>>> Kihwal
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> ________________________________
> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> > common-dev@hadoop.apache.org>
> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
> >;
> > "
> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> > mapreduce-dev@hadoop.apache.org>
> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > > >>>>> HADOOP-9845
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Hi,
> > > > >>>>>>
> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> HADOOP-9346.
> > > > >>>>>> For more detail, please see the JIRA ticket:
> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > > >>>>>>
> > > > >>>>>> - Tsuyoshi
> > > > >>>>>>
> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > > >>> tucu@cloudera.com>
> > > > >>>>> wrote:
> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > > >>>>>>>
> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > > benefits
> > > > >>> to
> > > > >>>>>>> justify the upgrade.
> > > > >>>>>>>
> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > > easier
> > > > >>> for
> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > > Hadoop
> > > > >>> 2.
> > > > >>>>> If
> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> > > support 2
> > > > >>>>>>> different versions and they my get in nasty waters due to
> > > classpath
> > > > >>>>> issues.
> > > > >>>>>>>
> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > > 2.1.0-beta
> > > > >>>>> branch
> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> > layer
> > > > >>>>> YARN-885).
> > > > >>>>>>>
> > > > >>>>>>> Now, to do this it will require a few things:
> > > > >>>>>>>
> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> > should
> > > > >>>>> install
> > > > >>>>>>> locally protobuf 2.5.0
> > > > >>>>>>>
> > > > >>>>>>> Thanks.
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Alejandro
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alejandro
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Alejandro
> > > > >>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
protoc 2.5.0.
thx
On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> I can take care of re-installing 2.4 and installing 2.5 in a different
> location. This would fix 2.0 branch builds as well.
> Thoughts?
>
> -Giri
>
>
> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Giri,
> >
> > first of all, thanks for installing protoc 2.5.0.
> >
> > I didn't know we were installing them as the only version and not driven
> by
> > env/path settings.
> >
> > Now we have a bit of a problem, precommit builds are broken because of
> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >
> > We have to options:
> >
> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> > follow up issues.
> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> 2.5.0
> > coexisting
> >
> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> > then merge it to the other branches.
> >
> > We need to sort this out quickly as precommits are not working.
> >
> > I'll wait till 3PM today for objections to option #1, if none I'll
> commit
> > it to trunk.
> >
> > Thanks.
> >
> > Alejandro
> >
> >
> >
> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> > path.
> > > If we still need 2.4 I may have to install it. Let me know
> > >
> > > -Giri
> > >
> > >
> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
> we
> > > can
> > > > use and env to set it in the jobs?
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> gkesavan@hortonworks.com
> > >
> > > > wrote:
> > > >
> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > > >
> > > > >
> > > > >
> > > > > -Giri
> > > > >
> > > > >
> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > > gkesavan@hortonworks.com> wrote:
> > > > >
> > > > >> Alejandro,
> > > > >>
> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > > >>
> > > > >> -Giri
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >>
> > > > >>> pinging again, I need help from somebody with sudo access to the
> > > hadoop
> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> > hours
> > > to
> > > > >>> set
> > > > >>> up myself.
> > > > >>>
> > > > >>> Please!!!
> > > > >>>
> > > > >>> thx
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> > hadoop
> > > > >>>> jenkins boxes.
> > > > >>>>
> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > > >>>>
> > > > >>>> Thx
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> eclark@apache.org>
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
> > have
> > > > the
> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> > running.
> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> 2.5.x
> > on
> > > > >>>>> the same class path.
> > > > >>>>>
> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> classes
> > we
> > > > >>>>> could use. With that said HBase is getting pretty close to a
> > > rather
> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > > sooner
> > > > >>>>> rather than later would be great. If we could get this into
> > 2.1.0
> > > it
> > > > >>>>> would be great as that would allow us to have a pretty easy
> story
> > > to
> > > > >>>>> users with regards to protobuf version.
> > > > >>>>>
> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> kihwal@yahoo-inc.com
> > >
> > > > >>> wrote:
> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> > See
> > > > >>>>> HADOOP-9672.
> > > > >>>>>> The version we are using has memory leak and inefficiency
> > issues.
> > > > >>> We've
> > > > >>>>> seen users running into it.
> > > > >>>>>>
> > > > >>>>>> Kihwal
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> ________________________________
> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> > common-dev@hadoop.apache.org>
> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
> >;
> > "
> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> > mapreduce-dev@hadoop.apache.org>
> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > > >>>>> HADOOP-9845
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Hi,
> > > > >>>>>>
> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> HADOOP-9346.
> > > > >>>>>> For more detail, please see the JIRA ticket:
> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > > >>>>>>
> > > > >>>>>> - Tsuyoshi
> > > > >>>>>>
> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > > >>> tucu@cloudera.com>
> > > > >>>>> wrote:
> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > > >>>>>>>
> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > > benefits
> > > > >>> to
> > > > >>>>>>> justify the upgrade.
> > > > >>>>>>>
> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > > easier
> > > > >>> for
> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > > Hadoop
> > > > >>> 2.
> > > > >>>>> If
> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> > > support 2
> > > > >>>>>>> different versions and they my get in nasty waters due to
> > > classpath
> > > > >>>>> issues.
> > > > >>>>>>>
> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > > 2.1.0-beta
> > > > >>>>> branch
> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> > layer
> > > > >>>>> YARN-885).
> > > > >>>>>>>
> > > > >>>>>>> Now, to do this it will require a few things:
> > > > >>>>>>>
> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> > should
> > > > >>>>> install
> > > > >>>>>>> locally protobuf 2.5.0
> > > > >>>>>>>
> > > > >>>>>>> Thanks.
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Alejandro
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alejandro
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Alejandro
> > > > >>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
protoc 2.5.0.
thx
On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> I can take care of re-installing 2.4 and installing 2.5 in a different
> location. This would fix 2.0 branch builds as well.
> Thoughts?
>
> -Giri
>
>
> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Giri,
> >
> > first of all, thanks for installing protoc 2.5.0.
> >
> > I didn't know we were installing them as the only version and not driven
> by
> > env/path settings.
> >
> > Now we have a bit of a problem, precommit builds are broken because of
> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >
> > We have to options:
> >
> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> > follow up issues.
> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> 2.5.0
> > coexisting
> >
> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> > then merge it to the other branches.
> >
> > We need to sort this out quickly as precommits are not working.
> >
> > I'll wait till 3PM today for objections to option #1, if none I'll
> commit
> > it to trunk.
> >
> > Thanks.
> >
> > Alejandro
> >
> >
> >
> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> > path.
> > > If we still need 2.4 I may have to install it. Let me know
> > >
> > > -Giri
> > >
> > >
> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
> we
> > > can
> > > > use and env to set it in the jobs?
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> gkesavan@hortonworks.com
> > >
> > > > wrote:
> > > >
> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > > >
> > > > >
> > > > >
> > > > > -Giri
> > > > >
> > > > >
> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > > gkesavan@hortonworks.com> wrote:
> > > > >
> > > > >> Alejandro,
> > > > >>
> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > > >>
> > > > >> -Giri
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >>
> > > > >>> pinging again, I need help from somebody with sudo access to the
> > > hadoop
> > > > >>> jenkins boxes to do this or to get sudo access for a couple of
> > hours
> > > to
> > > > >>> set
> > > > >>> up myself.
> > > > >>>
> > > > >>> Please!!!
> > > > >>>
> > > > >>> thx
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> > hadoop
> > > > >>>> jenkins boxes.
> > > > >>>>
> > > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > > >>>>
> > > > >>>> Thx
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> eclark@apache.org>
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> In HBase land we've pretty well discovered that we'll need to
> > have
> > > > the
> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> > running.
> > > > >>>>> That is to say there are issues with ever having 2.4.x and
> 2.5.x
> > on
> > > > >>>>> the same class path.
> > > > >>>>>
> > > > >>>>> Upgrading to 2.5.x would be great, as it brings some new
> classes
> > we
> > > > >>>>> could use. With that said HBase is getting pretty close to a
> > > rather
> > > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > > sooner
> > > > >>>>> rather than later would be great. If we could get this into
> > 2.1.0
> > > it
> > > > >>>>> would be great as that would allow us to have a pretty easy
> story
> > > to
> > > > >>>>> users with regards to protobuf version.
> > > > >>>>>
> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> kihwal@yahoo-inc.com
> > >
> > > > >>> wrote:
> > > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> > See
> > > > >>>>> HADOOP-9672.
> > > > >>>>>> The version we are using has memory leak and inefficiency
> > issues.
> > > > >>> We've
> > > > >>>>> seen users running into it.
> > > > >>>>>>
> > > > >>>>>> Kihwal
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> ________________________________
> > > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
> > common-dev@hadoop.apache.org>
> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
> >;
> > "
> > > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
> > mapreduce-dev@hadoop.apache.org>
> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > > >>>>> HADOOP-9845
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Hi,
> > > > >>>>>>
> > > > >>>>>> About Hadoop, Harsh is dealing with this problem in
> HADOOP-9346.
> > > > >>>>>> For more detail, please see the JIRA ticket:
> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > > >>>>>>
> > > > >>>>>> - Tsuyoshi
> > > > >>>>>>
> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > > >>> tucu@cloudera.com>
> > > > >>>>> wrote:
> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > > >>>>>>>
> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > > benefits
> > > > >>> to
> > > > >>>>>>> justify the upgrade.
> > > > >>>>>>>
> > > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > > easier
> > > > >>> for
> > > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > > Hadoop
> > > > >>> 2.
> > > > >>>>> If
> > > > >>>>>>> we do the upgrade later, downstream projects will have to
> > > support 2
> > > > >>>>>>> different versions and they my get in nasty waters due to
> > > classpath
> > > > >>>>> issues.
> > > > >>>>>>>
> > > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > > 2.1.0-beta
> > > > >>>>> branch
> > > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> > layer
> > > > >>>>> YARN-885).
> > > > >>>>>>>
> > > > >>>>>>> Now, to do this it will require a few things:
> > > > >>>>>>>
> > > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> > should
> > > > >>>>> install
> > > > >>>>>>> locally protobuf 2.5.0
> > > > >>>>>>>
> > > > >>>>>>> Thanks.
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Alejandro
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alejandro
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Alejandro
> > > > >>
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I can take care of re-installing 2.4 and installing 2.5 in a different
location. This would fix 2.0 branch builds as well.
Thoughts?
-Giri
On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Giri,
>
> first of all, thanks for installing protoc 2.5.0.
>
> I didn't know we were installing them as the only version and not driven by
> env/path settings.
>
> Now we have a bit of a problem, precommit builds are broken because of
> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>
> We have to options:
>
> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> follow up issues.
> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
> coexisting
>
> My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> then merge it to the other branches.
>
> We need to sort this out quickly as precommits are not working.
>
> I'll wait till 3PM today for objections to option #1, if none I'll commit
> it to trunk.
>
> Thanks.
>
> Alejandro
>
>
>
> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
> > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> path.
> > If we still need 2.4 I may have to install it. Let me know
> >
> > -Giri
> >
> >
> > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> > can
> > > use and env to set it in the jobs?
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gkesavan@hortonworks.com
> >
> > > wrote:
> > >
> > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > >
> > > >
> > > >
> > > > -Giri
> > > >
> > > >
> > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > gkesavan@hortonworks.com> wrote:
> > > >
> > > >> Alejandro,
> > > >>
> > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > >>
> > > >> -Giri
> > > >>
> > > >>
> > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >>
> > > >>> pinging again, I need help from somebody with sudo access to the
> > hadoop
> > > >>> jenkins boxes to do this or to get sudo access for a couple of
> hours
> > to
> > > >>> set
> > > >>> up myself.
> > > >>>
> > > >>> Please!!!
> > > >>>
> > > >>> thx
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > >>>> wrote:
> > > >>>
> > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> hadoop
> > > >>>> jenkins boxes.
> > > >>>>
> > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > >>>>
> > > >>>> Thx
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > > >>> wrote:
> > > >>>>
> > > >>>>> In HBase land we've pretty well discovered that we'll need to
> have
> > > the
> > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> running.
> > > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x
> on
> > > >>>>> the same class path.
> > > >>>>>
> > > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes
> we
> > > >>>>> could use. With that said HBase is getting pretty close to a
> > rather
> > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > sooner
> > > >>>>> rather than later would be great. If we could get this into
> 2.1.0
> > it
> > > >>>>> would be great as that would allow us to have a pretty easy story
> > to
> > > >>>>> users with regards to protobuf version.
> > > >>>>>
> > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <kihwal@yahoo-inc.com
> >
> > > >>> wrote:
> > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> See
> > > >>>>> HADOOP-9672.
> > > >>>>>> The version we are using has memory leak and inefficiency
> issues.
> > > >>> We've
> > > >>>>> seen users running into it.
> > > >>>>>>
> > > >>>>>> Kihwal
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> ________________________________
> > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > >>>>>> To: "common-dev@hadoop.apache.org" <
> common-dev@hadoop.apache.org>
> > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>;
> "
> > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > >>>>> mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>
> > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > >>>>> HADOOP-9845
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > > >>>>>> For more detail, please see the JIRA ticket:
> > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > >>>>>>
> > > >>>>>> - Tsuyoshi
> > > >>>>>>
> > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > >>> tucu@cloudera.com>
> > > >>>>> wrote:
> > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > >>>>>>>
> > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > benefits
> > > >>> to
> > > >>>>>>> justify the upgrade.
> > > >>>>>>>
> > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > easier
> > > >>> for
> > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > Hadoop
> > > >>> 2.
> > > >>>>> If
> > > >>>>>>> we do the upgrade later, downstream projects will have to
> > support 2
> > > >>>>>>> different versions and they my get in nasty waters due to
> > classpath
> > > >>>>> issues.
> > > >>>>>>>
> > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > 2.1.0-beta
> > > >>>>> branch
> > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> layer
> > > >>>>> YARN-885).
> > > >>>>>>>
> > > >>>>>>> Now, to do this it will require a few things:
> > > >>>>>>>
> > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> should
> > > >>>>> install
> > > >>>>>>> locally protobuf 2.5.0
> > > >>>>>>>
> > > >>>>>>> Thanks.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Alejandro
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Alejandro
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Alejandro
> > > >>
> > > >>
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I can take care of re-installing 2.4 and installing 2.5 in a different
location. This would fix 2.0 branch builds as well.
Thoughts?
-Giri
On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Giri,
>
> first of all, thanks for installing protoc 2.5.0.
>
> I didn't know we were installing them as the only version and not driven by
> env/path settings.
>
> Now we have a bit of a problem, precommit builds are broken because of
> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>
> We have to options:
>
> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> follow up issues.
> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
> coexisting
>
> My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> then merge it to the other branches.
>
> We need to sort this out quickly as precommits are not working.
>
> I'll wait till 3PM today for objections to option #1, if none I'll commit
> it to trunk.
>
> Thanks.
>
> Alejandro
>
>
>
> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
> > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> path.
> > If we still need 2.4 I may have to install it. Let me know
> >
> > -Giri
> >
> >
> > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> > can
> > > use and env to set it in the jobs?
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gkesavan@hortonworks.com
> >
> > > wrote:
> > >
> > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > >
> > > >
> > > >
> > > > -Giri
> > > >
> > > >
> > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > gkesavan@hortonworks.com> wrote:
> > > >
> > > >> Alejandro,
> > > >>
> > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > >>
> > > >> -Giri
> > > >>
> > > >>
> > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >>
> > > >>> pinging again, I need help from somebody with sudo access to the
> > hadoop
> > > >>> jenkins boxes to do this or to get sudo access for a couple of
> hours
> > to
> > > >>> set
> > > >>> up myself.
> > > >>>
> > > >>> Please!!!
> > > >>>
> > > >>> thx
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > >>>> wrote:
> > > >>>
> > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> hadoop
> > > >>>> jenkins boxes.
> > > >>>>
> > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > >>>>
> > > >>>> Thx
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > > >>> wrote:
> > > >>>>
> > > >>>>> In HBase land we've pretty well discovered that we'll need to
> have
> > > the
> > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> running.
> > > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x
> on
> > > >>>>> the same class path.
> > > >>>>>
> > > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes
> we
> > > >>>>> could use. With that said HBase is getting pretty close to a
> > rather
> > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > sooner
> > > >>>>> rather than later would be great. If we could get this into
> 2.1.0
> > it
> > > >>>>> would be great as that would allow us to have a pretty easy story
> > to
> > > >>>>> users with regards to protobuf version.
> > > >>>>>
> > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <kihwal@yahoo-inc.com
> >
> > > >>> wrote:
> > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> See
> > > >>>>> HADOOP-9672.
> > > >>>>>> The version we are using has memory leak and inefficiency
> issues.
> > > >>> We've
> > > >>>>> seen users running into it.
> > > >>>>>>
> > > >>>>>> Kihwal
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> ________________________________
> > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > >>>>>> To: "common-dev@hadoop.apache.org" <
> common-dev@hadoop.apache.org>
> > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>;
> "
> > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > >>>>> mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>
> > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > >>>>> HADOOP-9845
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > > >>>>>> For more detail, please see the JIRA ticket:
> > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > >>>>>>
> > > >>>>>> - Tsuyoshi
> > > >>>>>>
> > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > >>> tucu@cloudera.com>
> > > >>>>> wrote:
> > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > >>>>>>>
> > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > benefits
> > > >>> to
> > > >>>>>>> justify the upgrade.
> > > >>>>>>>
> > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > easier
> > > >>> for
> > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > Hadoop
> > > >>> 2.
> > > >>>>> If
> > > >>>>>>> we do the upgrade later, downstream projects will have to
> > support 2
> > > >>>>>>> different versions and they my get in nasty waters due to
> > classpath
> > > >>>>> issues.
> > > >>>>>>>
> > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > 2.1.0-beta
> > > >>>>> branch
> > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> layer
> > > >>>>> YARN-885).
> > > >>>>>>>
> > > >>>>>>> Now, to do this it will require a few things:
> > > >>>>>>>
> > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> should
> > > >>>>> install
> > > >>>>>>> locally protobuf 2.5.0
> > > >>>>>>>
> > > >>>>>>> Thanks.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Alejandro
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Alejandro
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Alejandro
> > > >>
> > > >>
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I can take care of re-installing 2.4 and installing 2.5 in a different
location. This would fix 2.0 branch builds as well.
Thoughts?
-Giri
On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Giri,
>
> first of all, thanks for installing protoc 2.5.0.
>
> I didn't know we were installing them as the only version and not driven by
> env/path settings.
>
> Now we have a bit of a problem, precommit builds are broken because of
> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>
> We have to options:
>
> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> follow up issues.
> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
> coexisting
>
> My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> then merge it to the other branches.
>
> We need to sort this out quickly as precommits are not working.
>
> I'll wait till 3PM today for objections to option #1, if none I'll commit
> it to trunk.
>
> Thanks.
>
> Alejandro
>
>
>
> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
> > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> path.
> > If we still need 2.4 I may have to install it. Let me know
> >
> > -Giri
> >
> >
> > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> > can
> > > use and env to set it in the jobs?
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gkesavan@hortonworks.com
> >
> > > wrote:
> > >
> > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > >
> > > >
> > > >
> > > > -Giri
> > > >
> > > >
> > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > gkesavan@hortonworks.com> wrote:
> > > >
> > > >> Alejandro,
> > > >>
> > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > >>
> > > >> -Giri
> > > >>
> > > >>
> > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >>
> > > >>> pinging again, I need help from somebody with sudo access to the
> > hadoop
> > > >>> jenkins boxes to do this or to get sudo access for a couple of
> hours
> > to
> > > >>> set
> > > >>> up myself.
> > > >>>
> > > >>> Please!!!
> > > >>>
> > > >>> thx
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > >>>> wrote:
> > > >>>
> > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> hadoop
> > > >>>> jenkins boxes.
> > > >>>>
> > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > >>>>
> > > >>>> Thx
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > > >>> wrote:
> > > >>>>
> > > >>>>> In HBase land we've pretty well discovered that we'll need to
> have
> > > the
> > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> running.
> > > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x
> on
> > > >>>>> the same class path.
> > > >>>>>
> > > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes
> we
> > > >>>>> could use. With that said HBase is getting pretty close to a
> > rather
> > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > sooner
> > > >>>>> rather than later would be great. If we could get this into
> 2.1.0
> > it
> > > >>>>> would be great as that would allow us to have a pretty easy story
> > to
> > > >>>>> users with regards to protobuf version.
> > > >>>>>
> > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <kihwal@yahoo-inc.com
> >
> > > >>> wrote:
> > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> See
> > > >>>>> HADOOP-9672.
> > > >>>>>> The version we are using has memory leak and inefficiency
> issues.
> > > >>> We've
> > > >>>>> seen users running into it.
> > > >>>>>>
> > > >>>>>> Kihwal
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> ________________________________
> > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > >>>>>> To: "common-dev@hadoop.apache.org" <
> common-dev@hadoop.apache.org>
> > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>;
> "
> > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > >>>>> mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>
> > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > >>>>> HADOOP-9845
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > > >>>>>> For more detail, please see the JIRA ticket:
> > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > >>>>>>
> > > >>>>>> - Tsuyoshi
> > > >>>>>>
> > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > >>> tucu@cloudera.com>
> > > >>>>> wrote:
> > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > >>>>>>>
> > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > benefits
> > > >>> to
> > > >>>>>>> justify the upgrade.
> > > >>>>>>>
> > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > easier
> > > >>> for
> > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > Hadoop
> > > >>> 2.
> > > >>>>> If
> > > >>>>>>> we do the upgrade later, downstream projects will have to
> > support 2
> > > >>>>>>> different versions and they my get in nasty waters due to
> > classpath
> > > >>>>> issues.
> > > >>>>>>>
> > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > 2.1.0-beta
> > > >>>>> branch
> > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> layer
> > > >>>>> YARN-885).
> > > >>>>>>>
> > > >>>>>>> Now, to do this it will require a few things:
> > > >>>>>>>
> > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> should
> > > >>>>> install
> > > >>>>>>> locally protobuf 2.5.0
> > > >>>>>>>
> > > >>>>>>> Thanks.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Alejandro
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Alejandro
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Alejandro
> > > >>
> > > >>
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I can take care of re-installing 2.4 and installing 2.5 in a different
location. This would fix 2.0 branch builds as well.
Thoughts?
-Giri
On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Giri,
>
> first of all, thanks for installing protoc 2.5.0.
>
> I didn't know we were installing them as the only version and not driven by
> env/path settings.
>
> Now we have a bit of a problem, precommit builds are broken because of
> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>
> We have to options:
>
> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
> follow up issues.
> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
> coexisting
>
> My take would be to commit HADOOP-9845 in trunk, iron out any issues an
> then merge it to the other branches.
>
> We need to sort this out quickly as precommits are not working.
>
> I'll wait till 3PM today for objections to option #1, if none I'll commit
> it to trunk.
>
> Thanks.
>
> Alejandro
>
>
>
> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
> > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
> path.
> > If we still need 2.4 I may have to install it. Let me know
> >
> > -Giri
> >
> >
> > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> > can
> > > use and env to set it in the jobs?
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gkesavan@hortonworks.com
> >
> > > wrote:
> > >
> > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > > >
> > > >
> > > >
> > > > -Giri
> > > >
> > > >
> > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > > gkesavan@hortonworks.com> wrote:
> > > >
> > > >> Alejandro,
> > > >>
> > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > > >>
> > > >> -Giri
> > > >>
> > > >>
> > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >>
> > > >>> pinging again, I need help from somebody with sudo access to the
> > hadoop
> > > >>> jenkins boxes to do this or to get sudo access for a couple of
> hours
> > to
> > > >>> set
> > > >>> up myself.
> > > >>>
> > > >>> Please!!!
> > > >>>
> > > >>> thx
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > >>>> wrote:
> > > >>>
> > > >>>> To move forward with this we need protoc 2.5.0 in the apache
> hadoop
> > > >>>> jenkins boxes.
> > > >>>>
> > > >>>> Who can help with this? I assume somebody at Y!, right?
> > > >>>>
> > > >>>> Thx
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > > >>> wrote:
> > > >>>>
> > > >>>>> In HBase land we've pretty well discovered that we'll need to
> have
> > > the
> > > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> running.
> > > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x
> on
> > > >>>>> the same class path.
> > > >>>>>
> > > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes
> we
> > > >>>>> could use. With that said HBase is getting pretty close to a
> > rather
> > > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> > sooner
> > > >>>>> rather than later would be great. If we could get this into
> 2.1.0
> > it
> > > >>>>> would be great as that would allow us to have a pretty easy story
> > to
> > > >>>>> users with regards to protobuf version.
> > > >>>>>
> > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <kihwal@yahoo-inc.com
> >
> > > >>> wrote:
> > > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro.
> See
> > > >>>>> HADOOP-9672.
> > > >>>>>> The version we are using has memory leak and inefficiency
> issues.
> > > >>> We've
> > > >>>>> seen users running into it.
> > > >>>>>>
> > > >>>>>> Kihwal
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> ________________________________
> > > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > > >>>>>> To: "common-dev@hadoop.apache.org" <
> common-dev@hadoop.apache.org>
> > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>;
> "
> > > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > > >>>>> mapreduce-dev@hadoop.apache.org" <
> mapreduce-dev@hadoop.apache.org>
> > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > > >>>>> HADOOP-9845
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > > >>>>>> For more detail, please see the JIRA ticket:
> > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > > >>>>>>
> > > >>>>>> - Tsuyoshi
> > > >>>>>>
> > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > > >>> tucu@cloudera.com>
> > > >>>>> wrote:
> > > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > > >>>>>>>
> > > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> > benefits
> > > >>> to
> > > >>>>>>> justify the upgrade.
> > > >>>>>>>
> > > >>>>>>> Doing the upgrade now, with the first beta, will make things
> > easier
> > > >>> for
> > > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> > Hadoop
> > > >>> 2.
> > > >>>>> If
> > > >>>>>>> we do the upgrade later, downstream projects will have to
> > support 2
> > > >>>>>>> different versions and they my get in nasty waters due to
> > classpath
> > > >>>>> issues.
> > > >>>>>>>
> > > >>>>>>> I've locally tested the patch in a pseudo deployment of
> > 2.1.0-beta
> > > >>>>> branch
> > > >>>>>>> and it works fine (something is broken in trunk in the RPC
> layer
> > > >>>>> YARN-885).
> > > >>>>>>>
> > > >>>>>>> Now, to do this it will require a few things:
> > > >>>>>>>
> > > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > > >>>>>>> * A follow up email to dev@ aliases indicating developers
> should
> > > >>>>> install
> > > >>>>>>> locally protobuf 2.5.0
> > > >>>>>>>
> > > >>>>>>> Thanks.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Alejandro
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Alejandro
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Alejandro
> > > >>
> > > >>
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
first of all, thanks for installing protoc 2.5.0.
I didn't know we were installing them as the only version and not driven by
env/path settings.
Now we have a bit of a problem, precommit builds are broken because of
mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
We have to options:
1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
follow up issues.
2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
coexisting
My take would be to commit HADOOP-9845 in trunk, iron out any issues an
then merge it to the other branches.
We need to sort this out quickly as precommits are not working.
I'll wait till 3PM today for objections to option #1, if none I'll commit
it to trunk.
Thanks.
Alejandro
On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
> If we still need 2.4 I may have to install it. Let me know
>
> -Giri
>
>
> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> can
> > use and env to set it in the jobs?
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> > wrote:
> >
> > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > >
> > >
> > >
> > > -Giri
> > >
> > >
> > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > gkesavan@hortonworks.com> wrote:
> > >
> > >> Alejandro,
> > >>
> > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > >>
> > >> -Giri
> > >>
> > >>
> > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >>
> > >>> pinging again, I need help from somebody with sudo access to the
> hadoop
> > >>> jenkins boxes to do this or to get sudo access for a couple of hours
> to
> > >>> set
> > >>> up myself.
> > >>>
> > >>> Please!!!
> > >>>
> > >>> thx
> > >>>
> > >>>
> > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > >>>> wrote:
> > >>>
> > >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> > >>>> jenkins boxes.
> > >>>>
> > >>>> Who can help with this? I assume somebody at Y!, right?
> > >>>>
> > >>>> Thx
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> In HBase land we've pretty well discovered that we'll need to have
> > the
> > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> > >>>>> the same class path.
> > >>>>>
> > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> > >>>>> could use. With that said HBase is getting pretty close to a
> rather
> > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> sooner
> > >>>>> rather than later would be great. If we could get this into 2.1.0
> it
> > >>>>> would be great as that would allow us to have a pretty easy story
> to
> > >>>>> users with regards to protobuf version.
> > >>>>>
> > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> > >>> wrote:
> > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> > >>>>> HADOOP-9672.
> > >>>>>> The version we are using has memory leak and inefficiency issues.
> > >>> We've
> > >>>>> seen users running into it.
> > >>>>>>
> > >>>>>> Kihwal
> > >>>>>>
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > >>>>> HADOOP-9845
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > >>>>>> For more detail, please see the JIRA ticket:
> > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > >>>>>>
> > >>>>>> - Tsuyoshi
> > >>>>>>
> > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > >>> tucu@cloudera.com>
> > >>>>> wrote:
> > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > >>>>>>>
> > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> benefits
> > >>> to
> > >>>>>>> justify the upgrade.
> > >>>>>>>
> > >>>>>>> Doing the upgrade now, with the first beta, will make things
> easier
> > >>> for
> > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> Hadoop
> > >>> 2.
> > >>>>> If
> > >>>>>>> we do the upgrade later, downstream projects will have to
> support 2
> > >>>>>>> different versions and they my get in nasty waters due to
> classpath
> > >>>>> issues.
> > >>>>>>>
> > >>>>>>> I've locally tested the patch in a pseudo deployment of
> 2.1.0-beta
> > >>>>> branch
> > >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> > >>>>> YARN-885).
> > >>>>>>>
> > >>>>>>> Now, to do this it will require a few things:
> > >>>>>>>
> > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > >>>>>>> * A follow up email to dev@ aliases indicating developers should
> > >>>>> install
> > >>>>>>> locally protobuf 2.5.0
> > >>>>>>>
> > >>>>>>> Thanks.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Alejandro
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Alejandro
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Alejandro
> > >>
> > >>
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
first of all, thanks for installing protoc 2.5.0.
I didn't know we were installing them as the only version and not driven by
env/path settings.
Now we have a bit of a problem, precommit builds are broken because of
mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
We have to options:
1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
follow up issues.
2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
coexisting
My take would be to commit HADOOP-9845 in trunk, iron out any issues an
then merge it to the other branches.
We need to sort this out quickly as precommits are not working.
I'll wait till 3PM today for objections to option #1, if none I'll commit
it to trunk.
Thanks.
Alejandro
On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
> If we still need 2.4 I may have to install it. Let me know
>
> -Giri
>
>
> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> can
> > use and env to set it in the jobs?
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> > wrote:
> >
> > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > >
> > >
> > >
> > > -Giri
> > >
> > >
> > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > gkesavan@hortonworks.com> wrote:
> > >
> > >> Alejandro,
> > >>
> > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > >>
> > >> -Giri
> > >>
> > >>
> > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >>
> > >>> pinging again, I need help from somebody with sudo access to the
> hadoop
> > >>> jenkins boxes to do this or to get sudo access for a couple of hours
> to
> > >>> set
> > >>> up myself.
> > >>>
> > >>> Please!!!
> > >>>
> > >>> thx
> > >>>
> > >>>
> > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > >>>> wrote:
> > >>>
> > >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> > >>>> jenkins boxes.
> > >>>>
> > >>>> Who can help with this? I assume somebody at Y!, right?
> > >>>>
> > >>>> Thx
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> In HBase land we've pretty well discovered that we'll need to have
> > the
> > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> > >>>>> the same class path.
> > >>>>>
> > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> > >>>>> could use. With that said HBase is getting pretty close to a
> rather
> > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> sooner
> > >>>>> rather than later would be great. If we could get this into 2.1.0
> it
> > >>>>> would be great as that would allow us to have a pretty easy story
> to
> > >>>>> users with regards to protobuf version.
> > >>>>>
> > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> > >>> wrote:
> > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> > >>>>> HADOOP-9672.
> > >>>>>> The version we are using has memory leak and inefficiency issues.
> > >>> We've
> > >>>>> seen users running into it.
> > >>>>>>
> > >>>>>> Kihwal
> > >>>>>>
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > >>>>> HADOOP-9845
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > >>>>>> For more detail, please see the JIRA ticket:
> > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > >>>>>>
> > >>>>>> - Tsuyoshi
> > >>>>>>
> > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > >>> tucu@cloudera.com>
> > >>>>> wrote:
> > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > >>>>>>>
> > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> benefits
> > >>> to
> > >>>>>>> justify the upgrade.
> > >>>>>>>
> > >>>>>>> Doing the upgrade now, with the first beta, will make things
> easier
> > >>> for
> > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> Hadoop
> > >>> 2.
> > >>>>> If
> > >>>>>>> we do the upgrade later, downstream projects will have to
> support 2
> > >>>>>>> different versions and they my get in nasty waters due to
> classpath
> > >>>>> issues.
> > >>>>>>>
> > >>>>>>> I've locally tested the patch in a pseudo deployment of
> 2.1.0-beta
> > >>>>> branch
> > >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> > >>>>> YARN-885).
> > >>>>>>>
> > >>>>>>> Now, to do this it will require a few things:
> > >>>>>>>
> > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > >>>>>>> * A follow up email to dev@ aliases indicating developers should
> > >>>>> install
> > >>>>>>> locally protobuf 2.5.0
> > >>>>>>>
> > >>>>>>> Thanks.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Alejandro
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Alejandro
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Alejandro
> > >>
> > >>
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
first of all, thanks for installing protoc 2.5.0.
I didn't know we were installing them as the only version and not driven by
env/path settings.
Now we have a bit of a problem, precommit builds are broken because of
mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
We have to options:
1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
follow up issues.
2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
coexisting
My take would be to commit HADOOP-9845 in trunk, iron out any issues an
then merge it to the other branches.
We need to sort this out quickly as precommits are not working.
I'll wait till 3PM today for objections to option #1, if none I'll commit
it to trunk.
Thanks.
Alejandro
On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
> If we still need 2.4 I may have to install it. Let me know
>
> -Giri
>
>
> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> can
> > use and env to set it in the jobs?
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> > wrote:
> >
> > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > >
> > >
> > >
> > > -Giri
> > >
> > >
> > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > gkesavan@hortonworks.com> wrote:
> > >
> > >> Alejandro,
> > >>
> > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > >>
> > >> -Giri
> > >>
> > >>
> > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >>
> > >>> pinging again, I need help from somebody with sudo access to the
> hadoop
> > >>> jenkins boxes to do this or to get sudo access for a couple of hours
> to
> > >>> set
> > >>> up myself.
> > >>>
> > >>> Please!!!
> > >>>
> > >>> thx
> > >>>
> > >>>
> > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > >>>> wrote:
> > >>>
> > >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> > >>>> jenkins boxes.
> > >>>>
> > >>>> Who can help with this? I assume somebody at Y!, right?
> > >>>>
> > >>>> Thx
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> In HBase land we've pretty well discovered that we'll need to have
> > the
> > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> > >>>>> the same class path.
> > >>>>>
> > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> > >>>>> could use. With that said HBase is getting pretty close to a
> rather
> > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> sooner
> > >>>>> rather than later would be great. If we could get this into 2.1.0
> it
> > >>>>> would be great as that would allow us to have a pretty easy story
> to
> > >>>>> users with regards to protobuf version.
> > >>>>>
> > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> > >>> wrote:
> > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> > >>>>> HADOOP-9672.
> > >>>>>> The version we are using has memory leak and inefficiency issues.
> > >>> We've
> > >>>>> seen users running into it.
> > >>>>>>
> > >>>>>> Kihwal
> > >>>>>>
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > >>>>> HADOOP-9845
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > >>>>>> For more detail, please see the JIRA ticket:
> > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > >>>>>>
> > >>>>>> - Tsuyoshi
> > >>>>>>
> > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > >>> tucu@cloudera.com>
> > >>>>> wrote:
> > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > >>>>>>>
> > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> benefits
> > >>> to
> > >>>>>>> justify the upgrade.
> > >>>>>>>
> > >>>>>>> Doing the upgrade now, with the first beta, will make things
> easier
> > >>> for
> > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> Hadoop
> > >>> 2.
> > >>>>> If
> > >>>>>>> we do the upgrade later, downstream projects will have to
> support 2
> > >>>>>>> different versions and they my get in nasty waters due to
> classpath
> > >>>>> issues.
> > >>>>>>>
> > >>>>>>> I've locally tested the patch in a pseudo deployment of
> 2.1.0-beta
> > >>>>> branch
> > >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> > >>>>> YARN-885).
> > >>>>>>>
> > >>>>>>> Now, to do this it will require a few things:
> > >>>>>>>
> > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > >>>>>>> * A follow up email to dev@ aliases indicating developers should
> > >>>>> install
> > >>>>>>> locally protobuf 2.5.0
> > >>>>>>>
> > >>>>>>> Thanks.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Alejandro
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Alejandro
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Alejandro
> > >>
> > >>
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
first of all, thanks for installing protoc 2.5.0.
I didn't know we were installing them as the only version and not driven by
env/path settings.
Now we have a bit of a problem, precommit builds are broken because of
mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
We have to options:
1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
follow up issues.
2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and 2.5.0
coexisting
My take would be to commit HADOOP-9845 in trunk, iron out any issues an
then merge it to the other branches.
We need to sort this out quickly as precommits are not working.
I'll wait till 3PM today for objections to option #1, if none I'll commit
it to trunk.
Thanks.
Alejandro
On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
> If we still need 2.4 I may have to install it. Let me know
>
> -Giri
>
>
> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > thanks giri, how do we set 2.4 or 2.5., what is the path to both so we
> can
> > use and env to set it in the jobs?
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> > wrote:
> >
> > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> > >
> > >
> > >
> > > -Giri
> > >
> > >
> > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > > gkesavan@hortonworks.com> wrote:
> > >
> > >> Alejandro,
> > >>
> > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> > >>
> > >> -Giri
> > >>
> > >>
> > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >>
> > >>> pinging again, I need help from somebody with sudo access to the
> hadoop
> > >>> jenkins boxes to do this or to get sudo access for a couple of hours
> to
> > >>> set
> > >>> up myself.
> > >>>
> > >>> Please!!!
> > >>>
> > >>> thx
> > >>>
> > >>>
> > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > >>>> wrote:
> > >>>
> > >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> > >>>> jenkins boxes.
> > >>>>
> > >>>> Who can help with this? I assume somebody at Y!, right?
> > >>>>
> > >>>> Thx
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> In HBase land we've pretty well discovered that we'll need to have
> > the
> > >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> > >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> > >>>>> the same class path.
> > >>>>>
> > >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> > >>>>> could use. With that said HBase is getting pretty close to a
> rather
> > >>>>> large release (0.96.0 aka The Singularity) so getting this in
> sooner
> > >>>>> rather than later would be great. If we could get this into 2.1.0
> it
> > >>>>> would be great as that would allow us to have a pretty easy story
> to
> > >>>>> users with regards to protobuf version.
> > >>>>>
> > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> > >>> wrote:
> > >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> > >>>>> HADOOP-9672.
> > >>>>>> The version we are using has memory leak and inefficiency issues.
> > >>> We've
> > >>>>> seen users running into it.
> > >>>>>>
> > >>>>>> Kihwal
> > >>>>>>
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> > >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> > >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> > >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> > >>>>> HADOOP-9845
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > >>>>>> For more detail, please see the JIRA ticket:
> > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> > >>>>>>
> > >>>>>> - Tsuyoshi
> > >>>>>>
> > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> > >>> tucu@cloudera.com>
> > >>>>> wrote:
> > >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> > >>>>>>>
> > >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> benefits
> > >>> to
> > >>>>>>> justify the upgrade.
> > >>>>>>>
> > >>>>>>> Doing the upgrade now, with the first beta, will make things
> easier
> > >>> for
> > >>>>>>> downstream projects (like HBase) using protobuf and adopting
> Hadoop
> > >>> 2.
> > >>>>> If
> > >>>>>>> we do the upgrade later, downstream projects will have to
> support 2
> > >>>>>>> different versions and they my get in nasty waters due to
> classpath
> > >>>>> issues.
> > >>>>>>>
> > >>>>>>> I've locally tested the patch in a pseudo deployment of
> 2.1.0-beta
> > >>>>> branch
> > >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> > >>>>> YARN-885).
> > >>>>>>>
> > >>>>>>> Now, to do this it will require a few things:
> > >>>>>>>
> > >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> > >>>>>>> * A follow up email to dev@ aliases indicating developers should
> > >>>>> install
> > >>>>>>> locally protobuf 2.5.0
> > >>>>>>>
> > >>>>>>> Thanks.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Alejandro
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Alejandro
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Alejandro
> > >>
> > >>
> >
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
If we still need 2.4 I may have to install it. Let me know
-Giri
On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can
> use and env to set it in the jobs?
>
> thx
>
> Alejandro
> (phone typing)
>
> On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> wrote:
>
> > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >
> >
> >
> > -Giri
> >
> >
> > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> Alejandro,
> >>
> >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >>
> >> -Giri
> >>
> >>
> >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >>
> >>> pinging again, I need help from somebody with sudo access to the hadoop
> >>> jenkins boxes to do this or to get sudo access for a couple of hours to
> >>> set
> >>> up myself.
> >>>
> >>> Please!!!
> >>>
> >>> thx
> >>>
> >>>
> >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>>> wrote:
> >>>
> >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> >>>> jenkins boxes.
> >>>>
> >>>> Who can help with this? I assume somebody at Y!, right?
> >>>>
> >>>> Thx
> >>>>
> >>>>
> >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> >>> wrote:
> >>>>
> >>>>> In HBase land we've pretty well discovered that we'll need to have
> the
> >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >>>>> the same class path.
> >>>>>
> >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> >>>>> could use. With that said HBase is getting pretty close to a rather
> >>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
> >>>>> rather than later would be great. If we could get this into 2.1.0 it
> >>>>> would be great as that would allow us to have a pretty easy story to
> >>>>> users with regards to protobuf version.
> >>>>>
> >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> >>> wrote:
> >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> >>>>> HADOOP-9672.
> >>>>>> The version we are using has memory leak and inefficiency issues.
> >>> We've
> >>>>> seen users running into it.
> >>>>>>
> >>>>>> Kihwal
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >>>>> HADOOP-9845
> >>>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >>>>>> For more detail, please see the JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >>>>>>
> >>>>>> - Tsuyoshi
> >>>>>>
> >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >>> tucu@cloudera.com>
> >>>>> wrote:
> >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>>>>>>
> >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
> >>> to
> >>>>>>> justify the upgrade.
> >>>>>>>
> >>>>>>> Doing the upgrade now, with the first beta, will make things easier
> >>> for
> >>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
> >>> 2.
> >>>>> If
> >>>>>>> we do the upgrade later, downstream projects will have to support 2
> >>>>>>> different versions and they my get in nasty waters due to classpath
> >>>>> issues.
> >>>>>>>
> >>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >>>>> branch
> >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> >>>>> YARN-885).
> >>>>>>>
> >>>>>>> Now, to do this it will require a few things:
> >>>>>>>
> >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >>>>>>> * A follow up email to dev@ aliases indicating developers should
> >>>>> install
> >>>>>>> locally protobuf 2.5.0
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Alejandro
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
If we still need 2.4 I may have to install it. Let me know
-Giri
On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can
> use and env to set it in the jobs?
>
> thx
>
> Alejandro
> (phone typing)
>
> On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> wrote:
>
> > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >
> >
> >
> > -Giri
> >
> >
> > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> Alejandro,
> >>
> >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >>
> >> -Giri
> >>
> >>
> >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >>
> >>> pinging again, I need help from somebody with sudo access to the hadoop
> >>> jenkins boxes to do this or to get sudo access for a couple of hours to
> >>> set
> >>> up myself.
> >>>
> >>> Please!!!
> >>>
> >>> thx
> >>>
> >>>
> >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>>> wrote:
> >>>
> >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> >>>> jenkins boxes.
> >>>>
> >>>> Who can help with this? I assume somebody at Y!, right?
> >>>>
> >>>> Thx
> >>>>
> >>>>
> >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> >>> wrote:
> >>>>
> >>>>> In HBase land we've pretty well discovered that we'll need to have
> the
> >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >>>>> the same class path.
> >>>>>
> >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> >>>>> could use. With that said HBase is getting pretty close to a rather
> >>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
> >>>>> rather than later would be great. If we could get this into 2.1.0 it
> >>>>> would be great as that would allow us to have a pretty easy story to
> >>>>> users with regards to protobuf version.
> >>>>>
> >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> >>> wrote:
> >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> >>>>> HADOOP-9672.
> >>>>>> The version we are using has memory leak and inefficiency issues.
> >>> We've
> >>>>> seen users running into it.
> >>>>>>
> >>>>>> Kihwal
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >>>>> HADOOP-9845
> >>>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >>>>>> For more detail, please see the JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >>>>>>
> >>>>>> - Tsuyoshi
> >>>>>>
> >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >>> tucu@cloudera.com>
> >>>>> wrote:
> >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>>>>>>
> >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
> >>> to
> >>>>>>> justify the upgrade.
> >>>>>>>
> >>>>>>> Doing the upgrade now, with the first beta, will make things easier
> >>> for
> >>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
> >>> 2.
> >>>>> If
> >>>>>>> we do the upgrade later, downstream projects will have to support 2
> >>>>>>> different versions and they my get in nasty waters due to classpath
> >>>>> issues.
> >>>>>>>
> >>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >>>>> branch
> >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> >>>>> YARN-885).
> >>>>>>>
> >>>>>>> Now, to do this it will require a few things:
> >>>>>>>
> >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >>>>>>> * A follow up email to dev@ aliases indicating developers should
> >>>>> install
> >>>>>>> locally protobuf 2.5.0
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Alejandro
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default path.
If we still need 2.4 I may have to install it. Let me know
-Giri
On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can
> use and env to set it in the jobs?
>
> thx
>
> Alejandro
> (phone typing)
>
> On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com>
> wrote:
>
> > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >
> >
> >
> > -Giri
> >
> >
> > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> >> Alejandro,
> >>
> >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >>
> >> -Giri
> >>
> >>
> >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >>
> >>> pinging again, I need help from somebody with sudo access to the hadoop
> >>> jenkins boxes to do this or to get sudo access for a couple of hours to
> >>> set
> >>> up myself.
> >>>
> >>> Please!!!
> >>>
> >>> thx
> >>>
> >>>
> >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>>> wrote:
> >>>
> >>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
> >>>> jenkins boxes.
> >>>>
> >>>> Who can help with this? I assume somebody at Y!, right?
> >>>>
> >>>> Thx
> >>>>
> >>>>
> >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
> >>> wrote:
> >>>>
> >>>>> In HBase land we've pretty well discovered that we'll need to have
> the
> >>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >>>>> the same class path.
> >>>>>
> >>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
> >>>>> could use. With that said HBase is getting pretty close to a rather
> >>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
> >>>>> rather than later would be great. If we could get this into 2.1.0 it
> >>>>> would be great as that would allow us to have a pretty easy story to
> >>>>> users with regards to protobuf version.
> >>>>>
> >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> >>> wrote:
> >>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
> >>>>> HADOOP-9672.
> >>>>>> The version we are using has memory leak and inefficiency issues.
> >>> We've
> >>>>> seen users running into it.
> >>>>>>
> >>>>>> Kihwal
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >>>>> HADOOP-9845
> >>>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >>>>>> For more detail, please see the JIRA ticket:
> >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >>>>>>
> >>>>>> - Tsuyoshi
> >>>>>>
> >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >>> tucu@cloudera.com>
> >>>>> wrote:
> >>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>>>>>>
> >>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
> >>> to
> >>>>>>> justify the upgrade.
> >>>>>>>
> >>>>>>> Doing the upgrade now, with the first beta, will make things easier
> >>> for
> >>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
> >>> 2.
> >>>>> If
> >>>>>>> we do the upgrade later, downstream projects will have to support 2
> >>>>>>> different versions and they my get in nasty waters due to classpath
> >>>>> issues.
> >>>>>>>
> >>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >>>>> branch
> >>>>>>> and it works fine (something is broken in trunk in the RPC layer
> >>>>> YARN-885).
> >>>>>>>
> >>>>>>> Now, to do this it will require a few things:
> >>>>>>>
> >>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >>>>>>> * A follow up email to dev@ aliases indicating developers should
> >>>>> install
> >>>>>>> locally protobuf 2.5.0
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Alejandro
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can use and env to set it in the jobs?
thx
Alejandro
(phone typing)
On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com> wrote:
> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>
>
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> Alejandro,
>>
>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>
>> -Giri
>>
>>
>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> pinging again, I need help from somebody with sudo access to the hadoop
>>> jenkins boxes to do this or to get sudo access for a couple of hours to
>>> set
>>> up myself.
>>>
>>> Please!!!
>>>
>>> thx
>>>
>>>
>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>
>>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
>>>> jenkins boxes.
>>>>
>>>> Who can help with this? I assume somebody at Y!, right?
>>>>
>>>> Thx
>>>>
>>>>
>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>>> wrote:
>>>>
>>>>> In HBase land we've pretty well discovered that we'll need to have the
>>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>>>>> the same class path.
>>>>>
>>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
>>>>> could use. With that said HBase is getting pretty close to a rather
>>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
>>>>> rather than later would be great. If we could get this into 2.1.0 it
>>>>> would be great as that would allow us to have a pretty easy story to
>>>>> users with regards to protobuf version.
>>>>>
>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>>> wrote:
>>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
>>>>> HADOOP-9672.
>>>>>> The version we are using has memory leak and inefficiency issues.
>>> We've
>>>>> seen users running into it.
>>>>>>
>>>>>> Kihwal
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>>>>> HADOOP-9845
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>>>>> For more detail, please see the JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>
>>>>>> - Tsuyoshi
>>>>>>
>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>> tucu@cloudera.com>
>>>>> wrote:
>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>>>>
>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>>> to
>>>>>>> justify the upgrade.
>>>>>>>
>>>>>>> Doing the upgrade now, with the first beta, will make things easier
>>> for
>>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
>>> 2.
>>>>> If
>>>>>>> we do the upgrade later, downstream projects will have to support 2
>>>>>>> different versions and they my get in nasty waters due to classpath
>>>>> issues.
>>>>>>>
>>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>>>>> branch
>>>>>>> and it works fine (something is broken in trunk in the RPC layer
>>>>> YARN-885).
>>>>>>>
>>>>>>> Now, to do this it will require a few things:
>>>>>>>
>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>> * A follow up email to dev@ aliases indicating developers should
>>>>> install
>>>>>>> locally protobuf 2.5.0
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Alejandro
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can use and env to set it in the jobs?
thx
Alejandro
(phone typing)
On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com> wrote:
> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>
>
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> Alejandro,
>>
>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>
>> -Giri
>>
>>
>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> pinging again, I need help from somebody with sudo access to the hadoop
>>> jenkins boxes to do this or to get sudo access for a couple of hours to
>>> set
>>> up myself.
>>>
>>> Please!!!
>>>
>>> thx
>>>
>>>
>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>
>>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
>>>> jenkins boxes.
>>>>
>>>> Who can help with this? I assume somebody at Y!, right?
>>>>
>>>> Thx
>>>>
>>>>
>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>>> wrote:
>>>>
>>>>> In HBase land we've pretty well discovered that we'll need to have the
>>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>>>>> the same class path.
>>>>>
>>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
>>>>> could use. With that said HBase is getting pretty close to a rather
>>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
>>>>> rather than later would be great. If we could get this into 2.1.0 it
>>>>> would be great as that would allow us to have a pretty easy story to
>>>>> users with regards to protobuf version.
>>>>>
>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>>> wrote:
>>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
>>>>> HADOOP-9672.
>>>>>> The version we are using has memory leak and inefficiency issues.
>>> We've
>>>>> seen users running into it.
>>>>>>
>>>>>> Kihwal
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>>>>> HADOOP-9845
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>>>>> For more detail, please see the JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>
>>>>>> - Tsuyoshi
>>>>>>
>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>> tucu@cloudera.com>
>>>>> wrote:
>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>>>>
>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>>> to
>>>>>>> justify the upgrade.
>>>>>>>
>>>>>>> Doing the upgrade now, with the first beta, will make things easier
>>> for
>>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
>>> 2.
>>>>> If
>>>>>>> we do the upgrade later, downstream projects will have to support 2
>>>>>>> different versions and they my get in nasty waters due to classpath
>>>>> issues.
>>>>>>>
>>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>>>>> branch
>>>>>>> and it works fine (something is broken in trunk in the RPC layer
>>>>> YARN-885).
>>>>>>>
>>>>>>> Now, to do this it will require a few things:
>>>>>>>
>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>> * A follow up email to dev@ aliases indicating developers should
>>>>> install
>>>>>>> locally protobuf 2.5.0
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Alejandro
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can use and env to set it in the jobs?
thx
Alejandro
(phone typing)
On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com> wrote:
> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>
>
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> Alejandro,
>>
>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>
>> -Giri
>>
>>
>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> pinging again, I need help from somebody with sudo access to the hadoop
>>> jenkins boxes to do this or to get sudo access for a couple of hours to
>>> set
>>> up myself.
>>>
>>> Please!!!
>>>
>>> thx
>>>
>>>
>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>
>>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
>>>> jenkins boxes.
>>>>
>>>> Who can help with this? I assume somebody at Y!, right?
>>>>
>>>> Thx
>>>>
>>>>
>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>>> wrote:
>>>>
>>>>> In HBase land we've pretty well discovered that we'll need to have the
>>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>>>>> the same class path.
>>>>>
>>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
>>>>> could use. With that said HBase is getting pretty close to a rather
>>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
>>>>> rather than later would be great. If we could get this into 2.1.0 it
>>>>> would be great as that would allow us to have a pretty easy story to
>>>>> users with regards to protobuf version.
>>>>>
>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>>> wrote:
>>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
>>>>> HADOOP-9672.
>>>>>> The version we are using has memory leak and inefficiency issues.
>>> We've
>>>>> seen users running into it.
>>>>>>
>>>>>> Kihwal
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>>>>> HADOOP-9845
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>>>>> For more detail, please see the JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>
>>>>>> - Tsuyoshi
>>>>>>
>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>> tucu@cloudera.com>
>>>>> wrote:
>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>>>>
>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>>> to
>>>>>>> justify the upgrade.
>>>>>>>
>>>>>>> Doing the upgrade now, with the first beta, will make things easier
>>> for
>>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
>>> 2.
>>>>> If
>>>>>>> we do the upgrade later, downstream projects will have to support 2
>>>>>>> different versions and they my get in nasty waters due to classpath
>>>>> issues.
>>>>>>>
>>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>>>>> branch
>>>>>>> and it works fine (something is broken in trunk in the RPC layer
>>>>> YARN-885).
>>>>>>>
>>>>>>> Now, to do this it will require a few things:
>>>>>>>
>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>> * A follow up email to dev@ aliases indicating developers should
>>>>> install
>>>>>>> locally protobuf 2.5.0
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Alejandro
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
thanks giri, how do we set 2.4 or 2.5., what is the path to both so we can use and env to set it in the jobs?
thx
Alejandro
(phone typing)
On Aug 9, 2013, at 23:10, Giridharan Kesavan <gk...@hortonworks.com> wrote:
> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>
>
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> Alejandro,
>>
>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>
>> -Giri
>>
>>
>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> pinging again, I need help from somebody with sudo access to the hadoop
>>> jenkins boxes to do this or to get sudo access for a couple of hours to
>>> set
>>> up myself.
>>>
>>> Please!!!
>>>
>>> thx
>>>
>>>
>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>
>>>> To move forward with this we need protoc 2.5.0 in the apache hadoop
>>>> jenkins boxes.
>>>>
>>>> Who can help with this? I assume somebody at Y!, right?
>>>>
>>>> Thx
>>>>
>>>>
>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>>> wrote:
>>>>
>>>>> In HBase land we've pretty well discovered that we'll need to have the
>>>>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>>>>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>>>>> the same class path.
>>>>>
>>>>> Upgrading to 2.5.x would be great, as it brings some new classes we
>>>>> could use. With that said HBase is getting pretty close to a rather
>>>>> large release (0.96.0 aka The Singularity) so getting this in sooner
>>>>> rather than later would be great. If we could get this into 2.1.0 it
>>>>> would be great as that would allow us to have a pretty easy story to
>>>>> users with regards to protobuf version.
>>>>>
>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>>> wrote:
>>>>>> Sorry to hijack the thread but, I also wanted to mention Avro. See
>>>>> HADOOP-9672.
>>>>>> The version we are using has memory leak and inefficiency issues.
>>> We've
>>>>> seen users running into it.
>>>>>>
>>>>>> Kihwal
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>>>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>>>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>>>>> HADOOP-9845
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>>>>> For more detail, please see the JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>>>>
>>>>>> - Tsuyoshi
>>>>>>
>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>> tucu@cloudera.com>
>>>>> wrote:
>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>>>>
>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>>> to
>>>>>>> justify the upgrade.
>>>>>>>
>>>>>>> Doing the upgrade now, with the first beta, will make things easier
>>> for
>>>>>>> downstream projects (like HBase) using protobuf and adopting Hadoop
>>> 2.
>>>>> If
>>>>>>> we do the upgrade later, downstream projects will have to support 2
>>>>>>> different versions and they my get in nasty waters due to classpath
>>>>> issues.
>>>>>>>
>>>>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>>>>> branch
>>>>>>> and it works fine (something is broken in trunk in the RPC layer
>>>>> YARN-885).
>>>>>>>
>>>>>>> Now, to do this it will require a few things:
>>>>>>>
>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>>>>> * A follow up email to dev@ aliases indicating developers should
>>>>> install
>>>>>>> locally protobuf 2.5.0
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Alejandro
>>>>
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
-Giri
On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Alejandro,
>
> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> pinging again, I need help from somebody with sudo access to the hadoop
>> jenkins boxes to do this or to get sudo access for a couple of hours to
>> set
>> up myself.
>>
>> Please!!!
>>
>> thx
>>
>>
>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > To move forward with this we need protoc 2.5.0 in the apache hadoop
>> > jenkins boxes.
>> >
>> > Who can help with this? I assume somebody at Y!, right?
>> >
>> > Thx
>> >
>> >
>> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>> wrote:
>> >
>> >> In HBase land we've pretty well discovered that we'll need to have the
>> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> >> the same class path.
>> >>
>> >> Upgrading to 2.5.x would be great, as it brings some new classes we
>> >> could use. With that said HBase is getting pretty close to a rather
>> >> large release (0.96.0 aka The Singularity) so getting this in sooner
>> >> rather than later would be great. If we could get this into 2.1.0 it
>> >> would be great as that would allow us to have a pretty easy story to
>> >> users with regards to protobuf version.
>> >>
>> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>> wrote:
>> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> >> HADOOP-9672.
>> >> > The version we are using has memory leak and inefficiency issues.
>> We've
>> >> seen users running into it.
>> >> >
>> >> > Kihwal
>> >> >
>> >> >
>> >> > ________________________________
>> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> >> > Sent: Thursday, August 8, 2013 1:59 AM
>> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> >> HADOOP-9845
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> >> > For more detail, please see the JIRA ticket:
>> >> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >> >
>> >> > - Tsuyoshi
>> >> >
>> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> tucu@cloudera.com>
>> >> wrote:
>> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >> >>
>> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>> to
>> >> >> justify the upgrade.
>> >> >>
>> >> >> Doing the upgrade now, with the first beta, will make things easier
>> for
>> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
>> 2.
>> >> If
>> >> >> we do the upgrade later, downstream projects will have to support 2
>> >> >> different versions and they my get in nasty waters due to classpath
>> >> issues.
>> >> >>
>> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> >> branch
>> >> >> and it works fine (something is broken in trunk in the RPC layer
>> >> YARN-885).
>> >> >>
>> >> >> Now, to do this it will require a few things:
>> >> >>
>> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> >> * A follow up email to dev@ aliases indicating developers should
>> >> install
>> >> >> locally protobuf 2.5.0
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> --
>> >> >> Alejandro
>> >>
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
-Giri
On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Alejandro,
>
> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> pinging again, I need help from somebody with sudo access to the hadoop
>> jenkins boxes to do this or to get sudo access for a couple of hours to
>> set
>> up myself.
>>
>> Please!!!
>>
>> thx
>>
>>
>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > To move forward with this we need protoc 2.5.0 in the apache hadoop
>> > jenkins boxes.
>> >
>> > Who can help with this? I assume somebody at Y!, right?
>> >
>> > Thx
>> >
>> >
>> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>> wrote:
>> >
>> >> In HBase land we've pretty well discovered that we'll need to have the
>> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> >> the same class path.
>> >>
>> >> Upgrading to 2.5.x would be great, as it brings some new classes we
>> >> could use. With that said HBase is getting pretty close to a rather
>> >> large release (0.96.0 aka The Singularity) so getting this in sooner
>> >> rather than later would be great. If we could get this into 2.1.0 it
>> >> would be great as that would allow us to have a pretty easy story to
>> >> users with regards to protobuf version.
>> >>
>> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>> wrote:
>> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> >> HADOOP-9672.
>> >> > The version we are using has memory leak and inefficiency issues.
>> We've
>> >> seen users running into it.
>> >> >
>> >> > Kihwal
>> >> >
>> >> >
>> >> > ________________________________
>> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> >> > Sent: Thursday, August 8, 2013 1:59 AM
>> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> >> HADOOP-9845
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> >> > For more detail, please see the JIRA ticket:
>> >> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >> >
>> >> > - Tsuyoshi
>> >> >
>> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> tucu@cloudera.com>
>> >> wrote:
>> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >> >>
>> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>> to
>> >> >> justify the upgrade.
>> >> >>
>> >> >> Doing the upgrade now, with the first beta, will make things easier
>> for
>> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
>> 2.
>> >> If
>> >> >> we do the upgrade later, downstream projects will have to support 2
>> >> >> different versions and they my get in nasty waters due to classpath
>> >> issues.
>> >> >>
>> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> >> branch
>> >> >> and it works fine (something is broken in trunk in the RPC layer
>> >> YARN-885).
>> >> >>
>> >> >> Now, to do this it will require a few things:
>> >> >>
>> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> >> * A follow up email to dev@ aliases indicating developers should
>> >> install
>> >> >> locally protobuf 2.5.0
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> --
>> >> >> Alejandro
>> >>
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
-Giri
On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Alejandro,
>
> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>
> -Giri
>
>
> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> pinging again, I need help from somebody with sudo access to the hadoop
>> jenkins boxes to do this or to get sudo access for a couple of hours to
>> set
>> up myself.
>>
>> Please!!!
>>
>> thx
>>
>>
>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > To move forward with this we need protoc 2.5.0 in the apache hadoop
>> > jenkins boxes.
>> >
>> > Who can help with this? I assume somebody at Y!, right?
>> >
>> > Thx
>> >
>> >
>> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org>
>> wrote:
>> >
>> >> In HBase land we've pretty well discovered that we'll need to have the
>> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> >> the same class path.
>> >>
>> >> Upgrading to 2.5.x would be great, as it brings some new classes we
>> >> could use. With that said HBase is getting pretty close to a rather
>> >> large release (0.96.0 aka The Singularity) so getting this in sooner
>> >> rather than later would be great. If we could get this into 2.1.0 it
>> >> would be great as that would allow us to have a pretty easy story to
>> >> users with regards to protobuf version.
>> >>
>> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
>> wrote:
>> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> >> HADOOP-9672.
>> >> > The version we are using has memory leak and inefficiency issues.
>> We've
>> >> seen users running into it.
>> >> >
>> >> > Kihwal
>> >> >
>> >> >
>> >> > ________________________________
>> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> >> > Sent: Thursday, August 8, 2013 1:59 AM
>> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> >> HADOOP-9845
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> >> > For more detail, please see the JIRA ticket:
>> >> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >> >
>> >> > - Tsuyoshi
>> >> >
>> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> tucu@cloudera.com>
>> >> wrote:
>> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >> >>
>> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits
>> to
>> >> >> justify the upgrade.
>> >> >>
>> >> >> Doing the upgrade now, with the first beta, will make things easier
>> for
>> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
>> 2.
>> >> If
>> >> >> we do the upgrade later, downstream projects will have to support 2
>> >> >> different versions and they my get in nasty waters due to classpath
>> >> issues.
>> >> >>
>> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> >> branch
>> >> >> and it works fine (something is broken in trunk in the RPC layer
>> >> YARN-885).
>> >> >>
>> >> >> Now, to do this it will require a few things:
>> >> >>
>> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> >> * A follow up email to dev@ aliases indicating developers should
>> >> install
>> >> >> locally protobuf 2.5.0
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> --
>> >> >> Alejandro
>> >>
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Alejandro,
I'm upgrading protobuf on slaves hadoop1-hadoop9.
-Giri
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
>
> Please!!!
>
> thx
>
>
> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > To move forward with this we need protoc 2.5.0 in the apache hadoop
> > jenkins boxes.
> >
> > Who can help with this? I assume somebody at Y!, right?
> >
> > Thx
> >
> >
> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> >
> >> In HBase land we've pretty well discovered that we'll need to have the
> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >> the same class path.
> >>
> >> Upgrading to 2.5.x would be great, as it brings some new classes we
> >> could use. With that said HBase is getting pretty close to a rather
> >> large release (0.96.0 aka The Singularity) so getting this in sooner
> >> rather than later would be great. If we could get this into 2.1.0 it
> >> would be great as that would allow us to have a pretty easy story to
> >> users with regards to protobuf version.
> >>
> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> wrote:
> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> >> HADOOP-9672.
> >> > The version we are using has memory leak and inefficiency issues.
> We've
> >> seen users running into it.
> >> >
> >> > Kihwal
> >> >
> >> >
> >> > ________________________________
> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >> > Sent: Thursday, August 8, 2013 1:59 AM
> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >> HADOOP-9845
> >> >
> >> >
> >> > Hi,
> >> >
> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >> > For more detail, please see the JIRA ticket:
> >> > https://issues.apache.org/jira/browse/HADOOP-9346
> >> >
> >> > - Tsuyoshi
> >> >
> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >> wrote:
> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >> >>
> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> >> justify the upgrade.
> >> >>
> >> >> Doing the upgrade now, with the first beta, will make things easier
> for
> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
> 2.
> >> If
> >> >> we do the upgrade later, downstream projects will have to support 2
> >> >> different versions and they my get in nasty waters due to classpath
> >> issues.
> >> >>
> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >> branch
> >> >> and it works fine (something is broken in trunk in the RPC layer
> >> YARN-885).
> >> >>
> >> >> Now, to do this it will require a few things:
> >> >>
> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> >> * A follow up email to dev@ aliases indicating developers should
> >> install
> >> >> locally protobuf 2.5.0
> >> >>
> >> >> Thanks.
> >> >>
> >> >> --
> >> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
thx rajiv, i'll follow up with you offline
cheers
Alejandro
(phone typing)
On Aug 9, 2013, at 18:45, Rajiv Chittajallu <ra...@yahoo-inc.com> wrote:
>
>
> On Aug 9, 2013, at 14:43, "Roman Shaposhnik" <rv...@apache.org> wrote:
>
>> On Fri, Aug 9, 2013 at 2:28 PM, Doug Cutting <cu...@apache.org> wrote:
>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> pinging again, I need help from somebody with sudo access to the hadoop
>>>> jenkins boxes to do this or to get sudo access for a couple of hours to set
>>>> up myself.
>>>
>>> Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
>>>
>>> http://wiki.apache.org/general/Jenkins
>>
>> Last time I did I was told that those were special boxes donated by Yahoo! and
>> they required a different protocol for getting onto them. In fact, I
>> was under the
>> impression that even ASF INFRA guys couldn't admin those boxes. Not sure
>> whether it has changed -- hence asking here first.
>
> Yahoo hosts build slaves for Hadoop and other ASF projects. Since Hadoop development community requested to keep slaves separated from other ASF projects, Infra team doesn't want to maintain it.
>
> Machines in Y! data center (minerva, vesta, etc) are in apache.org domain and are managed by ASF infra. The rest, asf*.ygridcore.net are managed by me.
>
> Most of the accounts those boxes are not Yahoos. Any commiter, independent of their
> employer, can get access and the expectation the it would only be used for builds. This restriction was put in to stop people from using these as their personal dev boxes.
>
> I apologize for not responding soon. Alejandro did send me an email about this. It's been a crazy week.
>
>
>>
>> Thanks,
>> Roman.
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Rajiv Chittajallu <ra...@yahoo-inc.com>.
On Aug 9, 2013, at 14:43, "Roman Shaposhnik" <rv...@apache.org> wrote:
> On Fri, Aug 9, 2013 at 2:28 PM, Doug Cutting <cu...@apache.org> wrote:
>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> pinging again, I need help from somebody with sudo access to the hadoop
>>> jenkins boxes to do this or to get sudo access for a couple of hours to set
>>> up myself.
>>
>> Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
>>
>> http://wiki.apache.org/general/Jenkins
>
> Last time I did I was told that those were special boxes donated by Yahoo! and
> they required a different protocol for getting onto them. In fact, I
> was under the
> impression that even ASF INFRA guys couldn't admin those boxes. Not sure
> whether it has changed -- hence asking here first.
Yahoo hosts build slaves for Hadoop and other ASF projects. Since Hadoop development community requested to keep slaves separated from other ASF projects, Infra team doesn't want to maintain it.
Machines in Y! data center (minerva, vesta, etc) are in apache.org domain and are managed by ASF infra. The rest, asf*.ygridcore.net are managed by me.
Most of the accounts those boxes are not Yahoos. Any commiter, independent of their
employer, can get access and the expectation the it would only be used for builds. This restriction was put in to stop people from using these as their personal dev boxes.
I apologize for not responding soon. Alejandro did send me an email about this. It's been a crazy week.
>
> Thanks,
> Roman.
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Roman Shaposhnik <rv...@apache.org>.
On Fri, Aug 9, 2013 at 2:28 PM, Doug Cutting <cu...@apache.org> wrote:
> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> pinging again, I need help from somebody with sudo access to the hadoop
>> jenkins boxes to do this or to get sudo access for a couple of hours to set
>> up myself.
>
> Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
>
> http://wiki.apache.org/general/Jenkins
Last time I did I was told that those were special boxes donated by Yahoo! and
they required a different protocol for getting onto them. In fact, I
was under the
impression that even ASF INFRA guys couldn't admin those boxes. Not sure
whether it has changed -- hence asking here first.
Thanks,
Roman.
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Doug Cutting <cu...@apache.org>.
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
http://wiki.apache.org/general/Jenkins
Doug
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Roman Shaposhnik <rv...@apache.org>.
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
Speaking of which -- I'd be totally willing to volunteer as a back up
for Giri for cases like these. I'm already maintaining Bigtop's Jenkins
infrastructure. Please let me know if sharing creds (or something
along those lines) would be feasible.
Thanks,
Roman.
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Doug Cutting <cu...@apache.org>.
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
http://wiki.apache.org/general/Jenkins
Doug
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Roman Shaposhnik <rv...@apache.org>.
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
Speaking of which -- I'd be totally willing to volunteer as a back up
for Giri for cases like these. I'm already maintaining Bigtop's Jenkins
infrastructure. Please let me know if sharing creds (or something
along those lines) would be feasible.
Thanks,
Roman.
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Alejandro,
I'm upgrading protobuf on slaves hadoop1-hadoop9.
-Giri
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
>
> Please!!!
>
> thx
>
>
> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > To move forward with this we need protoc 2.5.0 in the apache hadoop
> > jenkins boxes.
> >
> > Who can help with this? I assume somebody at Y!, right?
> >
> > Thx
> >
> >
> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> >
> >> In HBase land we've pretty well discovered that we'll need to have the
> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >> the same class path.
> >>
> >> Upgrading to 2.5.x would be great, as it brings some new classes we
> >> could use. With that said HBase is getting pretty close to a rather
> >> large release (0.96.0 aka The Singularity) so getting this in sooner
> >> rather than later would be great. If we could get this into 2.1.0 it
> >> would be great as that would allow us to have a pretty easy story to
> >> users with regards to protobuf version.
> >>
> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> wrote:
> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> >> HADOOP-9672.
> >> > The version we are using has memory leak and inefficiency issues.
> We've
> >> seen users running into it.
> >> >
> >> > Kihwal
> >> >
> >> >
> >> > ________________________________
> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >> > Sent: Thursday, August 8, 2013 1:59 AM
> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >> HADOOP-9845
> >> >
> >> >
> >> > Hi,
> >> >
> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >> > For more detail, please see the JIRA ticket:
> >> > https://issues.apache.org/jira/browse/HADOOP-9346
> >> >
> >> > - Tsuyoshi
> >> >
> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >> wrote:
> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >> >>
> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> >> justify the upgrade.
> >> >>
> >> >> Doing the upgrade now, with the first beta, will make things easier
> for
> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
> 2.
> >> If
> >> >> we do the upgrade later, downstream projects will have to support 2
> >> >> different versions and they my get in nasty waters due to classpath
> >> issues.
> >> >>
> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >> branch
> >> >> and it works fine (something is broken in trunk in the RPC layer
> >> YARN-885).
> >> >>
> >> >> Now, to do this it will require a few things:
> >> >>
> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> >> * A follow up email to dev@ aliases indicating developers should
> >> install
> >> >> locally protobuf 2.5.0
> >> >>
> >> >> Thanks.
> >> >>
> >> >> --
> >> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Doug Cutting <cu...@apache.org>.
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
Have you asked on builds@ or filed an INFRA Jira issue or asked on #asfinfra?
http://wiki.apache.org/general/Jenkins
Doug
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Alejandro,
I'm upgrading protobuf on slaves hadoop1-hadoop9.
-Giri
On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> pinging again, I need help from somebody with sudo access to the hadoop
> jenkins boxes to do this or to get sudo access for a couple of hours to set
> up myself.
>
> Please!!!
>
> thx
>
>
> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > To move forward with this we need protoc 2.5.0 in the apache hadoop
> > jenkins boxes.
> >
> > Who can help with this? I assume somebody at Y!, right?
> >
> > Thx
> >
> >
> > On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> >
> >> In HBase land we've pretty well discovered that we'll need to have the
> >> same version of protobuf that the HDFS/Yarn/MR servers are running.
> >> That is to say there are issues with ever having 2.4.x and 2.5.x on
> >> the same class path.
> >>
> >> Upgrading to 2.5.x would be great, as it brings some new classes we
> >> could use. With that said HBase is getting pretty close to a rather
> >> large release (0.96.0 aka The Singularity) so getting this in sooner
> >> rather than later would be great. If we could get this into 2.1.0 it
> >> would be great as that would allow us to have a pretty easy story to
> >> users with regards to protobuf version.
> >>
> >> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com>
> wrote:
> >> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> >> HADOOP-9672.
> >> > The version we are using has memory leak and inefficiency issues.
> We've
> >> seen users running into it.
> >> >
> >> > Kihwal
> >> >
> >> >
> >> > ________________________________
> >> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> >> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> >> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> >> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> >> > Sent: Thursday, August 8, 2013 1:59 AM
> >> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
> >> HADOOP-9845
> >> >
> >> >
> >> > Hi,
> >> >
> >> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> >> > For more detail, please see the JIRA ticket:
> >> > https://issues.apache.org/jira/browse/HADOOP-9346
> >> >
> >> > - Tsuyoshi
> >> >
> >> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >> wrote:
> >> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >> >>
> >> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> >> justify the upgrade.
> >> >>
> >> >> Doing the upgrade now, with the first beta, will make things easier
> for
> >> >> downstream projects (like HBase) using protobuf and adopting Hadoop
> 2.
> >> If
> >> >> we do the upgrade later, downstream projects will have to support 2
> >> >> different versions and they my get in nasty waters due to classpath
> >> issues.
> >> >>
> >> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> >> branch
> >> >> and it works fine (something is broken in trunk in the RPC layer
> >> YARN-885).
> >> >>
> >> >> Now, to do this it will require a few things:
> >> >>
> >> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> >> * A follow up email to dev@ aliases indicating developers should
> >> install
> >> >> locally protobuf 2.5.0
> >> >>
> >> >> Thanks.
> >> >>
> >> >> --
> >> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
> >
>
>
>
> --
> Alejandro
>
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
pinging again, I need help from somebody with sudo access to the hadoop
jenkins boxes to do this or to get sudo access for a couple of hours to set
up myself.
Please!!!
thx
On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> To move forward with this we need protoc 2.5.0 in the apache hadoop
> jenkins boxes.
>
> Who can help with this? I assume somebody at Y!, right?
>
> Thx
>
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> HADOOP-9672.
>> > The version we are using has memory leak and inefficiency issues. We've
>> seen users running into it.
>> >
>> > Kihwal
>> >
>> >
>> > ________________________________
>> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> > Sent: Thursday, August 8, 2013 1:59 AM
>> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> HADOOP-9845
>> >
>> >
>> > Hi,
>> >
>> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> > For more detail, please see the JIRA ticket:
>> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >
>> > - Tsuyoshi
>> >
>> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >>
>> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> >> justify the upgrade.
>> >>
>> >> Doing the upgrade now, with the first beta, will make things easier for
>> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
>> If
>> >> we do the upgrade later, downstream projects will have to support 2
>> >> different versions and they my get in nasty waters due to classpath
>> issues.
>> >>
>> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> branch
>> >> and it works fine (something is broken in trunk in the RPC layer
>> YARN-885).
>> >>
>> >> Now, to do this it will require a few things:
>> >>
>> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> * A follow up email to dev@ aliases indicating developers should
>> install
>> >> locally protobuf 2.5.0
>> >>
>> >> Thanks.
>> >>
>> >> --
>> >> Alejandro
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
pinging again, I need help from somebody with sudo access to the hadoop
jenkins boxes to do this or to get sudo access for a couple of hours to set
up myself.
Please!!!
thx
On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> To move forward with this we need protoc 2.5.0 in the apache hadoop
> jenkins boxes.
>
> Who can help with this? I assume somebody at Y!, right?
>
> Thx
>
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> HADOOP-9672.
>> > The version we are using has memory leak and inefficiency issues. We've
>> seen users running into it.
>> >
>> > Kihwal
>> >
>> >
>> > ________________________________
>> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> > Sent: Thursday, August 8, 2013 1:59 AM
>> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> HADOOP-9845
>> >
>> >
>> > Hi,
>> >
>> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> > For more detail, please see the JIRA ticket:
>> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >
>> > - Tsuyoshi
>> >
>> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >>
>> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> >> justify the upgrade.
>> >>
>> >> Doing the upgrade now, with the first beta, will make things easier for
>> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
>> If
>> >> we do the upgrade later, downstream projects will have to support 2
>> >> different versions and they my get in nasty waters due to classpath
>> issues.
>> >>
>> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> branch
>> >> and it works fine (something is broken in trunk in the RPC layer
>> YARN-885).
>> >>
>> >> Now, to do this it will require a few things:
>> >>
>> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> * A follow up email to dev@ aliases indicating developers should
>> install
>> >> locally protobuf 2.5.0
>> >>
>> >> Thanks.
>> >>
>> >> --
>> >> Alejandro
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
pinging again, I need help from somebody with sudo access to the hadoop
jenkins boxes to do this or to get sudo access for a couple of hours to set
up myself.
Please!!!
thx
On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> To move forward with this we need protoc 2.5.0 in the apache hadoop
> jenkins boxes.
>
> Who can help with this? I assume somebody at Y!, right?
>
> Thx
>
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> HADOOP-9672.
>> > The version we are using has memory leak and inefficiency issues. We've
>> seen users running into it.
>> >
>> > Kihwal
>> >
>> >
>> > ________________________________
>> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> > Sent: Thursday, August 8, 2013 1:59 AM
>> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> HADOOP-9845
>> >
>> >
>> > Hi,
>> >
>> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> > For more detail, please see the JIRA ticket:
>> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >
>> > - Tsuyoshi
>> >
>> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >>
>> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> >> justify the upgrade.
>> >>
>> >> Doing the upgrade now, with the first beta, will make things easier for
>> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
>> If
>> >> we do the upgrade later, downstream projects will have to support 2
>> >> different versions and they my get in nasty waters due to classpath
>> issues.
>> >>
>> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> branch
>> >> and it works fine (something is broken in trunk in the RPC layer
>> YARN-885).
>> >>
>> >> Now, to do this it will require a few things:
>> >>
>> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> * A follow up email to dev@ aliases indicating developers should
>> install
>> >> locally protobuf 2.5.0
>> >>
>> >> Thanks.
>> >>
>> >> --
>> >> Alejandro
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
pinging again, I need help from somebody with sudo access to the hadoop
jenkins boxes to do this or to get sudo access for a couple of hours to set
up myself.
Please!!!
thx
On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> To move forward with this we need protoc 2.5.0 in the apache hadoop
> jenkins boxes.
>
> Who can help with this? I assume somebody at Y!, right?
>
> Thx
>
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> > Sorry to hijack the thread but, I also wanted to mention Avro. See
>> HADOOP-9672.
>> > The version we are using has memory leak and inefficiency issues. We've
>> seen users running into it.
>> >
>> > Kihwal
>> >
>> >
>> > ________________________________
>> > From: Tsuyoshi OZAWA <oz...@gmail.com>
>> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> > Sent: Thursday, August 8, 2013 1:59 AM
>> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release,
>> HADOOP-9845
>> >
>> >
>> > Hi,
>> >
>> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> > For more detail, please see the JIRA ticket:
>> > https://issues.apache.org/jira/browse/HADOOP-9346
>> >
>> > - Tsuyoshi
>> >
>> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>> >>
>> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> >> justify the upgrade.
>> >>
>> >> Doing the upgrade now, with the first beta, will make things easier for
>> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
>> If
>> >> we do the upgrade later, downstream projects will have to support 2
>> >> different versions and they my get in nasty waters due to classpath
>> issues.
>> >>
>> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
>> branch
>> >> and it works fine (something is broken in trunk in the RPC layer
>> YARN-885).
>> >>
>> >> Now, to do this it will require a few things:
>> >>
>> >> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >> * A follow up email to dev@ aliases indicating developers should
>> install
>> >> locally protobuf 2.5.0
>> >>
>> >> Thanks.
>> >>
>> >> --
>> >> Alejandro
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
To move forward with this we need protoc 2.5.0 in the apache hadoop jenkins
boxes.
Who can help with this? I assume somebody at Y!, right?
Thx
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> HADOOP-9672.
> > The version we are using has memory leak and inefficiency issues. We've
> seen users running into it.
> >
> > Kihwal
> >
> >
> > ________________________________
> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > Sent: Thursday, August 8, 2013 1:59 AM
> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> >
> >
> > Hi,
> >
> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > For more detail, please see the JIRA ticket:
> > https://issues.apache.org/jira/browse/HADOOP-9346
> >
> > - Tsuyoshi
> >
> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>
> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> justify the upgrade.
> >>
> >> Doing the upgrade now, with the first beta, will make things easier for
> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
> If
> >> we do the upgrade later, downstream projects will have to support 2
> >> different versions and they my get in nasty waters due to classpath
> issues.
> >>
> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> branch
> >> and it works fine (something is broken in trunk in the RPC layer
> YARN-885).
> >>
> >> Now, to do this it will require a few things:
> >>
> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> * A follow up email to dev@ aliases indicating developers should
> install
> >> locally protobuf 2.5.0
> >>
> >> Thanks.
> >>
> >> --
> >> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
To move forward with this we need protoc 2.5.0 in the apache hadoop jenkins
boxes.
Who can help with this? I assume somebody at Y!, right?
Thx
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> HADOOP-9672.
> > The version we are using has memory leak and inefficiency issues. We've
> seen users running into it.
> >
> > Kihwal
> >
> >
> > ________________________________
> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > Sent: Thursday, August 8, 2013 1:59 AM
> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> >
> >
> > Hi,
> >
> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > For more detail, please see the JIRA ticket:
> > https://issues.apache.org/jira/browse/HADOOP-9346
> >
> > - Tsuyoshi
> >
> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>
> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> justify the upgrade.
> >>
> >> Doing the upgrade now, with the first beta, will make things easier for
> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
> If
> >> we do the upgrade later, downstream projects will have to support 2
> >> different versions and they my get in nasty waters due to classpath
> issues.
> >>
> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> branch
> >> and it works fine (something is broken in trunk in the RPC layer
> YARN-885).
> >>
> >> Now, to do this it will require a few things:
> >>
> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> * A follow up email to dev@ aliases indicating developers should
> install
> >> locally protobuf 2.5.0
> >>
> >> Thanks.
> >>
> >> --
> >> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Andrew Purtell <ap...@apache.org>.
On Thu, Aug 8, 2013 at 3:34 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> > In HBase land we've pretty well discovered that we'll need to have the
> > same version of protobuf that the HDFS/Yarn/MR servers are running.
> > That is to say there are issues with ever having 2.4.x and 2.5.x on
> > the same class path.
>
As long as Hadoop and HBase both are built using protoc = protobuf-java =
2.5.0 things seem ok. I've not done full ecosystem tests though. Would be
interesting to upgrade a build box, do global substitutions in Maven and
Ivy specs, and see what happens.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Andrew Purtell <ap...@apache.org>.
On Thu, Aug 8, 2013 at 3:34 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> > In HBase land we've pretty well discovered that we'll need to have the
> > same version of protobuf that the HDFS/Yarn/MR servers are running.
> > That is to say there are issues with ever having 2.4.x and 2.5.x on
> > the same class path.
>
As long as Hadoop and HBase both are built using protoc = protobuf-java =
2.5.0 things seem ok. I've not done full ecosystem tests though. Would be
interesting to upgrade a build box, do global substitutions in Maven and
Ivy specs, and see what happens.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Alejandro
(phone typing)
On Aug 8, 2013, at 15:34, Roman Shaposhnik <rv...@apache.org> wrote:
> I'd be huge +1 on a flag day migrating us to 2.5.x.
>
> We can run a set of validation tests in Bigtop on all components
> that could be affected by it. Hadoop and HBase are the obvious
> suspects. What else in the ecosystem could be affected?
>
> Thanks,
> Roman.
>
> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
>> In HBase land we've pretty well discovered that we'll need to have the
>> same version of protobuf that the HDFS/Yarn/MR servers are running.
>> That is to say there are issues with ever having 2.4.x and 2.5.x on
>> the same class path.
>>
>> Upgrading to 2.5.x would be great, as it brings some new classes we
>> could use. With that said HBase is getting pretty close to a rather
>> large release (0.96.0 aka The Singularity) so getting this in sooner
>> rather than later would be great. If we could get this into 2.1.0 it
>> would be great as that would allow us to have a pretty easy story to
>> users with regards to protobuf version.
>>
>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>>> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
>>> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>>>
>>> Kihwal
>>>
>>>
>>> ________________________________
>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>>> Sent: Thursday, August 8, 2013 1:59 AM
>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>>>
>>>
>>> Hi,
>>>
>>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>>> For more detail, please see the JIRA ticket:
>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>>
>>> - Tsuyoshi
>>>
>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>>
>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>>>> justify the upgrade.
>>>>
>>>> Doing the upgrade now, with the first beta, will make things easier for
>>>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>>>> we do the upgrade later, downstream projects will have to support 2
>>>> different versions and they my get in nasty waters due to classpath issues.
>>>>
>>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>>>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>>>
>>>> Now, to do this it will require a few things:
>>>>
>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>>> * A follow up email to dev@ aliases indicating developers should install
>>>> locally protobuf 2.5.0
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Roman Shaposhnik <rv...@apache.org>.
I'd be huge +1 on a flag day migrating us to 2.5.x.
We can run a set of validation tests in Bigtop on all components
that could be affected by it. Hadoop and HBase are the obvious
suspects. What else in the ecosystem could be affected?
Thanks,
Roman.
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
>> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
>> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>>
>> Kihwal
>>
>>
>> ________________________________
>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
>> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
>> Sent: Thursday, August 8, 2013 1:59 AM
>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>>
>>
>> Hi,
>>
>> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
>> For more detail, please see the JIRA ticket:
>> https://issues.apache.org/jira/browse/HADOOP-9346
>>
>> - Tsuyoshi
>>
>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>>
>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>>> justify the upgrade.
>>>
>>> Doing the upgrade now, with the first beta, will make things easier for
>>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>>> we do the upgrade later, downstream projects will have to support 2
>>> different versions and they my get in nasty waters due to classpath issues.
>>>
>>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>>
>>> Now, to do this it will require a few things:
>>>
>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>> * A follow up email to dev@ aliases indicating developers should install
>>> locally protobuf 2.5.0
>>>
>>> Thanks.
>>>
>>> --
>>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
To move forward with this we need protoc 2.5.0 in the apache hadoop jenkins
boxes.
Who can help with this? I assume somebody at Y!, right?
Thx
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> HADOOP-9672.
> > The version we are using has memory leak and inefficiency issues. We've
> seen users running into it.
> >
> > Kihwal
> >
> >
> > ________________________________
> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > Sent: Thursday, August 8, 2013 1:59 AM
> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> >
> >
> > Hi,
> >
> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > For more detail, please see the JIRA ticket:
> > https://issues.apache.org/jira/browse/HADOOP-9346
> >
> > - Tsuyoshi
> >
> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>
> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> justify the upgrade.
> >>
> >> Doing the upgrade now, with the first beta, will make things easier for
> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
> If
> >> we do the upgrade later, downstream projects will have to support 2
> >> different versions and they my get in nasty waters due to classpath
> issues.
> >>
> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> branch
> >> and it works fine (something is broken in trunk in the RPC layer
> YARN-885).
> >>
> >> Now, to do this it will require a few things:
> >>
> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> * A follow up email to dev@ aliases indicating developers should
> install
> >> locally protobuf 2.5.0
> >>
> >> Thanks.
> >>
> >> --
> >> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
To move forward with this we need protoc 2.5.0 in the apache hadoop jenkins
boxes.
Who can help with this? I assume somebody at Y!, right?
Thx
On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <ec...@apache.org> wrote:
> In HBase land we've pretty well discovered that we'll need to have the
> same version of protobuf that the HDFS/Yarn/MR servers are running.
> That is to say there are issues with ever having 2.4.x and 2.5.x on
> the same class path.
>
> Upgrading to 2.5.x would be great, as it brings some new classes we
> could use. With that said HBase is getting pretty close to a rather
> large release (0.96.0 aka The Singularity) so getting this in sooner
> rather than later would be great. If we could get this into 2.1.0 it
> would be great as that would allow us to have a pretty easy story to
> users with regards to protobuf version.
>
> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> > Sorry to hijack the thread but, I also wanted to mention Avro. See
> HADOOP-9672.
> > The version we are using has memory leak and inefficiency issues. We've
> seen users running into it.
> >
> > Kihwal
> >
> >
> > ________________________________
> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> > To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> > Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> > Sent: Thursday, August 8, 2013 1:59 AM
> > Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
> >
> >
> > Hi,
> >
> > About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> > For more detail, please see the JIRA ticket:
> > https://issues.apache.org/jira/browse/HADOOP-9346
> >
> > - Tsuyoshi
> >
> > On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
> >>
> >> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> >> justify the upgrade.
> >>
> >> Doing the upgrade now, with the first beta, will make things easier for
> >> downstream projects (like HBase) using protobuf and adopting Hadoop 2.
> If
> >> we do the upgrade later, downstream projects will have to support 2
> >> different versions and they my get in nasty waters due to classpath
> issues.
> >>
> >> I've locally tested the patch in a pseudo deployment of 2.1.0-beta
> branch
> >> and it works fine (something is broken in trunk in the RPC layer
> YARN-885).
> >>
> >> Now, to do this it will require a few things:
> >>
> >> * Make sure protobuf 2.5.0 is available in the jenkins box
> >> * A follow up email to dev@ aliases indicating developers should
> install
> >> locally protobuf 2.5.0
> >>
> >> Thanks.
> >>
> >> --
> >> Alejandro
>
--
Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Elliott Clark <ec...@apache.org>.
In HBase land we've pretty well discovered that we'll need to have the
same version of protobuf that the HDFS/Yarn/MR servers are running.
That is to say there are issues with ever having 2.4.x and 2.5.x on
the same class path.
Upgrading to 2.5.x would be great, as it brings some new classes we
could use. With that said HBase is getting pretty close to a rather
large release (0.96.0 aka The Singularity) so getting this in sooner
rather than later would be great. If we could get this into 2.1.0 it
would be great as that would allow us to have a pretty easy story to
users with regards to protobuf version.
On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>
> Kihwal
>
>
> ________________________________
> From: Tsuyoshi OZAWA <oz...@gmail.com>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Sent: Thursday, August 8, 2013 1:59 AM
> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>
>
> Hi,
>
> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> For more detail, please see the JIRA ticket:
> https://issues.apache.org/jira/browse/HADOOP-9346
>
> - Tsuyoshi
>
> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>
>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> justify the upgrade.
>>
>> Doing the upgrade now, with the first beta, will make things easier for
>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>> we do the upgrade later, downstream projects will have to support 2
>> different versions and they my get in nasty waters due to classpath issues.
>>
>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>
>> Now, to do this it will require a few things:
>>
>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> * A follow up email to dev@ aliases indicating developers should install
>> locally protobuf 2.5.0
>>
>> Thanks.
>>
>> --
>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Elliott Clark <ec...@apache.org>.
In HBase land we've pretty well discovered that we'll need to have the
same version of protobuf that the HDFS/Yarn/MR servers are running.
That is to say there are issues with ever having 2.4.x and 2.5.x on
the same class path.
Upgrading to 2.5.x would be great, as it brings some new classes we
could use. With that said HBase is getting pretty close to a rather
large release (0.96.0 aka The Singularity) so getting this in sooner
rather than later would be great. If we could get this into 2.1.0 it
would be great as that would allow us to have a pretty easy story to
users with regards to protobuf version.
On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>
> Kihwal
>
>
> ________________________________
> From: Tsuyoshi OZAWA <oz...@gmail.com>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Sent: Thursday, August 8, 2013 1:59 AM
> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>
>
> Hi,
>
> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> For more detail, please see the JIRA ticket:
> https://issues.apache.org/jira/browse/HADOOP-9346
>
> - Tsuyoshi
>
> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>
>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> justify the upgrade.
>>
>> Doing the upgrade now, with the first beta, will make things easier for
>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>> we do the upgrade later, downstream projects will have to support 2
>> different versions and they my get in nasty waters due to classpath issues.
>>
>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>
>> Now, to do this it will require a few things:
>>
>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> * A follow up email to dev@ aliases indicating developers should install
>> locally protobuf 2.5.0
>>
>> Thanks.
>>
>> --
>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Elliott Clark <ec...@apache.org>.
In HBase land we've pretty well discovered that we'll need to have the
same version of protobuf that the HDFS/Yarn/MR servers are running.
That is to say there are issues with ever having 2.4.x and 2.5.x on
the same class path.
Upgrading to 2.5.x would be great, as it brings some new classes we
could use. With that said HBase is getting pretty close to a rather
large release (0.96.0 aka The Singularity) so getting this in sooner
rather than later would be great. If we could get this into 2.1.0 it
would be great as that would allow us to have a pretty easy story to
users with regards to protobuf version.
On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <ki...@yahoo-inc.com> wrote:
> Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
> The version we are using has memory leak and inefficiency issues. We've seen users running into it.
>
> Kihwal
>
>
> ________________________________
> From: Tsuyoshi OZAWA <oz...@gmail.com>
> To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
> Sent: Thursday, August 8, 2013 1:59 AM
> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
>
>
> Hi,
>
> About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
> For more detail, please see the JIRA ticket:
> https://issues.apache.org/jira/browse/HADOOP-9346
>
> - Tsuyoshi
>
> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>>
>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
>> justify the upgrade.
>>
>> Doing the upgrade now, with the first beta, will make things easier for
>> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
>> we do the upgrade later, downstream projects will have to support 2
>> different versions and they my get in nasty waters due to classpath issues.
>>
>> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
>> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>>
>> Now, to do this it will require a few things:
>>
>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> * A follow up email to dev@ aliases indicating developers should install
>> locally protobuf 2.5.0
>>
>> Thanks.
>>
>> --
>> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Kihwal Lee <ki...@yahoo-inc.com>.
Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
The version we are using has memory leak and inefficiency issues. We've seen users running into it.
Kihwal
________________________________
From: Tsuyoshi OZAWA <oz...@gmail.com>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Sent: Thursday, August 8, 2013 1:59 AM
Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Kihwal Lee <ki...@yahoo-inc.com>.
Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
The version we are using has memory leak and inefficiency issues. We've seen users running into it.
Kihwal
________________________________
From: Tsuyoshi OZAWA <oz...@gmail.com>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Sent: Thursday, August 8, 2013 1:59 AM
Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Kihwal Lee <ki...@yahoo-inc.com>.
Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
The version we are using has memory leak and inefficiency issues. We've seen users running into it.
Kihwal
________________________________
From: Tsuyoshi OZAWA <oz...@gmail.com>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Sent: Thursday, August 8, 2013 1:59 AM
Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro
Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346
- Tsuyoshi
On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.
>
> As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
> justify the upgrade.
>
> Doing the upgrade now, with the first beta, will make things easier for
> downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
> we do the upgrade later, downstream projects will have to support 2
> different versions and they my get in nasty waters due to classpath issues.
>
> I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
> and it works fine (something is broken in trunk in the RPC layer YARN-885).
>
> Now, to do this it will require a few things:
>
> * Make sure protobuf 2.5.0 is available in the jenkins box
> * A follow up email to dev@ aliases indicating developers should install
> locally protobuf 2.5.0
>
> Thanks.
>
> --
> Alejandro