You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2013/08/13 05:35:07 UTC
[UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Jenkins is running a full test run on trunk using protoc 2.5.0.
https://builds.apache.org/job/Hadoop-trunk/480
And it seems go be going just fine.
If everything looks OK, I'm planing to backport HADOOP-9845 to the
2.1.0-beta branch midday PST tomorrow. This will normalize all builds
failures do the protoc mismatch.
Thanks.
Alejandro
On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> shooting to get it i n for 2.1.0.
>
> at moment is in trunk till the nightly finishes. then we'll decide
>
> in the mean time, you can have multiple versions installed in diff dirs
> and set the right one in the path
>
> thx
>
> Alejandro
> (phone typing)
>
> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Ok. After installing protobuf 2.5.0 I can compile trunk.
> > But now I cannot compile Hadoop-2 branches. None of them.
> > So if I switch between branches I need to reinstall protobuf?
> >
> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
> > versions?
> > I did not get definite impression there is.
> > If not it could be a pretty big disruption.
> >
> > Thanks,
> > --Konst
> >
> >
> >
> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
> >>
> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
> >> warning if you don't have it).
> >>
> >> We'd propagate this to the 2 branches once the precommit build is back
> to
> >> normal and see things are OK.
> >>
> >> Thanks.
> >>
> >>
> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>> wrote:
> >>
> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
> use
> >>> protoc 2.5.0.
> >>>
> >>> thx
> >>>
> >>>
> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> >>> gkesavan@hortonworks.com> wrote:
> >>>
> >>>> I can take care of re-installing 2.4 and installing 2.5 in a different
> >>>> location. This would fix 2.0 branch builds as well.
> >>>> Thoughts?
> >>>>
> >>>> -Giri
> >>>>
> >>>>
> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
> tucu@cloudera.com
> >>>>> wrote:
> >>>>
> >>>>> Giri,
> >>>>>
> >>>>> first of all, thanks for installing protoc 2.5.0.
> >>>>>
> >>>>> I didn't know we were installing them as the only version and not
> >>>> driven by
> >>>>> env/path settings.
> >>>>>
> >>>>> Now we have a bit of a problem, precommit builds are broken because
> of
> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
> >>>>>
> >>>>> We have to options:
> >>>>>
> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
> >> any
> >>>>> follow up issues.
> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
> >>>> 2.5.0
> >>>>> coexisting
> >>>>>
> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
> >> an
> >>>>> then merge it to the other branches.
> >>>>>
> >>>>> We need to sort this out quickly as precommits are not working.
> >>>>>
> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll
> >>>> commit
> >>>>> it to trunk.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Alejandro
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
> >>>>> gkesavan@hortonworks.com> wrote:
> >>>>>
> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
> >> default
> >>>>> path.
> >>>>>> If we still need 2.4 I may have to install it. Let me know
> >>>>>>
> >>>>>> -Giri
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
> >> so
> >>>> we
> >>>>>> can
> >>>>>>> use and env to set it in the jobs?
> >>>>>>>
> >>>>>>> thx
> >>>>>>>
> >>>>>>> Alejandro
> >>>>>>> (phone typing)
> >>>>>>>
> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
> >>>> gkesavan@hortonworks.com
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Giri
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
> >>>>>>>> gkesavan@hortonworks.com> wrote:
> >>>>>>>>
> >>>>>>>>> Alejandro,
> >>>>>>>>>
> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
> >>>>>>>>>
> >>>>>>>>> -Giri
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
> >>>>> tucu@cloudera.com
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
> >> the
> >>>>>> hadoop
> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
> >>>>> hours
> >>>>>> to
> >>>>>>>>>> set
> >>>>>>>>>> up myself.
> >>>>>>>>>>
> >>>>>>>>>> Please!!!
> >>>>>>>>>>
> >>>>>>>>>> thx
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
> >>>>>> tucu@cloudera.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
> >>>>> hadoop
> >>>>>>>>>>> jenkins boxes.
> >>>>>>>>>>>
> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
> >>>>>>>>>>>
> >>>>>>>>>>> Thx
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
> >>>> eclark@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
> >> to
> >>>>> have
> >>>>>>> the
> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
> >>>>> running.
> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
> >>>> 2.5.x
> >>>>> on
> >>>>>>>>>>>> the same class path.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
> >>>> classes
> >>>>> we
> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to
> >> a
> >>>>>> rather
> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
> >> in
> >>>>>> sooner
> >>>>>>>>>>>> rather than later would be great. If we could get this into
> >>>>> 2.1.0
> >>>>>> it
> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
> >>>> story
> >>>>>> to
> >>>>>>>>>>>> users with regards to protobuf version.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
> >>>> kihwal@yahoo-inc.com
> >>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
> >>>> Avro.
> >>>>> See
> >>>>>>>>>>>> HADOOP-9672.
> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
> >>>>> issues.
> >>>>>>>>>> We've
> >>>>>>>>>>>> seen users running into it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Kihwal
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ________________________________
> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
> >>>>> common-dev@hadoop.apache.org>
> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
> >> hdfs-dev@hadoop.apache.org
> >>>>> ;
> >>>>> "
> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
> >>>>> mapreduce-dev@hadoop.apache.org>
> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
> >> release,
> >>>>>>>>>>>> HADOOP-9845
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
> >>>> HADOOP-9346.
> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Tsuyoshi
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
> >>>>>>>>>> tucu@cloudera.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
> >> release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
> >>>>>> benefits
> >>>>>>>>>> to
> >>>>>>>>>>>>>> justify the upgrade.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
> >> things
> >>>>>> easier
> >>>>>>>>>> for
> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
> >> adopting
> >>>>>> Hadoop
> >>>>>>>>>> 2.
> >>>>>>>>>>>> If
> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
> >>>>>> support 2
> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
> >>>>>> classpath
> >>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
> >>>>>> 2.1.0-beta
> >>>>>>>>>>>> branch
> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
> >>>>> layer
> >>>>>>>>>>>> YARN-885).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Now, to do this it will require a few things:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
> >>>>> should
> >>>>>>>>>>>> install
> >>>>>>>>>>>>>> locally protobuf 2.5.0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Alejandro
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Alejandro
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Alejandro
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Alejandro
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
yep, will take of that. Also, Colin suggested me offline that we should
improve the protoc plugin to pick up the path from an ENV var if present to
simplify things for folks that need to build versions of hadoop using diff
versions of protoc. I'll work on this too.
thx
On Tue, Aug 13, 2013 at 4:41 PM, Steve Loughran <st...@hortonworks.com>wrote:
> On 13 August 2013 16:20, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Steve, this is a version issue, if you get protoc 2.5.0 in your PATH
> things
> > will work.
> >
>
> I assume that, but as the YARN docs still talk about 0.24, they need to be
> updated too
>
>
> >
> > Apologies for the the hiccups until we get all this sorted out, we had
> some
> > miscommunication on how to install protoc in the jenkins boxes and
> instead
> > having 2.4.1 and 2.5.0 side to side we got only 2.5.0.
> >
> > By tomorrow we should have things mostly sorted out.
> >
> > Thanks
> >
> >
> > On Tue, Aug 13, 2013 at 3:29 PM, Steve Loughran <stevel@hortonworks.com
> > >wrote:
> >
> > > On 13 August 2013 13:09, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> > >
> > > > There is no indication that protoc 2.5.0 is breaking anything.
> > > >
> > >
> > >
> > > clearly then this is not a stack trace:
> > >
> > > INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > Downloading:
> > >
> > >
> >
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > > Downloading:
> > >
> > >
> >
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > > Downloading:
> > >
> > >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > > Downloaded:
> > >
> > >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom(9
> > > KB at 185.9 KB/sec)
> > > Downloading:
> > >
> > >
> >
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > > Downloading:
> > >
> > >
> >
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > > Downloading:
> > >
> > >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > > Downloaded:
> > >
> > >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar(521
> > > KB at 7039.9 KB/sec)
> > > [INFO]
> > > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
> hadoop-common
> > > ---
> > > [INFO] Deleting
> > >
> > >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target
> > > [INFO]
> > > [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @
> hadoop-common
> > > ---
> > > [INFO] Executing tasks
> > >
> > > main:
> > > [mkdir] Created dir:
> > >
> > >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test-dir
> > > [mkdir] Created dir:
> > >
> > >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test/data
> > > [INFO] Executed tasks
> > > [INFO]
> > > [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc)
> @
> > > hadoop-common ---
> > > [WARNING] [protoc, --version] failed with error code 1
> > > [ERROR] protoc, could not get version
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Reactor Summary:
> > > [INFO]
> > >
> > >
> > > Assuming this is just a versioning issue, can you update the
> > documentation
> > > in the wiki &
> > >
> > >
> >
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/READMEto
> > > be consistent with the current protobuf requirements. I do want to
> > > follow the instructions not just because I am lazy, but because I want
> to
> > > manually test the installation process itself
> > >
> > > Once that's done I will try to follow these instructions to get
> protobuf
> > > 2.5 installed on my homebrew-managed mac.
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Steve Loughran <st...@hortonworks.com>.
On 13 August 2013 16:20, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Steve, this is a version issue, if you get protoc 2.5.0 in your PATH things
> will work.
>
I assume that, but as the YARN docs still talk about 0.24, they need to be
updated too
>
> Apologies for the the hiccups until we get all this sorted out, we had some
> miscommunication on how to install protoc in the jenkins boxes and instead
> having 2.4.1 and 2.5.0 side to side we got only 2.5.0.
>
> By tomorrow we should have things mostly sorted out.
>
> Thanks
>
>
> On Tue, Aug 13, 2013 at 3:29 PM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > On 13 August 2013 13:09, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> >
> > > There is no indication that protoc 2.5.0 is breaking anything.
> > >
> >
> >
> > clearly then this is not a stack trace:
> >
> > INFO]
> > ------------------------------------------------------------------------
> > [INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
> > [INFO]
> > ------------------------------------------------------------------------
> > Downloading:
> >
> >
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > Downloading:
> >
> >
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > Downloading:
> >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> > Downloaded:
> >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom(9
> > KB at 185.9 KB/sec)
> > Downloading:
> >
> >
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > Downloading:
> >
> >
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > Downloading:
> >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> > Downloaded:
> >
> >
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar(521
> > KB at 7039.9 KB/sec)
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-common
> > ---
> > [INFO] Deleting
> >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target
> > [INFO]
> > [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common
> > ---
> > [INFO] Executing tasks
> >
> > main:
> > [mkdir] Created dir:
> >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test-dir
> > [mkdir] Created dir:
> >
> >
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test/data
> > [INFO] Executed tasks
> > [INFO]
> > [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) @
> > hadoop-common ---
> > [WARNING] [protoc, --version] failed with error code 1
> > [ERROR] protoc, could not get version
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Reactor Summary:
> > [INFO]
> >
> >
> > Assuming this is just a versioning issue, can you update the
> documentation
> > in the wiki &
> >
> >
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/READMEto
> > be consistent with the current protobuf requirements. I do want to
> > follow the instructions not just because I am lazy, but because I want to
> > manually test the installation process itself
> >
> > Once that's done I will try to follow these instructions to get protobuf
> > 2.5 installed on my homebrew-managed mac.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Steve, this is a version issue, if you get protoc 2.5.0 in your PATH things
will work.
Apologies for the the hiccups until we get all this sorted out, we had some
miscommunication on how to install protoc in the jenkins boxes and instead
having 2.4.1 and 2.5.0 side to side we got only 2.5.0.
By tomorrow we should have things mostly sorted out.
Thanks
On Tue, Aug 13, 2013 at 3:29 PM, Steve Loughran <st...@hortonworks.com>wrote:
> On 13 August 2013 13:09, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > There is no indication that protoc 2.5.0 is breaking anything.
> >
>
>
> clearly then this is not a stack trace:
>
> INFO]
> ------------------------------------------------------------------------
> [INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> Downloading:
>
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloading:
>
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloading:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloaded:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom(9
> KB at 185.9 KB/sec)
> Downloading:
>
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloading:
>
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloading:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloaded:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar(521
> KB at 7039.9 KB/sec)
> [INFO]
> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-common
> ---
> [INFO] Deleting
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target
> [INFO]
> [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common
> ---
> [INFO] Executing tasks
>
> main:
> [mkdir] Created dir:
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test-dir
> [mkdir] Created dir:
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test/data
> [INFO] Executed tasks
> [INFO]
> [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) @
> hadoop-common ---
> [WARNING] [protoc, --version] failed with error code 1
> [ERROR] protoc, could not get version
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
>
>
> Assuming this is just a versioning issue, can you update the documentation
> in the wiki &
>
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/READMEto
> be consistent with the current protobuf requirements. I do want to
> follow the instructions not just because I am lazy, but because I want to
> manually test the installation process itself
>
> Once that's done I will try to follow these instructions to get protobuf
> 2.5 installed on my homebrew-managed mac.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Luke Lu <ll...@vicaya.com>.
I've verified that it's only a version issue (tested on a Ubuntu 12.04 VM)
: as long as you have 2.5.0 protoc, it works. BTW, the version check is a
little "too" strict. I had protobuf 2.5.1 (trunk) installed for 2.5 tests
and the exact check broke my build.
On Tue, Aug 13, 2013 at 3:29 PM, Steve Loughran <st...@hortonworks.com>wrote:
> On 13 August 2013 13:09, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > There is no indication that protoc 2.5.0 is breaking anything.
> >
>
>
> clearly then this is not a stack trace:
>
> INFO]
> ------------------------------------------------------------------------
> [INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> Downloading:
>
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloading:
>
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloading:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
> Downloaded:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom(9
> KB at 185.9 KB/sec)
> Downloading:
>
> https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloading:
>
> http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloading:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
> Downloaded:
>
> http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar(521
> KB at 7039.9 KB/sec)
> [INFO]
> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-common
> ---
> [INFO] Deleting
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target
> [INFO]
> [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common
> ---
> [INFO] Executing tasks
>
> main:
> [mkdir] Created dir:
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test-dir
> [mkdir] Created dir:
>
> /Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test/data
> [INFO] Executed tasks
> [INFO]
> [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) @
> hadoop-common ---
> [WARNING] [protoc, --version] failed with error code 1
> [ERROR] protoc, could not get version
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
>
>
> Assuming this is just a versioning issue, can you update the documentation
> in the wiki &
>
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/READMEto
> be consistent with the current protobuf requirements. I do want to
> follow the instructions not just because I am lazy, but because I want to
> manually test the installation process itself
>
> Once that's done I will try to follow these instructions to get protobuf
> 2.5 installed on my homebrew-managed mac.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Steve Loughran <st...@hortonworks.com>.
On 13 August 2013 13:09, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> There is no indication that protoc 2.5.0 is breaking anything.
>
clearly then this is not a stack trace:
INFO]
------------------------------------------------------------------------
[INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
Downloading:
https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
Downloading:
http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
Downloading:
http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom
Downloaded:
http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.pom(9
KB at 185.9 KB/sec)
Downloading:
https://repository.apache.org/content/repositories/snapshots/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
Downloading:
http://repository.jboss.org/nexus/content/groups/public/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
Downloading:
http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
Downloaded:
http://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar(521
KB at 7039.9 KB/sec)
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-common
---
[INFO] Deleting
/Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test-dir
[mkdir] Created dir:
/Users/stevel/Projects/hadoop-trunk/hadoop-common-project/hadoop-common/target/test/data
[INFO] Executed tasks
[INFO]
[INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) @
hadoop-common ---
[WARNING] [protoc, --version] failed with error code 1
[ERROR] protoc, could not get version
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
Assuming this is just a versioning issue, can you update the documentation
in the wiki &
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/READMEto
be consistent with the current protobuf requirements. I do want to
follow the instructions not just because I am lazy, but because I want to
manually test the installation process itself
Once that's done I will try to follow these instructions to get protobuf
2.5 installed on my homebrew-managed mac.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
There is no indication that protoc 2.5.0 is breaking anything.
Hadoop-trunk builds have been failing way before 1/2 way with:
-------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ? -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ?
-------
The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
the end of mapreduce tests.
Until then there were no failures at all.
I've increased heap size and tried a second run and the failure was earlier.
I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been
failing the same way in all the kept builds.
We need to fix Hadoop-trunk builds independently of this.
Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta branches
to get all the other jenkins jobs working?
I'll wait till tomorrow morning before proceeding.
Thx
On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Jenkins is running a full test run on trunk using protoc 2.5.0.
>
> https://builds.apache.org/job/Hadoop-trunk/480
>
> And it seems go be going just fine.
>
> If everything looks OK, I'm planing to backport HADOOP-9845 to the
> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
> failures do the protoc mismatch.
>
> Thanks.
>
> Alejandro
>
>
> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> shooting to get it i n for 2.1.0.
>>
>> at moment is in trunk till the nightly finishes. then we'll decide
>>
>> in the mean time, you can have multiple versions installed in diff dirs
>> and set the right one in the path
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
>> wrote:
>>
>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>> > But now I cannot compile Hadoop-2 branches. None of them.
>> > So if I switch between branches I need to reinstall protobuf?
>> >
>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>> > versions?
>> > I did not get definite impression there is.
>> > If not it could be a pretty big disruption.
>> >
>> > Thanks,
>> > --Konst
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>> >>
>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> >> warning if you don't have it).
>> >>
>> >> We'd propagate this to the 2 branches once the precommit build is back
>> to
>> >> normal and see things are OK.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >>> wrote:
>> >>
>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
>> use
>> >>> protoc 2.5.0.
>> >>>
>> >>> thx
>> >>>
>> >>>
>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>> >>> gkesavan@hortonworks.com> wrote:
>> >>>
>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a
>> different
>> >>>> location. This would fix 2.0 branch builds as well.
>> >>>> Thoughts?
>> >>>>
>> >>>> -Giri
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> >>>>> wrote:
>> >>>>
>> >>>>> Giri,
>> >>>>>
>> >>>>> first of all, thanks for installing protoc 2.5.0.
>> >>>>>
>> >>>>> I didn't know we were installing them as the only version and not
>> >>>> driven by
>> >>>>> env/path settings.
>> >>>>>
>> >>>>> Now we have a bit of a problem, precommit builds are broken because
>> of
>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >>>>>
>> >>>>> We have to options:
>> >>>>>
>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> >> any
>> >>>>> follow up issues.
>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> >>>> 2.5.0
>> >>>>> coexisting
>> >>>>>
>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> >> an
>> >>>>> then merge it to the other branches.
>> >>>>>
>> >>>>> We need to sort this out quickly as precommits are not working.
>> >>>>>
>> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>> >>>> commit
>> >>>>> it to trunk.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> >>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>
>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> >> default
>> >>>>> path.
>> >>>>>> If we still need 2.4 I may have to install it. Let me know
>> >>>>>>
>> >>>>>> -Giri
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> >>>> tucu@cloudera.com
>> >>>>>>> wrote:
>> >>>>>>
>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> >> so
>> >>>> we
>> >>>>>> can
>> >>>>>>> use and env to set it in the jobs?
>> >>>>>>>
>> >>>>>>> thx
>> >>>>>>>
>> >>>>>>> Alejandro
>> >>>>>>> (phone typing)
>> >>>>>>>
>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> >>>> gkesavan@hortonworks.com
>> >>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> -Giri
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Alejandro,
>> >>>>>>>>>
>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> >>>>>>>>>
>> >>>>>>>>> -Giri
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> >>>>> tucu@cloudera.com
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> >> the
>> >>>>>> hadoop
>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>> >>>>> hours
>> >>>>>> to
>> >>>>>>>>>> set
>> >>>>>>>>>> up myself.
>> >>>>>>>>>>
>> >>>>>>>>>> Please!!!
>> >>>>>>>>>>
>> >>>>>>>>>> thx
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> >>>>>> tucu@cloudera.com
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>> >>>>> hadoop
>> >>>>>>>>>>> jenkins boxes.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thx
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> >>>> eclark@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> >> to
>> >>>>> have
>> >>>>>>> the
>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> >>>>> running.
>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>> >>>> 2.5.x
>> >>>>> on
>> >>>>>>>>>>>> the same class path.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>> >>>> classes
>> >>>>> we
>> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> >> a
>> >>>>>> rather
>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> >> in
>> >>>>>> sooner
>> >>>>>>>>>>>> rather than later would be great. If we could get this into
>> >>>>> 2.1.0
>> >>>>>> it
>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>> >>>> story
>> >>>>>> to
>> >>>>>>>>>>>> users with regards to protobuf version.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> >>>> kihwal@yahoo-inc.com
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>> >>>> Avro.
>> >>>>> See
>> >>>>>>>>>>>> HADOOP-9672.
>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>> >>>>> issues.
>> >>>>>>>>>> We've
>> >>>>>>>>>>>> seen users running into it.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kihwal
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ________________________________
>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>> >>>>> common-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> >> hdfs-dev@hadoop.apache.org
>> >>>>> ;
>> >>>>> "
>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>> >>>>> mapreduce-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release,
>> >>>>>>>>>>>> HADOOP-9845
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>> >>>> HADOOP-9346.
>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - Tsuyoshi
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> >>>>>>>>>> tucu@cloudera.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> >>>>>> benefits
>> >>>>>>>>>> to
>> >>>>>>>>>>>>>> justify the upgrade.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> >> things
>> >>>>>> easier
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> >> adopting
>> >>>>>> Hadoop
>> >>>>>>>>>> 2.
>> >>>>>>>>>>>> If
>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>> >>>>>> support 2
>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>> >>>>>> classpath
>> >>>>>>>>>>>> issues.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>> >>>>>> 2.1.0-beta
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>> >>>>> layer
>> >>>>>>>>>>>> YARN-885).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Now, to do this it will require a few things:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>> >>>>> should
>> >>>>>>>>>>>> install
>> >>>>>>>>>>>>>> locally protobuf 2.5.0
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> Alejandro
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Alejandro
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Alejandro
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Alejandro
>> >>
>> >>
>> >>
>> >> --
>> >> Alejandro
>> >>
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
There is no indication that protoc 2.5.0 is breaking anything.
Hadoop-trunk builds have been failing way before 1/2 way with:
-------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ? -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ?
-------
The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
the end of mapreduce tests.
Until then there were no failures at all.
I've increased heap size and tried a second run and the failure was earlier.
I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been
failing the same way in all the kept builds.
We need to fix Hadoop-trunk builds independently of this.
Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta branches
to get all the other jenkins jobs working?
I'll wait till tomorrow morning before proceeding.
Thx
On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Jenkins is running a full test run on trunk using protoc 2.5.0.
>
> https://builds.apache.org/job/Hadoop-trunk/480
>
> And it seems go be going just fine.
>
> If everything looks OK, I'm planing to backport HADOOP-9845 to the
> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
> failures do the protoc mismatch.
>
> Thanks.
>
> Alejandro
>
>
> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> shooting to get it i n for 2.1.0.
>>
>> at moment is in trunk till the nightly finishes. then we'll decide
>>
>> in the mean time, you can have multiple versions installed in diff dirs
>> and set the right one in the path
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
>> wrote:
>>
>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>> > But now I cannot compile Hadoop-2 branches. None of them.
>> > So if I switch between branches I need to reinstall protobuf?
>> >
>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>> > versions?
>> > I did not get definite impression there is.
>> > If not it could be a pretty big disruption.
>> >
>> > Thanks,
>> > --Konst
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>> >>
>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> >> warning if you don't have it).
>> >>
>> >> We'd propagate this to the 2 branches once the precommit build is back
>> to
>> >> normal and see things are OK.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >>> wrote:
>> >>
>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
>> use
>> >>> protoc 2.5.0.
>> >>>
>> >>> thx
>> >>>
>> >>>
>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>> >>> gkesavan@hortonworks.com> wrote:
>> >>>
>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a
>> different
>> >>>> location. This would fix 2.0 branch builds as well.
>> >>>> Thoughts?
>> >>>>
>> >>>> -Giri
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> >>>>> wrote:
>> >>>>
>> >>>>> Giri,
>> >>>>>
>> >>>>> first of all, thanks for installing protoc 2.5.0.
>> >>>>>
>> >>>>> I didn't know we were installing them as the only version and not
>> >>>> driven by
>> >>>>> env/path settings.
>> >>>>>
>> >>>>> Now we have a bit of a problem, precommit builds are broken because
>> of
>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >>>>>
>> >>>>> We have to options:
>> >>>>>
>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> >> any
>> >>>>> follow up issues.
>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> >>>> 2.5.0
>> >>>>> coexisting
>> >>>>>
>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> >> an
>> >>>>> then merge it to the other branches.
>> >>>>>
>> >>>>> We need to sort this out quickly as precommits are not working.
>> >>>>>
>> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>> >>>> commit
>> >>>>> it to trunk.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> >>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>
>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> >> default
>> >>>>> path.
>> >>>>>> If we still need 2.4 I may have to install it. Let me know
>> >>>>>>
>> >>>>>> -Giri
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> >>>> tucu@cloudera.com
>> >>>>>>> wrote:
>> >>>>>>
>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> >> so
>> >>>> we
>> >>>>>> can
>> >>>>>>> use and env to set it in the jobs?
>> >>>>>>>
>> >>>>>>> thx
>> >>>>>>>
>> >>>>>>> Alejandro
>> >>>>>>> (phone typing)
>> >>>>>>>
>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> >>>> gkesavan@hortonworks.com
>> >>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> -Giri
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Alejandro,
>> >>>>>>>>>
>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> >>>>>>>>>
>> >>>>>>>>> -Giri
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> >>>>> tucu@cloudera.com
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> >> the
>> >>>>>> hadoop
>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>> >>>>> hours
>> >>>>>> to
>> >>>>>>>>>> set
>> >>>>>>>>>> up myself.
>> >>>>>>>>>>
>> >>>>>>>>>> Please!!!
>> >>>>>>>>>>
>> >>>>>>>>>> thx
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> >>>>>> tucu@cloudera.com
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>> >>>>> hadoop
>> >>>>>>>>>>> jenkins boxes.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thx
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> >>>> eclark@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> >> to
>> >>>>> have
>> >>>>>>> the
>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> >>>>> running.
>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>> >>>> 2.5.x
>> >>>>> on
>> >>>>>>>>>>>> the same class path.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>> >>>> classes
>> >>>>> we
>> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> >> a
>> >>>>>> rather
>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> >> in
>> >>>>>> sooner
>> >>>>>>>>>>>> rather than later would be great. If we could get this into
>> >>>>> 2.1.0
>> >>>>>> it
>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>> >>>> story
>> >>>>>> to
>> >>>>>>>>>>>> users with regards to protobuf version.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> >>>> kihwal@yahoo-inc.com
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>> >>>> Avro.
>> >>>>> See
>> >>>>>>>>>>>> HADOOP-9672.
>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>> >>>>> issues.
>> >>>>>>>>>> We've
>> >>>>>>>>>>>> seen users running into it.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kihwal
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ________________________________
>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>> >>>>> common-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> >> hdfs-dev@hadoop.apache.org
>> >>>>> ;
>> >>>>> "
>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>> >>>>> mapreduce-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release,
>> >>>>>>>>>>>> HADOOP-9845
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>> >>>> HADOOP-9346.
>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - Tsuyoshi
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> >>>>>>>>>> tucu@cloudera.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> >>>>>> benefits
>> >>>>>>>>>> to
>> >>>>>>>>>>>>>> justify the upgrade.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> >> things
>> >>>>>> easier
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> >> adopting
>> >>>>>> Hadoop
>> >>>>>>>>>> 2.
>> >>>>>>>>>>>> If
>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>> >>>>>> support 2
>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>> >>>>>> classpath
>> >>>>>>>>>>>> issues.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>> >>>>>> 2.1.0-beta
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>> >>>>> layer
>> >>>>>>>>>>>> YARN-885).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Now, to do this it will require a few things:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>> >>>>> should
>> >>>>>>>>>>>> install
>> >>>>>>>>>>>>>> locally protobuf 2.5.0
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> Alejandro
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Alejandro
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Alejandro
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Alejandro
>> >>
>> >>
>> >>
>> >> --
>> >> Alejandro
>> >>
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
There is no indication that protoc 2.5.0 is breaking anything.
Hadoop-trunk builds have been failing way before 1/2 way with:
-------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ? -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ?
-------
The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
the end of mapreduce tests.
Until then there were no failures at all.
I've increased heap size and tried a second run and the failure was earlier.
I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been
failing the same way in all the kept builds.
We need to fix Hadoop-trunk builds independently of this.
Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta branches
to get all the other jenkins jobs working?
I'll wait till tomorrow morning before proceeding.
Thx
On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Jenkins is running a full test run on trunk using protoc 2.5.0.
>
> https://builds.apache.org/job/Hadoop-trunk/480
>
> And it seems go be going just fine.
>
> If everything looks OK, I'm planing to backport HADOOP-9845 to the
> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
> failures do the protoc mismatch.
>
> Thanks.
>
> Alejandro
>
>
> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> shooting to get it i n for 2.1.0.
>>
>> at moment is in trunk till the nightly finishes. then we'll decide
>>
>> in the mean time, you can have multiple versions installed in diff dirs
>> and set the right one in the path
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
>> wrote:
>>
>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>> > But now I cannot compile Hadoop-2 branches. None of them.
>> > So if I switch between branches I need to reinstall protobuf?
>> >
>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>> > versions?
>> > I did not get definite impression there is.
>> > If not it could be a pretty big disruption.
>> >
>> > Thanks,
>> > --Konst
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>> >>
>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> >> warning if you don't have it).
>> >>
>> >> We'd propagate this to the 2 branches once the precommit build is back
>> to
>> >> normal and see things are OK.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >>> wrote:
>> >>
>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
>> use
>> >>> protoc 2.5.0.
>> >>>
>> >>> thx
>> >>>
>> >>>
>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>> >>> gkesavan@hortonworks.com> wrote:
>> >>>
>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a
>> different
>> >>>> location. This would fix 2.0 branch builds as well.
>> >>>> Thoughts?
>> >>>>
>> >>>> -Giri
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> >>>>> wrote:
>> >>>>
>> >>>>> Giri,
>> >>>>>
>> >>>>> first of all, thanks for installing protoc 2.5.0.
>> >>>>>
>> >>>>> I didn't know we were installing them as the only version and not
>> >>>> driven by
>> >>>>> env/path settings.
>> >>>>>
>> >>>>> Now we have a bit of a problem, precommit builds are broken because
>> of
>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >>>>>
>> >>>>> We have to options:
>> >>>>>
>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> >> any
>> >>>>> follow up issues.
>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> >>>> 2.5.0
>> >>>>> coexisting
>> >>>>>
>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> >> an
>> >>>>> then merge it to the other branches.
>> >>>>>
>> >>>>> We need to sort this out quickly as precommits are not working.
>> >>>>>
>> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>> >>>> commit
>> >>>>> it to trunk.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> >>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>
>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> >> default
>> >>>>> path.
>> >>>>>> If we still need 2.4 I may have to install it. Let me know
>> >>>>>>
>> >>>>>> -Giri
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> >>>> tucu@cloudera.com
>> >>>>>>> wrote:
>> >>>>>>
>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> >> so
>> >>>> we
>> >>>>>> can
>> >>>>>>> use and env to set it in the jobs?
>> >>>>>>>
>> >>>>>>> thx
>> >>>>>>>
>> >>>>>>> Alejandro
>> >>>>>>> (phone typing)
>> >>>>>>>
>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> >>>> gkesavan@hortonworks.com
>> >>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> -Giri
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Alejandro,
>> >>>>>>>>>
>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> >>>>>>>>>
>> >>>>>>>>> -Giri
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> >>>>> tucu@cloudera.com
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> >> the
>> >>>>>> hadoop
>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>> >>>>> hours
>> >>>>>> to
>> >>>>>>>>>> set
>> >>>>>>>>>> up myself.
>> >>>>>>>>>>
>> >>>>>>>>>> Please!!!
>> >>>>>>>>>>
>> >>>>>>>>>> thx
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> >>>>>> tucu@cloudera.com
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>> >>>>> hadoop
>> >>>>>>>>>>> jenkins boxes.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thx
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> >>>> eclark@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> >> to
>> >>>>> have
>> >>>>>>> the
>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> >>>>> running.
>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>> >>>> 2.5.x
>> >>>>> on
>> >>>>>>>>>>>> the same class path.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>> >>>> classes
>> >>>>> we
>> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> >> a
>> >>>>>> rather
>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> >> in
>> >>>>>> sooner
>> >>>>>>>>>>>> rather than later would be great. If we could get this into
>> >>>>> 2.1.0
>> >>>>>> it
>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>> >>>> story
>> >>>>>> to
>> >>>>>>>>>>>> users with regards to protobuf version.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> >>>> kihwal@yahoo-inc.com
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>> >>>> Avro.
>> >>>>> See
>> >>>>>>>>>>>> HADOOP-9672.
>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>> >>>>> issues.
>> >>>>>>>>>> We've
>> >>>>>>>>>>>> seen users running into it.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kihwal
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ________________________________
>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>> >>>>> common-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> >> hdfs-dev@hadoop.apache.org
>> >>>>> ;
>> >>>>> "
>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>> >>>>> mapreduce-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release,
>> >>>>>>>>>>>> HADOOP-9845
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>> >>>> HADOOP-9346.
>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - Tsuyoshi
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> >>>>>>>>>> tucu@cloudera.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> >>>>>> benefits
>> >>>>>>>>>> to
>> >>>>>>>>>>>>>> justify the upgrade.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> >> things
>> >>>>>> easier
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> >> adopting
>> >>>>>> Hadoop
>> >>>>>>>>>> 2.
>> >>>>>>>>>>>> If
>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>> >>>>>> support 2
>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>> >>>>>> classpath
>> >>>>>>>>>>>> issues.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>> >>>>>> 2.1.0-beta
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>> >>>>> layer
>> >>>>>>>>>>>> YARN-885).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Now, to do this it will require a few things:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>> >>>>> should
>> >>>>>>>>>>>> install
>> >>>>>>>>>>>>>> locally protobuf 2.5.0
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> Alejandro
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Alejandro
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Alejandro
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Alejandro
>> >>
>> >>
>> >>
>> >> --
>> >> Alejandro
>> >>
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: [UPDATE] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
There is no indication that protoc 2.5.0 is breaking anything.
Hadoop-trunk builds have been failing way before 1/2 way with:
-------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ? -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-client: ExecutionException;
nested exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ?
-------
The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
the end of mapreduce tests.
Until then there were no failures at all.
I've increased heap size and tried a second run and the failure was earlier.
I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been
failing the same way in all the kept builds.
We need to fix Hadoop-trunk builds independently of this.
Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta branches
to get all the other jenkins jobs working?
I'll wait till tomorrow morning before proceeding.
Thx
On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Jenkins is running a full test run on trunk using protoc 2.5.0.
>
> https://builds.apache.org/job/Hadoop-trunk/480
>
> And it seems go be going just fine.
>
> If everything looks OK, I'm planing to backport HADOOP-9845 to the
> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
> failures do the protoc mismatch.
>
> Thanks.
>
> Alejandro
>
>
> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> shooting to get it i n for 2.1.0.
>>
>> at moment is in trunk till the nightly finishes. then we'll decide
>>
>> in the mean time, you can have multiple versions installed in diff dirs
>> and set the right one in the path
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
>> wrote:
>>
>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>> > But now I cannot compile Hadoop-2 branches. None of them.
>> > So if I switch between branches I need to reinstall protobuf?
>> >
>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>> > versions?
>> > I did not get definite impression there is.
>> > If not it could be a pretty big disruption.
>> >
>> > Thanks,
>> > --Konst
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>> >
>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>> >>
>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
>> >> warning if you don't have it).
>> >>
>> >> We'd propagate this to the 2 branches once the precommit build is back
>> to
>> >> normal and see things are OK.
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com
>> >>> wrote:
>> >>
>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk
>> use
>> >>> protoc 2.5.0.
>> >>>
>> >>> thx
>> >>>
>> >>>
>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>> >>> gkesavan@hortonworks.com> wrote:
>> >>>
>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a
>> different
>> >>>> location. This would fix 2.0 branch builds as well.
>> >>>> Thoughts?
>> >>>>
>> >>>> -Giri
>> >>>>
>> >>>>
>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> >>>>> wrote:
>> >>>>
>> >>>>> Giri,
>> >>>>>
>> >>>>> first of all, thanks for installing protoc 2.5.0.
>> >>>>>
>> >>>>> I didn't know we were installing them as the only version and not
>> >>>> driven by
>> >>>>> env/path settings.
>> >>>>>
>> >>>>> Now we have a bit of a problem, precommit builds are broken because
>> of
>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >>>>>
>> >>>>> We have to options:
>> >>>>>
>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out
>> >> any
>> >>>>> follow up issues.
>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> >>>> 2.5.0
>> >>>>> coexisting
>> >>>>>
>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues
>> >> an
>> >>>>> then merge it to the other branches.
>> >>>>>
>> >>>>> We need to sort this out quickly as precommits are not working.
>> >>>>>
>> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll
>> >>>> commit
>> >>>>> it to trunk.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> >>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>
>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>> >> default
>> >>>>> path.
>> >>>>>> If we still need 2.4 I may have to install it. Let me know
>> >>>>>>
>> >>>>>> -Giri
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> >>>> tucu@cloudera.com
>> >>>>>>> wrote:
>> >>>>>>
>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>> >> so
>> >>>> we
>> >>>>>> can
>> >>>>>>> use and env to set it in the jobs?
>> >>>>>>>
>> >>>>>>> thx
>> >>>>>>>
>> >>>>>>> Alejandro
>> >>>>>>> (phone typing)
>> >>>>>>>
>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> >>>> gkesavan@hortonworks.com
>> >>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> -Giri
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Alejandro,
>> >>>>>>>>>
>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> >>>>>>>>>
>> >>>>>>>>> -Giri
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> >>>>> tucu@cloudera.com
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
>> >> the
>> >>>>>> hadoop
>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>> >>>>> hours
>> >>>>>> to
>> >>>>>>>>>> set
>> >>>>>>>>>> up myself.
>> >>>>>>>>>>
>> >>>>>>>>>> Please!!!
>> >>>>>>>>>>
>> >>>>>>>>>> thx
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> >>>>>> tucu@cloudera.com
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>> >>>>> hadoop
>> >>>>>>>>>>> jenkins boxes.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thx
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> >>>> eclark@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>> >> to
>> >>>>> have
>> >>>>>>> the
>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>> >>>>> running.
>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>> >>>> 2.5.x
>> >>>>> on
>> >>>>>>>>>>>> the same class path.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>> >>>> classes
>> >>>>> we
>> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to
>> >> a
>> >>>>>> rather
>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>> >> in
>> >>>>>> sooner
>> >>>>>>>>>>>> rather than later would be great. If we could get this into
>> >>>>> 2.1.0
>> >>>>>> it
>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>> >>>> story
>> >>>>>> to
>> >>>>>>>>>>>> users with regards to protobuf version.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> >>>> kihwal@yahoo-inc.com
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>> >>>> Avro.
>> >>>>> See
>> >>>>>>>>>>>> HADOOP-9672.
>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>> >>>>> issues.
>> >>>>>>>>>> We've
>> >>>>>>>>>>>> seen users running into it.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kihwal
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ________________________________
>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>> >>>>> common-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>> >> hdfs-dev@hadoop.apache.org
>> >>>>> ;
>> >>>>> "
>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>> >>>>> mapreduce-dev@hadoop.apache.org>
>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release,
>> >>>>>>>>>>>> HADOOP-9845
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>> >>>> HADOOP-9346.
>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - Tsuyoshi
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>> >>>>>>>>>> tucu@cloudera.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>> >> release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>> >>>>>> benefits
>> >>>>>>>>>> to
>> >>>>>>>>>>>>>> justify the upgrade.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>> >> things
>> >>>>>> easier
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>> >> adopting
>> >>>>>> Hadoop
>> >>>>>>>>>> 2.
>> >>>>>>>>>>>> If
>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>> >>>>>> support 2
>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>> >>>>>> classpath
>> >>>>>>>>>>>> issues.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>> >>>>>> 2.1.0-beta
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>> >>>>> layer
>> >>>>>>>>>>>> YARN-885).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Now, to do this it will require a few things:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>> >>>>> should
>> >>>>>>>>>>>> install
>> >>>>>>>>>>>>>> locally protobuf 2.5.0
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> Alejandro
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> Alejandro
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Alejandro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Alejandro
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Alejandro
>> >>
>> >>
>> >>
>> >> --
>> >> Alejandro
>> >>
>>
>
>
>
> --
> Alejandro
>
--
Alejandro