[UPDATE 2] Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845

[Alejandro Abdelnur <tu...@cloudera.com>]

I've filed https://issues.apache.org/jira/browse/HADOOP-9872 addressing the
following:

---------

   - handles protoc version correctly independently of the exit code
   - if HADOOP_PROTOC_PATH env var is defined, it uses it as the protoc
   executable * if HADOOP_PROTOC_PATH is not defined, it picks protoc from the
   PATH
   - documentation updated to reflect 2.5.0 is required
   - enforces the version of protoc and protobuf JAR are the same
   - Added to VersionInfo the protoc version used (sooner or later this
   will be useful for in a troubleshooting situation).

Luke Lu <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vicaya>
suggested
to make the version check for protoc lax (i.e. 2.5.*). While working on the
patch I've thought about that. But that would introduce a potential
mismatch between protoc and protobuff JAR.

Still If you want to use different version of protoc/protobuff from the one
defined in the POM, you can use the -Dprotobuf.version=#### to specify your
alternate version. But I would recommend not to do this, because if you
publish the artifacts to a Maven repo, the fact you used
-Dprotobuf.version=#### will be lost and the version defined in the POM
properties will be used (IMO Maven should use the effective POM on deploy,
but they don't).

---------

It would be great if a few people test the patch locally.

Once this is committed to trunk I'll bacport HADOOP-9845 & HADOOP-9872 to
all the 2 branches.

Thx.

On Tue, Aug 13, 2013 at 1:09 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

>
> There is no indication that protoc 2.5.0 is breaking anything.
>
> Hadoop-trunk builds have been failing way before 1/2 way with:
>
> -------
>
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
>
> -------
>
>
> The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards
> the end of mapreduce tests.
>
> Until then there were no failures at all.
>
> I've increased heap size and tried a second run and the failure was
> earlier.
>
> I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been
> failing the same way in all the kept builds.
>
> We need to fix Hadoop-trunk builds independently of this.
>
> Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta
> branches to get all the other jenkins jobs working?
>
> I'll wait till tomorrow morning before proceeding.
>
> Thx
>
>
>
>
> On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> Jenkins is running a full test run on trunk using protoc 2.5.0.
>>
>>   https://builds.apache.org/job/Hadoop-trunk/480
>>
>> And it seems go be going just fine.
>>
>> If everything looks OK, I'm planing to backport HADOOP-9845 to the
>> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds
>> failures do the protoc mismatch.
>>
>> Thanks.
>>
>> Alejandro
>>
>>
>> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> shooting to get it i n for 2.1.0.
>>>
>>> at moment is in trunk till the nightly finishes. then we'll decide
>>>
>>> in the mean time, you can have multiple versions installed in diff dirs
>>> and set the right one in the path
>>>
>>> thx
>>>
>>> Alejandro
>>> (phone typing)
>>>
>>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <sh...@gmail.com>
>>> wrote:
>>>
>>> > Ok. After installing protobuf 2.5.0 I can compile trunk.
>>> > But now I cannot compile Hadoop-2 branches. None of them.
>>> > So if I switch between branches I need to reinstall protobuf?
>>> >
>>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL
>>> > versions?
>>> > I did not get definite impression there is.
>>> > If not it could be a pretty big disruption.
>>> >
>>> > Thanks,
>>> > --Konst
>>> >
>>> >
>>> >
>>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> >wrote:
>>> >
>>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment).
>>> >>
>>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a
>>> >> warning if you don't have it).
>>> >>
>>> >> We'd propagate this to the 2 branches once the precommit build is
>>> back to
>>> >> normal and see things are OK.
>>> >>
>>> >> Thanks.
>>> >>
>>> >>
>>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <
>>> tucu@cloudera.com
>>> >>> wrote:
>>> >>
>>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make
>>> trunk use
>>> >>> protoc 2.5.0.
>>> >>>
>>> >>> thx
>>> >>>
>>> >>>
>>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
>>> >>> gkesavan@hortonworks.com> wrote:
>>> >>>
>>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a
>>> different
>>> >>>> location. This would fix 2.0 branch builds as well.
>>> >>>> Thoughts?
>>> >>>>
>>> >>>> -Giri
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <
>>> tucu@cloudera.com
>>> >>>>> wrote:
>>> >>>>
>>> >>>>> Giri,
>>> >>>>>
>>> >>>>> first of all, thanks for installing protoc 2.5.0.
>>> >>>>>
>>> >>>>> I didn't know we were installing them as the only version and not
>>> >>>> driven by
>>> >>>>> env/path settings.
>>> >>>>>
>>> >>>>> Now we have a bit of a problem, precommit builds are broken
>>> because of
>>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>>> >>>>>
>>> >>>>> We have to options:
>>> >>>>>
>>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron
>>> out
>>> >> any
>>> >>>>> follow up issues.
>>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1
>>> and
>>> >>>> 2.5.0
>>> >>>>> coexisting
>>> >>>>>
>>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any
>>> issues
>>> >> an
>>> >>>>> then merge it to the other branches.
>>> >>>>>
>>> >>>>> We need to sort this out quickly as precommits are not working.
>>> >>>>>
>>> >>>>> I'll wait till 3PM today  for objections to option #1, if none I'll
>>> >>>> commit
>>> >>>>> it to trunk.
>>> >>>>>
>>> >>>>> Thanks.
>>> >>>>>
>>> >>>>> Alejandro
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>>> >>>>> gkesavan@hortonworks.com> wrote:
>>> >>>>>
>>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the
>>> >> default
>>> >>>>> path.
>>> >>>>>> If we still need 2.4 I may have to install it. Let me know
>>> >>>>>>
>>> >>>>>> -Giri
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>>> >>>> tucu@cloudera.com
>>> >>>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both
>>> >> so
>>> >>>> we
>>> >>>>>> can
>>> >>>>>>> use and env to set it in the jobs?
>>> >>>>>>>
>>> >>>>>>> thx
>>> >>>>>>>
>>> >>>>>>> Alejandro
>>> >>>>>>> (phone typing)
>>> >>>>>>>
>>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>>> >>>> gkesavan@hortonworks.com
>>> >>>>>>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> -Giri
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>>> >>>>>>>> gkesavan@hortonworks.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Alejandro,
>>> >>>>>>>>>
>>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>>> >>>>>>>>>
>>> >>>>>>>>> -Giri
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>>> >>>>> tucu@cloudera.com
>>> >>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to
>>> >> the
>>> >>>>>> hadoop
>>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of
>>> >>>>> hours
>>> >>>>>> to
>>> >>>>>>>>>> set
>>> >>>>>>>>>> up myself.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Please!!!
>>> >>>>>>>>>>
>>> >>>>>>>>>> thx
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>>> >>>>>> tucu@cloudera.com
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache
>>> >>>>> hadoop
>>> >>>>>>>>>>> jenkins boxes.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thx
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>>> >>>> eclark@apache.org>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need
>>> >> to
>>> >>>>> have
>>> >>>>>>> the
>>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are
>>> >>>>> running.
>>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and
>>> >>>> 2.5.x
>>> >>>>> on
>>> >>>>>>>>>>>> the same class path.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new
>>> >>>> classes
>>> >>>>> we
>>> >>>>>>>>>>>> could use.  With that said HBase is getting pretty close to
>>> >> a
>>> >>>>>> rather
>>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this
>>> >> in
>>> >>>>>> sooner
>>> >>>>>>>>>>>> rather than later would be great.  If we could get this into
>>> >>>>> 2.1.0
>>> >>>>>> it
>>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy
>>> >>>> story
>>> >>>>>> to
>>> >>>>>>>>>>>> users with regards to protobuf version.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>>> >>>> kihwal@yahoo-inc.com
>>> >>>>>>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention
>>> >>>> Avro.
>>> >>>>> See
>>> >>>>>>>>>>>> HADOOP-9672.
>>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency
>>> >>>>> issues.
>>> >>>>>>>>>> We've
>>> >>>>>>>>>>>> seen users running into it.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Kihwal
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> ________________________________
>>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <oz...@gmail.com>
>>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" <
>>> >>>>> common-dev@hadoop.apache.org>
>>> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" <
>>> >> hdfs-dev@hadoop.apache.org
>>> >>>>> ;
>>> >>>>> "
>>> >>>>>>>>>>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>; "
>>> >>>>>>>>>>>> mapreduce-dev@hadoop.apache.org" <
>>> >>>>> mapreduce-dev@hadoop.apache.org>
>>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0
>>> >> release,
>>> >>>>>>>>>>>> HADOOP-9845
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Hi,
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in
>>> >>>> HADOOP-9346.
>>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket:
>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> - Tsuyoshi
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur <
>>> >>>>>>>>>> tucu@cloudera.com>
>>> >>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0
>>> >> release.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant
>>> >>>>>> benefits
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>>>> justify the upgrade.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make
>>> >> things
>>> >>>>>> easier
>>> >>>>>>>>>> for
>>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and
>>> >> adopting
>>> >>>>>> Hadoop
>>> >>>>>>>>>> 2.
>>> >>>>>>>>>>>> If
>>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to
>>> >>>>>> support 2
>>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to
>>> >>>>>> classpath
>>> >>>>>>>>>>>> issues.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of
>>> >>>>>> 2.1.0-beta
>>> >>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC
>>> >>>>> layer
>>> >>>>>>>>>>>> YARN-885).
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Now, to do this it will require a few things:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box
>>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers
>>> >>>>> should
>>> >>>>>>>>>>>> install
>>> >>>>>>>>>>>>>> locally protobuf 2.5.0
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Thanks.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> --
>>> >>>>>>>>>>>>>> Alejandro
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> --
>>> >>>>>>>>>>> Alejandro
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Alejandro
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Alejandro
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Alejandro
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Alejandro
>>> >>
>>>
>>
>>
>>
>> --
>> Alejandro
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro