You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by FavioVazquez <gi...@git.apache.org> on 2015/04/29 20:05:59 UTC
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
GitHub user FavioVazquez opened a pull request:
https://github.com/apache/spark/pull/5783
[SPARK-7238] Update protobuf-java version of com.google.protobuf dependancy
This upgrade is needed when building spark for CDH5 2.5.0-cdh5.3.3 due to incompatibilities in the protobuf version used by com.google.protobuf and the one used in hadoop. The default version of protobuf is set to 2.4.1 in the global properties, and this is stated in the pom.xml file:
<!-- In theory we need not directly depend on protobuf since Spark does not directly use it. However, when building with Hadoop/YARN 2.2 Maven doesn't correctly bump the protobuf version up from the one Mesos gives. For now we include this variable to explicitly bump the version when building with YARN. It would be nice to figure out why Maven can't resolve this correctly (like SBT does). -->
So this upgrade will only be affecting the com.google.protobuf version of java-protobuf. Tested for the Cloudera distribution 2.5.0-cdh5.3.3 using Mesos 0.22.0 in cluster mode.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/FavioVazquez/spark upgrade-protobuf-version
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5783.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5783
----
commit ec91ce3c405123818a4c56ef361d9cc82951677d
Author: FavioVazquez <fa...@gmail.com>
Date: 2015-04-29T17:58:09Z
- Updated protobuf-java version of com.google.protobuf dependancy to fix blocking error when connecting to HDFS via the Hadoop Cloudera HDFS CDH5 (fix for 2.5.0-cdh5.3.3 version)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97531892
For what it's worth, I think the right change would be to set all the default versions to match the `hadoop-2.2` profile, and make that profile empty (just keep it there for backwards compatibility reasons).
And then add a `hadoop-1` profile for those 2 people who still use hadoop 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97531483
You can just set `-Dprotobuf.version=2.5.0` in your command line. Or, more correctly, you should be enabling `-Phadoop-2.4` when compiling against CDH, which sets a bunch of other version too.
You inadvertently pointed at a real issue, though. The default Hadoop version in the build is 2.2.0, but the other versions (such as `protobuf.version`) don't match the versions in the hadoop-2.2 profile. @srowen?
Could you close this PR? If you want to take a stab at fixing the real issue, you can open a new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97525010
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by FavioVazquez <gi...@git.apache.org>.
Github user FavioVazquez commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97545038
Thank you for clearing that up for me. I'm doing the changes that you suggested and will make soon a pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97525776
This is not the correct way to do this. Different hadoop versions need different protobuf versions, which is why it's a variable and not a hardcoded value as you're doing. That allows you to control it on your build's command line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by FavioVazquez <gi...@git.apache.org>.
Github user FavioVazquez commented on the pull request:
https://github.com/apache/spark/pull/5783#issuecomment-97529157
I see. So it should be done in the command line when building? The problem is that the pre-compiled version for CDH doesn't work for the 2.5.0-cdh5.3.3 version because the protbuf inherits from the global property that is fixed for 2.4.1 and throws an error, doing this it worked.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7238] Update protobuf-java version of c...
Posted by FavioVazquez <gi...@git.apache.org>.
Github user FavioVazquez closed the pull request at:
https://github.com/apache/spark/pull/5783
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org