You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jaceklaskowski <gi...@git.apache.org> on 2016/01/09 12:36:29 UTC
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
GitHub user jaceklaskowski opened a pull request:
https://github.com/apache/spark/pull/10674
[SPARK-12736][CORE][DEPLOY] Standalone Master cannot be started due t…
…o NoClassDefFoundError: org/spark-project/guava/collect/Maps
/cc @srowen @rxin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jaceklaskowski/spark SPARK-12736
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10674.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10674
----
commit 16c228d9227731e1228e75b419b06bcf29e8765c
Author: Jacek Laskowski <ja...@japila.pl>
Date: 2016-01-09T11:34:07Z
[SPARK-12736][CORE][DEPLOY] Standalone Master cannot be started due to NoClassDefFoundError: org/spark-project/guava/collect/Maps
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10674#discussion_r49270677
--- Diff: network/common/pom.xml ---
@@ -55,6 +55,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
+ <scope>compile</scope>
--- End diff --
> Isn't compile the default scope? i.e. couldn't you leave out the explicit compile scope here and just not specify a scope?
> Isn't compile the default scope? i.e. couldn't you leave out the explicit compile scope here and just not specify a scope?
It's set to `provided` in the parent pom.xml: https://github.com/apache/spark/blob/b23c4521f5df905e4fe4d79dd5b670286e2697f7/pom.xml#L365
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170269576
**[Test build #49047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49047/consoleFull)** for PR 10674 at commit [`16c228d`](https://github.com/apache/spark/commit/16c228d9227731e1228e75b419b06bcf29e8765c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170479156
Yeah the original comment was helpful, though it also kind of led me to believe the setting was connected to the class existing in the public API. That bit would have to be changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170230870
I'm not sure about that; I believe Guava is supposed to be `provided` here. Does it fix the issue though? @vanzin how is this one supposed to work re: shading now that we don't need Guava in the public API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170268121
@jaceklaskowski, to help me understand the scope of this bug could you tell us how you built Spark in your failing example? Did you use the Maven or SBT builds? Which build profiles? I'm wondering because there it looks like there are some differences between how the two build tools handle the Guava dependency. Specifically, it looks like SBT always puts Guava in compile scope: https://github.com/apache/spark/blob/3d77cffec093bed4d330969f1a996f3358b9a772/pom.xml#L2302
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10674#discussion_r49266568
--- Diff: network/common/pom.xml ---
@@ -55,6 +55,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
+ <scope>compile</scope>
--- End diff --
Isn't compile the default scope? i.e. couldn't you leave out the explicit `compile` scope here and just not specify a scope?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170410974
The fix is correct; that shouldn't have been removed in the first place (sorry I missed that). The change in that pom is so that the files actually get packaged somewhere. Otherwise the shade plugin just changes references in existing classes, but no jar would actually include the shaded classes.
BTW it would be nice to restore the original comment that explained that too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170268746
I'm using the following commands to do the build:
```
$ ./dev/change-scala-version.sh 2.11
$ ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Dscala-2.11 -Phive -Phive-thriftserver -DskipTests clean install
```
(I've been using sbt but somehow it got "broken" lately, i.e. it doesn't build all modules - I'm going to report it later).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10674
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170238716
Yes, it does. I'm using the latest revision + the change. It's a serious issue since standalone Master cannot be started as of today.
I do not know how it's supposed to have been fixed, but that's exactly what helped to resolve the issue. Any help appreciated to make it better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170268730
**[Test build #49047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49047/consoleFull)** for PR 10674 at commit [`16c228d`](https://github.com/apache/spark/commit/16c228d9227731e1228e75b419b06bcf29e8765c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170247768
Yeah I think this is correct. The comment that was here previously suggested this was related to use of `Optional` in the public API, but I think it needs to stay in compile scope for the shading to work as desired. If there are no other thoughts today I'll merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170332485
I'm going to merge this to fix the issue; @vanzin maybe you can swoop in and double-check what I did here to make sure the `provided` vs `compile` scope for Guava is still as it needs to be now that there is no Guava in the API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/10674#discussion_r49266731
--- Diff: network/common/pom.xml ---
@@ -55,6 +55,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
+ <scope>compile</scope>
--- End diff --
Thought so, but the line has fixed standalone Master to start (after the line got removed in https://github.com/apache/spark/commit/659fd9d04b988d48960eac4f352ca37066f43f5c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170269614
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170267761
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170269616
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49047/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10674#discussion_r49270748
--- Diff: network/common/pom.xml ---
@@ -55,6 +55,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
+ <scope>compile</scope>
--- End diff --
Ah, gotcha; (this is why I shouldn't review before morning coffee).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10674#issuecomment-170228387
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org