You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bouk <gi...@git.apache.org> on 2014/05/02 21:45:08 UTC
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
GitHub user bouk opened a pull request:
https://github.com/apache/spark/pull/620
Reset the thread classLoader in the Mesos Executor
This is because Mesos calls it with a different environment or something, the result is that the Spark jar is missing and it can't load classes.
This fixes http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-td3510.html
I have no idea whether this is the right fix, I can only confirm that it fixes the issue for us.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Shopify/spark mesos-classloader-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/620.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #620
----
commit 40cd573f5b07fb6a1d00758be000b3f1938118be
Author: Bouke van der Bijl <bo...@gmail.com>
Date: 2014-05-02T19:43:14Z
Reset the thread classLoader in the Mesos Executor
This is because Mesos calls it with a different environment or
something, the result is that the Spark jar is missing and it can't load
classes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by bouk <gi...@git.apache.org>.
Github user bouk commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42213197
@pwendell I've updated it with your alternative fix, can confirm it still works
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42070660
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42070991
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42070675
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by airhorns <gi...@git.apache.org>.
Github user airhorns commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42080286
@bouk the spark contribution guidelines ask you make a JIRA https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42213555
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42095686
Would you mind trying an alternative fix?
```
val cls = Class.forName(name, true, Utils.getContextOrSparkClassLoader)
```
We recently did some clean-up of classloaders and I think we might have missed this case:
https://github.com/apache/spark/pull/398/files
https://issues.apache.org/jira/browse/SPARK-1480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42219393
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14667/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42080589
Would you mind printing the exception that happens when you don't have this fix? I'm wondering which instantiation fails. In theory I don't think we should have to do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42219390
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by bouk <gi...@git.apache.org>.
Github user bouk commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42088660
Note that exchanging the line https://github.com/apache/spark/blob/e53eb4f0159ebd4d72c4bbc3586fdfc66ccacab7/core/src/main/scala/org/apache/spark/SparkEnv.scala#L165 with
```java
val cls = Class.forName(name)
```
will also work. This confirms that it's a classLoader problem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42213570
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by bouk <gi...@git.apache.org>.
Github user bouk commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42088169
Mesos `stderr` without this patch on master
```
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0502 17:31:42.672224 14688 exec.cpp:131] Version: 0.18.0
I0502 17:31:42.674959 14707 exec.cpp:205] Executor registered on slave 20140501-182306-16842879-5050-10155-0
14/05/02 17:31:42 INFO MesosExecutorBackend: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/02 17:31:42 INFO MesosExecutorBackend: Registered with Mesos as executor ID 20140501-182306-16842879-5050-10155-0
14/05/02 17:31:43 INFO SecurityManager: Changing view acls to: vagrant
14/05/02 17:31:43 INFO SecurityManager: SecurityManager, is authentication enabled: false are ui acls enabled: false users with view permissions: Set(vagrant)
14/05/02 17:31:43 INFO Slf4jLogger: Slf4jLogger started
14/05/02 17:31:43 INFO Remoting: Starting remoting
14/05/02 17:31:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@vagrant.myshopify.io:50843]
14/05/02 17:31:43 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@vagrant.myshopify.io:50843]
java.lang.ClassNotFoundException: org/apache/spark/serializer/JavaSerializer
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:165)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:176)
at org.apache.spark.executor.Executor.<init>(Executor.scala:106)
at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:56)
Exception in thread "Thread-0" I0502 17:31:43.710039 14707 exec.cpp:412] Deactivating the executor libprocess
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by dkanoafry <gi...@git.apache.org>.
Github user dkanoafry commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42093656
fwiw, I experienced the same error with branch-1.0 and mesos 0.17, and applying the patch in this PR fixes the problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/620#issuecomment-42070992
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14624/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Reset the thread classLoader in the Mesos Exec...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/620
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---