You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by radekg <gi...@git.apache.org> on 2015/11/10 23:36:39 UTC
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
GitHub user radekg opened a pull request:
https://github.com/apache/spark/pull/9608
[SPARK-11638] [Mesos + Docker Bridge networking]: Run Spark on Mesos,…
… in Docker with Bridge networking
Provides `spark.driver.advertisedPort`, `spark.fileserver.advertisedPort`, `spark.broadcast.advertisedPort` and `spark.replClassServer.advertisedPort` settings to enable running Spark in Mesos on Docker with Bridge networking. Provides patches for Akka Remote to enable Spark driver advertisement using alternative host and port (https://issues.apache.org/jira/browse/SPARK-11638).
With these settings, it is possible to run Spark Master in a Docker container and have the executors running on Mesos talk back correctly to such Master.
The problem is discussed on the Mesos mailing list here: https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bFOtj5LJZFRH5u7ix-ghppFqKnVg9mkKctjg@mail.gmail.com%3E
To take advantage of these, patched Akka Remote is required. Patches for Akka available here: https://issues.apache.org/jira/browse/SPARK-11638.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/radekg/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9608.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9608
----
commit 43ef277123b848abd98691071e5094193eb18cc0
Author: radekg <ra...@gruchalski.com>
Date: 2015-11-10T22:33:36Z
[SPARK-11638] [Mesos + Docker Bridge networking]: Run Spark on Mesos, in Docker with Bridge networking
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167848079
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48416/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44533312
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
I'd appreciate some indication on how such test would look like. AFAIR the validity of `SPARK_PUBLIC_DNS` is not checked anywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r48771144
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -854,7 +854,8 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse(
--- End diff --
What's the rationale behind this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r47605974
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala ---
@@ -122,7 +122,8 @@ private[netty] class NettyRpcEnv(
@Nullable
override lazy val address: RpcAddress = {
- if (server != null) RpcAddress(host, server.getPort()) else null
+ if (server != null)
+ RpcAddress(host, conf.getInt("spark.driver.advertisedPort", server.getPort())) else null
--- End diff --
You've checked 0 previously but not here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-164606160
@dragos @tnachen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-178263971
@radekg @tnachen I'd just like to point out that Spark no longer uses Akka as its underlying RPC system, so a lot of the work done here may no longer apply.
Actually, what is the high level goal here? The JIRA is very detailed but I don't think it's very concise. You mentioned "it's possible to run Spark Master in a container". Did you mean the Spark driver? If that's the goal then we should look for a more isolated solution than to just a lot of the Spark core internals only to satisfy this narrow use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168351836
**[Test build #48573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48573/consoleFull)** for PR 9608 at commit [`f88ee99`](https://github.com/apache/spark/commit/f88ee99e36c6b6c6521d1e7d14a13e18114d17fe).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44533187
--- Diff: core/src/main/scala/org/apache/spark/HttpFileServer.scala ---
@@ -42,10 +42,11 @@ private[spark] class HttpFileServer(
fileDir.mkdir()
jarDir.mkdir()
logInfo("HTTP File server directory is " + baseDir)
- httpServer = new HttpServer(conf, baseDir, securityManager, requestedPort, "HTTP file server")
+ httpServer = new HttpServer(conf, baseDir, securityManager, requestedPort, "HTTP file server", Some("spark.fileserver.advertisedPort"))
--- End diff --
It will revert to the regular port setting: https://github.com/apache/spark/pull/9608/files#diff-25ede712da47f7ff04ddcffce9baee45R160
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44526017
--- Diff: core/src/main/scala/org/apache/spark/HttpFileServer.scala ---
@@ -42,10 +42,11 @@ private[spark] class HttpFileServer(
fileDir.mkdir()
jarDir.mkdir()
logInfo("HTTP File server directory is " + baseDir)
- httpServer = new HttpServer(conf, baseDir, securityManager, requestedPort, "HTTP file server")
+ httpServer = new HttpServer(conf, baseDir, securityManager, requestedPort, "HTTP file server", Some("spark.fileserver.advertisedPort"))
--- End diff --
what happens if there is no advertisedPort?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
The reason I'm asking is because bridge networking would solve a lot of other problems. So does that mean bridge networking on mesos and docker isn't supported right now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-169001637
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48764/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167648843
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168345176
**[Test build #48573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48573/consoleFull)** for PR 9608 at commit [`f88ee99`](https://github.com/apache/spark/commit/f88ee99e36c6b6c6521d1e7d14a13e18114d17fe).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168351854
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48573/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
Is this running on Mesos?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
@radekg, Thank you, I was starting to worry that is the case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168344020
I will need to verify this patch with torrent broadcast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-169001463
**[Test build #48764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48764/consoleFull)** for PR 9608 at commit [`04255b0`](https://github.com/apache/spark/commit/04255b0c7760319ba0f5111996b87f939af2f7fa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167648846
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48370/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44562422
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
Probably worth revisiting this part of the code. I see I should have just relied on `SPARK_LOCAL_HOSTNAME`, however, at first thought, I had trouble getting it to work (it was reset by some other code, without me explicitly calling it, to another value).
I admit - I do not understand the distinction between `SPARK_PUBLIC_DNS` and `SPARK_LOCAL_HOSTNAME`. `SPARK_PUBLIC_DNS` is used by the driver only and the other one is used for all other services. Why the difference?
It may be just my assumption but if I'm setting `SPARK_PUBLIC_DNS`, I'd like to have all services bound to `SPARK_PUBLIC_DNS`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168982029
**[Test build #48764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48764/consoleFull)** for PR 9608 at commit [`04255b0`](https://github.com/apache/spark/commit/04255b0c7760319ba0f5111996b87f939af2f7fa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168780854
I think it's also note documenting this of how it can possibly work with bridge mode. Also I think it's worth noting that user must explicitly map the ports before hand and also configure Spark to use those ports.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
Yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
@cherryii FYI, it's possible to achieve the same setup without using bridge networking, if you can use host network, it's perfectly fine to set this up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167648836
**[Test build #48370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48370/consoleFull)** for PR 9608 at commit [`946ec59`](https://github.com/apache/spark/commit/946ec59fc312e36270af7d7e9e4f444de66fa210).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44563045
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
However, I'll double check this and, if it works reliably with `SPARK_LOCAL_HOSTNAME`, remove this bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
I'm already using those three properties and the error I posted above is what happens in bridge networking mode.
Note: I'm using spark-submit ... --conf spark.driver.port=$PORT0 etc... <JAR> [app options]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44526124
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
might be worth having sanity checks here for ip validity, + some scalatests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
It's not going to work with bridge network. The options will work fine with host network. Bridge is a different story.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167643636
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44539552
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
sorry, not got much insight there, except failing fast on "" is good. It should really be the job of the code downstream to recognise and react to the usual config problems of trying to listen on an invalid hostname or port 0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-155590082
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
Was anything similar to this implemented in spark 2? I'm trying to run in a similar situation and can't seem to get spark to work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r48770592
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -245,10 +245,19 @@ object SparkEnv extends Logging {
val securityManager = new SecurityManager(conf)
+ val driverAdverisedData = if (isDriver) {
+ (conf.getOption("spark.driver.advertisedHost"),
+ conf.getOption("spark.driver.advertisedPort").map(_.toInt))
+ } else {
+ (None, None)
+ }
+
val actorSystemName = if (isDriver) driverActorSystemName else executorActorSystemName
// Create the ActorSystem for Akka and get the port it binds to.
val rpcEnv = RpcEnv.create(actorSystemName, hostname, port, conf, securityManager,
- clientMode = !isDriver)
+ clientMode = !isDriver,
+ advertisedHost = driverAdverisedData._1,
--- End diff --
typo: driverAdverisedData -> driverAdvertisedData
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167847920
**[Test build #48416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48416/consoleFull)** for PR 9608 at commit [`b712b8d`](https://github.com/apache/spark/commit/b712b8d3f0bd11575533af6bb5931df096bce239).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167645599
**[Test build #48370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48370/consoleFull)** for PR 9608 at commit [`946ec59`](https://github.com/apache/spark/commit/946ec59fc312e36270af7d7e9e4f444de66fa210).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r47593516
--- Diff: core/src/main/scala/org/apache/spark/HttpServer.scala ---
@@ -152,6 +153,17 @@ private[spark] class HttpServer(
}
}
+ private def getAdvertisedOrServerPort: Int = {
+ advertisedPortSetting match {
+ case Some(setting) =>
+ conf.getOption(setting).map(_.toInt).getOrElse(0) match {
+ case x if x > 0 => x
+ case _ => port
+ }
+ case None => port
+ }
+ }
+
/**
* Get the URI of this HTTP server (http://host:port or https://host:port)
--- End diff --
Should we be more clear here that this is the advertised Uri?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by cherryii <gi...@git.apache.org>.
Github user cherryii commented on the issue:
https://github.com/apache/spark/pull/9608
I am using docker, my container is a spark driver that only runs my job, my container's entrypoint is a script that runs spark submit. I use the client deploy-mode and set all the config values you mention above and get the error below when in bridge networking, spark starts and runs fine in host networking but will create other problems I'll have to deal with which are specific to my organization and deployment environment, bridge networking really would be the best option for me if I could get it working:
java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167826650
**[Test build #48416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48416/consoleFull)** for PR 9608 at commit [`b712b8d`](https://github.com/apache/spark/commit/b712b8d3f0bd11575533af6bb5931df096bce239).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168972745
@tnachen what would be the best place to publish such documentation? Regarding `worth nothing` comment. The whole concept behind this pr is to allow running spark master inside of a docker container on mesos and have the executors run on the same mesos infrastructure, outside of the container in which the master is running.
As such, this can only be achieved by knowing the ports on which the particular services will run inside of the mesos task. It's not possible otherwise as the ports have to be handed in to mesos at the time of the task submission. This also means that the ports can be dedicated to spark and the user is in full control of what ports will be available.
Does this clarify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r44597535
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -810,7 +810,7 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse( customHostname.getOrElse(InetAddresses.toUriString(localIpAddress)) )
--- End diff --
don't ask me about mesos internals; I work on YARN/hadoop stuff. I was just looking to see what was going to break there —and what are the usual general troublespots with apps (networking; configuration)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by 3ourroom <gi...@git.apache.org>.
Github user 3ourroom commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r47606048
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala ---
@@ -122,7 +122,8 @@ private[netty] class NettyRpcEnv(
@Nullable
override lazy val address: RpcAddress = {
- if (server != null) RpcAddress(host, server.getPort()) else null
+ if (server != null)
+ RpcAddress(host, conf.getInt("spark.driver.advertisedPort", server.getPort())) else null
--- End diff --
NAVER - http://www.naver.com/
--------------------------------------------
3ourroom@naver.com 님께 보내신 메일 <Re: [spark] [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spark on Mesos,… (#9608)> 이 다음과 같은 이유로 전송 실패했습니다.
--------------------------------------------
받는 사람이 회원님의 메일을 수신차단 하였습니다.
--------------------------------------------
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167643628
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r48834671
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -854,7 +854,8 @@ private[spark] object Utils extends Logging {
* Get the local machine's URI.
*/
def localHostNameForURI(): String = {
- customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
+ sys.env.get("SPARK_PUBLIC_DNS").getOrElse(
--- End diff --
@tnachen writing from what I remember the issue here was, I can retest if necessary but it's a rather long process:
`customHostname` comes from `SPARK_LOCAL_HOSTNAME `. When I was implementing this code, I tried setting the value of `SPARK_LOCAL_HOSTNAME ` to the same value as `SPARK_PUBLIC_DNS`. However, after getting to the `localHostNameForURI()` call, the `customHostname` value was no longer what I set `SPARK_LOCAL_HOSTNAME`. In other words, it has been changed by something from outside and I am not able to find it what happened.
Let me ask a question: under what circumstances the user would like to set `SPARK_LOCAL_HOSTNAME` to a different value than `SPARK_PUBLIC_DNS `?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9608#discussion_r48442211
--- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala ---
@@ -122,7 +122,8 @@ private[netty] class NettyRpcEnv(
@Nullable
override lazy val address: RpcAddress = {
- if (server != null) RpcAddress(host, server.getPort()) else null
+ if (server != null)
+ RpcAddress(host, conf.getInt("spark.driver.advertisedPort", server.getPort())) else null
--- End diff --
My assumption was that this was the safest behaviour for the Netty based rpc. If the `spark.driver.advertisedPort` setting is `0`, it's an operator error, no different than setting the value to, say, `1`. Not sure if there is any value in testing for `0` here. Ideas welcome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167848076
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
@cherryii feel free to reach out if you need additional help. I'm sure your issues can be sorted out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg closed the pull request at:
https://github.com/apache/spark/pull/9608
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-164947432
@radekg also please rebase to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-168351853
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-196016688
@tnachen Keeping up with the pace of changes to Spark code was a bit too much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
Are you deploying with Docker? If so, what problems are you facing when using host networking?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #9608: [SPARK-11638] [Mesos + Docker Bridge networking]: Run Spa...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the issue:
https://github.com/apache/spark/pull/9608
No matter if you was to run this in bridge or host mode, the best idea would be to request a number of ports. In case of `HOST` network only the `port` needs to be requested. If you can use Marathon, this is what the `portDefinitions` would look like:
```
"portDefinitions": [
{ "port": 0, "name": "spark.driver.port", ... },
{ "port": 0, "name": "spark.blockManager.port", ... },
{ "port": 0, "name": "spark.ui.port", ... }
]
```
And then, the following can be done when submitting the job:
```
spark-submit ... -Dspark.driver.port=$PORT0 -Dspark.blockManager.port=$PORT1 -Dspark.ui.port=$PORT2 ...
```
You may have to (I have no chance to test right now) request a port for `spark.shuffle.service.port` as it uses fixed `7337`, if you run multiple spark instances, it's best to give it a port elected by Mesos.
The respective documentation can be found here:
- https://mesosphere.github.io/marathon/docs/ports.html
- http://spark.apache.org/docs/latest/configuration.html
Hope this helps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-169001635
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167826085
I have 3 tests failing locally but I don't think these are related to my changes. `scalastyle` seems to be ok now. Failing tests:
```
- launch simple application with spark-submit *** FAILED ***
Process returned with exit code 1. See the log4j logs for more detail. (SparkSubmitSuite.scala:583)
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation': class file for jdk.Profile+Annotation not found
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Override.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Annotation.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Target.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/ElementType.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Retention.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/RetentionPolicy.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Override.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Annotation.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Target.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/ElementType.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Retention.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/RetentionPolicy.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
20 warnings
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation': class file for jdk.Profile+Annotation not found
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Override.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Annotation.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Target.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/ElementType.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Retention.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/RetentionPolicy.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Override.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Annotation.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Target.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/ElementType.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Retention.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/RetentionPolicy.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
20 warnings
- includes jars passed in through --jars *** FAILED ***
Process returned with exit code 1. See the log4j logs for more detail. (SparkSubmitSuite.scala:583)
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Integer.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Integer.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation': class file for jdk.Profile+Annotation not found
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Number.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Number.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Byte.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Byte.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Character.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Character.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Short.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Short.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
18 warnings
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Integer.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Integer.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation': class file for jdk.Profile+Annotation not found
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/AutoCloseable.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Number.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Number.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Byte.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Byte.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Character.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Character.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
warning: /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Short.class): major version 52 is newer than 51, the highest major version supported by this compiler.
It is recommended that the compiler be upgraded.
/Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Short.class): warning: Cannot find annotation method 'value()' in type 'Profile+Annotation'
18 warnings
- includes jars passed in through --packages *** FAILED ***
Process returned with exit code 1. See the log4j logs for more detail. (SparkSubmitSuite.scala:583)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/spark/pull/9608#issuecomment-167755475
I will take a look at those scalastyle errors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org