You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/12/06 23:41:01 UTC
spark git commit: [SPARK-18171][MESOS] Show correct framework address
in mesos master web ui when the advertised address is used
Repository: spark
Updated Branches:
refs/heads/master 7f31d378c -> a8ced76f1
[SPARK-18171][MESOS] Show correct framework address in mesos master web ui when the advertised address is used
## What changes were proposed in this pull request?
In [SPARK-4563](https://issues.apache.org/jira/browse/SPARK-4563) we added the support for the driver to advertise a different hostname/ip (`spark.driver.host` to the executors other than the hostname/ip the driver actually binds to (`spark.driver.bindAddress`). But in the mesos webui's frameworks page, it still shows the driver's binds hostname/ip (though the web ui link is correct). We should fix it to make them consistent.
Before:
After:
This PR doesn't affect the behavior on the spark side, only makes the display on the mesos master side more consistent.
## How was this patch tested?
Manual test.
- Build the package and build a docker image (spark:2.1-test)
``` sh
./dev/make-distribution.sh -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -Pmesos
```
- Then run the spark driver inside a docker container.
``` sh
docker run --rm -it \
--name=spark \
-p 30000-30010:30000-30010 \
-e LIBPROCESS_ADVERTISE_IP=172.17.42.1 \
-e LIBPROCESS_PORT=30000 \
-e MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos-1.0.0.so \
-e MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos-1.0.0.so \
-e SPARK_HOME=/opt/dist \
spark:2.1-test
```
- Inside the container, launch the spark driver, making use of the advertised address:
``` sh
/opt/dist/bin/spark-shell \
--master mesos://zk://172.17.42.1:2181/mesos \
--conf spark.driver.host=172.17.42.1 \
--conf spark.driver.bindAddress=172.17.0.1 \
--conf spark.driver.port=30001 \
--conf spark.driver.blockManager.port=30002 \
--conf spark.ui.port=30003 \
--conf spark.mesos.coarse=true \
--conf spark.cores.max=2 \
--conf spark.executor.cores=1 \
--conf spark.executor.memory=1g \
--conf spark.mesos.executor.docker.image=spark:2.1-test
```
- Run several spark jobs to ensure everything is running fine.
``` scala
val rdd = sc.textFile("file:///opt/dist/README.md")
rdd.cache().count
```
Author: Shuai Lin <li...@gmail.com>
Closes #15684 from lins05/spark-18171-show-correct-host-name-in-mesos-master-web-ui.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a8ced76f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a8ced76f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a8ced76f
Branch: refs/heads/master
Commit: a8ced76f16523c571284dccfc73d655e89ad570f
Parents: 7f31d37
Author: Shuai Lin <li...@gmail.com>
Authored: Wed Dec 7 07:40:54 2016 +0800
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Dec 7 07:40:54 2016 +0800
----------------------------------------------------------------------
.../apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a8ced76f/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
----------------------------------------------------------------------
diff --git a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala b/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
index 9cb6023..1d742fe 100644
--- a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
+++ b/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
@@ -80,6 +80,8 @@ trait MesosSchedulerUtils extends Logging {
frameworkId.foreach { id =>
fwInfoBuilder.setId(FrameworkID.newBuilder().setValue(id).build())
}
+ fwInfoBuilder.setHostname(Option(conf.getenv("SPARK_PUBLIC_DNS")).getOrElse(
+ conf.get(DRIVER_HOST_ADDRESS)))
conf.getOption("spark.mesos.principal").foreach { principal =>
fwInfoBuilder.setPrincipal(principal)
credBuilder.setPrincipal(principal)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org