SPARK-11638: Run Spark on Mesos, in Docker with Bridge networking

[Rad Gruchalski <ra...@gruchalski.com>]

Dear Team,  

We, Virdata, would like to present the result of the last few months of our work with Mesos and Spark. Our requirement was to run Spark on Mesos in Docker for multi-tenant.
This required adapting Spark to run in Docker with Bridge networking.

The result (and patches) of our work is presented in the following JIRA ticket: https://issues.apache.org/jira/browse/SPARK-11638. The PR is: https://github.com/apache/spark/pull/9608.

The Summary

Provides spark.driver.advertisedPort, spark.fileserver.advertisedPort, spark.broadcast.advertisedPort and spark.replClassServer.advertisedPort settings to enable running Spark in Mesos on Docker with Bridge networking. Provides patches for Akka Remote to enable Spark driver advertisement using alternative host and port.
With these settings, it is possible to run Spark Master in a Docker container and have the executors running on Mesos talk back correctly to such Master.

The problem is discussed on the Mesos mailing list here: https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bFOtj5LJZFRH5u7ix-ghppFqKnVg9mkKctjg@mail.gmail.com%3E

We would like to contribute this to Apache Spark.

Happy to provide any further information.










Kind regards,

Radek Gruchalski

radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.