Apache Spark on Raspberry Pi Cluster with Docker

[Mark Bonnekessel <ma...@mailbox.org>]

Hi,

we are trying to setup apache spark on a raspberry pi cluster for educational use.
Spark is installed in a docker container and all necessary ports are exposed.

After we start master and workers, all workers are listed as alive in the master web ui (http://master:8080 <http://master:8080/>).

I want to run the SimpleApp-Example from the spark-homepage (http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications <http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications>) on the cluster to verify that everything is working, but i cannot get it run.

I built an jar-file and submitted the application with spark-submit and get the following output:
spark-submit --master spark://master:6066 <spark://master:6066> --deploy-mode cluster --class SimpleApp target/simple-project-1.0.jar
Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/10/28 12:54:43 INFO RestSubmissionClient: Submitting a request to launch an application in spark://localhost:6066 <spark://localhost:6066>.
15/10/28 12:54:43 INFO RestSubmissionClient: Submission successfully created as driver-20151028115443-0002. Polling submission state...
15/10/28 12:54:43 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20151028115443-0002 in spark://localhost:6066 <spark://localhost:6066>.
15/10/28 12:54:43 INFO RestSubmissionClient: State of driver driver-20151028115443-0002 is now SUBMITTED.
15/10/28 12:54:43 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20151028115443-0002",
  "serverSparkVersion" : "1.5.1",
  "submissionId" : "driver-20151028115443-0002",
  "success" : true
}

The driver is created correctly, but it never starts the application.
What am i missing?

Regards,
Mark