You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2015/10/28 22:37:45 UTC
newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
Hi
I just created new cluster using the spark-c2 script from the
spark-1.5.1-bin-hadoop2.6 distribution. The master and slaves seem to be up
and running. I am having a heck of time figuring out how to submit apps. As
a test I compile the sample JavaSparkPi example. I have copied my jar file
to the master and want to run the application in cluster mode. My real app
will take a long time to complete. I do not want to wait around.
Any idea what the issue is?
Kind regards
Andy
http://spark.apache.org/docs/latest/submitting-applications.html
The following command works fine on my Mac, how ever when I run it on my
master I get the following warning. The app works correctly
[ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
org.apache.spark.examples.JavaSparkPi --master local[4]
sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out
15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1
15/10/28 21:07:11 WARN spark.SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the
spark config.
Adding num-exactors I still get the same warning. The app works correctly
$SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
--master local[4] --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
pi.numExecutor4.out
15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1
15/10/28 21:09:41 WARN spark.SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the
spark config.
I also tried variations on [ec2-user@ip-172-31-29-60 ~]$
$SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
--master spark://172.31.29.60:7077 --num-executors 4
sparkPi-1.0-SNAPSHOT.jar
15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1
15/10/28 21:14:48 WARN spark.SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the
spark config.
15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to:
ec2-user
15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to:
ec2-user
15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(ec2-user); users with modify permissions: Set(ec2-user)
15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/10/28 21:14:49 INFO Remoting: Starting remoting
15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@172.31.29.60:52405]
15/10/28 21:14:50 INFO util.Utils: Successfully started service
'sparkDriver' on port 52405.
15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker
15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster
15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
/mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d
15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
/mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084
15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with
capacity 530.0 MB
15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is
/mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a
73-83f0-bac856bc3548
15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server
15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/28 21:14:50 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:48262
15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file
server' on port 48262.
15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/28 21:14:50 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI' on
port 4040.
15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at
http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
15/10/28 21:14:50 INFO spark.SparkContext: Added JAR
file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
1446066890783
15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name
DAGScheduler for source because spark.app.id is not set.
15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to master
spark://172.31.29.60:7077...
15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught
exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@ae9a189 rejected from
java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1,
active threads = 1, queued tasks = 0, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread
PoolExecutor.java:2047)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369
)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.
java:112)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
AllMasters$1.apply(AppClient.scala:96)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
AllMasters$1.apply(AppClient.scala:95)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster
s(AppClient.scala:95)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$dep
loy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run
$1.apply$mcV$sp(AppClient.scala:132)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien
t.scala:124)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
at java.lang.Thread.run(Thread.java:745)
15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called
15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called
15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
/mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61a
b-4ea8-8c61-e7b9af8c481d
15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
/mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf
15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
/mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857
[ec2-user@ip-172-31-29-60 ~]$
[ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077
--num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
pi.numExecutor4.URILocalhost.out
15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1
15/10/28 21:22:42 WARN spark.SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the
spark config.
15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to:
ec2-user
15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to:
ec2-user
15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(ec2-user); users with modify permissions: Set(ec2-user)
15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/10/28 21:22:43 INFO Remoting: Starting remoting
15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@172.31.29.60:36217]
15/10/28 21:22:43 INFO util.Utils: Successfully started service
'sparkDriver' on port 36217.
15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker
15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster
15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
/mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815
15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
/mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31
15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with
capacity 530.0 MB
15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is
/mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48
fc-b142-8aeff83be686
15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server
15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/28 21:22:43 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:41335
15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file
server' on port 41335.
15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/28 21:22:44 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI' on
port 4040.
15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at
http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
15/10/28 21:22:44 INFO spark.SparkContext: Added JAR
file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
1446067364171
15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name
DAGScheduler for source because spark.app.id is not set.
15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to master
spark://localhost:7077...
15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect to
localhost:7077: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable remote
address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
5000 ms, all messages to this address will be delivered to dead letters.
Reason: Connection refused: localhost/127.0.0.1:7077
15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect to
master localhost:7077
akka.actor.ActorNotFound: Actor not found for:
ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
Path(/user/Master)]
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
5)
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
3)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE
xecutor.scala:67)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe
cutor.scala:82)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
cala:59)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
cala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute
(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s
cala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
at
akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo
rRefProvider.scala:87)
at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
at
akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin
ishTerminate(FaultHandling.scala:210)
at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
at akka.actor.ActorCell.terminate(ActorCell.scala:369)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis
patcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1
339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java
:107)
15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to master
spark://localhost:7077...
15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect to
localhost:7077: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable remote
address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
5000 ms, all messages to this address will be delivered to dead letters.
Reason: Connection refused: localhost/127.0.0.1:7077
15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect to
master localhost:7077
akka.actor.ActorNotFound: Actor not found for:
ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
Path(/user/Master)]
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
5)
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
3)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE
xecutor.scala:67)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe
cutor.scala:82)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
cala:59)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
cala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute
(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s
cala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
at
akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo
rRefProvider.scala:87)
at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
at
akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin
ishTerminate(FaultHandling.scala:210)
at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
at akka.actor.ActorCell.terminate(ActorCell.scala:369)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis
patcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1
339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java
:107)
15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to master
spark://localhost:7077...
15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught
exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@7a9a7ff1 rejected from
java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1,
active threads = 1, queued tasks = 0, completed tasks = 2]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread
PoolExecutor.java:2047)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369
)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.
java:112)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
AllMasters$1.apply(AppClient.scala:96)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
AllMasters$1.apply(AppClient.scala:95)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster
s(AppClient.scala:95)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$dep
loy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run
$1.apply$mcV$sp(AppClient.scala:132)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien
t.scala:124)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
at java.lang.Thread.run(Thread.java:745)
15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect to
localhost:7077: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable remote
address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
5000 ms, all messages to this address will be delivered to dead letters.
Reason: Connection refused: localhost/127.0.0.1:7077
15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called
15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called
15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
/mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48
fc-b142-8aeff83be686
15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
/mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8
15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
/mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e
[ec2-user@ip-172-31-29-60 ~]$
[ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077
--num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
pi.numExecutor4.URI172.31.29.60.cluster.out
Running Spark using the REST application submission protocol.
15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to
launch an application in spark://172.31.29.60:7077.
15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to
server spark://172.31.29.60:7077.
Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server.
Falling back to legacy submission gateway instead.
^C
[ec2-user@ip-172-31-29-60 ~]$
Re: newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Hi Robin and Sabarish
I figure out what the problem
To submit my java app so that it runs in cluster mode (ie. I can close my
laptop and go home) I need to do the following
1. make sure my jar file is available on all the slaves. Spark-submit will
cause my driver to run on a slave, It will not automatically copy my jar
file to slaves. I found placing the jar in hdfs the easiest way to handle
this
2. I needed to pass the command argument deploy-mode cluster
3. Use specify the path the jar file as a url hdfs://
Here are two tricks to figure out the correct URL for master
1. If you know the name of your cluster you can find the public DNS name for
your master. By default use port 7077
cd spark-1.5.1-bin-hadoop2.6/ec2
$ spark-ec2 get-master --region=us-west-1 streamingDC
Searching for existing cluster streamingDC in region us-west-1...
Found 1 master, 3 slaves.
ec2-54-251-207-123.us-west-1.compute.amazonaws.com
$
2. If you know the public DNS name of the master go to
http://mastername..compute.amazonaws.com/8080 . The Title should be the
correct url (ie. Port 7077)
On master
/root/ephemeral-hdfs/bin/hadoop fs -mkdir /home/ec2-user/sparkExamples
/root/ephemeral-hdfs/bin/hadoop fs -put sparkPi-1.0-SNAPSHOT.jar
/home/ec2-user/sparkExamples
/root/ephemeral-hdfs/bin/hadoop fs -ls /home/ec2-user/sparkExamples
$SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
--master spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077
--deploy-mode cluster
hdfs:///home/ec2-user/sparkExamples/sparkPi-1.0-SNAPSHOT.jar 100
Running Spark using the REST application submission protocol.
15/10/29 16:39:08 INFO rest.RestSubmissionClient: Submitting a request to
launch an application in
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
15/10/29 16:39:09 WARN rest.RestSubmissionClient: Unable to connect to
server spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
Warning: Master endpoint
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 was not a
REST server. Falling back to legacy submission gateway instead.
[ec2-user@ip-172-31-29-60 ~]$
I really appreciate everyone¹s help
Andy
Re: newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
Posted by Sabarish Sasidharan <sa...@manthan.com>.
One other option is to use AWS EMR and specify Spark as one of the
applications while launching. It will automatically install and configure
Spark. With that you can also use master as yarn-cluster for your jobs
(--master yarn-cluster)
Regards
Sab
On 29-Oct-2015 4:04 am, "Andy Davidson" <An...@santacruzintegration.com>
wrote:
> Hi Robin
>
> Okay this work a much better but not quite as I expected
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://
> ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077
> ./sparkPi-1.0-SNAPSHOT.jar 100 2>&1 | tee URL.7077.out
>
>
> Runs how ever I get a log msgs in my shell. Its running in client mode (as
> expected)
>
>
>
> I want to run headless. My real job will take a long time. I want to log
> out of the cluster while it is running.
>
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://
> ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 --deploy-mode
> cluster ./sparkPi-1.0-SNAPSHOT.jar 100 2>&1 | tee URL.7077.cluster.out
>
>
> Running Spark using the REST application submission protocol.
>
> 15/10/28 22:21:08 INFO rest.RestSubmissionClient: Submitting a request to
> launch an application in spark://
> ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
>
> 15/10/28 22:21:08 WARN rest.RestSubmissionClient: Unable to connect to
> server spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
>
> Warning: Master endpoint spark://
> ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 was not a REST
> server. Falling back to legacy submission gateway instead.
>
> 15/10/28 22:21:15 ERROR deploy.ClientEndpoint: Exception from cluster was:
> java.io.FileNotFoundException: /home/ec2-user/./sparkPi-1.0-SNAPSHOT.jar
> (No such file or directory)
>
> java.io.FileNotFoundException: /home/ec2-user/./sparkPi-1.0-SNAPSHOT.jar
> (No such file or directory)
>
> at java.io.FileInputStream.open0(Native Method)
>
> at java.io.FileInputStream.open(FileInputStream.java:195)
>
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>
> at
> org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:124)
>
> at
> org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:114)
>
> at org.spark-project.guava.io.ByteSource.copyTo(ByteSource.java:202)
>
> at org.spark-project.guava.io.Files.copy(Files.java:436)
>
> at
> org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:514)
>
> at org.apache.spark.util.Utils$.copyFile(Utils.scala:485)
>
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:562)
>
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:369)
>
> at org.apache.spark.deploy.worker.DriverRunner.org
> $apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150)
>
> at
> org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79)
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
> Any suggestions
>
> Andy
>
> For giggles I tried using port 6066. No luck
>
> From: Robin East <ro...@xense.co.uk>
> Date: Wednesday, October 28, 2015 at 2:53 PM
> To: Andrew Davidson <An...@SantaCruzIntegration.com>
> Cc: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: Re: newbie trouble submitting java app to AWS cluster I created
> using spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
>
> You'll actually be using standalone (the Hadoop reference in the binary is
> to client libraries included in the build). If you navigate to
> http://<master-host>:8080 you should teach the spark master UI. At the
> top you should see a spark://... Url. That is the url you need to pass as
> the --master parameter to spark-submit
>
> Sent from my iPhone
>
> On 28 Oct 2015, at 21:46, Andy Davidson <An...@SantaCruzIntegration.com>
> wrote:
>
> I forgot to mention. I do not have a preference for the cluster manager. I
> choose the spark-1.5.1-bin-hadoop2.6 distribution because I want to use
> hdfs. I assumed this distribution would use yarn.
>
> Thanks
>
> Andy
>
> From: Andrew Davidson <An...@SantaCruzIntegration.com>
> Date: Wednesday, October 28, 2015 at 2:37 PM
> To: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: newbie trouble submitting java app to AWS cluster I created
> using spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
>
> Hi
>
>
> I just created new cluster using the spark-c2 script from the spark-1.5.1-bin-hadoop2.6
> distribution. The master and slaves seem to be up and running. I am
> having a heck of time figuring out how to submit apps. As a test I compile
> the sample JavaSparkPi example. I have copied my jar file to the master and
> want to run the application in cluster mode. My real app will take a long
> time to complete. I do not want to wait around.
>
>
> Any idea what the issue is?
>
>
> Kind regards
>
>
> Andy
>
>
>
> http://spark.apache.org/docs/latest/submitting-applications.html
>
>
> The following command works fine on my Mac, how ever when I run it on my
> master I get the following warning. The app works correctly
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master local[4]
> sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out
>
> 15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:07:11 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
> Adding —num-exactors I still get the same warning. The app works correctly
>
>
> $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master local[4] --num-executors 4
> sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.numExecutor4.out
>
> 15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:09:41 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
> I also tried variations on [ec2-user@ip-172-31-29-60 ~]$
> $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
> --master spark://172.31.29.60:7077 --num-executors 4
> sparkPi-1.0-SNAPSHOT.jar
>
> 15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:14:48 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to:
> ec2-user
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to:
> ec2-user
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(ec2-user); users with modify permissions: Set(ec2-user)
>
> 15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/10/28 21:14:49 INFO Remoting: Starting remoting
>
> 15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@172.31.29.60:52405]
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service
> 'sparkDriver' on port 52405.
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster
>
> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory
> at /mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d
>
> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory
> at /mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084
>
> 15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with
> capacity 530.0 MB
>
> 15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a73-83f0-bac856bc3548
>
> 15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server
>
> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:48262
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 48262.
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>
> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI'
> on port 4040.
>
> 15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at
> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>
> 15/10/28 21:14:50 INFO spark.SparkContext: Added JAR
> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
> http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
> 1446066890783
>
> 15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name
> DAGScheduler for source because spark.app.id is not set.
>
> 15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to
> master spark://172.31.29.60:7077...
>
> 15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[appclient-registration-retry-thread,5,main]
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@ae9a189 rejected from
> java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1,
> active threads = 1, queued tasks = 0, completed tasks = 0]
>
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
>
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95)
>
> at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org
> $apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132)
>
> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61ab-4ea8-8c61-e7b9af8c481d
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077
> --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
> pi.numExecutor4.URILocalhost.out
>
> 15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:22:42 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to:
> ec2-user
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to:
> ec2-user
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(ec2-user); users with modify permissions: Set(ec2-user)
>
> 15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/10/28 21:22:43 INFO Remoting: Starting remoting
>
> 15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@172.31.29.60:36217]
>
> 15/10/28 21:22:43 INFO util.Utils: Successfully started service
> 'sparkDriver' on port 36217.
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster
>
> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory
> at /mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815
>
> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory
> at /mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31
>
> 15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with
> capacity 530.0 MB
>
> 15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc-b142-8aeff83be686
>
> 15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server
>
> 15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:22:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:41335
>
> 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 41335.
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>
> 15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:22:44 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
>
> 15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI'
> on port 4040.
>
> 15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at
> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>
> 15/10/28 21:22:44 INFO spark.SparkContext: Added JAR
> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
> http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
> 1446067364171
>
> 15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name
> DAGScheduler for source because spark.app.id is not set.
>
> 15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to
> master spark://localhost:7077...
>
> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect
> to localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable
> remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now
> gated for 5000 ms, all messages to this address will be delivered to dead
> letters. Reason: Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect
> to master localhost:7077
>
> akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
> Path(/user/Master)]
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
>
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
>
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
>
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
>
> at
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> 15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to
> master spark://localhost:7077...
>
> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect
> to localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable
> remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now
> gated for 5000 ms, all messages to this address will be delivered to dead
> letters. Reason: Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect
> to master localhost:7077
>
> akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
> Path(/user/Master)]
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
>
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
>
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
>
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
>
> at
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> 15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to
> master spark://localhost:7077...
>
> 15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[appclient-registration-retry-thread,5,main]
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@7a9a7ff1 rejected from
> java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1,
> active threads = 1, queued tasks = 0, completed tasks = 2]
>
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
>
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95)
>
> at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org
> $apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132)
>
> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect
> to localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable
> remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now
> gated for 5000 ms, all messages to this address will be delivered to dead
> letters. Reason: Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc-b142-8aeff83be686
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077
> --num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
> pi.numExecutor4.URI172.31.29.60.cluster.out
>
> Running Spark using the REST application submission protocol.
>
> 15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to
> launch an application in spark://172.31.29.60:7077.
>
> 15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to
> server spark://172.31.29.60:7077.
>
> Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server.
> Falling back to legacy submission gateway instead.
>
> ^C
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
Re: newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Hi Robin
Okay this work a much better but not quite as I expected
[ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
org.apache.spark.examples.JavaSparkPi --master
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077
./sparkPi-1.0-SNAPSHOT.jar 100 2>&1 | tee URL.7077.out
Runs how ever I get a log msgs in my shell. Its running in client mode (as
expected)
I want to run headless. My real job will take a long time. I want to log out
of the cluster while it is running.
[ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
org.apache.spark.examples.JavaSparkPi --master
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077
--deploy-mode cluster ./sparkPi-1.0-SNAPSHOT.jar 100 2>&1 | tee
URL.7077.cluster.out
Running Spark using the REST application submission protocol.
15/10/28 22:21:08 INFO rest.RestSubmissionClient: Submitting a request to
launch an application in
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
15/10/28 22:21:08 WARN rest.RestSubmissionClient: Unable to connect to
server spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077.
Warning: Master endpoint
spark://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:7077 was not a
REST server. Falling back to legacy submission gateway instead.
15/10/28 22:21:15 ERROR deploy.ClientEndpoint: Exception from cluster was:
java.io.FileNotFoundException: /home/ec2-user/./sparkPi-1.0-SNAPSHOT.jar (No
such file or directory)
java.io.FileNotFoundException: /home/ec2-user/./sparkPi-1.0-SNAPSHOT.jar (No
such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:124)
at
org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:114)
at org.spark-project.guava.io.ByteSource.copyTo(ByteSource.java:202)
at org.spark-project.guava.io.Files.copy(Files.java:436)
at
org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Util
s.scala:514)
at org.apache.spark.util.Utils$.copyFile(Utils.scala:485)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:562)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:369)
at
org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$D
riverRunner$$downloadUserJar(DriverRunner.scala:150)
at
org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:7
9)
[ec2-user@ip-172-31-29-60 ~]$
Any suggestions
Andy
For giggles I tried using port 6066. No luck
From: Robin East <ro...@xense.co.uk>
Date: Wednesday, October 28, 2015 at 2:53 PM
To: Andrew Davidson <An...@SantaCruzIntegration.com>
Cc: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re: newbie trouble submitting java app to AWS cluster I created
using spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
> You'll actually be using standalone (the Hadoop reference in the binary is to
> client libraries included in the build). If you navigate to
> http://<master-host>:8080 you should teach the spark master UI. At the top you
> should see a spark://... Url. That is the url you need to pass as the --master
> parameter to spark-submit
>
> Sent from my iPhone
>
> On 28 Oct 2015, at 21:46, Andy Davidson <An...@SantaCruzIntegration.com> wrote:
>
>> I forgot to mention. I do not have a preference for the cluster manager. I
>> choose the spark-1.5.1-bin-hadoop2.6 distribution because I want to use hdfs.
>> I assumed this distribution would use yarn.
>>
>> Thanks
>>
>> Andy
>>
>> From: Andrew Davidson <An...@SantaCruzIntegration.com>
>> Date: Wednesday, October 28, 2015 at 2:37 PM
>> To: "user@spark.apache.org" <us...@spark.apache.org>
>> Subject: newbie trouble submitting java app to AWS cluster I created using
>> spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
>>
>>> Hi
>>>
>>>
>>>
>>> I just created new cluster using the spark-c2 script from the
>>> spark-1.5.1-bin-hadoop2.6 distribution. The master and slaves seem to be up
>>> and running. I am having a heck of time figuring out how to submit apps. As
>>> a test I compile the sample JavaSparkPi example. I have copied my jar file
>>> to the master and want to run the application in cluster mode. My real app
>>> will take a long time to complete. I do not want to wait around.
>>>
>>>
>>>
>>> Any idea what the issue is?
>>>
>>>
>>>
>>> Kind regards
>>>
>>>
>>>
>>> Andy
>>>
>>>
>>>
>>>
>>>
>>> http://spark.apache.org/docs/latest/submitting-applications.html
>>>
>>>
>>> The following command works fine on my Mac, how ever when I run it on my
>>> master I get the following warning. The app works correctly
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
>>> org.apache.spark.examples.JavaSparkPi --master local[4]
>>> sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out
>>>
>>> 15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1
>>>
>>> 15/10/28 21:07:11 WARN spark.SparkConf:
>>>
>>> SPARK_WORKER_INSTANCES was detected (set to '1').
>>>
>>> This is deprecated in Spark 1.0+.
>>>
>>>
>>>
>>> Please instead use:
>>>
>>> - ./spark-submit with --num-executors to specify the number of executors
>>>
>>> - Or set SPARK_EXECUTOR_INSTANCES
>>>
>>> - spark.executor.instances to configure the number of instances in the
>>> spark config.
>>>
>>>
>>>
>>> Adding num-exactors I still get the same warning. The app works correctly
>>>
>>>
>>>
>>> $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
>>> --master local[4] --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
>>> pi.numExecutor4.out
>>>
>>> 15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1
>>>
>>> 15/10/28 21:09:41 WARN spark.SparkConf:
>>>
>>> SPARK_WORKER_INSTANCES was detected (set to '1').
>>>
>>> This is deprecated in Spark 1.0+.
>>>
>>>
>>>
>>> Please instead use:
>>>
>>> - ./spark-submit with --num-executors to specify the number of executors
>>>
>>> - Or set SPARK_EXECUTOR_INSTANCES
>>>
>>> - spark.executor.instances to configure the number of instances in the
>>> spark config.
>>>
>>>
>>>
>>> I also tried variations on [ec2-user@ip-172-31-29-60 ~]$
>>> $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
>>> --master spark://172.31.29.60:7077 --num-executors 4
>>> sparkPi-1.0-SNAPSHOT.jar
>>>
>>> 15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1
>>>
>>> 15/10/28 21:14:48 WARN spark.SparkConf:
>>>
>>> SPARK_WORKER_INSTANCES was detected (set to '1').
>>>
>>> This is deprecated in Spark 1.0+.
>>>
>>>
>>>
>>> Please instead use:
>>>
>>> - ./spark-submit with --num-executors to specify the number of executors
>>>
>>> - Or set SPARK_EXECUTOR_INSTANCES
>>>
>>> - spark.executor.instances to configure the number of instances in the
>>> spark config.
>>>
>>>
>>>
>>> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to:
>>> ec2-user
>>>
>>> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to:
>>> ec2-user
>>>
>>> 15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager:
>>> authentication disabled; ui acls disabled; users with view permissions:
>>> Set(ec2-user); users with modify permissions: Set(ec2-user)
>>>
>>> 15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
>>>
>>> 15/10/28 21:14:49 INFO Remoting: Starting remoting
>>>
>>> 15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses
>>> :[akka.tcp://sparkDriver@172.31.29.60:52405]
>>>
>>> 15/10/28 21:14:50 INFO util.Utils: Successfully started service
>>> 'sparkDriver' on port 52405.
>>>
>>> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker
>>>
>>> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster
>>>
>>> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
>>> /mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d
>>>
>>> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
>>> /mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084
>>>
>>> 15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with
>>> capacity 530.0 MB
>>>
>>> 15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is
>>> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a
>>> 73-83f0-bac856bc3548
>>>
>>> 15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server
>>>
>>> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>
>>> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
>>> SocketConnector@0.0.0.0:48262
>>>
>>> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file
>>> server' on port 48262.
>>>
>>> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>>>
>>> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>
>>> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
>>> SelectChannelConnector@0.0.0.0:4040
>>>
>>> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI' on
>>> port 4040.
>>>
>>> 15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at
>>> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>>>
>>> 15/10/28 21:14:50 INFO spark.SparkContext: Added JAR
>>> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
>>> http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
>>> 1446066890783
>>>
>>> 15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name
>>> DAGScheduler for source because spark.app.id is not set.
>>>
>>> 15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to master
>>> spark://172.31.29.60:7077...
>>>
>>> 15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught
>>> exception in thread Thread[appclient-registration-retry-thread,5,main]
>>>
>>> java.util.concurrent.RejectedExecutionException: Task
>>> java.util.concurrent.FutureTask@ae9a189 rejected from
>>> java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1,
>>> active threads = 1, queued tasks = 0, completed tasks = 0]
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread
>>> PoolExecutor.java:2047)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>>>
>>> at
>>>
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369>>>
)
>>>
>>> at
>>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.
>>> java:112)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
>>> AllMasters$1.apply(AppClient.scala:96)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
>>> AllMasters$1.apply(AppClient.scala:95)
>>>
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
>>> 244)
>>>
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
>>> 244)
>>>
>>> at
>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
>>> :33)
>>>
>>> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>>>
>>> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>>>
>>> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster
>>> s(AppClient.scala:95)
>>>
>>> at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org
>>> <http://ClientEndpoint.org>
>>> $apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(App
>>> Client.scala:121)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run
>>> $1.apply$mcV$sp(AppClient.scala:132)
>>>
>>> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien
>>> t.scala:124)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 301(ScheduledThreadPoolExecutor.java:180)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:294)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
>>> 42)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
>>> 17)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> 15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called
>>>
>>> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called
>>>
>>> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61a
>>> b-4ea8-8c61-e7b9af8c481d
>>>
>>> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf
>>>
>>> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$
>>>
>>>
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
>>> org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077
>>> --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
>>> pi.numExecutor4.URILocalhost.out
>>>
>>> 15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1
>>>
>>> 15/10/28 21:22:42 WARN spark.SparkConf:
>>>
>>> SPARK_WORKER_INSTANCES was detected (set to '1').
>>>
>>> This is deprecated in Spark 1.0+.
>>>
>>>
>>>
>>> Please instead use:
>>>
>>> - ./spark-submit with --num-executors to specify the number of executors
>>>
>>> - Or set SPARK_EXECUTOR_INSTANCES
>>>
>>> - spark.executor.instances to configure the number of instances in the
>>> spark config.
>>>
>>>
>>>
>>> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to:
>>> ec2-user
>>>
>>> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to:
>>> ec2-user
>>>
>>> 15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager:
>>> authentication disabled; ui acls disabled; users with view permissions:
>>> Set(ec2-user); users with modify permissions: Set(ec2-user)
>>>
>>> 15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
>>>
>>> 15/10/28 21:22:43 INFO Remoting: Starting remoting
>>>
>>> 15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses
>>> :[akka.tcp://sparkDriver@172.31.29.60:36217]
>>>
>>> 15/10/28 21:22:43 INFO util.Utils: Successfully started service
>>> 'sparkDriver' on port 36217.
>>>
>>> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker
>>>
>>> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster
>>>
>>> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
>>> /mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815
>>>
>>> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
>>> /mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31
>>>
>>> 15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with
>>> capacity 530.0 MB
>>>
>>> 15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is
>>> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48
>>> fc-b142-8aeff83be686
>>>
>>> 15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server
>>>
>>> 15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>
>>> 15/10/28 21:22:43 INFO server.AbstractConnector: Started
>>> SocketConnector@0.0.0.0:41335
>>>
>>> 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file
>>> server' on port 41335.
>>>
>>> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>>>
>>> 15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>
>>> 15/10/28 21:22:44 INFO server.AbstractConnector: Started
>>> SelectChannelConnector@0.0.0.0:4040
>>>
>>> 15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI' on
>>> port 4040.
>>>
>>> 15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at
>>> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>>>
>>> 15/10/28 21:22:44 INFO spark.SparkContext: Added JAR
>>> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
>>> http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
>>> 1446067364171
>>>
>>> 15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name
>>> DAGScheduler for source because spark.app.id is not set.
>>>
>>> 15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to master
>>> spark://localhost:7077...
>>>
>>> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect to
>>> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
>>> akka.tcp://sparkMaster@localhost:7077
>>>
>>> 15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable remote
>>> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
>>> 5000 ms, all messages to this address will be delivered to dead letters.
>>> Reason: Connection refused: localhost/127.0.0.1:7077
>>>
>>> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect to
>>> master localhost:7077
>>>
>>> akka.actor.ActorNotFound: Actor not found for:
>>> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
>>> Path(/user/Master)]
>>>
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
>>> 5)
>>>
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
>>> 3)
>>>
>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE
>>> xecutor.scala:67)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe
>>> cutor.scala:82)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
>>> cala:59)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
>>> cala:59)
>>>
>>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>
>>> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>>
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute
>>> (Future.scala:74)
>>>
>>> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>>>
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s
>>> cala:73)
>>>
>>> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>>>
>>> at
>>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>>>
>>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>>>
>>> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>>>
>>> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>>>
>>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>>>
>>> at
>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo
>>> rRefProvider.scala:87)
>>>
>>> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>>>
>>> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>>>
>>> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>>>
>>> at
>>> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin
>>> ishTerminate(FaultHandling.scala:210)
>>>
>>> at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>>>
>>> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>>>
>>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>>>
>>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>>>
>>> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>>>
>>> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>>>
>>> at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis
>>> patcher.scala:393)
>>>
>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>
>>> at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1
>>> 339)
>>>
>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>
>>> at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java
>>> :107)
>>>
>>> 15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to master
>>> spark://localhost:7077...
>>>
>>> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect to
>>> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
>>> akka.tcp://sparkMaster@localhost:7077
>>>
>>> 15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable remote
>>> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
>>> 5000 ms, all messages to this address will be delivered to dead letters.
>>> Reason: Connection refused: localhost/127.0.0.1:7077
>>>
>>> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect to
>>> master localhost:7077
>>>
>>> akka.actor.ActorNotFound: Actor not found for:
>>> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
>>> Path(/user/Master)]
>>>
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
>>> 5)
>>>
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6
>>> 3)
>>>
>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE
>>> xecutor.scala:67)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe
>>> cutor.scala:82)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
>>> cala:59)
>>>
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s
>>> cala:59)
>>>
>>> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>
>>> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>>
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute
>>> (Future.scala:74)
>>>
>>> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>>>
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s
>>> cala:73)
>>>
>>> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>>>
>>> at
>>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>>>
>>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>>>
>>> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>>>
>>> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>>>
>>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>>>
>>> at
>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo
>>> rRefProvider.scala:87)
>>>
>>> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>>>
>>> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>>>
>>> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>>>
>>> at
>>> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin
>>> ishTerminate(FaultHandling.scala:210)
>>>
>>> at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>>>
>>> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>>>
>>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>>>
>>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>>>
>>> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>>>
>>> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>>>
>>> at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis
>>> patcher.scala:393)
>>>
>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>
>>> at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1
>>> 339)
>>>
>>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>
>>> at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java
>>> :107)
>>>
>>> 15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to master
>>> spark://localhost:7077...
>>>
>>> 15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught
>>> exception in thread Thread[appclient-registration-retry-thread,5,main]
>>>
>>> java.util.concurrent.RejectedExecutionException: Task
>>> java.util.concurrent.FutureTask@7a9a7ff1 rejected from
>>> java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1,
>>> active threads = 1, queued tasks = 0, completed tasks = 2]
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread
>>> PoolExecutor.java:2047)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>>>
>>> at
>>>
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369>>>
)
>>>
>>> at
>>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.
>>> java:112)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
>>> AllMasters$1.apply(AppClient.scala:96)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister
>>> AllMasters$1.apply(AppClient.scala:95)
>>>
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
>>> 244)
>>>
>>> at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
>>> 244)
>>>
>>> at
>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
>>> :33)
>>>
>>> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>>>
>>> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>>>
>>> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster
>>> s(AppClient.scala:95)
>>>
>>> at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org
>>> <http://ClientEndpoint.org>
>>> $apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(App
>>> Client.scala:121)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run
>>> $1.apply$mcV$sp(AppClient.scala:132)
>>>
>>> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>>>
>>> at
>>> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien
>>> t.scala:124)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 301(ScheduledThreadPoolExecutor.java:180)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:294)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
>>> 42)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
>>> 17)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> 15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect to
>>> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
>>> akka.tcp://sparkMaster@localhost:7077
>>>
>>> 15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable remote
>>> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for
>>> 5000 ms, all messages to this address will be delivered to dead letters.
>>> Reason: Connection refused: localhost/127.0.0.1:7077
>>>
>>> 15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called
>>>
>>> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called
>>>
>>> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48
>>> fc-b142-8aeff83be686
>>>
>>> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8
>>>
>>> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
>>> /mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$
>>>
>>>
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
>>> org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077
>>> --num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
>>> pi.numExecutor4.URI172.31.29.60.cluster.out
>>>
>>> Running Spark using the REST application submission protocol.
>>>
>>> 15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to
>>> launch an application in spark://172.31.29.60:7077.
>>>
>>> 15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to
>>> server spark://172.31.29.60:7077.
>>>
>>> Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server.
>>> Falling back to legacy submission gateway instead.
>>>
>>> ^C
>>>
>>> [ec2-user@ip-172-31-29-60 ~]$
Re: newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
I forgot to mention. I do not have a preference for the cluster manager. I
choose the spark-1.5.1-bin-hadoop2.6 distribution because I want to use
hdfs. I assumed this distribution would use yarn.
Thanks
Andy
From: Andrew Davidson <An...@SantaCruzIntegration.com>
Date: Wednesday, October 28, 2015 at 2:37 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: newbie trouble submitting java app to AWS cluster I created using
spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution
> Hi
>
>
>
> I just created new cluster using the spark-c2 script from the
> spark-1.5.1-bin-hadoop2.6 distribution. The master and slaves seem to be up
> and running. I am having a heck of time figuring out how to submit apps. As a
> test I compile the sample JavaSparkPi example. I have copied my jar file to
> the master and want to run the application in cluster mode. My real app will
> take a long time to complete. I do not want to wait around.
>
>
>
> Any idea what the issue is?
>
>
>
> Kind regards
>
>
>
> Andy
>
>
>
>
>
> http://spark.apache.org/docs/latest/submitting-applications.html
>
>
> The following command works fine on my Mac, how ever when I run it on my
> master I get the following warning. The app works correctly
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master local[4]
> sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out
>
> 15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:07:11 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the spark
> config.
>
>
>
> Adding num-exactors I still get the same warning. The app works correctly
>
>
>
> $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
> --master local[4] --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
> pi.numExecutor4.out
>
> 15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:09:41 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the spark
> config.
>
>
>
> I also tried variations on [ec2-user@ip-172-31-29-60 ~]$
> $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi
> --master spark://172.31.29.60:7077 --num-executors 4 sparkPi-1.0-SNAPSHOT.jar
>
> 15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:14:48 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the spark
> config.
>
>
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to: ec2-user
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to:
> ec2-user
>
> 15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(ec2-user); users
> with modify permissions: Set(ec2-user)
>
> 15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/10/28 21:14:49 INFO Remoting: Starting remoting
>
> 15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@172.31.29.60:52405]
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'sparkDriver'
> on port 52405.
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster
>
> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
> /mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d
>
> 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at
> /mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084
>
> 15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with capacity
> 530.0 MB
>
> 15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a73
> -83f0-bac856bc3548
>
> 15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server
>
> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:48262
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 48262.
>
> 15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>
> 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:14:50 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
>
> 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at
> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>
> 15/10/28 21:14:50 INFO spark.SparkContext: Added JAR
> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
> http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
> 1446066890783
>
> 15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name DAGScheduler
> for source because spark.app.id is not set.
>
> 15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to master
> spark://172.31.29.60:7077...
>
> 15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception
> in thread Thread[appclient-registration-retry-thread,5,main]
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@ae9a189 rejected from
> java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1, active
> threads = 1, queued tasks = 0, completed tasks = 0]
>
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPo
> olExecutor.java:2047)
>
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.ja
> va:112)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl
> lMasters$1.apply(AppClient.scala:96)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl
> lMasters$1.apply(AppClient.scala:95)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24
> 4)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24
> 4)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:3
> 3)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(
> AppClient.scala:95)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deplo
> y$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1
> .apply$mcV$sp(AppClient.scala:132)
>
> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.
> scala:124)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$30
> 1(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sched
> uledThreadPoolExecutor.java:294)
>
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142>
)
>
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617>
)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61ab-
> 4ea8-8c61-e7b9af8c481d
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf
>
> 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory
> /mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077
> --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
> pi.numExecutor4.URILocalhost.out
>
> 15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1
>
> 15/10/28 21:22:42 WARN spark.SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '1').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the spark
> config.
>
>
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to: ec2-user
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to:
> ec2-user
>
> 15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(ec2-user); users
> with modify permissions: Set(ec2-user)
>
> 15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/10/28 21:22:43 INFO Remoting: Starting remoting
>
> 15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@172.31.29.60:36217]
>
> 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'sparkDriver'
> on port 36217.
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster
>
> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
> /mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815
>
> 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at
> /mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31
>
> 15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with capacity
> 530.0 MB
>
> 15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc
> -b142-8aeff83be686
>
> 15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server
>
> 15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:22:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:41335
>
> 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 41335.
>
> 15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>
> 15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/10/28 21:22:44 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
>
> 15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at
> http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040
>
> 15/10/28 21:22:44 INFO spark.SparkContext: Added JAR
> file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at
> http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp
> 1446067364171
>
> 15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name DAGScheduler
> for source because spark.app.id is not set.
>
> 15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to master
> spark://localhost:7077...
>
> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect to
> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable remote
> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000
> ms, all messages to this address will be delivered to dead letters. Reason:
> Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect to
> master localhost:7077
>
> akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
> Path(/user/Master)]
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExe
> cutor.scala:67)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecu
> tor.scala:82)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca
> la:59)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca
> la:59)
>
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(F
> uture.scala:74)
>
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.sca
> la:73)
>
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorR
> efProvider.scala:87)
>
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finis
> hTerminate(FaultHandling.scala:210)
>
> at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispa
> tcher.scala:393)
>
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133
> 9)
>
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:1
> 07)
>
> 15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to master
> spark://localhost:7077...
>
> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect to
> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable remote
> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000
> ms, all messages to this address will be delivered to dead letters. Reason:
> Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect to
> master localhost:7077
>
> akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/),
> Path(/user/Master)]
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExe
> cutor.scala:67)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecu
> tor.scala:82)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca
> la:59)
>
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca
> la:59)
>
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(F
> uture.scala:74)
>
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.sca
> la:73)
>
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorR
> efProvider.scala:87)
>
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finis
> hTerminate(FaultHandling.scala:210)
>
> at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispa
> tcher.scala:393)
>
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133
> 9)
>
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:1
> 07)
>
> 15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to master
> spark://localhost:7077...
>
> 15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception
> in thread Thread[appclient-registration-retry-thread,5,main]
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@7a9a7ff1 rejected from
> java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1,
> active threads = 1, queued tasks = 0, completed tasks = 2]
>
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPo
> olExecutor.java:2047)
>
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.ja
> va:112)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl
> lMasters$1.apply(AppClient.scala:96)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl
> lMasters$1.apply(AppClient.scala:95)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24
> 4)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24
> 4)
>
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:3
> 3)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(
> AppClient.scala:95)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deplo
> y$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1
> .apply$mcV$sp(AppClient.scala:132)
>
> at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
>
> at
> org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.
> scala:124)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$30
> 1(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sched
> uledThreadPoolExecutor.java:294)
>
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142>
)
>
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617>
)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect to
> localhost:7077: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> 15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable remote
> address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000
> ms, all messages to this address will be delivered to dead letters. Reason:
> Connection refused: localhost/127.0.0.1:7077
>
> 15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc
> -b142-8aeff83be686
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8
>
> 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory
> /mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e
>
> [ec2-user@ip-172-31-29-60 ~]$
>
>
>
> [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class
> org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077
> --num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee
> pi.numExecutor4.URI172.31.29.60.cluster.out
>
> Running Spark using the REST application submission protocol.
>
> 15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to
> launch an application in spark://172.31.29.60:7077.
>
> 15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to server
> spark://172.31.29.60:7077.
>
> Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server.
> Falling back to legacy submission gateway instead.
>
> ^C
>
> [ec2-user@ip-172-31-29-60 ~]$