You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Paul Magid <Pa...@toyota.com> on 2014/09/24 00:10:02 UTC
Worker Random Port
I am trying to get an edge server up and running connecting to our Spark 1.1 cluster. The edge server is in a different DMZ than the rest of the cluster and we have to specifically open firewall ports between the edge server and the rest of the cluster. I can log on to any node in the cluster (other than the edge) and submit code through spark-shell just fine. I have port 7077 from the edge to the master open (verified), and I have port 7078 open from the edge to all the workers (also verified). I have tried setting the worker port to not be dynamic by using SPARK_WORKER_PORT in the spark-env.sh but it does not seem to stop the dynamic port behavior. I have included the startup output when running spark-shell from the edge server in a different dmz and then from a node in the cluster. Any help greatly appreciated.
Paul Magid
Toyota Motor Sales IS Enterprise Architecture (EA)
Architect I R&D
Ph: 310-468-9091 (X69091)
PCN 1C2970, Mail Drop PN12
Running spark-shell from the edge server
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14/09/23 14:20:38 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:20:38 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:20:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, )
14/09/23 14:20:38 INFO HttpServer: Starting HTTP Server
14/09/23 14:20:39 INFO Utils: Successfully started service 'HTTP class server' on port 22788.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
14/09/23 14:20:42 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:20:42 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:20:42 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, )
14/09/23 14:20:43 INFO Slf4jLogger: Slf4jLogger started
14/09/23 14:20:43 INFO Remoting: Starting remoting
14/09/23 14:20:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@votlbdcd09.tms.toyota.com:32356]
14/09/23 14:20:43 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@votlbdcd09.tms.toyota.com:32356]
14/09/23 14:20:43 INFO Utils: Successfully started service 'sparkDriver' on port 32356.
14/09/23 14:20:43 INFO SparkEnv: Registering MapOutputTracker
14/09/23 14:20:43 INFO SparkEnv: Registering BlockManagerMaster
14/09/23 14:20:43 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140923142043-4454
14/09/23 14:20:43 INFO Utils: Successfully started service 'Connection manager for block manager' on port 48469.
14/09/23 14:20:43 INFO ConnectionManager: Bound socket to port 48469 with id = ConnectionManagerId(votlbdcd09.tms.toyota.com,48469)
14/09/23 14:20:43 INFO MemoryStore: MemoryStore started with capacity 265.9 MB
14/09/23 14:20:43 INFO BlockManagerMaster: Trying to register BlockManager
14/09/23 14:20:43 INFO BlockManagerMasterActor: Registering block manager votlbdcd09.tms.toyota.com:48469 with 265.9 MB RAM
14/09/23 14:20:43 INFO BlockManagerMaster: Registered BlockManager
14/09/23 14:20:43 INFO HttpFileServer: HTTP File server directory is /tmp/spark-888c359a-5a2a-4aaa-80e3-8009cdfa25c8
14/09/23 14:20:43 INFO HttpServer: Starting HTTP Server
14/09/23 14:20:43 INFO Utils: Successfully started service 'HTTP file server' on port 12470.
14/09/23 14:20:43 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/09/23 14:20:43 INFO SparkUI: Started SparkUI at http://votlbdcd09.tms.toyota.com:4040
14/09/23 14:20:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/09/23 14:20:44 INFO EventLoggingListener: Logging events to file:/user/spark/applicationHistory//spark-shell-1411507243973
14/09/23 14:20:44 INFO AppClient$ClientActor: Connecting to master spark://votlbdcd01.tms.toyota.com:7077...
14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
14/09/23 14:20:44 INFO SparkILoop: Created spark context..
14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140923142044-0006
Spark context available as sc.
scala> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/0 on worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/0 on hostPort votlbdcd03.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/1 on worker-20140923141811-votlbdcd08.tms.toyota.com-7078 (votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/1 on hostPort votlbdcd08.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/2 on worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/2 on hostPort votlbdcd02.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/3 on worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/3 on hostPort votlbdcd05.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/4 on worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/4 on hostPort votlbdcd06.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/5 on worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/5 on hostPort votlbdcd07.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/6 on worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/6 on hostPort votlbdcd04.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/3 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/6 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/0 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/2 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/4 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/5 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/1 is now RUNNING
scala> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/3 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/3 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/7 on worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/7 on hostPort votlbdcd05.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/7 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/6 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/6 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/8 on worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/8 on hostPort votlbdcd04.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/8 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/0 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/0 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/9 on worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/9 on hostPort votlbdcd03.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/9 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/2 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/2 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/10 on worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/10 on hostPort votlbdcd02.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/10 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/1 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/1 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/11 on worker-20140923141811-votlbdcd08.tms.toyota.com-7078 (votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/11 on hostPort votlbdcd08.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/11 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/4 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/4 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/12 on worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/12 on hostPort votlbdcd06.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/12 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/5 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/5 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/13 on worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/13 on hostPort votlbdcd07.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/13 is now RUNNING
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/7 is now EXITED (Command exited with code 1)
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/7 removed: Command exited with code 1
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/14 on worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/14 on hostPort votlbdcd05.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/14 is now RUNNING
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/8 is now EXITED (Command exited with code 1)
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/8 removed: Command exited with code 1
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added: app-20140923142044-0006/15 on worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923142044-0006/15 on hostPort votlbdcd04.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/15 is now RUNNING
14/09/23 14:22:31 INFO AppClient$ClientActor: Executor updated: app-20140923142044-0006/9 is now EXITED (Command exited with code 1)
14/09/23 14:22:31 INFO SparkDeploySchedulerBackend: Executor app-20140923142044-0006/9 removed: Command exited with code 1
14/09/23 14:22:31 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
14/09/23 14:22:31 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: Master removed our application: FAILED
Running spark-shell in the cluster
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14/09/23 14:47:26 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:47:26 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:47:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, )
14/09/23 14:47:26 INFO HttpServer: Starting HTTP Server
14/09/23 14:47:26 INFO Utils: Successfully started service 'HTTP class server' on port 23968.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
14/09/23 14:47:30 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:47:30 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:47:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, ); users with modify permissions: Set(root, )
14/09/23 14:47:31 INFO Slf4jLogger: Slf4jLogger started
14/09/23 14:47:31 INFO Remoting: Starting remoting
14/09/23 14:47:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@votlbdcd01.tms.toyota.com:44430]
14/09/23 14:47:31 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@votlbdcd01.tms.toyota.com:44430]
14/09/23 14:47:31 INFO Utils: Successfully started service 'sparkDriver' on port 44430.
14/09/23 14:47:31 INFO SparkEnv: Registering MapOutputTracker
14/09/23 14:47:31 INFO SparkEnv: Registering BlockManagerMaster
14/09/23 14:47:31 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140923144731-48c5
14/09/23 14:47:31 INFO Utils: Successfully started service 'Connection manager for block manager' on port 30579.
14/09/23 14:47:31 INFO ConnectionManager: Bound socket to port 30579 with id = ConnectionManagerId(votlbdcd01.tms.toyota.com,30579)
14/09/23 14:47:31 INFO MemoryStore: MemoryStore started with capacity 265.9 MB
14/09/23 14:47:31 INFO BlockManagerMaster: Trying to register BlockManager
14/09/23 14:47:31 INFO BlockManagerMasterActor: Registering block manager votlbdcd01.tms.toyota.com:30579 with 265.9 MB RAM
14/09/23 14:47:31 INFO BlockManagerMaster: Registered BlockManager
14/09/23 14:47:31 INFO HttpFileServer: HTTP File server directory is /tmp/spark-1a38cbd9-107b-40ce-bb28-f1574c9fbcda
14/09/23 14:47:31 INFO HttpServer: Starting HTTP Server
14/09/23 14:47:31 INFO Utils: Successfully started service 'HTTP file server' on port 31028.
14/09/23 14:47:31 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/09/23 14:47:31 INFO SparkUI: Started SparkUI at http://votlbdcd01.tms.toyota.com:4040
14/09/23 14:47:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/09/23 14:47:32 INFO EventLoggingListener: Logging events to file:/user/spark/applicationHistory//spark-shell-1411508851947
14/09/23 14:47:32 INFO AppClient$ClientActor: Connecting to master spark://votlbdcd01.tms.toyota.com:7077...
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
14/09/23 14:47:32 INFO SparkILoop: Created spark context..
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140923144732-0009
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/0 on worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/0 on hostPort votlbdcd03.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/1 on worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/1 on hostPort votlbdcd02.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/2 on worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/2 on hostPort votlbdcd05.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/3 on worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/3 on hostPort votlbdcd06.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/4 on worker-20140923144121-votlbdcd08.tms.toyota.com-7078 (votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/4 on hostPort votlbdcd08.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/5 on worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/5 on hostPort votlbdcd07.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: app-20140923144732-0009/6 on worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140923144732-0009/6 on hostPort votlbdcd04.tms.toyota.com:7078 with 8 cores, 25.0 GB RAM
Spark context available as sc.
scala> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/2 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/6 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/0 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/1 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/5 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/3 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: app-20140923144732-0009/4 is now RUNNING
14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd05.tms.toyota.com:34924/user/Executor#-195649138] with ID 2
14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd04.tms.toyota.com:22963/user/Executor#-2105826271] with ID 6
14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager votlbdcd05.tms.toyota.com:11677 with 12.9 GB RAM
14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager votlbdcd04.tms.toyota.com:14746 with 12.9 GB RAM
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd03.tms.toyota.com:19463/user/Executor#964509447] with ID 0
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd02.tms.toyota.com:47003/user/Executor#461654156] with ID 1
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd07.tms.toyota.com:27089/user/Executor#-397798004] with ID 5
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd08.tms.toyota.com:22617/user/Executor#-449311685] with ID 4
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@votlbdcd06.tms.toyota.com:43167/user/Executor#-1493582134] with ID 3
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager votlbdcd03.tms.toyota.com:13456 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager votlbdcd02.tms.toyota.com:33808 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager votlbdcd07.tms.toyota.com:34519 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager votlbdcd08.tms.toyota.com:14214 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager votlbdcd06.tms.toyota.com:45670 with 12.9 GB RAM
Re: Worker Random Port
Posted by Andrew Ash <an...@andrewash.com>.
Hi Paul,
There are several ports you need to configure in order to run in a tight
network environment. It sounds like you the DMZ that contains the spark
cluster is wide open internally, but you have to poke holes between that
and the driver.
You should take a look at the port configuration options at the bottom of
http://spark.apache.org/docs/latest/security.html for the ones you'll need
to set to make this configuration work. The ones that are to or from the
driver are the ones you should configure.
The other thing to consider is that some connections are from the workers
back to the driver (not all connections are initiated by the driver). Does
your configuration allow outbound connections from the DMZ -> edge server?
Good luck!
Andrew
On Tue, Sep 23, 2014 at 3:10 PM, Paul Magid <Pa...@toyota.com> wrote:
> I am trying to get an edge server up and running connecting to our Spark
> 1.1 cluster. The edge server is in a different DMZ than the rest of the
> cluster and we have to specifically open firewall ports between the edge
> server and the rest of the cluster. I can log on to any node in the
> cluster (other than the edge) and submit code through spark-shell just
> fine. I have port 7077 from the edge to the master open (verified), and I
> have port 7078 open from the edge to all the workers (also verified). I
> have tried setting the worker port to not be dynamic by using
> SPARK_WORKER_PORT in the spark-env.sh but it does not seem to stop the
> dynamic port behavior. I have included the startup output when running
> spark-shell from the edge server in a different dmz and then from a node in
> the cluster. Any help greatly appreciated.
>
>
>
> Paul Magid
>
> Toyota Motor Sales IS Enterprise Architecture (EA)
>
> Architect I R&D
>
> Ph: 310-468-9091 (X69091)
>
> PCN 1C2970, Mail Drop PN12
>
>
>
>
>
> Running spark-shell from the edge server
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> 14/09/23 14:20:38 INFO SecurityManager: Changing view acls to: root,
>
> 14/09/23 14:20:38 INFO SecurityManager: Changing modify acls to: root,
>
> 14/09/23 14:20:38 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root, ); users
> with modify permissions: Set(root, )
>
> 14/09/23 14:20:38 INFO HttpServer: Starting HTTP Server
>
> 14/09/23 14:20:39 INFO Utils: Successfully started service 'HTTP class
> server' on port 22788.
>
> Welcome to
>
> ____ __
>
> / __/__ ___ _____/ /__
>
> _\ \/ _ \/ _ `/ __/ '_/
>
> /___/ .__/\_,_/_/ /_/\_\ version 1.1.0
>
> /_/
>
>
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.7.0_55)
>
> Type in expressions to have them evaluated.
>
> Type :help for more information.
>
> 14/09/23 14:20:42 INFO SecurityManager: Changing view acls to: root,
>
> 14/09/23 14:20:42 INFO SecurityManager: Changing modify acls to: root,
>
> 14/09/23 14:20:42 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root, ); users
> with modify permissions: Set(root, )
>
> 14/09/23 14:20:43 INFO Slf4jLogger: Slf4jLogger started
>
> 14/09/23 14:20:43 INFO Remoting: Starting remoting
>
> 14/09/23 14:20:43 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@votlbdcd09.tms.toyota.com:32356]
>
> 14/09/23 14:20:43 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@votlbdcd09.tms.toyota.com:32356]
>
> 14/09/23 14:20:43 INFO Utils: Successfully started service 'sparkDriver'
> on port 32356.
>
> 14/09/23 14:20:43 INFO SparkEnv: Registering MapOutputTracker
>
> 14/09/23 14:20:43 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/09/23 14:20:43 INFO DiskBlockManager: Created local directory at
> /tmp/spark-local-20140923142043-4454
>
> 14/09/23 14:20:43 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 48469.
>
> 14/09/23 14:20:43 INFO ConnectionManager: Bound socket to port 48469 with
> id = ConnectionManagerId(votlbdcd09.tms.toyota.com,48469)
>
> 14/09/23 14:20:43 INFO MemoryStore: MemoryStore started with capacity
> 265.9 MB
>
> 14/09/23 14:20:43 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/09/23 14:20:43 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd09.tms.toyota.com:48469 with 265.9 MB RAM
>
> 14/09/23 14:20:43 INFO BlockManagerMaster: Registered BlockManager
>
> 14/09/23 14:20:43 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-888c359a-5a2a-4aaa-80e3-8009cdfa25c8
>
> 14/09/23 14:20:43 INFO HttpServer: Starting HTTP Server
>
> 14/09/23 14:20:43 INFO Utils: Successfully started service 'HTTP file
> server' on port 12470.
>
> 14/09/23 14:20:43 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/09/23 14:20:43 INFO SparkUI: Started SparkUI at
> http://votlbdcd09.tms.toyota.com:4040
>
> 14/09/23 14:20:43 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/09/23 14:20:44 INFO EventLoggingListener: Logging events to
> file:/user/spark/applicationHistory//spark-shell-1411507243973
>
> 14/09/23 14:20:44 INFO AppClient$ClientActor: Connecting to master
> spark://votlbdcd01.tms.toyota.com:7077...
>
> 14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
>
> 14/09/23 14:20:44 INFO SparkILoop: Created spark context..
>
> 14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: Connected to Spark
> cluster with app ID app-20140923142044-0006
>
> Spark context available as sc.
>
>
>
> scala> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/0 on
> worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (
> votlbdcd03.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/0 on hostPort votlbdcd03.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/1 on
> worker-20140923141811-votlbdcd08.tms.toyota.com-7078 (
> votlbdcd08.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/1 on hostPort votlbdcd08.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/2 on
> worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (
> votlbdcd02.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/2 on hostPort votlbdcd02.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/3 on
> worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (
> votlbdcd05.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/3 on hostPort votlbdcd05.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/4 on
> worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (
> votlbdcd06.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/4 on hostPort votlbdcd06.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/5 on
> worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (
> votlbdcd07.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/5 on hostPort votlbdcd07.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/6 on
> worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (
> votlbdcd04.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/6 on hostPort votlbdcd04.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/3 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/6 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/0 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/2 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/4 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/5 is now RUNNING
>
> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/1 is now RUNNING
>
>
>
>
>
> scala> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/3 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/3 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/7 on
> worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (
> votlbdcd05.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/7 on hostPort votlbdcd05.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/7 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/6 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/6 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/8 on
> worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (
> votlbdcd04.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/8 on hostPort votlbdcd04.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/8 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/0 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/0 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/9 on
> worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (
> votlbdcd03.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/9 on hostPort votlbdcd03.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/9 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/2 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/2 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/10 on
> worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (
> votlbdcd02.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/10 on hostPort votlbdcd02.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/10 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/1 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/1 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/11 on
> worker-20140923141811-votlbdcd08.tms.toyota.com-7078 (
> votlbdcd08.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/11 on hostPort votlbdcd08.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/11 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/4 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/4 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/12 on
> worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (
> votlbdcd06.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/12 on hostPort votlbdcd06.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/12 is now RUNNING
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/5 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/5 removed: Command exited with code 1
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/13 on
> worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (
> votlbdcd07.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/13 on hostPort votlbdcd07.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/13 is now RUNNING
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/7 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/7 removed: Command exited with code 1
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/14 on
> worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (
> votlbdcd05.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/14 on hostPort votlbdcd05.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/14 is now RUNNING
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/8 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/8 removed: Command exited with code 1
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added:
> app-20140923142044-0006/15 on
> worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (
> votlbdcd04.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923142044-0006/15 on hostPort votlbdcd04.tms.toyota.com:7078
> with 8 cores, 25.0 GB RAM
>
> 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/15 is now RUNNING
>
> 14/09/23 14:22:31 INFO AppClient$ClientActor: Executor updated:
> app-20140923142044-0006/9 is now EXITED (Command exited with code 1)
>
> 14/09/23 14:22:31 INFO SparkDeploySchedulerBackend: Executor
> app-20140923142044-0006/9 removed: Command exited with code 1
>
> 14/09/23 14:22:31 ERROR SparkDeploySchedulerBackend: Application has been
> killed. Reason: Master removed our application: FAILED
>
> 14/09/23 14:22:31 ERROR TaskSchedulerImpl: Exiting due to error from
> cluster scheduler: Master removed our application: FAILED
>
>
>
>
>
>
>
>
>
>
>
> Running spark-shell in the cluster
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
> 14/09/23 14:47:26 INFO SecurityManager: Changing view acls to: root,
>
> 14/09/23 14:47:26 INFO SecurityManager: Changing modify acls to: root,
>
> 14/09/23 14:47:26 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root, ); users
> with modify permissions: Set(root, )
>
> 14/09/23 14:47:26 INFO HttpServer: Starting HTTP Server
>
> 14/09/23 14:47:26 INFO Utils: Successfully started service 'HTTP class
> server' on port 23968.
>
> Welcome to
>
> ____ __
>
> / __/__ ___ _____/ /__
>
> _\ \/ _ \/ _ `/ __/ '_/
>
> /___/ .__/\_,_/_/ /_/\_\ version 1.1.0
>
> /_/
>
>
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.7.0_55)
>
> Type in expressions to have them evaluated.
>
> Type :help for more information.
>
> 14/09/23 14:47:30 INFO SecurityManager: Changing view acls to: root,
>
> 14/09/23 14:47:30 INFO SecurityManager: Changing modify acls to: root,
>
> 14/09/23 14:47:30 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root, ); users
> with modify permissions: Set(root, )
>
> 14/09/23 14:47:31 INFO Slf4jLogger: Slf4jLogger started
>
> 14/09/23 14:47:31 INFO Remoting: Starting remoting
>
> 14/09/23 14:47:31 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@votlbdcd01.tms.toyota.com:44430]
>
> 14/09/23 14:47:31 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@votlbdcd01.tms.toyota.com:44430]
>
> 14/09/23 14:47:31 INFO Utils: Successfully started service 'sparkDriver'
> on port 44430.
>
> 14/09/23 14:47:31 INFO SparkEnv: Registering MapOutputTracker
>
> 14/09/23 14:47:31 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/09/23 14:47:31 INFO DiskBlockManager: Created local directory at
> /tmp/spark-local-20140923144731-48c5
>
> 14/09/23 14:47:31 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 30579.
>
> 14/09/23 14:47:31 INFO ConnectionManager: Bound socket to port 30579 with
> id = ConnectionManagerId(votlbdcd01.tms.toyota.com,30579)
>
> 14/09/23 14:47:31 INFO MemoryStore: MemoryStore started with capacity
> 265.9 MB
>
> 14/09/23 14:47:31 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/09/23 14:47:31 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd01.tms.toyota.com:30579 with 265.9 MB RAM
>
> 14/09/23 14:47:31 INFO BlockManagerMaster: Registered BlockManager
>
> 14/09/23 14:47:31 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-1a38cbd9-107b-40ce-bb28-f1574c9fbcda
>
> 14/09/23 14:47:31 INFO HttpServer: Starting HTTP Server
>
> 14/09/23 14:47:31 INFO Utils: Successfully started service 'HTTP file
> server' on port 31028.
>
> 14/09/23 14:47:31 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/09/23 14:47:31 INFO SparkUI: Started SparkUI at
> http://votlbdcd01.tms.toyota.com:4040
>
> 14/09/23 14:47:31 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/09/23 14:47:32 INFO EventLoggingListener: Logging events to
> file:/user/spark/applicationHistory//spark-shell-1411508851947
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Connecting to master
> spark://votlbdcd01.tms.toyota.com:7077...
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
>
> 14/09/23 14:47:32 INFO SparkILoop: Created spark context..
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Connected to Spark
> cluster with app ID app-20140923144732-0009
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/0 on
> worker-20140923084845-votlbdcd03.tms.toyota.com-7078 (
> votlbdcd03.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/0 on hostPort votlbdcd03.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/1 on
> worker-20140923084651-votlbdcd02.tms.toyota.com-7078 (
> votlbdcd02.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/1 on hostPort votlbdcd02.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/2 on
> worker-20140923091151-votlbdcd05.tms.toyota.com-7078 (
> votlbdcd05.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/2 on hostPort votlbdcd05.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/3 on
> worker-20140923103250-votlbdcd06.tms.toyota.com-7078 (
> votlbdcd06.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/3 on hostPort votlbdcd06.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/4 on
> worker-20140923144121-votlbdcd08.tms.toyota.com-7078 (
> votlbdcd08.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/4 on hostPort votlbdcd08.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/5 on
> worker-20140923104019-votlbdcd07.tms.toyota.com-7078 (
> votlbdcd07.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/5 on hostPort votlbdcd07.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
> app-20140923144732-0009/6 on
> worker-20140923085411-votlbdcd04.tms.toyota.com-7078 (
> votlbdcd04.tms.toyota.com:7078) with 8 cores
>
> 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20140923144732-0009/6 on hostPort votlbdcd04.tms.toyota.com:7078 with
> 8 cores, 25.0 GB RAM
>
> Spark context available as sc.
>
>
>
> scala> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/2 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/6 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/0 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/1 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/5 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/3 is now RUNNING
>
> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
> app-20140923144732-0009/4 is now RUNNING
>
> 14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd05.tms.toyota.com:34924/user/Executor#-195649138]
> with ID 2
>
> 14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd04.tms.toyota.com:22963/user/Executor#-2105826271]
> with ID 6
>
> 14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd05.tms.toyota.com:11677 with 12.9 GB RAM
>
> 14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd04.tms.toyota.com:14746 with 12.9 GB RAM
>
> 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd03.tms.toyota.com:19463/user/Executor#964509447]
> with ID 0
>
> 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd02.tms.toyota.com:47003/user/Executor#461654156]
> with ID 1
>
> 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd07.tms.toyota.com:27089/user/Executor#-397798004]
> with ID 5
>
> 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd08.tms.toyota.com:22617/user/Executor#-449311685]
> with ID 4
>
> 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
> Actor[akka.tcp://
> sparkExecutor@votlbdcd06.tms.toyota.com:43167/user/Executor#-1493582134]
> with ID 3
>
> 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd03.tms.toyota.com:13456 with 12.9 GB RAM
>
> 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd02.tms.toyota.com:33808 with 12.9 GB RAM
>
> 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd07.tms.toyota.com:34519 with 12.9 GB RAM
>
> 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd08.tms.toyota.com:14214 with 12.9 GB RAM
>
> 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
> votlbdcd06.tms.toyota.com:45670 with 12.9 GB RAM
>