You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Matthew Goodman (JIRA)" <ji...@apache.org> on 2015/05/28 03:03:17 UTC
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not
py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562121#comment-14562121 ]
Matthew Goodman commented on SPARK-7909:
----------------------------------------
I have almost everything working, but I am getting hung-up on getting the pyspark binary to launch things correctly. When logging into ec2:
{code:title=PySpark Output Loop|borderStyle=solid}
root@ip-172-31-6-84 ~]$ ./spark/bin/pyspark
Python 3.4.3 |Continuum Analytics, Inc.| (default, Mar 6 2015, 12:03:53)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/05/28 00:58:28 INFO SparkContext: Running Spark version 1.4.0-SNAPSHOT
15/05/28 00:58:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/05/28 00:58:28 INFO SecurityManager: Changing view acls to: root
15/05/28 00:58:28 INFO SecurityManager: Changing modify acls to: root
15/05/28 00:58:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/05/28 00:58:29 INFO Slf4jLogger: Slf4jLogger started
15/05/28 00:58:29 INFO Remoting: Starting remoting
15/05/28 00:58:29 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@172.31.6.84:59125]
15/05/28 00:58:30 INFO Utils: Successfully started service 'sparkDriver' on port 59125.
15/05/28 00:58:30 INFO SparkEnv: Registering MapOutputTracker
15/05/28 00:58:30 INFO SparkEnv: Registering BlockManagerMaster
15/05/28 00:58:30 INFO DiskBlockManager: Created local directory at /mnt/spark/spark-985d5a6c-150e-40ad-875f-351733a40276/blockmgr-e36c9174-ff48-42e1-bbd0-c2b0649ab751
15/05/28 00:58:30 INFO DiskBlockManager: Created local directory at /mnt2/spark/spark-fb2a7e42-2998-4ad2-be5f-d25472727d57/blockmgr-660e39f5-3561-4bec-a042-7cab1ea8cf54
15/05/28 00:58:30 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/05/28 00:58:30 INFO HttpFileServer: HTTP File server directory is /mnt/spark/spark-985d5a6c-150e-40ad-875f-351733a40276/httpd-40a7ad26-25d9-482b-bc7f-68d9f126b32d
15/05/28 00:58:30 INFO HttpServer: Starting HTTP Server
15/05/28 00:58:30 INFO Server: jetty-8.y.z-SNAPSHOT
15/05/28 00:58:30 INFO AbstractConnector: Started SocketConnector@0.0.0.0:46564
15/05/28 00:58:30 INFO Utils: Successfully started service 'HTTP file server' on port 46564.
15/05/28 00:58:30 INFO SparkEnv: Registering OutputCommitCoordinator
15/05/28 00:58:30 INFO Server: jetty-8.y.z-SNAPSHOT
15/05/28 00:58:30 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/05/28 00:58:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/05/28 00:58:30 INFO SparkUI: Started SparkUI at http://ec2-52-24-65-198.us-west-2.compute.amazonaws.com:4040
15/05/28 00:58:30 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@ec2-52-24-65-198.us-west-2.compute.amazonaws.com:7077/user/Master...
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150528005831-0005
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/0 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/0 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/0 is now LOADING
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/0 is now RUNNING
15/05/28 00:58:31 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34430.
15/05/28 00:58:31 INFO NettyBlockTransferService: Server created on 34430
15/05/28 00:58:31 INFO BlockManagerMaster: Trying to register BlockManager
15/05/28 00:58:31 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.6.84:34430 with 265.4 MB RAM, BlockManagerId(driver, 172.31.6.84, 34430)
15/05/28 00:58:31 INFO BlockManagerMaster: Registered BlockManager
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.4.0-SNAPSHOT
/_/
Using Python version 3.4.3 (default, Mar 6 2015 12:03:53)
SparkContext available as sc, SQLContext available as sqlContext.
>>> 15/05/28 00:58:32 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/0 is now EXITED (Command exited with code 1)
15/05/28 00:58:32 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/0 removed: Command exited with code 1
15/05/28 00:58:32 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0
15/05/28 00:58:32 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/1 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:32 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/1 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:32 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/1 is now LOADING
15/05/28 00:58:32 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/1 is now RUNNING
15/05/28 00:58:34 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/1 is now EXITED (Command exited with code 1)
15/05/28 00:58:34 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/1 removed: Command exited with code 1
15/05/28 00:58:34 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1
15/05/28 00:58:34 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/2 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:34 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/2 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:34 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/2 is now LOADING
15/05/28 00:58:34 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/2 is now RUNNING
15/05/28 00:58:36 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/2 is now EXITED (Command exited with code 1)
15/05/28 00:58:36 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/2 removed: Command exited with code 1
15/05/28 00:58:36 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 2
15/05/28 00:58:36 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/3 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:36 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/3 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:36 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/3 is now RUNNING
15/05/28 00:58:36 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/3 is now LOADING
15/05/28 00:58:37 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/3 is now EXITED (Command exited with code 1)
15/05/28 00:58:37 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/3 removed: Command exited with code 1
15/05/28 00:58:37 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 3
15/05/28 00:58:37 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/4 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:37 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/4 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:37 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/4 is now LOADING
15/05/28 00:58:37 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/4 is now RUNNING
15/05/28 00:58:39 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/4 is now EXITED (Command exited with code 1)
15/05/28 00:58:39 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/4 removed: Command exited with code 1
15/05/28 00:58:39 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 4
15/05/28 00:58:39 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/5 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:39 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/5 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:39 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/5 is now LOADING
15/05/28 00:58:39 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/5 is now RUNNING
15/05/28 00:58:41 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/5 is now EXITED (Command exited with code 1)
15/05/28 00:58:41 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/5 removed: Command exited with code 1
15/05/28 00:58:41 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 5
15/05/28 00:58:41 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/6 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:41 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/6 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:41 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/6 is now LOADING
15/05/28 00:58:41 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/6 is now RUNNING
15/05/28 00:58:42 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/6 is now EXITED (Command exited with code 1)
15/05/28 00:58:42 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/6 removed: Command exited with code 1
15/05/28 00:58:42 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 6
15/05/28 00:58:42 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/7 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:42 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/7 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:42 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/7 is now LOADING
15/05/28 00:58:42 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/7 is now RUNNING
15/05/28 00:58:44 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/7 is now EXITED (Command exited with code 1)
15/05/28 00:58:44 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/7 removed: Command exited with code 1
15/05/28 00:58:44 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 7
15/05/28 00:58:44 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/8 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:44 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/8 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:44 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/8 is now LOADING
15/05/28 00:58:44 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/8 is now RUNNING
15/05/28 00:58:46 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/8 is now EXITED (Command exited with code 1)
15/05/28 00:58:46 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/8 removed: Command exited with code 1
15/05/28 00:58:46 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 8
15/05/28 00:58:46 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/9 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:46 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/9 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:46 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/9 is now LOADING
15/05/28 00:58:46 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/9 is now RUNNING
15/05/28 00:58:47 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/9 is now EXITED (Command exited with code 1)
15/05/28 00:58:47 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/9 removed: Command exited with code 1
15/05/28 00:58:47 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
15/05/28 00:58:47 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/10 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:47 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/10 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:47 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/10 is now RUNNING
15/05/28 00:58:47 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/10 is now LOADING
15/05/28 00:58:49 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/10 is now EXITED (Command exited with code 1)
15/05/28 00:58:49 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/10 removed: Command exited with code 1
15/05/28 00:58:49 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 10
15/05/28 00:58:49 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/11 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:49 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/11 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:49 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/11 is now LOADING
15/05/28 00:58:49 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/11 is now RUNNING
15/05/28 00:58:51 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/11 is now EXITED (Command exited with code 1)
15/05/28 00:58:51 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/11 removed: Command exited with code 1
15/05/28 00:58:51 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 11
15/05/28 00:58:51 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/12 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:51 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/12 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:51 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/12 is now LOADING
15/05/28 00:58:51 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/12 is now RUNNING
15/05/28 00:58:52 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/12 is now EXITED (Command exited with code 1)
15/05/28 00:58:52 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/12 removed: Command exited with code 1
15/05/28 00:58:52 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 12
15/05/28 00:58:52 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/13 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/13 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:52 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/13 is now LOADING
15/05/28 00:58:52 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/13 is now RUNNING
15/05/28 00:58:54 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/13 is now EXITED (Command exited with code 1)
15/05/28 00:58:54 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/13 removed: Command exited with code 1
15/05/28 00:58:54 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 13
15/05/28 00:58:54 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/14 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:54 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/14 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:54 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/14 is now LOADING
15/05/28 00:58:54 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/14 is now RUNNING
15/05/28 00:58:55 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/14 is now EXITED (Command exited with code 1)
15/05/28 00:58:55 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/14 removed: Command exited with code 1
15/05/28 00:58:55 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 14
15/05/28 00:58:55 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/15 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/15 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:55 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/15 is now RUNNING
15/05/28 00:58:55 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/15 is now LOADING
^C (ABORT CHARACTER)
KeyboardInterrupt
>>> 15/05/28 00:58:57 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/15 is now EXITED (Command exited with code 1)
15/05/28 00:58:57 INFO SparkDeploySchedulerBackend: Executor app-20150528005831-0005/15 removed: Command exited with code 1
15/05/28 00:58:57 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 15
15/05/28 00:58:57 INFO AppClient$ClientActor: Executor added: app-20150528005831-0005/16 on worker-20150527230803-172.31.13.150-50730 (172.31.13.150:50730) with 2 cores
15/05/28 00:58:57 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150528005831-0005/16 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB RAM
15/05/28 00:58:57 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/16 is now LOADING
15/05/28 00:58:57 INFO AppClient$ClientActor: Executor updated: app-20150528005831-0005/16 is now RUNNING
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/metrics/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/api,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/static,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/threadDump,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/job/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/job,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/json,null}
15/05/28 00:58:58 INFO ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs,null}
15/05/28 00:58:58 INFO SparkUI: Stopped Spark web UI at http://ec2-52-24-65-198.us-west-2.compute.amazonaws.com:4040
15/05/28 00:58:58 INFO DAGScheduler: Stopping DAGScheduler
15/05/28 00:58:58 INFO SparkDeploySchedulerBackend: Shutting down all executors
15/05/28 00:58:58 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
15/05/28 00:58:58 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/05/28 00:58:58 INFO Utils: path = /mnt/spark/spark-985d5a6c-150e-40ad-875f-351733a40276/blockmgr-e36c9174-ff48-42e1-bbd0-c2b0649ab751, already present as root for deletion.
15/05/28 00:58:58 INFO Utils: path = /mnt2/spark/spark-fb2a7e42-2998-4ad2-be5f-d25472727d57/blockmgr-660e39f5-3561-4bec-a042-7cab1ea8cf54, already present as root for deletion.
15/05/28 00:58:58 INFO MemoryStore: MemoryStore cleared
15/05/28 00:58:58 INFO BlockManager: BlockManager stopped
15/05/28 00:58:58 INFO BlockManagerMaster: BlockManagerMaster stopped
15/05/28 00:58:58 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/05/28 00:58:58 INFO SparkContext: Successfully stopped SparkContext
15/05/28 00:58:58 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/05/28 00:58:58 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/05/28 00:58:58 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
{code}
> spark-ec2 and associated tools not py3 ready
> --------------------------------------------
>
> Key: SPARK-7909
> URL: https://issues.apache.org/jira/browse/SPARK-7909
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Environment: ec2 python3
> Reporter: Matthew Goodman
>
> At present there is not a possible permutation of tools that supports Python3 on both the launching computer and running cluster. There are a couple problems involved:
> - There is no prebuilt spark binary with python3 support.
> - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements
> - Config files for cluster processes don't seem to make it to all nodes in a working format.
> I have fixes for some of this, but the config and running context debugging remains elusive to me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org