You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adrian Bridgett <ad...@opensignal.com> on 2016/10/05 17:05:53 UTC
mesos in spark 2.0.1 - must call stop() otherwise app hangs
I saw this originally with 2.0.0 but as 2.0.1 is freshly out I thought
I'd retry.
With pyspark apps, it seems that in 2.x you must call .stop() on the
spark context as otherwise the application doesn't stop (see log below -
first gap is when it finishes, second gap is when I hit ^C).
Thought I'd mention this in case a) it's a bug (certainly changed
behaviour since 1.6.1) and b) anyone else hits it.
Unique devices= 1243123123123
16/10/05 16:52:30 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping
the total amount of executors to 0
(hangs here - spark UI still running - show all executors there)
(after a while dynamic allocation kicks in which cleans up a little)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_7_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
(size: 3.7 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_7_piece0 on 10.1.200.212:38722 in memory (size: 3.7 KB, free:
912.3 MB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on 10.1.200.212:38722 in memory (size: 26.2 KB, free:
912.3 MB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-201-35.ec2.internal:31353 in memory (size:
26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-200-127.ec2.internal:11645 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-200-126.ec2.internal:17810 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-201-190.ec2.internal:37033 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-201-8.ec2.internal:14506 in memory (size:
26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-200-128.ec2.internal:36370 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-202-128.ec2.internal:34654 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_4_piece0 on ip-10-1-202-127.ec2.internal:28749 in memory
(size: 26.2 KB, free: 10.5 GB)
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3471
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3472
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3473
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3474
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3475
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3476
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3477
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3478
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3479
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3480
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3481
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3482
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3483
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3484
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3485
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3486
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3487
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3488
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3489
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3490
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3491
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3492
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3493
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 1
16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 2
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_6_piece0 on 10.1.200.212:38722 in memory (size: 9.9 KB, free:
912.3 MB)
16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
broadcast_6_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
(size: 9.9 KB, free: 10.5 GB)
(but app is still hanging around - not sure what the import sys is doing
here, but I've left it in case it explains anything!)
\u25bd
import sys
\u25bd
import sys
(and here is the ^C)
^C16/10/05 16:55:21 INFO spark.SparkContext: Invoking stop() from
shutdown hook
16/10/05 16:55:21 INFO server.ServerConnector: Stopped
ServerConnector@70087549{HTTP/1.1}{0.0.0.0:4040}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@22516b56{/stages/stage/kill,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6b5c5804{/api,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@98268{/,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@59865b04{/static,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5a5272d3{/executors/threadDump/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1ade2c09{/executors/threadDump,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7ff4c458{/executors/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@667cb568{/executors,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@60441110{/environment/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@39cb8097{/environment,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@16b1e403{/storage/rdd/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6f8d2666{/storage/rdd,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7455a06e{/storage/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5e0e2b61{/storage,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@5615ea0e{/stages/pool/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@57598349{/stages/pool,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6a38093f{/stages/stage/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@2b9460e5{/stages/stage,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6c3dbe02{/stages/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1079b7a{/stages,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@432049a9{/jobs/job/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6144d55d{/jobs/job,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7c4232ba{/jobs/json,null,UNAVAILABLE}
16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@291d6f18{/jobs,null,UNAVAILABLE}
16/10/05 16:55:21 INFO ui.SparkUI: Stopped Spark web UI at
http://10.1.200.212:4040
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend:
Shutting down all executors
16/10/05 16:55:21 INFO
cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each
executor to shut down
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 8 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 6 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 7 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 1 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 5 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 2 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 4 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 3 is now TASK_FINISHED
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 0 is now TASK_FINISHED
I1005 16:55:21.862349 16747 sched.cpp:1903] Asked to stop the driver
I1005 16:55:21.862777 16390 sched.cpp:1143] Stopping framework
'cd7fd4fe-1eff-438c-87f9-348461e6a640-1683'
16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend:
driver.run() returned with code DRIVER_STOPPED
16/10/05 16:55:21 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
16/10/05 16:55:21 INFO memory.MemoryStore: MemoryStore cleared
16/10/05 16:55:21 INFO storage.BlockManager: BlockManager stopped
16/10/05 16:55:21 INFO storage.BlockManagerMaster: BlockManagerMaster
stopped
16/10/05 16:55:21 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
16/10/05 16:55:21 INFO spark.SparkContext: Successfully stopped SparkContext
16/10/05 16:55:21 INFO util.ShutdownHookManager: Shutdown hook called
16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b/pyspark-28e7d692-a344-4d0f-9495-3304ccaa520e
16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b
And now a run with sc.stop():
16/10/05 16:57:52 INFO scheduler.DAGScheduler: Job 2 finished: count at
NativeMethodAccessorImpl.java:-2, took 12.148737 s
Unique devices= 1243123123123
16/10/05 16:57:53 INFO server.ServerConnector: Stopped
ServerConnector@6395b5{HTTP/1.1}{0.0.0.0:4040}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@41d00db1{/stages/stage/kill,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@6dc9bd9a{/api,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@62c55d05{/,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@69c56749{/static,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@2271d89a{/executors/threadDump/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7a7ebfb8{/executors/threadDump,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@4469ec58{/executors/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@231d0c59{/executors,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@352daf52{/environment/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@72798a1f{/environment,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1def5f95{/storage/rdd/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@697feb2d{/storage/rdd,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@7056f8aa{/storage/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@44056f98{/storage,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@48e1d5a{/stages/pool/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@54d9ad20{/stages/pool,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@3921b8da{/stages/stage/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@630479fa{/stages/stage,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1e69dddc{/stages/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@37eb5fc9{/stages,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@3feca272{/jobs/job/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@fae47cc{/jobs/job,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@59587d7{/jobs/json,null,UNAVAILABLE}
16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
o.s.j.s.ServletContextHandler@1fe2fda3{/jobs,null,UNAVAILABLE}
16/10/05 16:57:53 INFO ui.SparkUI: Stopped Spark web UI at
http://10.1.200.212:4040
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping
the total amount of executors to 0
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend:
Shutting down all executors
16/10/05 16:57:53 INFO
cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each
executor to shut down
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 6 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 1 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 7 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 8 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 2 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 3 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 4 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 5 is now TASK_FINISHED
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
task 0 is now TASK_FINISHED
I1005 16:57:53.946076 17178 sched.cpp:1903] Asked to stop the driver
I1005 16:57:53.946187 17215 sched.cpp:1143] Stopping framework
'cd7fd4fe-1eff-438c-87f9-348461e6a640-1685'
16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend:
driver.run() returned with code DRIVER_STOPPED
16/10/05 16:57:53 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
16/10/05 16:57:53 INFO memory.MemoryStore: MemoryStore cleared
16/10/05 16:57:53 INFO storage.BlockManager: BlockManager stopped
16/10/05 16:57:53 INFO storage.BlockManagerMaster: BlockManagerMaster
stopped
16/10/05 16:57:53 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
16/10/05 16:57:53 INFO spark.SparkContext: Successfully stopped SparkContext
16/10/05 16:57:54 INFO util.ShutdownHookManager: Shutdown hook called
16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711/pyspark-04c90784-4350-4660-bac8-106fdfc7e6cf
16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711
Code was this (sorry, C&P job - could kill off the sys of course!)
import sys
from pyspark.sql import SparkSession
sc= SparkSession\
.builder\
.appName("spark2test")\
.getOrCreate()
df = sc.read.load('/prod/data/core/filtered/parquet/2016-08-01')
print("Rowcount=", df.count())
print("Unique devices=",df.select('device_id_time').distinct().count())
#sc.stop()
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Timothy Chen <tn...@gmail.com>.
Been noticing the same issue with Yarn as well (also pyspark).
So I don't think it's Mesos related.
Tim
> On Oct 5, 2016, at 10:05 AM, Adrian Bridgett <ad...@opensignal.com> wrote:
>
> I saw this originally with 2.0.0 but as 2.0.1 is freshly out I thought I'd retry.
>
> With pyspark apps, it seems that in 2.x you must call .stop() on the spark context as otherwise the application doesn't stop (see log below - first gap is when it finishes, second gap is when I hit ^C).
>
> Thought I'd mention this in case a) it's a bug (certainly changed behaviour since 1.6.1) and b) anyone else hits it.
>
>
> Unique devices= 1243123123123
> 16/10/05 16:52:30 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0
>
> (hangs here - spark UI still running - show all executors there)
> (after a while dynamic allocation kicks in which cleans up a little)
>
>
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory (size: 3.7 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on 10.1.200.212:38722 in memory (size: 3.7 KB, free: 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on 10.1.200.212:38722 in memory (size: 26.2 KB, free: 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-201-35.ec2.internal:31353 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-200-127.ec2.internal:11645 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-200-126.ec2.internal:17810 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-201-190.ec2.internal:37033 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-201-8.ec2.internal:14506 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-200-128.ec2.internal:36370 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-202-128.ec2.internal:34654 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-1-202-127.ec2.internal:28749 in memory (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3471
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3472
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3473
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3474
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3475
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3476
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3477
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3478
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3479
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3480
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3481
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3482
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3483
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3484
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3485
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3486
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3487
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3488
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3489
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3490
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3491
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3492
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3493
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 1
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 2
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on 10.1.200.212:38722 in memory (size: 9.9 KB, free: 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory (size: 9.9 KB, free: 10.5 GB)
>
> (but app is still hanging around - not sure what the import sys is doing here, but I've left it in case it explains anything!)
>
>
>
>
> ▽
> import sys
>
>
> ▽
> import sys
> (and here is the ^C)
> ^C16/10/05 16:55:21 INFO spark.SparkContext: Invoking stop() from shutdown hook
> 16/10/05 16:55:21 INFO server.ServerConnector: Stopped ServerConnector@70087549{HTTP/1.1}{0.0.0.0:4040}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@22516b56{/stages/stage/kill,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6b5c5804{/api,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@98268{/,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@59865b04{/static,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5a5272d3{/executors/threadDump/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1ade2c09{/executors/threadDump,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7ff4c458{/executors/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@667cb568{/executors,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@60441110{/environment/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@39cb8097{/environment,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@16b1e403{/storage/rdd/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6f8d2666{/storage/rdd,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7455a06e{/storage/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5e0e2b61{/storage,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5615ea0e{/stages/pool/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@57598349{/stages/pool,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6a38093f{/stages/stage/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2b9460e5{/stages/stage,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6c3dbe02{/stages/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1079b7a{/stages,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@432049a9{/jobs/job/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6144d55d{/jobs/job,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7c4232ba{/jobs/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@291d6f18{/jobs,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO ui.SparkUI: Stopped Spark web UI at http://10.1.200.212:4040
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Shutting down all executors
> 16/10/05 16:55:21 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 8 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 6 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 7 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 5 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 2 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 4 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 3 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_FINISHED
> I1005 16:55:21.862349 16747 sched.cpp:1903] Asked to stop the driver
> I1005 16:55:21.862777 16390 sched.cpp:1143] Stopping framework 'cd7fd4fe-1eff-438c-87f9-348461e6a640-1683'
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: driver.run() returned with code DRIVER_STOPPED
> 16/10/05 16:55:21 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
> 16/10/05 16:55:21 INFO memory.MemoryStore: MemoryStore cleared
> 16/10/05 16:55:21 INFO storage.BlockManager: BlockManager stopped
> 16/10/05 16:55:21 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
> 16/10/05 16:55:21 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
> 16/10/05 16:55:21 INFO spark.SparkContext: Successfully stopped SparkContext
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Shutdown hook called
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b/pyspark-28e7d692-a344-4d0f-9495-3304ccaa520e
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b
>
> And now a run with sc.stop():
> 16/10/05 16:57:52 INFO scheduler.DAGScheduler: Job 2 finished: count at NativeMethodAccessorImpl.java:-2, took 12.148737 s
> Unique devices= 1243123123123
> 16/10/05 16:57:53 INFO server.ServerConnector: Stopped ServerConnector@6395b5{HTTP/1.1}{0.0.0.0:4040}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@41d00db1{/stages/stage/kill,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6dc9bd9a{/api,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@62c55d05{/,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@69c56749{/static,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2271d89a{/executors/threadDump/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7a7ebfb8{/executors/threadDump,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4469ec58{/executors/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@231d0c59{/executors,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@352daf52{/environment/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72798a1f{/environment,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1def5f95{/storage/rdd/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@697feb2d{/storage/rdd,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7056f8aa{/storage/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@44056f98{/storage,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@48e1d5a{/stages/pool/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@54d9ad20{/stages/pool,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3921b8da{/stages/stage/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@630479fa{/stages/stage,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1e69dddc{/stages/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@37eb5fc9{/stages,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3feca272{/jobs/job/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@fae47cc{/jobs/job,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@59587d7{/jobs/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1fe2fda3{/jobs,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO ui.SparkUI: Stopped Spark web UI at http://10.1.200.212:4040
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Shutting down all executors
> 16/10/05 16:57:53 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 6 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 7 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 8 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 2 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 3 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 4 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 5 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_FINISHED
> I1005 16:57:53.946076 17178 sched.cpp:1903] Asked to stop the driver
> I1005 16:57:53.946187 17215 sched.cpp:1143] Stopping framework 'cd7fd4fe-1eff-438c-87f9-348461e6a640-1685'
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: driver.run() returned with code DRIVER_STOPPED
> 16/10/05 16:57:53 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
> 16/10/05 16:57:53 INFO memory.MemoryStore: MemoryStore cleared
> 16/10/05 16:57:53 INFO storage.BlockManager: BlockManager stopped
> 16/10/05 16:57:53 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
> 16/10/05 16:57:53 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
> 16/10/05 16:57:53 INFO spark.SparkContext: Successfully stopped SparkContext
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Shutdown hook called
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711/pyspark-04c90784-4350-4660-bac8-106fdfc7e6cf
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711
>
> Code was this (sorry, C&P job - could kill off the sys of course!)
>
> import sys
> from pyspark.sql import SparkSession
>
> sc= SparkSession\
> .builder\
> .appName("spark2test")\
> .getOrCreate()
>
> df = sc.read.load('/prod/data/core/filtered/parquet/2016-08-01')
> print("Rowcount=", df.count())
> print("Unique devices=",df.select('device_id_time').distinct().count())
> #sc.stop()
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Adrian Bridgett <ad...@opensignal.com>.
Thanks for the clarification Sean, much appreciated!
On 06/10/2016 09:18, Sean Owen wrote:
> Yes you should do that. The examples, with one exception, do show
> this, and it's always been the intended behavior. I guess it's no
> surprise to me because any 'context' object in any framework generally
> has to be shutdown for reasons like this.
>
> We need to update the one example. The twist is error handling though,
> yeah, because you need to stop() even if an exception occurs. Easy
> enough with "finally", but I guess people don't do that. It'd be nice
> to get rid of this non-daemon thread if possible
>
> On Thu, Oct 6, 2016 at 9:02 AM Adrian Bridgett <adrian@opensignal.com
> <ma...@opensignal.com>> wrote:
>
> Just one question - what about errors? Should we be wrapping our
> entire
> code in a ...finally spark.stop() clause (as per
> http://spark.apache.org/docs/latest/programming-guide.html#unit-testing)?
>
> BTW the .stop() requirement was news to quite a few people here, maybe
> it'd be a good idea to shout more loudly about in the initial
> quickstart/examples?
>
> Cheers,
>
> Adrian
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
>
--
*Adrian Bridgett* | Sysadmin Engineer, OpenSignal
<http://www.opensignal.com>
_____________________________________________________
Office: 3rd Floor, The Angel Office, 2 Angel Square, London, EC1V 1NY
Phone #: +44 777-377-8251
Skype: abridgett |@adrianbridgett <http://twitter.com/adrianbridgett>|
LinkedIn link <https://uk.linkedin.com/in/abridgett>
_____________________________________________________
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Sean Owen <so...@cloudera.com>.
Yes you should do that. The examples, with one exception, do show this, and
it's always been the intended behavior. I guess it's no surprise to me
because any 'context' object in any framework generally has to be shutdown
for reasons like this.
We need to update the one example. The twist is error handling though,
yeah, because you need to stop() even if an exception occurs. Easy enough
with "finally", but I guess people don't do that. It'd be nice to get rid
of this non-daemon thread if possible
On Thu, Oct 6, 2016 at 9:02 AM Adrian Bridgett <ad...@opensignal.com>
wrote:
> Just one question - what about errors? Should we be wrapping our entire
> code in a ...finally spark.stop() clause (as per
> http://spark.apache.org/docs/latest/programming-guide.html#unit-testing)?
>
> BTW the .stop() requirement was news to quite a few people here, maybe
> it'd be a good idea to shout more loudly about in the initial
> quickstart/examples?
>
> Cheers,
>
> Adrian
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Adrian Bridgett <ad...@opensignal.com>.
Just one question - what about errors? Should we be wrapping our entire
code in a ...finally spark.stop() clause (as per
http://spark.apache.org/docs/latest/programming-guide.html#unit-testing)?
BTW the .stop() requirement was news to quite a few people here, maybe
it'd be a good idea to shout more loudly about in the initial
quickstart/examples?
Cheers,
Adrian
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Adrian Bridgett <ad...@opensignal.com>.
Fab thanks all - I'll ensure we fix our code :-)
On 05/10/2016 18:10, Sean Owen wrote:
> Being discussed as we speak at
> https://issues.apache.org/jira/browse/SPARK-17707
>
> Calling stop() is definitely the right thing to do and always has been
> (see examples), but, may be possible to get rid of the new non-daemon
> thread preventing shutdown to make it possible to still get away
> without it.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: mesos in spark 2.0.1 - must call stop() otherwise app hangs
Posted by Sean Owen <so...@cloudera.com>.
Being discussed as we speak at
https://issues.apache.org/jira/browse/SPARK-17707
Calling stop() is definitely the right thing to do and always has been (see
examples), but, may be possible to get rid of the new non-daemon thread
preventing shutdown to make it possible to still get away without it.
On Wed, Oct 5, 2016 at 6:06 PM Adrian Bridgett <ad...@opensignal.com>
wrote:
> I saw this originally with 2.0.0 but as 2.0.1 is freshly out I thought
> I'd retry.
>
> With pyspark apps, it seems that in 2.x you must call .stop() on the
> spark context as otherwise the application doesn't stop (see log below -
> first gap is when it finishes, second gap is when I hit ^C).
>
> Thought I'd mention this in case a) it's a bug (certainly changed
> behaviour since 1.6.1) and b) anyone else hits it.
>
>
> Unique devices= 1243123123123
> 16/10/05 16:52:30 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping
> the total amount of executors to 0
>
> (hangs here - spark UI still running - show all executors there)
> (after a while dynamic allocation kicks in which cleans up a little)
>
>
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_7_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
> (size: 3.7 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_7_piece0 on 10.1.200.212:38722 in memory (size: 3.7 KB, free:
> 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on 10.1.200.212:38722 in memory (size: 26.2 KB, free:
> 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-201-35.ec2.internal:31353 in memory (size:
> 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-200-127.ec2.internal:11645 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-200-126.ec2.internal:17810 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-201-190.ec2.internal:37033 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-201-8.ec2.internal:14506 in memory (size:
> 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-200-128.ec2.internal:36370 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-202-128.ec2.internal:34654 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_4_piece0 on ip-10-1-202-127.ec2.internal:28749 in memory
> (size: 26.2 KB, free: 10.5 GB)
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3471
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3472
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3473
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3474
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3475
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3476
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3477
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3478
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3479
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3480
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3481
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3482
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3483
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3484
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3485
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3486
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3487
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3488
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3489
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3490
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3491
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3492
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned accumulator 3493
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 1
> 16/10/05 16:53:21 INFO spark.ContextCleaner: Cleaned shuffle 2
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_6_piece0 on 10.1.200.212:38722 in memory (size: 9.9 KB, free:
> 912.3 MB)
> 16/10/05 16:53:21 INFO storage.BlockManagerInfo: Removed
> broadcast_6_piece0 on ip-10-1-202-125.ec2.internal:18594 in memory
> (size: 9.9 KB, free: 10.5 GB)
>
> (but app is still hanging around - not sure what the import sys is doing
> here, but I've left it in case it explains anything!)
>
>
>
>
> ▽
> import sys
>
>
> ▽
> import sys
> (and here is the ^C)
> ^C16/10/05 16:55:21 INFO spark.SparkContext: Invoking stop() from
> shutdown hook
> 16/10/05 16:55:21 INFO server.ServerConnector: Stopped
> ServerConnector@70087549{HTTP/1.1}{0.0.0.0:4040}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@22516b56
> {/stages/stage/kill,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6b5c5804{/api,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@98268{/,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@59865b04{/static,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@5a5272d3
> {/executors/threadDump/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1ade2c09
> {/executors/threadDump,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7ff4c458{/executors/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@667cb568{/executors,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@60441110{/environment/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@39cb8097{/environment,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@16b1e403{/storage/rdd/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6f8d2666{/storage/rdd,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7455a06e{/storage/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@5e0e2b61{/storage,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@5615ea0e{/stages/pool/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@57598349{/stages/pool,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6a38093f
> {/stages/stage/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@2b9460e5{/stages/stage,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6c3dbe02{/stages/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1079b7a{/stages,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@432049a9{/jobs/job/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6144d55d{/jobs/job,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7c4232ba{/jobs/json,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@291d6f18{/jobs,null,UNAVAILABLE}
> 16/10/05 16:55:21 INFO ui.SparkUI: Stopped Spark web UI at
> http://10.1.200.212:4040
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend:
> Shutting down all executors
> 16/10/05 16:55:21 INFO
> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each
> executor to shut down
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 8 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 6 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 7 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 1 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 5 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 2 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 4 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 3 is now TASK_FINISHED
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 0 is now TASK_FINISHED
> I1005 16:55:21.862349 16747 sched.cpp:1903] Asked to stop the driver
> I1005 16:55:21.862777 16390 sched.cpp:1143] Stopping framework
> 'cd7fd4fe-1eff-438c-87f9-348461e6a640-1683'
> 16/10/05 16:55:21 INFO mesos.MesosCoarseGrainedSchedulerBackend:
> driver.run() returned with code DRIVER_STOPPED
> 16/10/05 16:55:21 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 16/10/05 16:55:21 INFO memory.MemoryStore: MemoryStore cleared
> 16/10/05 16:55:21 INFO storage.BlockManager: BlockManager stopped
> 16/10/05 16:55:21 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 16/10/05 16:55:21 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 16/10/05 16:55:21 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Shutdown hook called
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory
>
> /tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b/pyspark-28e7d692-a344-4d0f-9495-3304ccaa520e
> 16/10/05 16:55:21 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-3d420117-9442-4375-91ac-d54038e2c18b
>
> And now a run with sc.stop():
> 16/10/05 16:57:52 INFO scheduler.DAGScheduler: Job 2 finished: count at
> NativeMethodAccessorImpl.java:-2, took 12.148737 s
> Unique devices= 1243123123123
> 16/10/05 16:57:53 INFO server.ServerConnector: Stopped
> ServerConnector@6395b5{HTTP/1.1}{0.0.0.0:4040}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@41d00db1
> {/stages/stage/kill,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6dc9bd9a{/api,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@62c55d05{/,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@69c56749{/static,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@2271d89a
> {/executors/threadDump/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7a7ebfb8
> {/executors/threadDump,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@4469ec58{/executors/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@231d0c59{/executors,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@352daf52{/environment/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@72798a1f{/environment,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1def5f95{/storage/rdd/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@697feb2d{/storage/rdd,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7056f8aa{/storage/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@44056f98{/storage,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@48e1d5a{/stages/pool/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@54d9ad20{/stages/pool,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@3921b8da
> {/stages/stage/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@630479fa{/stages/stage,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1e69dddc{/stages/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@37eb5fc9{/stages,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@3feca272{/jobs/job/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@fae47cc{/jobs/job,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@59587d7{/jobs/json,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1fe2fda3{/jobs,null,UNAVAILABLE}
> 16/10/05 16:57:53 INFO ui.SparkUI: Stopped Spark web UI at
> http://10.1.200.212:4040
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Capping
> the total amount of executors to 0
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend:
> Shutting down all executors
> 16/10/05 16:57:53 INFO
> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each
> executor to shut down
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 6 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 1 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 7 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 8 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 2 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 3 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 4 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 5 is now TASK_FINISHED
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend: Mesos
> task 0 is now TASK_FINISHED
> I1005 16:57:53.946076 17178 sched.cpp:1903] Asked to stop the driver
> I1005 16:57:53.946187 17215 sched.cpp:1143] Stopping framework
> 'cd7fd4fe-1eff-438c-87f9-348461e6a640-1685'
> 16/10/05 16:57:53 INFO mesos.MesosCoarseGrainedSchedulerBackend:
> driver.run() returned with code DRIVER_STOPPED
> 16/10/05 16:57:53 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 16/10/05 16:57:53 INFO memory.MemoryStore: MemoryStore cleared
> 16/10/05 16:57:53 INFO storage.BlockManager: BlockManager stopped
> 16/10/05 16:57:53 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 16/10/05 16:57:53 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 16/10/05 16:57:53 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Shutdown hook called
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory
>
> /tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711/pyspark-04c90784-4350-4660-bac8-106fdfc7e6cf
> 16/10/05 16:57:54 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-db6e7cc5-21d1-4895-a51f-ad797989d711
>
> Code was this (sorry, C&P job - could kill off the sys of course!)
>
> import sys
> from pyspark.sql import SparkSession
>
> sc= SparkSession\
> .builder\
> .appName("spark2test")\
> .getOrCreate()
>
> df = sc.read.load('/prod/data/core/filtered/parquet/2016-08-01')
> print("Rowcount=", df.count())
> print("Unique devices=",df.select('device_id_time').distinct().count())
> #sc.stop()
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>