You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "DjvuLee (JIRA)" <ji...@apache.org> on 2017/07/27 09:38:00 UTC
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time

    [ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103005#comment-16103005 ] 

DjvuLee commented on SPARK-21547:
---------------------------------

17/07/27 11:29:51 INFO TaskSetManager: Finished task 169.0 in stage 1504.0 (TID 1504369) in 43975 ms on n6-195-137.byted.org (999/1000)
17/07/27 11:29:55 INFO TaskSetManager: Finished task 882.0 in stage 1504.0 (TID 1504905) in 44153 ms on n6-195-137.byted.org (1000/1000)
17/07/27 11:29:55 INFO YarnScheduler: Removed TaskSet 1504.0, whose tasks have all completed, from pool
17/07/27 11:29:55 INFO DAGScheduler: ResultStage 1504 (call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230) finished in 457.863 s
17/07/27 11:29:55 INFO DAGScheduler: Job 1504 finished: call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230, took 457.877969 s
17/07/27 11:30:02 INFO JobScheduler: Added jobs for time 1501126200000 ms
17/07/27 11:30:32 INFO JobScheduler: Added jobs for time 1501126230000 ms
17/07/27 11:31:02 INFO JobScheduler: Added jobs for time 1501126260000 ms
17/07/27 11:31:32 INFO JobScheduler: Added jobs for time 1501126290000 ms
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906391
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906392
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906396
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906402
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906404
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492509
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492508
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492507
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492506
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492505
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492504
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492503
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492502
...
7/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906397
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906398
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906395
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906399
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906403
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906400
17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906401
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on 10.6.131.75:23734 in memory (size: 35.9 KB, free: 2.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-157-227.byted.org:13090 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-157-158.byted.org:21120 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n6-195-150.byted.org:13277 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-156-165.byted.org:35355 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n6-132-023.byted.org:52521 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-136-133.byted.org:25696 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-150-029.byted.org:34673 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-148-038.byted.org:22503 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-150-038.byted.org:28209 in memory (size: 35.9 KB, free: 9.4 GB)

...

17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-163-151.byted.org:33703 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-148-028.byted.org:36086 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-151-039.byted.org:21081 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-157-167.byted.org:29370 in memory (size: 35.9 KB, free: 9.4 GB)
17/07/27 11:32:02 INFO JobScheduler: Added jobs for time 1501126320000 ms
17/07/27 11:32:32 INFO JobScheduler: Added jobs for time 1501126350000 ms
17/07/27 11:32:45 INFO JobScheduler: Finished job streaming job 1501116960000 ms.0 from job set of time 1501116960000 ms
17/07/27 11:32:45 INFO JobScheduler: Total delay: 9405.183 s for time 1501116960000 ms (execution: 1169.595 s)
17/07/27 11:32:45 INFO JobScheduler: Starting job streaming job 1501117530000 ms.0 from job set of time 1501117530000 ms
17/07/27 11:32:45 INFO PythonRDD: Removing RDD 6998 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 6998
17/07/27 11:32:45 INFO PythonRDD: Removing RDD 7001 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 7001
17/07/27 11:32:45 INFO PythonRDD: Removing RDD 6998 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 6998
17/07/27 11:32:45 INFO PythonRDD: Removing RDD 7001 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 7001
17/07/27 11:32:45 INFO MapPartitionsRDD: Removing RDD 6997 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 6997
17/07/27 11:32:45 INFO MapPartitionsRDD: Removing RDD 7000 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 7000
17/07/27 11:32:45 INFO KafkaRDD: Removing RDD 6996 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 6996
17/07/27 11:32:45 INFO KafkaRDD: Removing RDD 6999 from persistence list
17/07/27 11:32:45 INFO BlockManager: Removing RDD 6999
17/07/27 11:32:45 INFO ReceivedBlockTracker: Deleting batches:
17/07/27 11:32:45 INFO InputInfoTracker: remove old batch metadata: 1501116870000 ms 1501116900000 ms
17/07/27 11:32:45 INFO SparkContext: Starting job: call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230
17/07/27 11:32:45 INFO DAGScheduler: Got job 1505 (call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230) with 1000 output partitions
17/07/27 11:32:45 INFO DAGScheduler: Final stage: ResultStage 1505 (call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230)
17/07/27 11:32:45 INFO DAGScheduler: Parents of final stage: List()
17/07/27 11:32:45 INFO DAGScheduler: Missing parents: List()
17/07/27 11:32:45 INFO DAGScheduler: Submitting ResultStage 1505 (PythonRDD[8411] at call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230), which ha
17/07/27 11:32:45 INFO MemoryStore: Block broadcast_1505 stored as values in memory (estimated size 81.4 KB, free 2.4 GB)
17/07/27 11:32:45 INFO MemoryStore: Block broadcast_1505_piece0 stored as bytes in memory (estimated size 35.9 KB, free 2.4 GB)
17/07/27 11:32:45 INFO BlockManagerInfo: Added broadcast_1505_piece0 in memory on 10.6.131.75:23734 (size: 35.9 KB, free: 2.4 GB)
17/07/27 11:32:45 INFO SparkContext: Created broadcast 1505 from broadcast at DAGScheduler.scala:1012

> Spark cleaner cost too many time
> --------------------------------
>
>                 Key: SPARK-21547
>                 URL: https://issues.apache.org/jira/browse/SPARK-21547
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.0.0
>            Reporter: DjvuLee
>
> Spark Streaming sometime cost so many time deal with cleaning, and this can become worse when enable the dynamic allocation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org