You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:14:05 UTC
[jira] [Resolved] (SPARK-20139) Spark UI reports partial success
for completed stage while log shows all tasks are finished
[ https://issues.apache.org/jira/browse/SPARK-20139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-20139.
----------------------------------
Resolution: Incomplete
> Spark UI reports partial success for completed stage while log shows all tasks are finished
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-20139
> URL: https://issues.apache.org/jira/browse/SPARK-20139
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.1.0
> Reporter: Etti Gur
> Priority: Major
> Labels: bulk-closed
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Spark UI reports partial success for completed stage while log shows all tasks are finished - i.e.:
> We have a stage that is presented under completed stages on spark UI,
> but the successful tasks are shown like so: (146372/524964) not as you'd expect (524964/524964)
> Looking at the application master log shows all tasks in that stage are successful:
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 522973.0 in stage 0.0 (TID 522973) in 1163910 ms on ip-10-1-15-34.ec2.internal (executor 116) (524963/524964)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 12508.0 in stage 2.0 (TID 537472) in 241250 ms on ip-10-1-15-14.ec2.internal (executor 38) (20234/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 12465.0 in stage 2.0 (TID 537429) in 241994 ms on ip-10-1-15-106.ec2.internal (executor 133) (20235/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 15079.0 in stage 2.0 (TID 540043) in 202889 ms on ip-10-1-15-173.ec2.internal (executor 295) (20236/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 19828.0 in stage 2.0 (TID 544792) in 137845 ms on ip-10-1-15-147.ec2.internal (executor 43) (20237/20262)
> 17/03/29 09:45:50 INFO TaskSetManager: Finished task 19072.0 in stage 2.0 (TID 544036) in 147363 ms on ip-10-1-15-19.ec2.internal (executor 175) (20238/20262)
> 17/03/29 09:45:50 INFO TaskSetManager: Finished task 524146.0 in stage 0.0 (TID 524146) in 889950 ms on ip-10-1-15-72.ec2.internal (executor 74) (524964/524964)
> Also in the log we get an error:
> 17/03/29 08:24:16 ERROR LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
> This looks like the stage is indeed completed with all its tasks but UI shows like not all tasks really finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org