You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sai Nikhilesh Kasturi (Jira)" <ji...@apache.org> on 2020/03/30 16:38:00 UTC
[jira] [Commented] (SPARK-19764) Executors hang with supposedly
running task that are really finished.
[ https://issues.apache.org/jira/browse/SPARK-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071122#comment-17071122 ]
Sai Nikhilesh Kasturi commented on SPARK-19764:
-----------------------------------------------
Hello, I am new to spark but I am also facing the same issue. I have 8 workers running in parallel, 7 of them are completing the task in time but 8 worker is getting stuck and not completing that stage.
> Executors hang with supposedly running task that are really finished.
> ---------------------------------------------------------------------
>
> Key: SPARK-19764
> URL: https://issues.apache.org/jira/browse/SPARK-19764
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
> Affects Versions: 2.0.2
> Environment: Ubuntu 16.04 LTS
> OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
> Spark 2.0.2 - Spark Cluster Manager
> Reporter: Ari Gesher
> Priority: Major
> Attachments: SPARK-19764.tgz, driver-log-stderr.log, executor-2.log, netty-6153.jpg
>
>
> We've come across a job that won't finish. Running on a six-node cluster, each of the executors end up with 5-7 tasks that are never marked as completed.
> Here's an excerpt from the web UI:
> ||Index ▴||ID||Attempt||Status||Locality Level||Executor ID / Host||Launch Time||Duration||Scheduler Delay||Task Deserialization Time||GC Time||Result Serialization Time||Getting Result Time||Peak Execution Memory||Shuffle Read Size / Records||Errors||
> |105 | 1131 | 0 | SUCCESS |PROCESS_LOCAL |4 / 172.31.24.171 | 2017/02/27 22:51:36 | 1.9 min | 9 ms | 4 ms | 0.7 s | 2 ms| 6 ms| 384.1 MB| 90.3 MB / 572 | |
> |106| 1168| 0| RUNNING |ANY| 2 / 172.31.16.112| 2017/02/27 22:53:25| 6.5 h |0 ms| 0 ms| 1 s |0 ms| 0 ms| |384.1 MB |98.7 MB / 624 | |
> However, the Executor reports the task as finished:
> {noformat}
> 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168)
> 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). 2633558 bytes result sent via BlockManager)
> {noformat}
> As does the driver log:
> {noformat}
> 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168)
> 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). 2633558 bytes result sent via BlockManager)
> {noformat}
> Full log from this executor and the {{stderr}} from {{app-20170227223614-0001/2/stderr}} attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org