You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Aaron Pfeifer (Jira)" <ji...@apache.org> on 2020/05/22 00:40:00 UTC
[jira] [Created] (TEZ-4187) Shuffle hangs occasionally after fetch
failure with multiple input vertices
Aaron Pfeifer created TEZ-4187:
----------------------------------
Summary: Shuffle hangs occasionally after fetch failure with multiple input vertices
Key: TEZ-4187
URL: https://issues.apache.org/jira/browse/TEZ-4187
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.2
Environment: Tez 0.9.2
Hadoop 2.9.2
Debian 9.12
Reporter: Aaron Pfeifer
Attachments: DAG.png, task.log
Hi folks!
There appears to be an issue where the ShuffleSchedulerCallable is not properly interrupted when a fetch failure has occurred from one of multiple input vertices. When this happens, our logs show the ShuffleScheduler (for the input vertex which did *not* set the throwable) running forever in this loop: https://github.com/apache/tez/blob/10cb3519bd34389210e6511a2ba291b52dcda081/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleScheduler.java#L1122-L1127
I've attached logs demonstrating this issue and what our DAG looks like. I can reproduce this almost every time. It always fails on a vertex with more than 1 input.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)