[jira] [Updated] (TEZ-3293) Fetch failures can cause a shuffle hang waiting for memory merge that never starts

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated TEZ-3293:
----------------------------
    Priority: Critical  (was: Major)

> Fetch failures can cause a shuffle hang waiting for memory merge that never starts
> ----------------------------------------------------------------------------------
>
>                 Key: TEZ-3293
>                 URL: https://issues.apache.org/jira/browse/TEZ-3293
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.7.1, 0.8.3
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>
> Tez jobs can hang in shuffle waiting for a memory merge that never starts.  When a MapOutput is reserved it increments usedMemory but when it is unreserved it decrements usedMemory _and_ commitMemory.  If enough shuffle failures occur of sufficient size then commitMemory may never reach the merge threshold even after all outstanding transfers have committed and thus hang the shuffle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)