[jira] [Commented] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close

["Chris Nauroth (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534501#comment-17534501 ] 

Chris Nauroth commented on MAPREDUCE-7369:
------------------------------------------

Hello [~stevel@apache.org] and [~groot]. I wanted to call out a potential unintended side effect in behavior resulting from this change.

Right now, "liveness" inherently demands some sort of forward progress by the task, like continuous processing of input records or manual interaction with the {{Progressable}} interface if the job has been custom coded to do so. It is not sufficient that the OS process is merely alive and sending pings. That means that in some gray failure scenarios, where the process is running and pinging but not making meaningful progress (e.g. horrendously degraded but not completely failed disk), the framework gets to detect that, kill the task, and schedule a reattempt elsewhere. Speculative execution helps with this too, but if users have disabled speculative execution for some reason, then this is an important protection mechanism.

I understand what motivates this change: slow large rename operations against cloud storage that don't appear to signal meaningful progress. However, if we equate ping with liveness, then I think we also lose the resilience described above. For that reason, I would suggest putting the new behavior behind a configuration flag, something like the 001 patch file attached to the issue here rather than the current GitHub PR.

Ideally, we'd get these long renames interacting with the {{Progressable}} somehow, but that's a much bigger topic.

Thanks!

> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7369
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.3.1
>            Reporter: Prabhu Joseph
>            Assignee: Ashutosh Gupta
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: MAPREDUCE-7369.001.patch
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close. MultipleOutputs#closes takes more time when there are multiple files to be closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1634949471086_61268_m_001115_0: AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right timeout value. The timeout can be disabled with 0 but that might lead to hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by ApplicationMaster. It expects the status information which won't be send during MultipleOutputs#close. This jira is to add a config which considers the ping from task as part of Task Liveliness Check in the ApplicationMaster.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org