You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Kuhu Shukla (JIRA)" <ji...@apache.org> on 2018/11/26 21:36:00 UTC
[jira] [Commented] (MAPREDUCE-7164) FileOutputCommitter does not
report progress while merging paths.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699606#comment-16699606 ]
Kuhu Shukla commented on MAPREDUCE-7164:
----------------------------------------
Would it be useful to add progress() functionality to JobContext , similar to the TaskAttemptContext? If not, the change to mergePaths() simply calling progress gets tricky as there are paths like commitJobInternal() that only have the job context to pass on to this method. I do not think it would be ok to report progress only when the task context is being passed. I can do an instanceof check but that seems like a bad choice.
> FileOutputCommitter does not report progress while merging paths.
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-7164
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7164
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.3, 2.8.5, 2.9.2
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Priority: Major
>
> In cases where the rename and merge path logic takes more time than usual, the committer does not report progress and can cause job failure. This behavior was not present in Hadoop 1.x. This JIRA will fix it so that the old behavior for 1.x is restored.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org