You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "yinan zhan (Jira)" <ji...@apache.org> on 2021/03/17 03:14:00 UTC

[jira] [Comment Edited] (HADOOP-17584) s3a committer may commit more data

    [ https://issues.apache.org/jira/browse/HADOOP-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303051#comment-17303051 ] 

yinan zhan edited comment on HADOOP-17584 at 3/17/21, 3:13 AM:
---------------------------------------------------------------

I think using jobid not jobId/appAttemptId has side effect when am fail and abort job has not run.

protected Path getJobAttemptPath(int appAttemptId) {
     return getMagicJobAttemptPath(getUUID(), getOutputPath());
 }

jobId/appAttemptId replace jobid maybe a good solution.


was (Author: adol):
I think using jobid not appAttemptId has side effect when am fail and abort job has not run.

jobId/appAttemptId replace jobid maybe a good solution.

> s3a committer may commit more data
> ----------------------------------
>
>                 Key: HADOOP-17584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17584
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: yinan zhan
>            Priority: Major
>
> s3a magic committer isRecoverySupported() is false, so will restart all task after application master restart for am jvm crashed, leaving pendingset in magic path not to clear. pendingset name format is jobAttemptPath + taskAttemptID.getTaskID() + ".pendingset", and jobAttemptPath is actually jobIdPath not JobAttemptIdPath in s3a magic committer. These pendingset files are overwritted by new task commit.
> But if in new am attempt, a speculative task overcomes origin task, so pendingset file in last attempt may be hold for job commit, the data for commit is wrong



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org