You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2020/11/09 19:30:00 UTC
[jira] [Resolved] (HADOOP-17066) S3A staging committer committing
duplicate files
[ https://issues.apache.org/jira/browse/HADOOP-17066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-17066.
-------------------------------------
Resolution: Duplicate
HADOOP-17318 covers the duplicate job problem everywhere in the committer and, combined with a change in spark, should go away.
This is an intermittent issue as it depends on the timing you launch stages and, for task staging dir conflict, whether two tasks attempts of conflicting jobs are launched at the same time.
> S3A staging committer committing duplicate files
> ------------------------------------------------
>
> Key: HADOOP-17066
> URL: https://issues.apache.org/jira/browse/HADOOP-17066
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0, 3.2.1, 3.1.3
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> SPARK-39111 reporting concurrent jobs double writing files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org