You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2019/07/15 10:32:00 UTC
[jira] [Resolved] (HADOOP-16428) Distcp don't make use of S3a
Committers, be it magic or staging
[ https://issues.apache.org/jira/browse/HADOOP-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-16428.
-------------------------------------
Resolution: Won't Fix
Release Note:
closing as WONTFIX on the basis that this would be a fundamental rewrite of DistCp, which, as it is a critical path of many workflows, is not something I'd rush to do. You'd do it in its own module/project.
HADOOP-15281 will actually be faster than postponing upload commits until any job commit
> Distcp don't make use of S3a Committers, be it magic or staging
> ---------------------------------------------------------------
>
> Key: HADOOP-16428
> URL: https://issues.apache.org/jira/browse/HADOOP-16428
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools/distcp
> Affects Versions: 3.1.1
> Reporter: Sahil Kaw
> Priority: Minor
>
> Currently, I don't see Distcp make use of S3a Committers, be it Magic or Staging and I have noticed most of the jobs which use MapReduce frameworks use S3 committers except distcp. Distcp makes use of the FileOutputCommitter even if S3a committer parameters are specified in the core-site.xml. Is this by design? If yes, can someone please explain the reason for that. Are there any limitations or potential risks of using S3a committers with Distcp?
> I know there is a "-direct" option that can be used with the FileOutputCommitter in order to avoid renaming while committing fr object Stores. But if anyone can put some light on the current limitation of S3a committers with distcp and reason for choosing FileOutputCommitters for Distcp over S3a committers, it would be helpful. Thanks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org