You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/07/27 14:00:00 UTC
[jira] [Commented] (HADOOP-17628) Distcp contract test is really
slow with ABFS and S3A; timing out
[ https://issues.apache.org/jira/browse/HADOOP-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388082#comment-17388082 ]
Steve Loughran commented on HADOOP-17628:
-----------------------------------------
working on this as its taking so long the ROI will be immediate for us developers
* s3a to to -direct always
* width and depth of dir tree massively reduced
* removed the ContractSecureDistCp because new abfs stores are secure only; superfluous
Side issue, distcp with iterator is tangibly slower on abfs rather than distcp. That's why it was always the testDistCpWithIterator test which times out.
This also implies that other code which uses listIterator &c is underperformant. As it is adopted in more places (spark, iceberg), needs attention.
> Distcp contract test is really slow with ABFS and S3A; timing out
> -----------------------------------------------------------------
>
> Key: HADOOP-17628
> URL: https://issues.apache.org/jira/browse/HADOOP-17628
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/s3, test, tools/distcp
> Affects Versions: 3.4.0
> Reporter: Bilahari T H
> Priority: Minor
>
> The test case testDistCpWithIterator in AbstractContractDistCpTest is consistently timing out.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org