[jira] [Work logged] (HADOOP-17628) Distcp contract test is really slow with ABFS and S3A; timing out

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-17628?focusedWorklogId=628478&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628478 ]

ASF GitHub Bot logged work on HADOOP-17628:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 27/Jul/21 14:14
            Start Date: 27/Jul/21 14:14
    Worklog Time Spent: 10m 
      Work Description: steveloughran opened a new pull request #3240:
URL: https://github.com/apache/hadoop/pull/3240


   
   Changes
   
   * subclass can declare whether or not -direct should be default
   * all tests then switch to that
   * major cut back on default depth/width of directories
   * if you set the file size of "scale.test.distcp.file.size.kb" to 0
     the large file test case is skipped.
   * aggressive cutback on all needless mkdir() calls.
   
   S3A suite
   * declares -direct always
   
   ABFS suites
   * deletes superfluous ITestAbfsFileSystemContractSecureDistCp
   * uses abfs scale test timeout
   * only runs with -Dscale
   
   All these changes bring execution time down from 4 min to 2 min against each store.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 628478)
    Remaining Estimate: 0h
            Time Spent: 10m

> Distcp contract test is really slow with ABFS and S3A; timing out
> -----------------------------------------------------------------
>
>                 Key: HADOOP-17628
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17628
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure, fs/s3, test, tools/distcp
>    Affects Versions: 3.4.0
>            Reporter: Bilahari T H
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The test case testDistCpWithIterator in AbstractContractDistCpTest is consistently timing out.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org