You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/06/25 19:20:00 UTC
[jira] [Commented] (HADOOP-15384) distcp numListstatusThreads
option doesn't get to -delete scan
[ https://issues.apache.org/jira/browse/HADOOP-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522708#comment-16522708 ]
Steve Loughran commented on HADOOP-15384:
-----------------------------------------
Patch 001: passes thread count down, logs it, and splits time for reporting listing into source and dest
This has tangible improvements when using object stores as a destination, though the mimicking of directories trees can still make distcp to some remote stores (s3, swift) still tangibly awful. This is all you can do short of a complete rewrite. Which I don't intend to propose for the following reason: distcp is a complex and critical part of too many people's workflows.
Testing: ran {{ITestS3AContractDistCp}} against S3 Ireland. No new tests, as the distcp contract tests were already setting num threads: I have set the thread count to 40 though, that being the maximum.
Here's the output of the relevant phase of {{testUpdateDeepDirectoryStructureToRemote}}
{code}
2018-06-25 19:35:18,047 [Thread-139] INFO mapred.CopyCommitter (CopyCommitter.java:deleteMissing(387)) - -delete option is enabled. About to remove entries from target that are missing in source
2018-06-25 19:35:18,062 [Thread-139] INFO mapred.CopyCommitter (CopyCommitter.java:deleteMissing(396)) - Source listing completed in 0:00:00.015
2018-06-25 19:35:18,063 [Thread-139] INFO mapred.CopyCommitter (CopyCommitter.java:listTargetFiles(554)) - Scanning destination directory s3a://hwdev-steve-ireland-new/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir with thread count: 40
2018-06-25 19:35:19,872 [Thread-139] INFO tools.SimpleCopyListing (SimpleCopyListing.java:printStats(608)) - Paths (files+dirs) cnt = 11; dirCnt = 5
2018-06-25 19:35:19,872 [Thread-139] INFO tools.SimpleCopyListing (SimpleCopyListing.java:doBuildListing(402)) - Build file listing completed.
2018-06-25 19:35:19,886 [Thread-139] INFO tools.DistCp (CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 19:35:19,899 [Thread-139] INFO tools.DistCp (CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 19:35:19,913 [Thread-139] INFO mapred.CopyCommitter (CopyCommitter.java:deleteMissing(415)) - Destination listing completed in 0:00:01.851
{code}
And for {{ITestAzureNativeContractDistCp}}
{code}
2018-06-25 20:11:44,992 INFO [Thread-147]: mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2018-06-25 20:11:44,992 INFO [Thread-147]: mapred.CopyCommitter (CopyCommitter.java:concatFileChunks(210)) - concat file chunks ...
2018-06-25 20:11:45,405 INFO [Thread-147]: mapred.CopyCommitter (CopyCommitter.java:deleteMissing(387)) - -delete option is enabled. About to remove entries from target that are missing in source
2018-06-25 20:11:45,418 INFO [Thread-147]: mapred.CopyCommitter (CopyCommitter.java:deleteMissing(396)) - Source listing completed in 0:00:00.013
2018-06-25 20:11:45,419 INFO [Thread-147]: mapred.CopyCommitter (CopyCommitter.java:listTargetFiles(554)) - Scanning destination directory wasb://contract@contender.blob.core.windows.net/test/ITestAzureNativeContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/outputDir/inputDir with thread count: 40
2018-06-25 20:11:46,338 INFO [Thread-147]: tools.SimpleCopyListing (SimpleCopyListing.java:printStats(608)) - Paths (files+dirs) cnt = 11; dirCnt = 5
2018-06-25 20:11:46,338 INFO [Thread-147]: tools.SimpleCopyListing (SimpleCopyListing.java:doBuildListing(402)) - Build file listing completed.
2018-06-25 20:11:46,351 INFO [Thread-147]: tools.DistCp (CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 20:11:46,365 INFO [Thread-147]: tools.DistCp (CopyListing.java:buildListing(94)) - Number of paths in the copy list: 11
2018-06-25 20:11:46,377 INFO [Thread-147]: mapred.CopyCommitter (CopyCommitter.java:deleteMissing(415)) - Destination listing completed in 0:00:00.959
{code}
A small bird fell out the sky, deceased, during the S3A run. It didn't happen on a rerun —I'm assuming unrelated. If more wild animals die during S3 integration tests then it'd be something to consider a significant regression in the AWS SDK
+ [~ehiggs], [~fabbri]
> distcp numListstatusThreads option doesn't get to -delete scan
> --------------------------------------------------------------
>
> Key: HADOOP-15384
> URL: https://issues.apache.org/jira/browse/HADOOP-15384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: tools/distcp
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15384-001.patch
>
>
> The distcp {{numListstatusThreads}} option isn't used when configuring the GlobbedCopyListing used in {{CopyComitter.deleteMissing()}}
> This means that for large scans of object stores, performance is significantly worse.
> Fix: pass the option down from the task conf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org