[jira] [Commented] (FLINK-39811) S5cmd effective concurrency capped by static part size for small downloads

["Piotr Nowojski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-39811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085701#comment-18085701 ] 

Piotr Nowojski commented on FLINK-39811:
----------------------------------------

merged commit 8ca97a3 and 8ca97a3^ into apache:master

> S5cmd effective concurrency capped by static part size for small downloads
> --------------------------------------------------------------------------
>
>                 Key: FLINK-39811
>                 URL: https://issues.apache.org/jira/browse/FLINK-39811
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystems
>            Reporter: Efrat Levitan
>            Assignee: Efrat Levitan
>            Priority: Major
>              Labels: pull-request-available
>
> Unlike the upload path (recoverableWriter) which sets {{S3_MULTIPART_MIN_PART_SIZE}} of [5 MiB|https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L93], we don't set any part size for s5cmd so downloads part size fall back to its default of [50MiB|https://github.com/peak/s5cmd/blob/54d6a8a955688f07e5acc40d61f9c42ceac6c33b/command/cp.go#L30].
> As a result, the effective concurrency is capped to batchSize / 50MiB, underutilizing the worker pool for downloaded batchSizes below (50*concurrency MiB)
> - s3 sdk source [ref|https://github.com/aws/aws-sdk-go/blob/070853e88d22854d2355c2543d0958a5f76ad407/service/s3/s3manager/download.go#L329-L338] N(=concurrency) channels are initialized but only (batchsize/partsize) of them are assigned a range to read.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)