You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/03/03 16:08:51 UTC

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818809522



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Not sure, this might be another issue for multipart copies. Threshold to use multipart copy vs atomic copy is 5GB, using the upload buffer size (defaults to 5MB or 64MB depending on the environment) doesn't seem to make sense, right? That generates far more parts than necessary. Thoughts?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org