You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/03/02 16:34:10 UTC

[GitHub] [beam] spatel11 opened a new pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

spatel11 opened a new pull request #16987:
URL: https://github.com/apache/beam/pull/16987


   This PR fixes the AWS2 S3FileSystem multipart copy.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [x] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819788258



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Thanks for helping me!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058145488


   Hmm, yes, something went wrong... I'll have a look.
   Generally I'd recommend doing `git rebase -i master` on your branch (having pulled the latest master before), and then squash all consecutive commits by marking the respective lines with `f` (fix).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818825768



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Mmm.  Yes I think you're right.  This link here also seems to back up this idea: https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html
   
   > If objects are PUT using a multipart upload, it’s a good practice to GET them in the same part sizes (or at least aligned to part boundaries) for best performance.
   
   Since the threshold for multipartCopy is 5GB, we could use that in place of the `uploadBufferSizeBytes` (in both aws1 and aws2 filesystems).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819738296



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Yep, my test job that was failing before got done a few minutes ago.
   
   This particular job produced 75 files ranging from 3.6 - 5.5 gb (so it's exercising both copy branches).
   
   All files were copied from the temp location successfully (as opposed to before when it would fail on the multipart copy).
   
   Furthermore, the files seemed structurally ok (I crawled them using AWS Glue and ran some simple queries).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058141982


   Mmm.  Now it reads like I have made a bunch of extra changes . . . 
   
   I feel like I may have done something incorrect
   
   I did the following things:
   squash:
   ```
   git reset $(git merge-base master $(git branch --show-current))
   git add -A
   git commit -m "BEAM-14011 squashed"
   ```
   
   rebase:
   ```
   git rebase master
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aromanenko-dev commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819792771



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Perhaps, it would be useful to make it possible to configure the size of a part with multipart copy with CLI options (if not yet, sorry, I didn't check).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/06e7c201dbdd8fe37d308b0bad2b1684e85e1dc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (06e7c20) will **decrease** coverage by `0.02%`.
   > The diff coverage is `75.86%`.
   
   > :exclamation: Current head f209267 differs from pull request most recent head af74d39. Consider uploading reports for the commit af74d39 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16987      +/-   ##
   ==========================================
   - Coverage   73.83%   73.80%   -0.03%     
   ==========================================
     Files         664      663       -1     
     Lines       87272    87136     -136     
   ==========================================
   - Hits        64434    64314     -120     
   - Misses      21735    21738       +3     
   + Partials     1103     1084      -19     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (ø)` | |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (-1.17%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (ø)` | |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (ø)` | |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | `40.42% <0.00%> (-18.44%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | `25.67% <0.00%> (-13.52%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <0.00%> (-0.55%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [7 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [06e7c20...af74d39](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818814475



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Btw, here's docs for respective AWS limits: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
   5MB is the lower limit, so things will certainly work. But performance might suffer




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819818393



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Yes, I had the same thought but didn't wanna expand the scope of this fix. It's not yet possible to do that using S3Options. But I think it makes sense to add that similar to the config of uploadBufferSize.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r817882143



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -515,15 +516,17 @@ CompleteMultipartUploadResponse multipartCopy(
     if (objectSize == 0) {
       final UploadPartCopyRequest uploadPartCopyRequest =
           UploadPartCopyRequest.builder()
-              .bucket(sourcePath.getBucket())
-              .key(sourcePath.getKey())
+              .bucket(destinationPath.getBucket())
+              .key(destinationPath.getKey())

Review comment:
       It was suggested on the jira to use the non deprecated api, so I'll make an update shortly




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058184094


   I _think_ I got it this time.  My apologies.  Typically I don't squash the branch, I just do a squash merge to master.  This is the first time I've used rebase 😬 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aromanenko-dev merged pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #16987:
URL: https://github.com/apache/beam/pull/16987


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/3cd1f7f949bd476abb11bdb0b368a2f12a496cd1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3cd1f7f) will **increase** coverage by `0.00%`.
   > The diff coverage is `75.86%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #16987   +/-   ##
   =======================================
     Coverage   73.80%   73.80%           
   =======================================
     Files         663      663           
     Lines       87056    87136   +80     
   =======================================
   + Hits        64251    64314   +63     
   - Misses      21721    21738   +17     
     Partials     1084     1084           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (-0.27%)` | :arrow_down: |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (ø)` | |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (-1.21%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (+7.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (+0.21%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3cd1f7f...f209267](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r817889943



##########
File path: CHANGES.md
##########
@@ -116,6 +116,8 @@
 
 ## Bugfixes
 
+* Fix S3 copy for large objects ([BEAM-14011](https://issues.apache.org/jira/browse/BEAM-14011))

Review comment:
       This is for beam 2.37, which as I understand it is already cut/in progress.  So this will have to move to the 2.38 section




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aromanenko-dev edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
aromanenko-dev edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058204579


   @spatel11 Thanks, not a problem! In Beam, we tend to avoid the "merge master to feature branch" commits and make PR commits atomic as much as it possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aromanenko-dev commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058204579


   @spatel11 Thanks, not a problem! In Beam, we tend to avoid a merge master to feature branch commits and make PR commits atomic as much as it possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818960140



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       updated to use the MAX_COPY_OBJECT_SIZE_BYTES instead of the upload buffer size




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r817901191



##########
File path: CHANGES.md
##########
@@ -116,6 +116,8 @@
 
 ## Bugfixes
 
+* Fix S3 copy for large objects ([BEAM-14011](https://issues.apache.org/jira/browse/BEAM-14011))

Review comment:
       moved.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r817893443



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -515,15 +516,17 @@ CompleteMultipartUploadResponse multipartCopy(
     if (objectSize == 0) {
       final UploadPartCopyRequest uploadPartCopyRequest =
           UploadPartCopyRequest.builder()
-              .bucket(sourcePath.getBucket())
-              .key(sourcePath.getKey())
+              .bucket(destinationPath.getBucket())
+              .key(destinationPath.getKey())

Review comment:
       this has been done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819472165



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       I can't find any good recommendations regarding part sizes of multipart copy... my gut feeling is that using the max possible we're on the other far end.  But it should be fine.
   Have you been able to run some tests against S3 @spatel11?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/06e7c201dbdd8fe37d308b0bad2b1684e85e1dc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (06e7c20) will **decrease** coverage by `0.02%`.
   > The diff coverage is `75.86%`.
   
   > :exclamation: Current head f209267 differs from pull request most recent head af74d39. Consider uploading reports for the commit af74d39 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16987      +/-   ##
   ==========================================
   - Coverage   73.83%   73.80%   -0.03%     
   ==========================================
     Files         664      663       -1     
     Lines       87272    87136     -136     
   ==========================================
   - Hits        64434    64314     -120     
   - Misses      21735    21738       +3     
   + Partials     1103     1084      -19     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (ø)` | |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (-1.17%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (ø)` | |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (ø)` | |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | `40.42% <0.00%> (-18.44%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | `25.67% <0.00%> (-13.52%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <0.00%> (-0.55%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [7 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [06e7c20...af74d39](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818809522



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Not sure, this might be another issue for multipart copies. Threshold to use multipart copy vs atomic copy is 5GB, using the upload buffer size (defaults to 5MB or 64MB depending on the environment) doesn't seem to make sense, right? That generates far more parts than necessary. Thoughts?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/3cd1f7f949bd476abb11bdb0b368a2f12a496cd1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3cd1f7f) will **increase** coverage by `0.00%`.
   > The diff coverage is `75.86%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #16987   +/-   ##
   =======================================
     Coverage   73.80%   73.80%           
   =======================================
     Files         663      663           
     Lines       87056    87136   +80     
   =======================================
   + Hits        64251    64314   +63     
   - Misses      21721    21738   +17     
     Partials     1084     1084           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (-0.27%)` | :arrow_down: |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (ø)` | |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (-1.21%)` | :arrow_down: |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (+7.14%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (+0.21%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <0.00%> (-1.71%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [1 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3cd1f7f...f209267](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/06e7c201dbdd8fe37d308b0bad2b1684e85e1dc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (06e7c20) will **decrease** coverage by `0.02%`.
   > The diff coverage is `75.86%`.
   
   > :exclamation: Current head f209267 differs from pull request most recent head 133f569. Consider uploading reports for the commit 133f569 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16987      +/-   ##
   ==========================================
   - Coverage   73.83%   73.80%   -0.03%     
   ==========================================
     Files         664      663       -1     
     Lines       87272    87136     -136     
   ==========================================
   - Hits        64434    64314     -120     
   - Misses      21735    21738       +3     
   + Partials     1103     1084      -19     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (ø)` | |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (-1.17%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (ø)` | |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (ø)` | |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | `40.42% <0.00%> (-18.44%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | `25.67% <0.00%> (-13.52%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <0.00%> (-0.55%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [7 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [06e7c20...133f569](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1057127968


   R: @aromanenko-dev R: @mosche 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058150615


   @spatel11 That should work, please do a `git rebase -i master` on your branch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819753338



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Awesome, great to hear!
   Thanks so much for your contribution @spatel11 .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] spatel11 commented on a change in pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r819738296



##########
File path: sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
                         Math.min(objectSize - 1, bytePosition + uploadBufferSizeBytes - 1)))

Review comment:
       Yep, my test job that was failing before got done a few minutes ago.
   
   This particular job produced 75 files ranging from 3.6 - 5.5 gb (so it's exercising both copy branches).
   
   All files were copied from the temp location successfully (as opposed to before when it would fail on the multipart copy).  And on the files larger than 5gb, I can see the -partnumber on the entity tag, indicating a multipart object.
   
   Furthermore, the files seemed structurally ok (I crawled them using AWS Glue and ran some simple queries).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058059257


   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #16987:
URL: https://github.com/apache/beam/pull/16987#issuecomment-1058159732


   # [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#16987](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f209267) into [master](https://codecov.io/gh/apache/beam/commit/06e7c201dbdd8fe37d308b0bad2b1684e85e1dc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (06e7c20) will **decrease** coverage by `0.02%`.
   > The diff coverage is `75.86%`.
   
   > :exclamation: Current head f209267 differs from pull request most recent head 45c8464. Consider uploading reports for the commit 45c8464 to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/16987/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #16987      +/-   ##
   ==========================================
   - Coverage   73.83%   73.80%   -0.03%     
   ==========================================
     Files         664      663       -1     
     Lines       87272    87136     -136     
   ==========================================
   - Hits        64434    64314     -120     
   - Misses      21735    21738       +3     
   + Partials     1103     1084      -19     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.63% <82.79%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.45% <40.00%> (ø)` | |
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `60.37% <47.82%> (-1.17%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.02% <77.77%> (ø)` | |
   | [...s/interactive/dataproc/dataproc\_cluster\_manager.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kYXRhcHJvYy9kYXRhcHJvY19jbHVzdGVyX21hbmFnZXIucHk=) | `88.57% <85.13%> (ø)` | |
   | [sdks/python/apache\_beam/io/filesystem.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZXN5c3RlbS5weQ==) | `89.41% <100.00%> (ø)` | |
   | [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `81.39% <100.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | `40.42% <0.00%> (-18.44%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | `25.67% <0.00%> (-13.52%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <0.00%> (-0.55%)` | :arrow_down: |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.14% <0.00%> (-0.50%)` | :arrow_down: |
   | ... and [7 more](https://codecov.io/gh/apache/beam/pull/16987/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [06e7c20...45c8464](https://codecov.io/gh/apache/beam/pull/16987?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org