You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "KKcorps (via GitHub)" <gi...@apache.org> on 2026/10/26 11:20:31 UTC
[PR] Upload S3 multipart parts in parallel [pinot]
KKcorps opened a new pull request, #20044:
URL: https://github.com/apache/pinot/pull/20044
`S3PinotFS` sends multipart parts one at a time, so a large file uploads at single-stream speed. In our case that was a 14 GB upsert snapshot uploaded from a minion.
This adds `multiPartUploadParallelism` (default `1`, which keeps today's sequential behaviour). With a higher value, parts go out on a bounded thread pool that lives only for that upload.
Each part reads its own byte range of the file, so the SDK can reopen it on a retry instead of relying on a shared stream. Part uploads now also refresh credentials on 401 or 403, and only one refresh happens across parallel parts. A failed part cancels the remaining parts and aborts the upload.
Tested with a new mock-based `S3PinotFSMultipartUploadTest` at parallelism 1, 4 and 32, which checks part order, byte ranges and abort on failure. The s3mock `testMultiPartUpload` now runs with parallelism 2. I could not run it locally because there is no Docker here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upload S3 multipart parts in parallel [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20044:
URL: https://github.com/apache/pinot/pull/20044#issuecomment-6417105770
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/20044?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 59.11%. Comparing base ([`c5b8107`](https://app.codecov.io/gh/apache/pinot/commit/c5b8107fa6a0e0bcf7f1b401cc2aa827a7e32e74?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`71228b8`](https://app.codecov.io/gh/apache/pinot/commit/71228b8a99f2e5dcf084b20804f1cfb78c0a54eb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20044 +/- ##
=============================================
+ Coverage 41.85% 59.11% +17.25%
+ Complexity 1518 1 -1517
=============================================
Files 3558 2743 -815
Lines 236346 173346 -63000
Branches 37954 28615 -9339
=============================================
+ Hits 98924 102474 +3550
+ Misses 128630 62225 -66405
+ Partials 8792 8647 -145
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.11% <ø> (+17.25%)` | :arrow_up: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.11% <ø> (+17.25%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.11% <ø> (+17.26%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `59.11% <ø> (?)` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/20044/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/20044?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upload S3 multipart parts in parallel [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps closed pull request #20044: Upload S3 multipart parts in parallel
URL: https://github.com/apache/pinot/pull/20044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org