You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "suraj-goel (via GitHub)" <gi...@apache.org> on 2025/01/27 11:40:48 UTC

[I] Add the capability to speed up S3 uploads using AWS transfer manager (druid)

suraj-goel opened a new issue, #17669:
URL: https://github.com/apache/druid/issues/17669

   ### Description
   
   Please describe the feature or change with as much detail as possible. 
   
   If you have a detailed implementation in mind and wish to contribute that implementation yourself, and the change that
   you are planning would require a 'Design Review' tag because it introduces or changes some APIs, or it is large and
   imposes lasting consequences on the codebase, please open a Proposal instead.
   
   * Use [AWS Transfer Manager](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html) to speed up uploads to S3.
   
   
   ### Motivation
   
   Please provide the following for the desired feature or change:
   - A detailed description of the intended use case, if applicable
      - This will improve upload speed to S3.
   - Rationale for why the desired feature/change would be beneficial
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Add the capability to speed up S3 uploads using AWS transfer manager (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on issue #17669:
URL: https://github.com/apache/druid/issues/17669#issuecomment-2617776115

   I was working on something similar a while back. Maybe these changes will help you https://github.com/apache/druid/pull/14834. What I found was that the speed improvements weren't a lot for MSQ because transfer manager in the SDK v1(?) didn't parallelize if the downloads were chunked. It helped a bit when segments were getting downloaded and uploaded. However, for some reason, segments uploaded without the transfer manager were slowly downloaded when downloaded with the transfer manager. 
   As such, it would be really cool if you can benchmark the performance of the changes, along with speedups/slowdown when downloaded and uploaded pre-post your patch. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Add the capability to speed up S3 uploads using AWS transfer manager (druid)

Posted by "suraj-goel (via GitHub)" <gi...@apache.org>.
suraj-goel commented on issue #17669:
URL: https://github.com/apache/druid/issues/17669#issuecomment-2617954468

   > I was working on something similar a while back. Maybe these changes will help you [#14834](https://github.com/apache/druid/pull/14834). What I found was that the speed improvements weren't a lot for MSQ because transfer manager in the SDK v1(?) didn't parallelize if the downloads were chunked. It helped a bit when segments were getting downloaded and uploaded. However, for some reason, segments uploaded without the transfer manager were slowly downloaded when downloaded with the transfer manager. As such, it would be really cool if you can benchmark the performance of the changes, along with speedups/slowdown when downloaded and uploaded pre-post your patch.
   
   @LakshSingla, Thanks for the suggestions. 
   Right now, My changes are focussed on optimizing the uploads.
   I will try to come up with some existing benchmarks or create a new one for transfer manager.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Add the capability to speed up S3 uploads using AWS transfer manager (druid)

Posted by "suraj-goel (via GitHub)" <gi...@apache.org>.
suraj-goel commented on issue #17669:
URL: https://github.com/apache/druid/issues/17669#issuecomment-2615531124

   I will work on this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Add the capability to speed up S3 uploads using AWS transfer manager (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm closed issue #17669: Add the capability to speed up S3 uploads using AWS transfer manager
URL: https://github.com/apache/druid/issues/17669


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org