[I] Performance low on AWS S3 deepstorage data pull on fresh historical (druid)

["applike-ss (via GitHub)" <gi...@apache.org>]

applike-ss opened a new issue, #17809:
URL: https://github.com/apache/druid/issues/17809

   My performance is comparibly low when using AWS S3 as my deep storage and then using a fresh historical that doesn't hold any data on its volumes yet.
   I do get around 50MB/s, while when using the aws-cli on the same pod for downloading the complete deep storage folder it is more like 400-500MB/s.
   
   ### Affected Version
   
   v32.0.0
   
   ### Description
   
   Please include as much detailed information about the problem as possible.
   - Cluster size: 
   any
   
   - Configurations in use
   Tried many different ones with these settings changes
   - druid.segmentCache.numLoadingThreads (1-100)
   - xms/xmx/XX:MaxDirectMemorySize java settings
   - druid.segmentCache.numBootstrapThreads (1-100)
   - aws.maxConnections=500 (likely non-existent option)
   - druid.s3.maxConnections=500 (likely non-existent option)
   - druid.server.http.numThreads (60-200)
    
   - Steps to reproduce the problem
   add s3 as deep storage, empty your historical in use, restart it, see slow download speeds 
   
   With these settings I do get very high download speeds with aws-cli, even though it is just python code:
   ```
   $ aws configure set default.s3.multipart_threshold 64MB
   $ aws configure set default.s3.multipart_chunksize 16MB
   $ aws configure set default.s3.max_concurrent_requests 100
   $ aws configure set default.s3.max_queue_size 10000
   $ aws configure set default.s3.max_bandwidth 1024MB/s
   $ aws configure set default.s3.preferred_transfer_client crt #<- most effective change
   ```
   
   Neither CPU nor Memory or Network seem too much utilized during druids segment download, so i don't know where to dig in deeper. Any help is appreciated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [I] Performance low on AWS S3 deepstorage data pull on fresh historical (druid)

["applike-ss (via GitHub)" <gi...@apache.org>]

applike-ss closed issue #17809: Performance low on AWS S3 deepstorage data pull on fresh historical
URL: https://github.com/apache/druid/issues/17809


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [I] Performance low on AWS S3 deepstorage data pull on fresh historical (druid)

["applike-ss (via GitHub)" <gi...@apache.org>]

applike-ss commented on issue #17809:
URL: https://github.com/apache/druid/issues/17809#issuecomment-2736583131

   i was able to fix this by setting `druid.coordinator.loadqueuepeon.http.batchSize` on the coordinator to 10 (unset before).
   
   Furthermore i set `druid.segmentCache.numLoadingThreads=10` and `druid.server.http.numThreads=25` on historicals.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org