[GitHub] [pinot] jasperjiaguo commented on issue #8910: High Disk Utilization when Reloading Segments for a Large Table

[GitBox <gi...@apache.org>]

jasperjiaguo commented on issue #8910:
URL: https://github.com/apache/pinot/issues/8910#issuecomment-1158173016

   Hello @ankitsultana imiting concurrency and throttling are both implemented. You can take a look at this feature https://github.com/apache/pinot/pull/8753 and also this one https://github.com/apache/pinot/pull/8694. Basically what we do here is:
   
   https://github.com/apache/pinot/pull/8753:
   1. When your segment is **unencrypted**, stream download and untar, so that server wouldn't write the tar ball to disk, as will save around 1/3 of bytes written to disk.
   config `pinot.server.instance.segment.stream.download.untar`
   2. When the above feature is on, control download rate per segment , in bytes/sec
   config `pinot.server.instance.segment.stream.download.untar.rate.limit.bytes.per.sec`
   
   https://github.com/apache/pinot/pull/8694:
   At the table level control the number of segments downloading in parallel.
   `pinot.server.instance.table.level.max.parallel.segment.downloads`
   
   These two features work **independently**. But when they are configured together, you essentially have the ability to control the per table level download rate, and avoid saturating the disk. Please let me know if you have other questions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org