[GitHub] [incubator-druid] himanshug opened a new pull request #7919: disable all compression in intermediate segment persists while ingestion

[GitBox <gi...@apache.org>]

himanshug opened a new pull request #7919: disable all compression in intermediate segment persists while ingestion
URL: https://github.com/apache/incubator-druid/pull/7919
 
 
   **Motivation:**
   All Druid ingestion tasks consume excessive amount of off-heap memory during final merge of intermediate persisted segments. A good fraction of that memory goes towards holding many 64K decompression buffers, recently reported in #7824 and I have seen this behavior on many different clusters in the past. Most of these 64K decompression buffers can be avoided if compression was disabled on intermediate persisted segments which is not really required on those intermediate segments but on the final segment that gets pushed.
   
   **Changes Made:**
   This patch disables compression on intermediate persisted segments for all ingestion tasks. However, in case this causes problems, a configuration "indexSpecForIntermediatePersists" could be used inside the task tuningConfig to revert the behavior on specific clusters.
   This config is intentionally undocumented to not complicate user document as I don't foresee it being used really. Most of the changes in patch are actually to add this config.
   
   **Testing:**
   Existing unit and integration tests should cover all changes here.
   
   **Release Notes:**
   This PR must be mentioned in the release notes to inform users about this change in behavior which doesn't change the end result but does change some of the intermediate processing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org