[GitHub] Ben-Zvi opened a new pull request #1438: DRILL-6566: Reduce Hash Agg Batch size and estimate when low available memory

[GitBox <gi...@apache.org>]

Ben-Zvi opened a new pull request #1438: DRILL-6566: Reduce Hash Agg Batch size and estimate when low available memory
URL: https://github.com/apache/drill/pull/1438
 
 
   (1) First commit just renamed MAX_BATCH_SIZE to MAX_BATCH_ROW_COUNT in order to avoid confusion over "size".
   (2) 2nd commit: Addressing two issues: The configuration batch size (default 16M) is taken as is (by the memory manager). And the (outgoing) batch size estimates are created early (when the outgoing is empty), based on 64K rows per batch.
      The change: Taking the Hash-Agg memory limit into account, and planning for multiple batches, the configured size (e.g. 16M) may be reduced to allow for the needed number of batches (this new size is given to the memory manager).
   Later when the estimates are made, that (possibly reduced) size is used to reduce the estimates, if needed.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services