[jira] [Created] (DRILL-2178) Change the outgoing record batch size in PartitionSender

["Venki Korukanti (JIRA)" <ji...@apache.org>]

Venki Korukanti created DRILL-2178:
--------------------------------------

             Summary: Change the outgoing record batch size in PartitionSender
                 Key: DRILL-2178
                 URL: https://issues.apache.org/jira/browse/DRILL-2178
             Project: Apache Drill
          Issue Type: Bug
            Reporter: Venki Korukanti
            Assignee: Venki Korukanti


Currently we allocate memory for vectors in partition sender OutgoingRecordBatch using allocateNew() which for most ValueVectors allocates space for 4096 record capacity, but we flush the current record batch as soon as we reach 1000 records causing wasted memory. Automatic resizing kicks in after flushing few record batches, but auto resize always doubles or halves the capacity. This cause the buffer record capacity to flip between  512 and 2048.

This JIRA is to:
1. Decide on the outgoing record batch depending upon the number of receivers of partition sender. Default value is 1024, but when the number of receivers exceeds 1000 change it to 512.
2. Allocate value vector space for storing the outgoing record batch size decided in (1). For this we make use of {{AllocationHelper.allocate(ValueVector v, int valueCount, int bytesPerValue, int repeatedPerTop)}}. {{bytesPerValue}} and {{repeatedPerTop}} is currently hard coded to 50 and 10, but this shouldn't matter as these values are applicable for variable and repeated vectors which have realloc facility if they run out of space.




 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)