[jira] [Updated] (DRILL-6113) Limit batch size for Merge Receiver

["Pritesh Maker (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pritesh Maker updated DRILL-6113:
---------------------------------
    Fix Version/s:     (was: 1.14.0)

> Limit batch size for Merge Receiver
> -----------------------------------
>
>                 Key: DRILL-6113
>                 URL: https://issues.apache.org/jira/browse/DRILL-6113
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Flow
>    Affects Versions: 1.12.0
>            Reporter: Padma Penumarthy
>            Assignee: Padma Penumarthy
>            Priority: Major
>
> Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of varying width, it is difficult to predict the output batch size (in terms of memory) for this operator. Change this to derive row count based on actual memory available. We are introducing a new option called outputBatchSize to limit the batch size of each operator. Use the memory configured from that. Figure out the average row width of outgoing batch based on averages of batches coming from incoming streams. Limit the row count based on memory available to use and average row width.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)