You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by "Padma Penumarthy (JIRA)" <ji...@apache.org> on 2018/01/26 22:21:00 UTC

[jira] [Created] (DRILL-6113) Limit batch size for Merge Receiver

Padma Penumarthy created DRILL-6113:
---------------------------------------

             Summary: Limit batch size for Merge Receiver
                 Key: DRILL-6113
                 URL: https://issues.apache.org/jira/browse/DRILL-6113
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Flow
    Affects Versions: 1.12.0
            Reporter: Padma Penumarthy
            Assignee: Padma Penumarthy
             Fix For: 1.13.0


Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of varying width, it is difficult to predict the output batch size (in terms of memory) for this operator. Change this to derive row count based on actual memory available. We are introducing a new option called outputBatchSize to limit the batch size of each operator. Use the memory configured from that. Figure out the average row width of outgoing batch based on averages of batches coming from incoming streams. Limit the row count based on memory available to use and average row width.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)