[jira] [Created] (DRILL-7801) Changing the scope of output_batch_size

["Artem Trush (Jira)" <ji...@apache.org>]

Artem Trush created DRILL-7801:
----------------------------------

             Summary: Changing the scope of output_batch_size
                 Key: DRILL-7801
                 URL: https://issues.apache.org/jira/browse/DRILL-7801
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.17.0, 1.16.0, 1.15.0, 1.14.0
            Reporter: Artem Trush
            Assignee: Artem Trush
             Fix For: 1.18.0


{{*Drill.exec.memory.operator.output_batch_size*}} parameter caused problems with the execution speed of certain queries, in particular, it led to situations where the number of batch was equal to the number of records, such as 99890 batch and 99890 records.

After comparing drill 1.13, where the query is executed in a few minutes, and 1.16, where the query is executed in a few hours, I came to the following conclusion.
The problem is in the formation of the size of the record batch transmitted between operators.


For example, lets take a look at *{{ProjectRecordBatch}}* .
We have incoming batch that comes from another operator with 2000 records inside.

*Drill 1.13*
We have function *_doWork_*. There is simple logic inside. This function is calling every time when we have incoming batch in Project operator.
In a few words outgoing batch size depends on just incoming batch size. And in most cases value of outgoing batch size equal to incoming batch size. So 2000 will come, 2000 will go.
{{}}
{code:java}
final int outputRecords = projector.projectRecords(0, incomingRecordCount, 0);{code}
As we can see outputRecords depends just on incomingRecords.

*Drill 1.16*
Now we have a memoryManager which takes as parameter our option outgoing_batch_size.
Lets look at this function doWork again.
Firstly, we got this
{{}}
{code:java}
//calculate the output row count
memoryManager.update();{code}
Inside we have
( getOutputBatchSize() is our config and batchSizer.rowCount() is incoming batch size)
{code:java}
//if rowWidth is not zero, set the output row count in the sizer
setOutputRowCount(getOutputBatchSize(), rowWidth);
// if more rows can be allowed than the incoming row count, then set the
// output row count to the incoming row count.
outPutRowCount = Math.min(getOutputRowCount(), batchSizer.rowCount()); {code}
Back to the function _*doWork*_.
memoryManager.update() fills variable called maxOuputRecordCount. Here
{{}}
{code:java}
int maxOuputRecordCount = memoryManager.getOutputRowCount();{code}

And the main difference between 13 and 16 with using a new parameter
{code:java}
final int outputRecords = projector.projectRecords(this.incoming,0, maxOuputRecordCount, 0); 
{code}
If maxOutputRecordCount smaller than incomingBatch size, then number of outputRecords will decrease and the number of batches will increase. So will come 2000, will go 600 600 600 .. or another value depends of output_batch_size.

As you could see in both cases the number of output batches is always not bigger than number of incoming batches. And the same rule is following in every operator with memoryManager.


This leads to a situation where the number of batches grows excessively. A simple solution to this problem is to increase the value for *{{drill.exec.memory.operator.output_batch_size}}* . However, because this option is set at the system level, changing it results in *{{RESOURCE ERROR: One or more nodes ran out of memory}}* in other queries.

My suggestion is to change the scope of *{{drill.exec.memory.operator.output_batch_size}}* from system to system and session. Which will allow you to increase this option only for problematic requests, without affecting the work of all others. As for me I don't see any reason to prevent this change. If you have any information about the negative effects of changing the scope of this parameter, please share it.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)