[jira] [Comment Edited] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648544#comment-15648544 ] 

Paul Rogers edited comment on DRILL-5014 at 11/8/16 7:29 PM:
-------------------------------------------------------------

The issue actually seems more complex. ESB seems to have evolved to have several different ways to spill that all interact in complex ways.

On the one hand. ESB tries to operate within a defined memory "budget." In this model, ESB accumulates batches until the memory used exceeds 95% of the allowed maximum. Then, ESB spills half the batches. Simple enough.

The maximum memory comes from two sources: either a limit set by the Foreman or a limit set by the "parent allocator" in the operator tree. (One part of the code uses one limit, another part uses the other limit.)

Then, the two config vars above were added. They create different ideas about limits and when to spill, causing the unexpected behavior described above.

What is needed is a rationalized design. Is it the memory limit or spill counts that determine spilling? What if spill counts conflict with memory? If memory-based spilling conflicts with the spill parameters?


was (Author: paul-rogers):
The issue actually seems more complex. ESB seems to have evolved to have several different ways to spill that all interact in complex ways.

On the one hand. ESB tries to operate within a defined memory "budget." In this model, ESB accumulates batches until the memory used exceeds 95% of the allowed maximum. Then, ESB spills half the batches. Simple enough.

Then, the two config vars above were added. They create different ideas about limits and when to spill, causing the unexpected behavior described above.

What is needed is a rationalized design. Is it the memory limit or spill counts that determine spilling? What if spill counts conflict with memory? If memory-based spilling conflicts with the spill parameters?

> ExternalSortBatch cache size, spill count differs from config setting
> ---------------------------------------------------------------------
>
>                 Key: DRILL-5014
>                 URL: https://issues.apache.org/jira/browse/DRILL-5014
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Priority: Minor
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to remain within a defined memory footprint. Spilling occurs based on a number of factors. Among those are two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to spill per spill event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to accumulate in memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 200 batches, then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill size, **NOT** the number set in the threshold. As a result, the ESB operator will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the batch size (used in spill decisions) matches the actual spill count (as implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)