[jira] [Commented] (DRILL-6429) SortImpl Should Not Use BufferAllocator.setLenient()

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481337#comment-16481337 ] 

Paul Rogers commented on DRILL-6429:
------------------------------------

Bit of the back-story here. Originally, there was no `setLenient()`. We added it just for sort.

Drill's allocator model enforces strict per-operator limits. Until a year ago, operators made no real attempt to honor the limit; so the limit was the only tool available to prevent run-away operator usage. Of course, the limit was of magical value because, for most operators, it was set at 10 GB...

For sort, we did our best to modify the code to honor those limits for spilling, measuring input batch size and so on. As a result, the operator itself enforces its own limit.

Now, it is nearly impossible to get everything right since the Sort has no control over its input batch size. The result is that when the moon is full and the starts are aligned just right, the Sort will exceed the hard-coded limit by some few percent.

Should we punish the user by killing their query? We decided not to. So, we introduced the lenient mode.

The thought is, since most operators made no attempt at all to limit memory, it makes no sense to punish the user because Sort happened to be off by a few percent.

Given all this, what is the reason to change our thinking now? Can the sort do a better job of predicting memory usage? Does sort now have control over its input batch size?

> SortImpl Should Not Use BufferAllocator.setLenient()
> ----------------------------------------------------
>
>                 Key: DRILL-6429
>                 URL: https://issues.apache.org/jira/browse/DRILL-6429
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Timothy Farkas
>            Assignee: Timothy Farkas
>            Priority: Major
>
> The Sort operator should always obey its memory limit. This is required for resource management.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)