[jira] [Commented] (DRILL-5493) Managed External Sort + CTAS partition by results in "Unable to allocate sv2 vector" error

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053483#comment-16053483 ] 

Paul Rogers commented on DRILL-5493:
------------------------------------

Basics:

{code}
Config: memory limit = 63,161,283, spill file size = 268435456, spill batch size = 8388608, merge limit = 2147483647, merge batch size = 16777216
...
Actual batch schema & sizes {
  col1(type: VARCHAR, std col. size: 54, actual col. size: 7404, total size: 268730368, data size: 242606878, row capacity: 32768, density: -5)
  Records: 32768, Total size: 268,795,904, Gross row width:8203, Net row width:7404, Density:0}
{code}

The two critical numbers are:

{code}
memory limit = 63,161,283
Total size: 268,795,904
{code}

The sort is running in frag:0:0, which is the screen fragment. A downstream batch has created a batch of size 270 MB. But, the sort has been given only 63 MB. Clearly, the sort cannot make progress.

So, this is an expected error given the casual way that Drill currently creates batches of sizes that are completely independent of memory limits applied to operators. This is a bug in the Drill execution engine in general, not a bug in the external sort in particular.

> Managed External Sort + CTAS partition by results in "Unable to allocate sv2 vector" error
> ------------------------------------------------------------------------------------------
>
>                 Key: DRILL-5493
>                 URL: https://issues.apache.org/jira/browse/DRILL-5493
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.10.0
>            Reporter: Rahul Challapalli
>            Assignee: Paul Rogers
>         Attachments: 26ee07bb-81ff-1c10-9003-90510f4b8e1d.sys.drill, drillbit.log
>
>
> Config :
> {code}
> git.commit.id.abbrev=1e0a14c
> No of nodes : 1
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> Assertions Enabled : true
> {code}
> The below query fails during the CTAS phase (the explicit order by in the query runs fine)
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_query` = 17;
> create table dfs.drillTestDir.xsort_ctas4 partition by (col1) as select columns[0] as col1 from (select * from dfs.`/drill/testdata/resource-manager/wide-to-zero` order by columns[0]);
> Error: RESOURCE ERROR: Unable to allocate sv2 buffer
> Fragment 0:0
> [Error Id: 24ae2ec8-ac2a-45c3-b550-43c12764165d on qa-node190.qa.lab:31010] (state=,code=0)
> {code}
> I attached the logs and profiles. The data is too large to attach to a jira.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)