[jira] [Comment Edited] (DRILL-5453) Managed External Sort : Sorting on a lot of columns is taking unreasonably long time

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067605#comment-16067605 ] 

Paul Rogers edited comment on DRILL-5453 at 6/29/17 2:37 AM:
-------------------------------------------------------------

This bug is a tough one. There is certainly a bug, but it is not in the external sort. It is, instead, either a user error, or an unfortunate consequence of how the text reader works.

When the query runs, we get this error:

{code}
  Records: 8096, Total size: 678,068,224, Data size: 389425696, Gross row width: 83754, Net row width: 48101, Density: 1}
{code}

Not sure how this worked on the test machine with the memory assigned.

The memory per node is only 482,344,960. Even if the sort is given the entire amount (which it is, because the query is single-fragment), one cannot fit 678 MB of data in a 482 MB container. Given that, the very next allocation (the SV2) fails.

There is nothing the sort can do in this case. The solution is to modify the scan operator to create much smaller record batches. That solution is coming in the form of DRILL-5211. So, I've associated this ticket with that project.



was (Author: paul-rogers):
This bug is a tough one. There is certainly a bug, but it is not in the external sort. It is, instead, either a user error, or an unfortunate consequence of how the text reader works.

When the query runs, we get this error:

{code}
  Records: 8096, Total size: 678,068,224, Data size: 389425696, Gross row width: 83754, Net row width: 48101, Density: 1}
{code}

The memory per node is only 482,344,960. Even if the sort is given the entire amount (which it is, because the query is single-fragment), one cannot fit 678 MB of data in a 482 MB container. Given that, the very next allocation (the SV2) fails.

There is nothing the sort can do in this case. The solution is to modify the scan operator to create much smaller record batches. That solution is coming in the form of DRILL-5211. So, I've associated this ticket with that project.


> Managed External Sort : Sorting on a lot of columns is taking unreasonably long time
> ------------------------------------------------------------------------------------
>
>                 Key: DRILL-5453
>                 URL: https://issues.apache.org/jira/browse/DRILL-5453
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.10.0
>            Reporter: Rahul Challapalli
>            Assignee: Paul Rogers
>         Attachments: drill5453.sys.drill
>
>
> The below query ran for ~16hrs before I cancelled it.
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.memory.max_query_memory_per_node` = 482344960;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select count(*) from (select * from dfs.`/drill/testdata/resource-manager/3500cols.tbl` order by columns[450],columns[330],columns[230],columns[220],columns[110],columns[90],columns[80],columns[70],columns[40],columns[10],columns[20],columns[30],columns[40],columns[50], columns[454],columns[413],columns[940],columns[834],columns[73],columns[140],columns[104],columns[2222],columns[30],columns[2420],columns[1520], columns[1410], columns[1110],columns[1290],columns[2380],columns[705],columns[45],columns[1054],columns[2430],columns[420],columns[404],columns[3350], columns[3333],columns[153],columns[356],columns[84],columns[745],columns[1450],columns[103],columns[2065],columns[343],columns[3420],columns[530], columns[3210] ) d where d.col433 = 'sjka skjf';
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {code}
> The data set and the logs are too large to attach to a jira. But below is a description of the data
> {code}
> No of records : 1,000,000
> No of columns : 3500
> Length of each column : < 50
> {code}
> The profile is attached and I will give my analysis on why I think its an un-reasonable amount of time soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)