[jira] [Commented] (DRILL-5474) Spill directory is not being cleaned up immediately after cancellation

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083079#comment-16083079 ] 

Paul Rogers commented on DRILL-5474:
------------------------------------

The stack trace shown is a specific case of DRILL-5060. The code which writes a spill file does not check if the query has been cancelled. This behavior has existed in the original "unmanaged" sort as well as the "managed" version.

When the sort succeeds, it deletes the spill files immediately before returning {{NONE}}. When the sort (or query) fails, the sort relies on the {{FragementExecutor}} to call the {{close()}} method which will delete the spill files.

> Spill directory is not being cleaned up immediately after cancellation
> ----------------------------------------------------------------------
>
>                 Key: DRILL-5474
>                 URL: https://issues.apache.org/jira/browse/DRILL-5474
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.10.0
>            Reporter: Rahul Challapalli
>            Assignee: Paul Rogers
>
> I allowed the below query to run until it stats spilling. Now I cancelled the query and the sqlline prompt returned immediately. From this point, it took ~50 seconds for this spill directory to be deleted and during this time, the spill file constantly grew in size
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.disable_exchanges` = true;
> alter session set `planner.width.max_per_query` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 1052428800;
> alter session set `planner.enable_decimal_data_type` = true;
> select count(*) from (
>   select * from dfs.`/drill/testdata/resource-manager/all_types_large` d1
>   order by d1.map.missing, d1.missing12.x, d1.missing1, d1.missing2, d1.missing3, d1.missing4,
>     d1.missing5, d1.missing6, d1.missing7, d1.missing8, d1.missing9, d1.missing10, d1.missing11,
>     d1.missing12.x, d1.missing13, d1.missing14, d1.missing15, d1.missing16, d1.missing17, d1.missing18,
>     d1.missing19, d1.missing20, d1.missing21, d1.missing22, d1.missing23, d1.missing24, d1.missing25,
>     d1.missing26, d1.missing27, d1.missing28, d1.`missing 29`, d1.missing30, d1.missing31, d1.missing32,
>     d1.missing33, d1.missing34, d1.m1
> ) d where d.missing3 is false;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)