[jira] [Commented] (DRILL-5083) RecordIterator can sometimes restart a query on close

["Roman Kulyk (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173529#comment-16173529 ] 

Roman Kulyk commented on DRILL-5083:
------------------------------------

[~khfaraaz],
Sorry for the late answer I missed your message. 

To reproduce this issue I used the next env:
1 node cluster (16G RAM, 2 cores)

I used tpcds_sf100 2 query:

{code:sql}
WITH wscs AS (SELECT sold_date_sk, sales_price FROM (SELECT ws_sold_date_sk sold_date_sk, ws_ext_sales_price sales_price FROM web_sales) UNION ALL (SELECT cs_sold_date_sk sold_date_sk, cs_ext_sales_price sales_price FROM catalog_sales)), wswscs AS (SELECT d_week_seq, Sum(CASE WHEN ( d_day_name = 'Sunday' ) THEN sales_price ELSE NULL END) sun_sales, Sum(CASE WHEN ( d_day_name = 'Monday' ) THEN sales_price ELSE NULL END) mon_sales, Sum(CASE WHEN ( d_day_name = 'Tuesday' ) THEN sales_price ELSE NULL END) tue_sales, Sum(CASE WHEN ( d_day_name = 'Wednesday' ) THEN sales_price ELSE NULL END) wed_sales, Sum(CASE WHEN ( d_day_name = 'Thursday' ) THEN sales_price ELSE NULL END) thu_sales, Sum(CASE WHEN ( d_day_name = 'Friday' ) THEN sales_price ELSE NULL END) fri_sales, Sum(CASE WHEN ( d_day_name = 'Saturday' ) THEN sales_price ELSE NULL END) sat_sales FROM wscs, date_dim WHERE d_date_sk = sold_date_sk GROUP BY d_week_seq) SELECT d_week_seq1, Round(sun_sales1 / sun_sales2, 2), Round(mon_sales1 / mon_sales2, 2), Round(tue_sales1 / tue_sales2, 2), Round(wed_sales1 / wed_sales2, 2), Round(thu_sales1 / thu_sales2, 2), Round(fri_sales1 / fri_sales2, 2), Round(sat_sales1 / sat_sales2, 2) FROM (SELECT wswscs.d_week_seq d_week_seq1, sun_sales sun_sales1, mon_sales mon_sales1, tue_sales tue_sales1, wed_sales wed_sales1, thu_sales thu_sales1, fri_sales fri_sales1, sat_sales sat_sales1 FROM wswscs, date_dim WHERE date_dim.d_week_seq = wswscs.d_week_seq AND d_year = 1998) y, (SELECT wswscs.d_week_seq d_week_seq2, sun_sales sun_sales2, mon_sales mon_sales2, tue_sales tue_sales2, wed_sales wed_sales2, thu_sales thu_sales2, fri_sales fri_sales2, sat_sales sat_sales2 FROM wswscs, date_dim WHERE date_dim.d_week_seq = wswscs.d_week_seq AND d_year = 1998 + 1) z WHERE d_week_seq1 = d_week_seq2 - 53 ORDER BY d_week_seq1 limit 1;
{code}

on views which were created from the next dataset (all files were in parquet format):
{code:xml}
hadoop fs -du -h /drill/testdata/
1.4 G    /drill/testdata/catalog_sales
22.4 M   /drill/testdata/date_dim
734.0 M  /drill/testdata/web_sales
{code}


To get a reproduce you should change the next properties before you run the query:

{code:sql}
alter session set `planner.enable_hashjoin` = false;
alter session set `planner.enable_hashagg` = false;
alter session set `planner.enable_mergejoin` = true;
alter session set `planner.memory.max_query_memory_per_node` = 1048576;
{code}

After that, you should run the query (tpcds_sf100 2) and interrupt it with "Ctrl + c" at the right moment. There are 2 ways to find a moment to get the reproduce:

1) In my case, if I will not interrupt the query, it will fail after 1 min 40 sec. To get hang in CANCELLATION_REQUESTED state I need to make "Ctrl + c" after ~1 min 30 sec. So I think you can scale time for the situation on your cluster.

2) You can open 2nd terminal window with "tail -f /opt/mapr/drill/log/drillbit.log" and you should interrupt the query before the first fragment will change state to FINISHED. 
To make this you should run the query for the first time. The query should fail and you will be able to see a number of operators which were running before some fragment finishes (as an example you can see a required number of operators in the red circles in Reproduce5083.jpg attachment). When you will run the query in the second time, just wait for a required number in the 2nd terminal and interrupt query with "Ctrl + c".

You can reproduce this issue in Drill before 368bc38b1 and you can verify it on the latest apache master.


> RecordIterator can sometimes restart a query on close
> -----------------------------------------------------
>
>                 Key: DRILL-5083
>                 URL: https://issues.apache.org/jira/browse/DRILL-5083
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Roman Kulyk
>            Priority: Minor
>              Labels: ready-to-commit
>             Fix For: 1.11.0
>
>         Attachments: DrillOperatorErrorHandlingRedesign.pdf, Reproduce5083.jpg
>
>
> This one is very confusing...
> In a test with a MergeJoin and external sort, operators are stacked something like this:
> {code}
> Screen
> - MergeJoin
> - - External Sort
> ...
> {code}
> Using the injector to force a OOM in spill, the external sort threw a UserException up the stack. This was handed by:
> {code}
> IteratorValidatorBatchIterator.next( )
> RecordIterator.clearInflightBatches( )
> RecordIterator.close( )
> MergeJoinBatch.close( )
> {code}
> Which does the following:
> {code}
>       // Check whether next() should even have been called in current state.
>       if (null != exceptionState) {
>         throw new IllegalStateException(
> {code}
> But, the exceptionState is set, so we end up throwing an IllegalStateException during cleanup.
> Seems the code should agree: if {{next( )}} will be called during cleanup, then {{next( )}} should gracefully handle that case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)