[jira] [Created] (IMPALA-6257) TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan hangs in S3 build

["Tianyi Wang (JIRA)" <ji...@apache.org>]

Tianyi Wang created IMPALA-6257:
-----------------------------------

             Summary: TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan hangs in S3 build
                 Key: IMPALA-6257
                 URL: https://issues.apache.org/jira/browse/IMPALA-6257
             Project: IMPALA
          Issue Type: Bug
    Affects Versions: Impala 2.11.0
            Reporter: Tianyi Wang
            Priority: Blocker


In a recent S3 build, TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan failed because a fragment never finishes. The error message is:
{noformat}
07:49:57  TestKuduMemLimits.test_low_mem_limit_low_selectivity_scan[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none-1] 
07:49:57 query_test/test_kudu.py:1062: in test_low_mem_limit_low_selectivity_scan
07:49:57     v.wait_for_metric("impala-server.num-fragments-in-flight", 0, timeout=30)
07:49:57 verifiers/metric_verifier.py:62: in wait_for_metric
07:49:58     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
07:49:58 common/impala_service.py:89: in wait_for_metric_value
07:49:58     (metric_name, expected_value, timeout)
07:49:58 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 30s
{noformat}
Looking at the coordinator log, the query hangs for 30 seconds before cancelled by the client:
{noformat}
I1127 07:30:37.677126  5939 coordinator.cc:789] Coordinator waiting for backends to finish, 1 remaining
I1127 07:30:37.677865  5939 coordinator.cc:794] All backends finished successfully.
I1127 07:30:37.677877  5939 coordinator.cc:1090] Release admssion control resources for query 4044d47830c1cc98:8a63d27b00000000
I1127 07:30:37.677942  6043 data-stream-mgr.cc:238] DeregisterRecvr(): fragment_instance_id=4044d47830c1cc98:8a63d27b00000000, node=1
I1127 07:30:37.677963  6043 data-stream-recvr.cc:235] cancelled stream: fragment_instance_id_=4044d47830c1cc98:8a63d27b00000000 node_id=1
I1127 07:30:37.678045  6043 query-state.cc:385] Instance completed. instance_id=4044d47830c1cc98:8a63d27b00000000 #in-flight=1 status=OK
I1127 07:30:37.678073  6043 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=4044d47830c1cc98:8a63d27b00000000 refcnt=2
I1127 07:31:08.084748  5939 impala-hs2-server.cc:699] CloseOperation(): query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084779  5939 impala-server.cc:992] UnregisterQuery(): query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084790  5939 impala-server.cc:1075] Cancel(): query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084808  5939 coordinator.cc:895] Cancel() query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084827  5939 coordinator.cc:905] CancelBackends() query_id=4044d47830c1cc98:8a63d27b00000000, tried to cancel 0 backends
I1127 07:31:08.090203  5939 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=4044d47830c1cc98:8a63d27b00000000 refcnt=1
I1127 07:31:08.101332  5939 impala-hs2-server.cc:381] CloseSession(): request=TCloseSessionReq {
{noformat}
Compared to the same but successful query, the #in-flight differs:
{noformat}
I1128 07:35:45.863986  7571 coordinator.cc:789] Coordinator waiting for backends to finish, 1 remaining
I1128 07:35:45.864734  7571 coordinator.cc:794] All backends finished successfully.
I1128 07:35:45.864749  7571 coordinator.cc:1090] Release admssion control resources for query 6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.864823  7671 data-stream-mgr.cc:238] DeregisterRecvr(): fragment_instance_id=6f43945cef3a95be:744fd2a300000000, node=1
I1128 07:35:45.864845  7671 data-stream-recvr.cc:235] cancelled stream: fragment_instance_id_=6f43945cef3a95be:744fd2a300000000 node_id=1
I1128 07:35:45.864953  7671 query-state.cc:385] Instance completed. instance_id=6f43945cef3a95be:744fd2a300000000 #in-flight=0 status=OK
I1128 07:35:45.864982  7671 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=6f43945cef3a95be:744fd2a300000000 refcnt=2
I1128 07:35:45.944633  7571 impala-hs2-server.cc:699] CloseOperation(): query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944655  7571 impala-server.cc:992] UnregisterQuery(): query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944666  7571 impala-server.cc:1075] Cancel(): query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944691  7571 coordinator.cc:895] Cancel() query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944710  7571 coordinator.cc:905] CancelBackends() query_id=6f43945cef3a95be:744fd2a300000000, tried to cancel 0 backends
I1128 07:35:45.949873  7571 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=6f43945cef3a95be:744fd2a300000000 refcnt=1
I1128 07:35:45.959841  7571 impala-hs2-server.cc:381] CloseSession(): request=TCloseSessionReq {
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)