[jira] [Commented] (IMPALA-11285) TestAsyncLoadData.test_async_load fails on S3

["Quanlong Huang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681847#comment-17681847 ] 

Quanlong Huang commented on IMPALA-11285:
-----------------------------------------

The original error message "'HiveServer2Cursor' object has no attribute 'id'" is fixed by IMPALA-11159. It hides the actual error.

Recently saw this test failed again and got the actual error:
{code:java}
metadata/test_load.py:226: in test_async_load
    self.wait_for_state(handle, finished_state, 10, client=client)
common/impala_test_suite.py:1166: in wait_for_state
    self.wait_for_any_state(handle, [expected_state], timeout, client)
common/impala_test_suite.py:1184: in wait_for_any_state
    raise Timeout(timeout_msg)
E   Timeout: query 'e94fb03ebaf9e842:56bc753500000000' did not reach one of the expected states [4], last known state 3 {code}
Note that this is a different issue apart from IMPALA-10999 since this is a timeout in wait_for_state(). IMPALA-10999 is an assertion error.
{code:java}
enum QueryState {
  CREATED = 0 
  INITIALIZED = 1 
  COMPILED = 2 
  RUNNING = 3 
  FINISHED = 4 
  EXCEPTION = 5 
} {code}
Based on the thrift spec in common/thrift/beeswax.thrift, the expected state is FINISHED and the last know state is RUNNING.

Looking into the logs, the query takes a little bit longer than 10s to finish:
{noformat}
# grep e94fb03ebaf9e842 logs/ee_tests/impalad.INFO
I0127 12:25:32.921113  7726 impala-server.cc:1361] e94fb03ebaf9e842:56bc753500000000] Registered query query_id=e94fb03ebaf9e842:56bc753500000000 session_id=2b4a951b4b8d2be6:af116cdbec14a481
I0127 12:25:32.921283  7726 Frontend.java:1916] e94fb03ebaf9e842:56bc753500000000] Analyzing query: load data inpath '/test-warehouse/test_load_staging_beeswax_True'           into table test_async_load_898a2f19.test_load_nopart_beeswax_True db: functional
I0127 12:25:32.921301  7726 Frontend.java:1925] e94fb03ebaf9e842:56bc753500000000] The original executor group sets from executor membership snapshot: [TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:)]
I0127 12:25:32.921320  7726 RequestPoolService.java:194] e94fb03ebaf9e842:56bc753500000000] Default pool only, scheduler allocation is not specified.
I0127 12:25:32.921344  7726 Frontend.java:1946] e94fb03ebaf9e842:56bc753500000000] A total of 2 executor group sets to be considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:small, max_mem_limit:67108864), TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:large, max_mem_limit:9223372036854775807)]
I0127 12:25:32.921361  7726 Frontend.java:1961] e94fb03ebaf9e842:56bc753500000000] Consider executor group set: TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:small, max_mem_limit:67108864)
I0127 12:25:33.072446  7726 AnalysisContext.java:484] e94fb03ebaf9e842:56bc753500000000] Analysis took 150 ms
I0127 12:25:33.072506  7726 BaseAuthorizationChecker.java:113] e94fb03ebaf9e842:56bc753500000000] Authorization check took 0 ms
I0127 12:25:33.072523  7726 Frontend.java:2061] e94fb03ebaf9e842:56bc753500000000] Analysis and authorization finished.
I0127 12:25:33.075359  7726 Frontend.java:2036] e94fb03ebaf9e842:56bc753500000000] Selected executor group: TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, exec_group_name_prefix:small, max_mem_limit:67108864), reason: query is not auto-scalable
I0127 12:25:33.076525  8804 debug-util.cc:427] e94fb03ebaf9e842:56bc753500000000] Debug Action: CRS_DELAY_BEFORE_LOAD_DATA:SLEEP@3000 sleeping for 3000 ms
I0127 12:25:43.094301  7726 impala-server.cc:1445] UnregisterQuery(): query_id=e94fb03ebaf9e842:56bc753500000000
I0127 12:26:08.213359  8804 ImpaladCatalog.java:223] e94fb03ebaf9e842:56bc753500000000] Adding: TABLE:test_async_load_898a2f19.test_load_nopart_beeswax_true version: 6204 size: 3866
I0127 12:26:08.214099 23736 impala-server.cc:1477] Query successfully unregistered: query_id=e94fb03ebaf9e842:56bc753500000000 {noformat}
The call of wait_for_state() in this test is just to make sure the query succeed. So we can simply bump the timeout to fix it.

> TestAsyncLoadData.test_async_load fails on S3
> ---------------------------------------------
>
>                 Key: IMPALA-11285
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11285
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 4.1.0
>            Reporter: Laszlo Gaal
>            Priority: Blocker
>              Labels: broken-build
>
>  TestAsyncLoadData.test_async_load throws an error when running on S3.
> Backtrace:{code}
> metadata/test_load.py:184: in test_async_load
>     self.wait_for_state(handle, finished_state, 10, client=client)
> common/impala_test_suite.py:1084: in wait_for_state
>     self.wait_for_any_state(handle, [expected_state], timeout, client)
> common/impala_test_suite.py:1100: in wait_for_any_state
>     "last known state {2}".format(handle.get_handle().id, expected_states,
> E   AttributeError: 'HiveServer2Cursor' object has no attribute 'id'
> {code}
> Could be a test problem, or a dependency problem as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org