You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/10/23 00:10:00 UTC
[jira] [Commented] (IMPALA-10967) Load data should handle AWS
NLB-type timeout
[ https://issues.apache.org/jira/browse/IMPALA-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433189#comment-17433189 ]
ASF subversion and git services commented on IMPALA-10967:
----------------------------------------------------------
Commit 975883c47035843398ee99a21fa132f67a0d4954 in impala's branch refs/heads/master from Qifan Chen
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=975883c ]
IMPALA-10811 RPC to submit query getting stuck for AWS NLB forever
This patch addresses Impala client hang due to AWS network load balancer
timeout which is fixed at 350s. When some long DDL operations are
executing and the timeout happens, AWS silently drops the connection and
the Impala client enters the hang state.
The fix maintains the current TCLIService protocol between the client
and Impala server and is applicable to the following Impala clients
which issue thrift RPC ExecuteStatement() followed by repeated call to
GetOperationStatus() (HS2, Impyla and HUE) or a variant of it (Beeswax)
to Impala backend.
1. HS2
2. Beeswax
3. Impyla
4. HUE
In the fix, the backend method ClientRequestState::ExecDdlRequest()
can start a new thread in 'async_exec_thread_' for ExecDdlRequestImpl()
which executes most of the DDLs asynchronously. This thread is waited
for in the wait thread 'wait_thread_'. Since the wait thread also runs
asynchronously, the execution of the DDLs will not cause a wait on the
Impala client. Thus the Impala client can keep checking its execution
status via GetOperationStatus() without long waiting, say more than
350s.
As an optimization, the above asynchronous mode is not applied to the
execution of certain DDLs that run very low risks of long execution.
1. Operations that do not access catalog service;
2. COMPUTE STATS as the stats computation queries already run
asynchronously.
External behavior change:
1. A new field with name "DDL execution mode:" is added to the
summary section in the runtime profile, next to "DDL Type". This
field takes either 'asynchronous' or 'synchronous' as value.
2. A new query option 'enable_async_ddl_execution', default to true,
is added. It can be set to false to turn off the patch.
Limitations:
This patch does not handle potential AWS NLB-type time out for LOAD
DATA (IMPALA-10967).
Testing:
1. Added new async. DDL unit tests with HS2, HS2-HTTP, Beeswax and
JDBC clients.
2. Ran core tests successfully.
Change-Id: Ib57e86926a233ef13d27a9ec8d9c36d33a88a44e
Reviewed-on: http://gerrit.cloudera.org:8080/17872
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Load data should handle AWS NLB-type timeout
> --------------------------------------------
>
> Key: IMPALA-10967
> URL: https://issues.apache.org/jira/browse/IMPALA-10967
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Qifan Chen
> Assignee: Qifan Chen
> Priority: Major
>
> Currently, since Impala handles the load data statement request in a single thread, the client can experience AWS NLB-type timeout (see IMPALA-10811) if the data loading takes more than 350s to complete.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org