[jira] [Commented] (IMPALA-10219) Add a query option to simulate catalogd HDFS listing delays

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218778#comment-17218778 ] 

ASF subversion and git services commented on IMPALA-10219:
----------------------------------------------------------

Commit 15c3b13e9730479e096275d974000ae9fe8fbb83 in impala's branch refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=15c3b13 ]

IMPALA-10219: Expose DEBUG_ACTION query option in catalog

This patches enables DEBUG_ACTION in the catalog service's
java code. Specifically, DEBUG_ACTION query option is now
exposed to TResetMetadataRequest and TExecDdlRequest
so that we can inject delays while executing refresh
or ddl statements.

For example,
1. To inject a delay of 100ms per HDFS list operation
during refresh statement set the following query option:

set debug_action=catalogd_refresh_hdfs_listing_delay:SLEEP@100;

2. To inject a delay of 100ms in alter table recover
partitions statement:

set debug_action=catalogd_table_recover_delay:SLEEP@100;

3. To inject a delay of 100ms in compute stats statement

set debug_action=catalogd_update_stats_delay:SLEEP@100;

Note that this option only adds the delay during the
update_stats phase of the compute stats execution.

Testing:
1. Added a test which sets the query option and makes
sure that command takes more time than without query option.
2. Added unit tests for the debugAction implementation
logic.

Change-Id: Ia7196b1ce76415a5faf3fa8575a26d22b2bf50b1
Reviewed-on: http://gerrit.cloudera.org:8080/16548
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Add a query option to simulate catalogd HDFS listing delays
> -----------------------------------------------------------
>
>                 Key: IMPALA-10219
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10219
>             Project: IMPALA
>          Issue Type: Sub-task
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>            Priority: Minor
>
> This parent issue (IMPALA-6671) caused serious query backlog on large setups where namenode response times are slower for whatever reasons. While you can tune the namenode to some extent it is still problematic that Impala HDFS operations which happen while holding the table lock block other unrelated queries.
> In order to simulate such problems in the product it would be nice to introduce a query option which adds a artificial delay in the RPCs to namenode when the table is being loaded. A query option is preferred over service level configuration since, that way it is easier to model a slow blocking query and a unrelated fast query in the test suite.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org