You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/06/13 06:37:19 UTC

[impala] 05/07: IMPALA-7154: Increase metastore client time out for tests

This is an automated email from the ASF dual-hosted git repository.

tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit e3117aa6645fec89ef587d6e57b38b981080cb03
Author: Vihang Karajgaonkar <vi...@cloudera.com>
AuthorDate: Tue Jun 11 14:24:26 2019 -0700

    IMPALA-7154: Increase metastore client time out for tests
    
    Some of the e2e tests create tables and databases on S3. HMS operations
    like dropDatabaseCascade can take significant time (3-5 min) in such
    tests. If the HMS Client times out, it retries for a configurable number
    of times. Unfortunately, HMS client is not smart enough to understand
    which operations should be retried. Eg, a drop database operation which
    is timedout on the client side, should not be retried since the server
    thread may have succeeded before the next retry. This would cause the
    client to receive a NoSuchObjectException which would look like a flaky
    test failure on the Impala side.
    
    This patch increases the metastore client.socket.timeout for
    the tests so that such flaky test failures are reduced.
    
    Note:
    1. The timeout value of 10 min was chosen based on looking at logs for
    one such failure where the HMS API took about 3.5 min. Once this patch
    is deployed we should continue to see if such failures reoccur and
    readjust the client timeout value accordingly.
    
    Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
    Reviewed-on: http://gerrit.cloudera.org:8080/13586
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 fe/src/test/resources/hive-site.xml.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/fe/src/test/resources/hive-site.xml.py b/fe/src/test/resources/hive-site.xml.py
index e46adc3..d8acccd 100644
--- a/fe/src/test/resources/hive-site.xml.py
+++ b/fe/src/test/resources/hive-site.xml.py
@@ -31,8 +31,15 @@ CONFIG.update({
   # IMPALA-781: Impala doesn't support the default LazyBinaryColumnarSerde for RCFile.
   'hive.default.rcfile.serde': 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe',
 
-  'hive.metastore.client.connect.retry.delay': '0',
-  'hive.metastore.client.socket.timeout': '120',
+  # IMPALA-7154: Some of the HMS operations on S3 can take long time and any timeouts
+  # on the HMS Client side can result in flaky test failures
+  # Following configuration specifies the time in seconds between successive retry
+  # attempts by metastore client in case of failures. By default, metastore client
+  # retries once based on the config value of hive.metastore.failure.retries
+  'hive.metastore.client.connect.retry.delay': '1',
+  # set the metastore client timeout to 10 min
+  'hive.metastore.client.socket.timeout': '600',
+
   'hive.metastore.uris': 'thrift://${INTERNAL_LISTEN_HOST}:9083',
   'hive.metastore.warehouse.dir': '${WAREHOUSE_LOCATION_PREFIX}/test-warehouse',