[impala] branch master updated: IMPALA-9165: Add back hard kill to kill-hbase.sh

[jo...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 4c09975  IMPALA-9165: Add back hard kill to kill-hbase.sh
4c09975 is described below

commit 4c09975c14f624028100e9940526a111897846cb
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Fri Nov 22 11:05:20 2019 -0800

    IMPALA-9165: Add back hard kill to kill-hbase.sh
    
    The fix for IMPALA-9150 changed kill-hbase.sh to use HBase's
    stop-hbase.sh script. Around this time, the GVO timeout issues
    started. GVO can reuse machines, so we don't know what state
    they may be in. If something failed to kill HBase processes,
    the next job would need to be able to kill them even without
    access to the last run's files / logs.
    
    This restores the original kill logic to kill-hbase.sh, after
    trying a graceful shutdown using HBase's stop-hbase.sh script.
    The original kill logic doesn't rely on anything from the
    filesystem to know about the existence of processes, so it
    would handle machine reuse.
    
    This also changes our Jenkins test scripts to shut down the
    minicluster at the end.
    
    Testing:
     - Started with a running minicluster, ran bin/clean.sh,
       then ran testdata/bin/kill-all.sh and verified that the
       java processes were gone
    
    Change-Id: Ie2f0b342bcd1d8abea8ef923adbb54a14518a7a6
    Reviewed-on: http://gerrit.cloudera.org:8080/14789
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/jenkins/all-tests.sh                   |  3 +++
 bin/jenkins/dockerized-impala-run-tests.sh | 10 +++++++++-
 testdata/bin/kill-hbase.sh                 |  8 ++++++--
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/bin/jenkins/all-tests.sh b/bin/jenkins/all-tests.sh
index b531804..21c3acc 100644
--- a/bin/jenkins/all-tests.sh
+++ b/bin/jenkins/all-tests.sh
@@ -67,5 +67,8 @@ if ! bin/run-all-tests.sh; then
   RET_CODE=1
 fi
 
+# Shutdown minicluster at the end
+testdata/bin/kill-all.sh
+
 bin/jenkins/finalize.sh
 exit $RET_CODE
diff --git a/bin/jenkins/dockerized-impala-run-tests.sh b/bin/jenkins/dockerized-impala-run-tests.sh
index e097fcc..45f7e86 100755
--- a/bin/jenkins/dockerized-impala-run-tests.sh
+++ b/bin/jenkins/dockerized-impala-run-tests.sh
@@ -92,4 +92,12 @@ export BE_TEST=false
 # to make them start up dockerised clusters and will probably make more assumptions
 # about the cluster being tested.
 export CLUSTER_TEST=false
-./bin/run-all-tests.sh
+RET_CODE=0
+if ! ./bin/run-all-tests.sh; then
+  RET_CODE=1
+fi
+
+# Shutdown minicluster at the end
+./testdata/bin/kill-all.sh
+
+exit $RET_CODE
diff --git a/testdata/bin/kill-hbase.sh b/testdata/bin/kill-hbase.sh
index e404aef..47298c8 100755
--- a/testdata/bin/kill-hbase.sh
+++ b/testdata/bin/kill-hbase.sh
@@ -23,10 +23,14 @@ setup_report_build_error
 
 DIR=$(dirname "$0")
 echo Stopping Hbase
-# Use the stop-hbase.sh script provided by HBase. This does a more graceful shutdown
-# that using our kill-java-service.sh script.
+# First, use the stop-hbase.sh script provided by HBase. This does a more graceful
+# shutdown than using our kill-java-service.sh script.
 ${HBASE_HOME}/bin/stop-hbase.sh
 
+# Second, also do a harder shutdown in case there is anything that is not covered
+# by the stop-hbase.sh script. Kill region server first, then hmaster, and zookeeper.
+"$DIR"/kill-java-service.sh -c HRegionServer -c HMaster -c HQuorumPeer -s 2
+
 # Clear up data so that zookeeper/hbase won't do recovery when it starts.
 # TODO: is this still needed when using bin/stop-hbase.sh?
 rm -rf /tmp/hbase-*