[incubator-kyuubi] branch branch-1.6 updated: [KYUUBI #3272] Synchronize graceful shutdown with main stop sequence

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git


The following commit(s) were added to refs/heads/branch-1.6 by this push:
     new fe79deee2 [KYUUBI #3272] Synchronize graceful shutdown with main stop sequence
fe79deee2 is described below

commit fe79deee24e75197b87ffd54006bcc4790f825bb
Author: Brandon Grams <br...@zillowgroup.com>
AuthorDate: Mon Aug 22 15:14:13 2022 +0800

    [KYUUBI #3272] Synchronize graceful shutdown with main stop sequence
    
    ### _Why are the changes needed?_
    
    Fixes #3272
    
    Kyuubi server stop sequence doesn't respect graceful shutdown triggered in the discovery clients' `DeRegisterWatcher` loops. This leads to frontend service components being stopped before sessions close, resetting connections and leading to client disruption.
    
    PR synchronizes these threads to ensure that service components wait for client sessions to close before continuing the stop sequence.
    * Existing service discovery deregistration functionality is maintained to ensure that new connections are established with healthy instances once the stop sequence has begun.
    * Existing Hadoop shutdown hooks ensure that the watcher can be timed out, and are configurable via `hadoop.service.shutdown.timeout`. The Kyuubi server will always shut down within 30 seconds when using default config.
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    
    - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
    
    Closes #3291 from bgrams/fix/graceful-shutdown.
    
    Closes #3272
    
    ffec2a2a [Brandon Grams] Synchronize graceful session shutdown
    
    Authored-by: Brandon Grams <br...@zillowgroup.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
    (cherry picked from commit bc168527177899e2a72c50cb8f4477c15b4f9452)
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala     | 1 +
 .../src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala  | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala b/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala
index c2727a48f..31618e465 100644
--- a/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala
+++ b/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala
@@ -32,6 +32,7 @@ class KyuubiServiceDiscovery(
     if (!isServerLost.get()) {
       discoveryClient.deregisterService()
       discoveryClient.closeClient()
+      gracefulShutdownLatch.await() // wait for graceful shutdown triggered by watcher
     } else {
       warn(s"The Zookeeper ensemble is LOST")
     }
diff --git a/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala b/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala
index 3b1f5c8f1..bdb9b12fe 100644
--- a/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala
+++ b/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala
@@ -17,6 +17,7 @@
 
 package org.apache.kyuubi.ha.client
 
+import java.util.concurrent.CountDownLatch
 import java.util.concurrent.atomic.AtomicBoolean
 
 import org.apache.kyuubi.Logging
@@ -34,6 +35,7 @@ abstract class ServiceDiscovery(
     name: String,
     val fe: FrontendService) extends AbstractService(name) {
 
+  protected val gracefulShutdownLatch = new CountDownLatch(1)
   protected val isServerLost = new AtomicBoolean(false)
 
   /**
@@ -68,6 +70,7 @@ abstract class ServiceDiscovery(
       Thread.sleep(1000 * 60)
     }
     isServerLost.set(isLost)
+    gracefulShutdownLatch.countDown()
     fe.serverable.stop()
   }
 }