[kafka] branch trunk updated: KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (#12830)

[ij...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ijuma pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new fe375dce547 KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (#12830)
fe375dce547 is described below

commit fe375dce54717f0699e9ab860c4a33669d017f22
Author: Greg Harris <gr...@aiven.io>
AuthorDate: Sat Apr 15 12:21:56 2023 -0700

    KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (#12830)
    
    The GRACEFUL_SHUTDOWN_TIMEOUT_MS for the Trogdor JsonRestServer is 100ms.
    In heavily loaded CI environments, this timeout can be exceeded. When this happens,
    it causes the jettyServer.stop() and jettyServer.destroy() calls to throw exceptions, which
    prevents shutdownExecutor.shutdown() from running. This has the effect of causing the JsonRestServer::waitForShutdown method to block for 1 day, which exceeds the 120s
    timeout on the CoordinatorTest (and any other test relying on MiniTrogdorCluster).
    
    This change makes it such that the graceful shutdown timeout is less likely to be exceeded,
    and when it is, the timeout does not cause the waitForShutdown method to block for much
    longer than the graceful shutdown timeout.
    
    Reviewers: Ismael Juma <is...@juma.me.uk>
---
 .../java/org/apache/kafka/trogdor/rest/JsonRestServer.java     | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java b/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
index e5388f8a6f9..cc3bbc0cc6d 100644
--- a/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
+++ b/trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java
@@ -56,7 +56,7 @@ import java.util.concurrent.TimeUnit;
 public class JsonRestServer {
     private static final Logger log = LoggerFactory.getLogger(JsonRestServer.class);
 
-    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 100;
+    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 10 * 1000;
 
     private final ScheduledExecutorService shutdownExecutor;
 
@@ -142,9 +142,13 @@ public class JsonRestServer {
                 } catch (Exception e) {
                     log.error("Unable to stop REST server", e);
                 } finally {
-                    jettyServer.destroy();
+                    try {
+                        jettyServer.destroy();
+                    } catch (Exception e) {
+                        log.error("Unable to destroy REST server", e);
+                    }
+                    shutdownExecutor.shutdown();
                 }
-                shutdownExecutor.shutdown();
                 return null;
             });
         }