[pulsar] 32/38: [broker] Increase timeout for loading topics (#6750)

[zh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhaijia pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 121cf08922987d98a54a9655e169a9651b09486d
Author: Addison Higham <ad...@gmail.com>
AuthorDate: Thu May 7 01:55:14 2020 -0600

    [broker] Increase timeout for loading topics (#6750)
    
    In #6489, a timeout was introduced to make sure calls into the
    BrokerService finish or error out. However, this timeout is too low by
    default when loading topics that have many replicated clusters.
    
    Loading replicated topics is quite an expensive operation, involve
    global ZK lookups and the start of many sub-processes. While we would
    hope it finishes in 60 seconds we want to safe.
    
    Long term, it may make sense to break out this operation into more
    steps where each step can have it's own timeout
    
    Co-authored-by: Addison Higham <ah...@instructure.com>(cherry picked from commit 6854b007aaba90128530808035d0402b27e93846)
---
 .../main/java/org/apache/pulsar/broker/ServiceConfiguration.java    | 6 ++++++
 .../main/java/org/apache/pulsar/broker/service/BrokerService.java   | 3 ++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 8d9a2b7..6b92061 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -215,6 +215,12 @@ public class ServiceConfiguration implements PulsarConfiguration {
     private long brokerShutdownTimeoutMs = 60000;
 
     @FieldContext(
+            category = CATEGORY_SERVER,
+            doc = "Amount of seconds to timeout when loading a topic. In situations with many geo-replicated clusters, this may need raised."
+    )
+    private long topicLoadTimeoutSeconds = 60;
+
+    @FieldContext(
         category = CATEGORY_POLICIES,
         doc = "Enable backlog quota check. Enforces actions on topic when the quota is reached"
     )
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 808f0c3..7a36d0b 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -851,7 +851,8 @@ public class BrokerService implements Closeable, ZooKeeperCacheListener<Policies
     protected CompletableFuture<Optional<Topic>> loadOrCreatePersistentTopic(final String topic,
             boolean createIfMissing) throws RuntimeException {
         checkTopicNsOwnership(topic);
-        final CompletableFuture<Optional<Topic>> topicFuture = futureWithDeadline();
+        final CompletableFuture<Optional<Topic>> topicFuture = futureWithDeadline(pulsar.getConfiguration().getTopicLoadTimeoutSeconds(),
+                TimeUnit.SECONDS, new TimeoutException("Failed to load topic within timeout"));
         if (!pulsar.getConfiguration().isEnablePersistentTopics()) {
             if (log.isDebugEnabled()) {
                 log.debug("Broker is unable to load persistent topic {}", topic);