You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/08 01:24:11 UTC

[GitHub] [pulsar] Demogorgon314 opened a new pull request, #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Demogorgon314 opened a new pull request, #18810:
URL: https://github.com/apache/pulsar/pull/18810

   Master Issue: https://github.com/apache/pulsar/issues/16691
   
   ### Motivation
   
   We will start raising PRs to implement PIP-192, https://github.com/apache/pulsar/issues/16691
   
   ### Modifications
   
   This PR should not impact the existing broker load balance behavior.
   
   For the PIP-192 project, this PR
   * Implemented broker registry.
   * Added units test.
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/Demogorgon314/pulsar/pull/7
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1378379113

   nit: 
   
   Why not use LoadDataStore<BrokerLoadData> + TableView<BrokerLookupData> for this broker registry service? 
   
   LoadDataStore<BrokerLoadData>.keySet() can return available brokers.
   
   TableView<BrokerLookupData>.get(broker) can return the pulsar version and other lookup info.
   
   This will further reduce the metadata store dependency.
   
   Sorry for this random idea, and I just wanted to share it here. We may need a separate discussion(a PIP) to think deeper if we agree with this direction.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1048036339


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   It will block the current thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1045546483


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;

Review Comment:
   What's the consistency model of `LockManager`? Does BrokerRegistry guarantee eventual consistency?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1060350244


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1060349676


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }
+    }
+
+    private void handleMetadataStoreNotification(Notification t) {
+        if (!isVerifiedNotification(t)) {
+            return;
+        }
+        try {
+            if (log.isDebugEnabled()) {
+                log.debug("Handle notification: [{}]", t);
+            }
+            if (listeners.isEmpty()) {
+                return;
+            }
+            this.scheduler.submit(() -> {
+                String brokerId = t.getPath().substring(LOOKUP_DATA_PATH.length() + 1);
+                for (BiConsumer<String, NotificationType> listener : listeners) {
+                    listener.accept(brokerId, t.getType());
+                }
+            });
+        } catch (RejectedExecutionException e) {
+            // Executor is shutting down
+        }

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1067328842


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   I see. LGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1068833098


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt ->

Review Comment:
   I see. Thanks for the explanation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046667491


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;

Review Comment:
   The `BrokerRegistry#lookup` method will always use the latest value, and the `LockManager` will ensure the same broker address will not register to zk twice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1048054033


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   It is used in `BrokerVersionFilter`, you can check the POC code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1060350109


##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryTest.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LoadManager;
+import org.apache.pulsar.broker.loadbalance.ResourceUnit;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Unit test for {@link BrokerRegistry}.
+ */
+@Slf4j
+@Test(groups = "broker")
+public class BrokerRegistryTest {

Review Comment:
   I have added a state field to store the registry state, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1049052538


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   
   One alternative to optimizing BrokerVersionFilter is adding the pulsar version in the znode path.
   
   /loadbalance/brokers/{lookup-address}-{pulsar-version}
   
   By this design, we can get available brokers and their pulsar versions by a single getChildren() call(and they are cached together).
   
   We may have some more complex zk path designs(e.g. nested zk path), but I think the above path design is good enough for our use case.
   
   
   
   
   
   
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047897296


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataCache;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    private volatile CompletableFuture<Void> cacheReloadFuture;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataCache = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+        this.cacheReloadFuture = CompletableFuture.completedFuture(null);
+        this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            // Update all lookup data to cache
+            this.reloadAllBrokerLookupCacheAsync()
+                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+            this.started.set(true);
+        } catch (MetadataStoreException | ExecutionException | InterruptedException | TimeoutException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {

Review Comment:
   I think getBrokerId() is fine too.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   I see that we maintain an internal cache with ttl inside LockManagerImpl.
   It appears that we don't need to maintain `brokerLookupDataMap` separately. Can you clarify why we need `brokerLookupDataMap`?
   
   
   ref:
   ```
   @Override
       public CompletableFuture<List<String>> listLocks(String path) {
           return cache.getChildren(path);
       }
   
   
       @Override
       public CompletableFuture<List<String>> getChildren(String path) {
           return store.getChildren(path);
       }
   
   @Override
       public final CompletableFuture<List<String>> getChildren(String path) {
           if (!isValidPath(path)) {
               return FutureUtil.failedFuture(new MetadataStoreException.InvalidPathException(path));
           }
           return childrenCache.get(path);
       }
   
   this.childrenCache = Caffeine.newBuilder()
                   .refreshAfterWrite(CACHE_REFRESH_TIME_MILLIS, TimeUnit.MILLISECONDS)
                   .expireAfterWrite(CACHE_REFRESH_TIME_MILLIS * 2, TimeUnit.MILLISECONDS)
                   .buildAsync(new AsyncCacheLoader<String, List<String>>() {
                       @Override
                       public CompletableFuture<List<String>> asyncLoad(String key, Executor executor) {
                           return getChildrenFromStore(key);
                       }
   
                       @Override
                       public CompletableFuture<List<String>> asyncReload(String key, List<String> oldValue,
                               Executor executor) {
                           if (isConnected) {
                               return getChildrenFromStore(key);
                           } else {
                               // Do not refresh if we're not connected
                               return CompletableFuture.completedFuture(oldValue);
                           }
                       }
                   });
   
   ```
   
   Also, the broker delete notification is already handled by
   
   ```
   private void handleDataNotification(Notification n) {
           if (n.getType() == NotificationType.Deleted) {
               ResourceLockImpl<T> lock = locks.get(n.getPath());
               if (lock != null) {
                   lock.lockWasInvalidated();
               }
           }
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047979094


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Can't we get the value by `readLock()`? 
   
   readLock reads from`MetadataCache<T> cache;`
   
   for path : listLocks()
    action.accept(path, readLock(path))



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1055019638


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   The selection method will be like this:
   ```java
   public CompletableFuture<Optional<String>> selectAsync(ServiceUnitId bundle) {
   return brokerRegistry.getAvailableBrokerLookupDataAsync()
             .thenCompose(availableBrokers -> {
                 LoadManagerContext context = this.getContext();
     
                 // Filter out brokers that do not meet the rules.
                 List<BrokerFilter> filterPipeline = getBrokerFilterPipeline();
                 Map<String, BrokerLookupData> availableBrokerCandidates = new HashMap<>(availableBrokers);
                 try {
                     for (final BrokerFilter filter : filterPipeline) {
                         filter.filter(availableBrokerCandidates, context);
                     }
                 } catch (BrokerFilterException e) {
                     availableBrokerCandidates = availableBrokers;
                 }
                 if (availableBrokerCandidates.isEmpty()) {
                     return CompletableFuture.completedFuture(Optional.empty());
                 }
     
                 return CompletableFuture.completedFuture(
                         brokerSelectionStrategy.select(availableBrokerCandidates.keySet(), bundle, context));
             });
   }
   ```
   So the filter will not be blocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1411415628

   @BewareMyPower Fixed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Technoboy- commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092788832


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    protected enum State {
+        Init,
+        Started,
+        Registered,
+        Closed
+    }
+
+    private State state;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+        this.state = State.Init;
+    }
+
+    @Override
+    public synchronized void start() throws PulsarServerException {
+        if (this.state != State.Init) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.state = State.Started;
+            this.register();
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.state == State.Started || this.state == State.Registered;
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {

Review Comment:
   Why make this method to be `public` ?  we can register more times, right ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1053937530


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Or can we add a method to let the registry return the map of lookup data? It will be a little heavy since each select broker must return this map.
   
   ```
   public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync()
   ```
   
   And change the broker filter method as:
   
   ```
   List<String> filter(Map<String, BrokerLookupData> availableBrokerLookupData, LoadManagerContext context) throws BrokerFilterException;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1045944377


##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryTest.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import org.apache.bookkeeper.util.ZkUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LoadManager;
+import org.apache.pulsar.broker.loadbalance.ResourceUnit;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+/**
+ * Unit test for {@link BrokerRegistry}.
+ */
+public class BrokerRegistryTest {
+
+    private ExecutorService executor;
+
+    private LocalBookkeeperEnsemble bkEnsemble;
+
+    private PulsarService pulsar1;
+
+    private PulsarService pulsar2;
+
+    // Make sure the load manager don't register itself to `/loadbalance/brokers/{lookupServiceAddress}`
+    public static class MockLoadManager implements LoadManager {
+
+        @Override
+        public void start() throws PulsarServerException {
+            // No-op
+        }
+
+        @Override
+        public boolean isCentralized() {
+            return false;
+        }
+
+        @Override
+        public Optional<ResourceUnit> getLeastLoaded(ServiceUnitId su) throws Exception {
+            return Optional.empty();
+        }
+
+        @Override
+        public LoadManagerReport generateLoadReport() throws Exception {
+            return null;
+        }
+
+        @Override
+        public void setLoadReportForceUpdateFlag() {
+            // No-op
+        }
+
+        @Override
+        public void writeLoadReportOnZookeeper() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public void writeResourceQuotasToZooKeeper() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public List<Metrics> getLoadBalancingMetrics() {
+            return null;
+        }
+
+        @Override
+        public void doLoadShedding() {
+            // No-op
+        }
+
+        @Override
+        public void doNamespaceBundleSplit() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public void disableBroker() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public Set<String> getAvailableBrokers() throws Exception {
+            return null;
+        }
+
+        @Override
+        public CompletableFuture<Set<String>> getAvailableBrokersAsync() {
+            return null;
+        }
+
+        @Override
+        public void stop() throws PulsarServerException {
+            // No-op
+        }
+
+        @Override
+        public void initialize(PulsarService pulsar) {
+            // No-op
+        }
+    }
+
+    @BeforeMethod
+    void setup() throws Exception {
+        executor = new ThreadPoolExecutor(5, 20, 30, TimeUnit.SECONDS,
+                new LinkedBlockingQueue<>());
+        // Start local bookkeeper ensemble
+        bkEnsemble = new LocalBookkeeperEnsemble(3, 0, () -> 0);
+        bkEnsemble.start();
+
+        // Start broker 1
+        ServiceConfiguration config1 = new ServiceConfiguration();
+        config1.setLoadBalancerEnabled(false);
+        config1.setLoadManagerClassName(MockLoadManager.class.getName());
+        config1.setClusterName("use");
+        config1.setWebServicePort(Optional.of(0));
+        config1.setMetadataStoreUrl("zk:127.0.0.1" + ":" + bkEnsemble.getZookeeperPort());
+        config1.setBrokerShutdownTimeoutMs(0L);
+        config1.setBrokerServicePort(Optional.of(0));
+        config1.setAdvertisedAddress("localhost");
+        createCluster(bkEnsemble.getZkClient(), config1);
+        pulsar1 = new PulsarService(config1);
+        pulsar1.start();
+
+        // Start broker 2
+        ServiceConfiguration config2 = new ServiceConfiguration();
+        config2.setLoadBalancerEnabled(false);
+        config2.setLoadManagerClassName(MockLoadManager.class.getName());
+        config2.setClusterName("use");
+        config2.setWebServicePort(Optional.of(0));
+        config2.setMetadataStoreUrl("zk:127.0.0.1" + ":" + bkEnsemble.getZookeeperPort());
+        config2.setBrokerShutdownTimeoutMs(0L);
+        config2.setBrokerServicePort(Optional.of(0));
+        config2.setAdvertisedAddress("localhost");
+        pulsar2 = new PulsarService(config2);
+        pulsar2.start();
+    }
+
+    @AfterMethod(alwaysRun = true)
+    void shutdown() throws Exception {
+        executor.shutdownNow();
+
+        pulsar2.close();
+        pulsar1.close();
+
+        bkEnsemble.stop();
+    }
+

Review Comment:
   Sure! I'll add more unit tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046665315


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   > Can we just return the cached list, future.complete(Lists.newArrayList(this.brokerLookupDataMap.keySet()));?
   > Why do we need to use brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH) every time?
   > If this listLocks is for a more consistent getAvailableBrokers read, then why do we cache the broker list in > brokerLookupDataMap and update the entries via the zk listener?
   
   The `brokerLookupDataLockManager` also used a cache, so we can use the `listLocks ` here, and the `brokerLookupDataMap` is used to store the broker lookup address and lookup data for broker filter to filter out the older version broker. 
   
   So I don't think it needs strong consistency for `brokerLookupDataMap`.
   
   > Do we need to update the brokerLookupDataMap cache if the return list differs from brokerLookupDataMap.keySet() ?
   
   Yes, we can trigger the cache reload when the cache is different.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046752418


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataCache;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    private volatile CompletableFuture<Void> cacheReloadFuture;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataCache = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+        this.cacheReloadFuture = CompletableFuture.completedFuture(null);
+        this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            // Update all lookup data to cache
+            this.reloadAllBrokerLookupCacheAsync()
+                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+            this.started.set(true);
+        } catch (MetadataStoreException | ExecutionException | InterruptedException | TimeoutException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {

Review Comment:
   I have been thinking about changing the method name to `getBrokerId()`, since the function of `LookupServiceAddress` is the unique identifier.
   
   @heesung-sn What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059490454


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }

Review Comment:
   Don't we need to reset `started.set(false)` when an exception is thrown?
   
   We don't expect this `register() / unregister()` (or `start() / close()` in similar classes) to be called competitively many times from multiple threads. What's the practice in Pulsar for concurrency control in such cases? (I think the simple `synchronized` is enough in these cases)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092807827


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    protected enum State {
+        Init,
+        Started,
+        Registered,
+        Closed
+    }
+
+    private State state;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+        this.state = State.Init;
+    }
+
+    @Override
+    public synchronized void start() throws PulsarServerException {
+        if (this.state != State.Init) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.state = State.Started;
+            this.register();
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.state == State.Started || this.state == State.Registered;
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (this.state == State.Started) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Registered;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (this.state == State.Registered) {
+            try {
+                this.brokerLookupDataLock.release()
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Started;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Technoboy- merged pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- merged PR #18810:
URL: https://github.com/apache/pulsar/pull/18810


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047964792


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   The `LockManagerImpl ` cache only has key list(`LookupServiceAddress`), but we don't have a cache to traverse the `key, value`, see:
   https://github.com/apache/pulsar/pull/18810/files#diff-830131453bc160555aeefc19581054f593d97be9848c4070f75a2031316b09fdR184-R186
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059310205


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }

Review Comment:
   ```suggestion
           if (started.compareAndSet(false, true)) {
               pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
               try {
                   this.register();
               } catch (MetadataStoreException e) {
                   throw new PulsarServerException(e);
               }
           }
   ```



##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryTest.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LoadManager;
+import org.apache.pulsar.broker.loadbalance.ResourceUnit;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Unit test for {@link BrokerRegistry}.
+ */
+@Slf4j
+@Test(groups = "broker")
+public class BrokerRegistryTest {

Review Comment:
   Please also add tests for the behavior of a closed registry. Make sure the resources are released, the containers are cleaned up.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }

Review Comment:
   Is it possible to change registered to AtomicBoolean? So that we don't need the `synchronized`



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }
+    }
+
+    private void handleMetadataStoreNotification(Notification t) {
+        if (!isVerifiedNotification(t)) {
+            return;
+        }
+        try {
+            if (log.isDebugEnabled()) {
+                log.debug("Handle notification: [{}]", t);
+            }
+            if (listeners.isEmpty()) {
+                return;
+            }
+            this.scheduler.submit(() -> {
+                String brokerId = t.getPath().substring(LOOKUP_DATA_PATH.length() + 1);
+                for (BiConsumer<String, NotificationType> listener : listeners) {
+                    listener.accept(brokerId, t.getType());
+                }
+            });
+        } catch (RejectedExecutionException e) {
+            // Executor is shutting down
+        }

Review Comment:
   We don't need to perform the listeners if the registry is closed, right?



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }

Review Comment:
   ```suggestion
            if (started.compareAndSet(true, false)) {
               try {
                   this.unregister();
                   brokerLookupDataLockManager.close();
                   scheduler.shutdown();
                   this.listeners.clear();
               } catch (Exception ex) {
                   if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
                       throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
                   } else {
                       throw new PulsarServerException(MetadataStoreException.unwrap(ex));
                   }
               }
           }
   ```



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }

Review Comment:
   It's better to check if the registry is started or not.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1053789262


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Actually, we have concerns about updating the znode path design here.
   
   Unless necessary, we want to keep this path design for the following reasons.
   
   1. This path design change can further complicate version compatibility(uneasy to maintain the path design for different pulsar versions).
   2. During version upgrade, if the shutdown is ungraceful, the new path design could incur two locks for the same broker. However, currently, the restart will be blocked as the lingering lock will block it.
   
   
   
   We could revisit the original proposal:
   
   ```
   foreach (){
      for path : listLocks()
         var future  = readLock(path)
         if(future.isDone && !future.isCompletedExceptionally){
            action.accept(path, readLock(path))
         } else {
         // ignore blocking
        }
      }
   }
   ```
   
   Here the `readLock` will return the up-to-date lock value (maximum 5 mins delay, except the first time) as the cache refreshes after 5mins, once the key-value is stored.
   In this way, we can make this foreach non-blocking. When the `readLock` cache is initially empty, it will return the empty available broker set. In this case. the client should retry.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1060230963


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }

Review Comment:
   I noticed no `synchronized` for this method.
   
   > What's the practice in Pulsar for concurrency control in such cases?
   
   Most places change to the CLOSING state from READY, and then to the CLOSED from CLOSING after all the tasks are done. And we can try multiple times to close the resource from CLOSING to CLOSED.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1061258577


##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryTest.java:
##########
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LoadManager;
+import org.apache.pulsar.broker.loadbalance.ResourceUnit;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Unit test for {@link BrokerRegistry}.
+ */
+@Slf4j
+@Test(groups = "broker")
+public class BrokerRegistryTest {

Review Comment:
   Ping @codelipenghui 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1382617644

   > nit: 
   > 
   > Why not use `LoadDataStore<BrokerLoadData> + TableView<BrokerLookupData>` for this broker registry service? 
   > 
   > `LoadDataStore<BrokerLoadData>.keySet()` can return available brokers.
   > 
   > `TableView<BrokerLookupData>.get(broker)` can return the pulsar version and other lookup info.
   > 
   > This will further reduce the metadata store dependency.
   > 
   > Sorry for this random idea, and I just wanted to share it here. We may need a separate discussion(a PIP) to think deeper if we agree with this direction.
   
   @heesung-sn The lookup data will not be too heavy for Zookeeper, right? Maybe we can do some tests to find out is necessary to use `LoadDataStore` to store the lookup data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Technoboy- commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092786983


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    protected enum State {
+        Init,
+        Started,
+        Registered,
+        Closed
+    }
+
+    private State state;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+        this.state = State.Init;
+    }
+
+    @Override
+    public synchronized void start() throws PulsarServerException {
+        if (this.state != State.Init) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.state = State.Started;
+            this.register();
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.state == State.Started || this.state == State.Registered;
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (this.state == State.Started) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Registered;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (this.state == State.Registered) {
+            try {
+                this.brokerLookupDataLock.release()
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Started;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }

Review Comment:
   Duplicate codes. Please add a private method named `checkState()` for line 168~170.
   And replace them for `lookupAsync`, `getAvailableBrokersAsync`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Technoboy- commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092793012


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    protected enum State {
+        Init,
+        Started,
+        Registered,
+        Closed
+    }
+
+    private State state;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+        this.state = State.Init;
+    }
+
+    @Override
+    public synchronized void start() throws PulsarServerException {
+        if (this.state != State.Init) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.state = State.Started;
+            this.register();
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.state == State.Started || this.state == State.Registered;
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (this.state == State.Started) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Registered;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (this.state == State.Registered) {
+            try {
+                this.brokerLookupDataLock.release()
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                this.state = State.Started;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        if (!this.isStarted()) {
+            return FutureUtil.failedFuture(new IllegalStateException("The registry already closed."));
+        }
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public synchronized void listen(BiConsumer<String, NotificationType> listener) {

Review Comment:
   `listen` -> addListener



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Technoboy- commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092788832


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    protected enum State {
+        Init,
+        Started,
+        Registered,
+        Closed
+    }
+
+    private State state;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+        this.state = State.Init;
+    }
+
+    @Override
+    public synchronized void start() throws PulsarServerException {
+        if (this.state != State.Init) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.state = State.Started;
+            this.register();
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.state == State.Started || this.state == State.Registered;
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {

Review Comment:
   Why make this method to be `public` ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1413686368

   This PR introduced a new flaky test #19401 .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092790855


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataCache;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    private volatile CompletableFuture<Void> cacheReloadFuture;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataCache = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+        this.cacheReloadFuture = CompletableFuture.completedFuture(null);
+        this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            // Update all lookup data to cache
+            this.reloadAllBrokerLookupCacheAsync()
+                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+            this.started.set(true);
+        } catch (MetadataStoreException | ExecutionException | InterruptedException | TimeoutException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {

Review Comment:
   Use `brokerId` as the name may be more generic? Since the `getBrokerId ` method depends on the implementation. And it can use `LookupServiceAddress` as the broker id, or we can support other formats of broker id in the future if necessary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 closed pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 closed pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager
URL: https://github.com/apache/pulsar/pull/18810


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] BewareMyPower commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1411406079

   > org.apache.pulsar.broker.loadbalance.extensions.BrokerRegistryTest.MockLoadManager is not abstract and does not override abstract method setNamespaceBundleAffinity(java.lang.String,java.lang.String) in org.apache.pulsar.broker.loadbalance.LoadManager
   
   Please fix the style check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1054781077


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   This looks ok to me, but don't we still need to handle the CompletableFuture (how can we make `filter ` non-blocking after `getAvailableBrokerLookupDataAsync`?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1049052538


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   One alternative to optimizing BrokerVersionFilter is adding the pulsar version in the znode path.
   
   /loadbalance/brokers/{lookup-address}-{pulsar-version}
   
   By this design, we can get available brokers and their pulsar versions by a single getChildren() call(and they are cached together).
   
   We may consider more complex zk path designs(e.g. nested zk path), but I think the above path design is good enough for our use case.
   
   
   
   
   
   
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1049166450


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   This is a good solution. I will implement it in the broker filter PR. In this PR, I removed this cache.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046665593


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {

Review Comment:
   Updated.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047964792


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   The `LockManagerImpl ` cache only has key(`LookupServiceAddress`) list, we don't have a cache to traverse the `key, value`, see:
   https://github.com/apache/pulsar/pull/18810/files#diff-830131453bc160555aeefc19581054f593d97be9848c4070f75a2031316b09fdR184-R186
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1045513312


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Can we just return the cached list, `future.complete(Lists.newArrayList(this.brokerLookupDataMap.keySet()));`?
   Why do we need to use `brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)` every time?
   
   If this `listLocks` is for a more consistent `getAvailableBrokers` read, then why do we cache the broker list in `brokerLookupDataMap` and update the entries via the `zk listener`?
   
   Do we need to update the brokerLookupDataMap cache if the return list differs from brokerLookupDataMap.keySet() ?
   
   



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {

Review Comment:
   what if `brokerLookupDataLock.release(..)` fails (but we pre-set `registered` to false in this case)?



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"

Review Comment:
   can't we use `pulsar.getLookupServiceAddress()` instead?



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {

Review Comment:
   what if `brokerLookupDataLockManager .acquireLock(..)` fails (but we pre-set `registered` to true in this case)?



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)
+                .thenAccept(listLocks -> future.complete(Lists.newArrayList(listLocks)))
+                .exceptionally(ex -> {
+                    Throwable realCause = FutureUtil.unwrapCompletionException(ex);
+                    log.warn("Error when trying to get active brokers", realCause);
+                    future.complete(Lists.newArrayList(this.brokerLookupDataMap.keySet()));
+                    return null;
+                });
+        return future;
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    @Override
+    public void forEach(BiConsumer<String, BrokerLookupData> action) {
+        this.brokerLookupDataMap.forEach(action);
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        try {
+            this.unregister();
+        } catch (MetadataStoreException ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }
+
+        this.listeners.clear();
+    }
+
+    private void handleMetadataStoreNotification(Notification t) {
+        if (t.getPath().startsWith(LOOKUP_DATA_PATH) && t.getPath().length() > LOOKUP_DATA_PATH.length()) {
+            try {
+                if (log.isDebugEnabled()) {
+                    log.debug("Handle notification: [{}]", t);
+                }
+                this.scheduler.submit(() -> {
+                    String lookupServiceAddress = t.getPath().substring(LOOKUP_DATA_PATH.length() + 1);
+                    this.updateBrokerLookupDataToLocalCache(lookupServiceAddress, t.getType());
+                    for (BiConsumer<String, NotificationType> listener : listeners) {
+                        listener.accept(lookupServiceAddress, t.getType());
+                    }
+                });
+            } catch (RejectedExecutionException e) {
+                // Executor is shutting down
+            }
+        }
+    }
+
+    private void updateBrokerLookupDataToLocalCache(String lookupServiceAddress, NotificationType type) {
+        switch (type) {
+            case Created, Modified, ChildrenChanged -> {
+                try {
+                    Optional<BrokerLookupData> lookupData =
+                            brokerLookupDataLockManager.readLock(keyPath(lookupServiceAddress))
+                                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                    if (lookupData.isEmpty()) {
+                        brokerLookupDataMap.remove(lookupServiceAddress);
+                        log.info("[{}] Broker lookup data is not present", lookupServiceAddress);
+                        break;
+                    }
+                    brokerLookupDataMap.put(lookupServiceAddress, lookupData.get());
+                } catch (Exception e) {
+                    log.warn("Error reading broker data from cache for broker - [{}], [{}]",

Review Comment:
   In this case, this `brokerLookupDataMap` cache can be inconsistent from others. Can we make this cache update eventually consistent?



##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryTest.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import org.apache.bookkeeper.util.ZkUtils;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.LoadManager;
+import org.apache.pulsar.broker.loadbalance.ResourceUnit;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.stats.Metrics;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooKeeper;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+/**
+ * Unit test for {@link BrokerRegistry}.
+ */
+public class BrokerRegistryTest {
+
+    private ExecutorService executor;
+
+    private LocalBookkeeperEnsemble bkEnsemble;
+
+    private PulsarService pulsar1;
+
+    private PulsarService pulsar2;
+
+    // Make sure the load manager don't register itself to `/loadbalance/brokers/{lookupServiceAddress}`
+    public static class MockLoadManager implements LoadManager {
+
+        @Override
+        public void start() throws PulsarServerException {
+            // No-op
+        }
+
+        @Override
+        public boolean isCentralized() {
+            return false;
+        }
+
+        @Override
+        public Optional<ResourceUnit> getLeastLoaded(ServiceUnitId su) throws Exception {
+            return Optional.empty();
+        }
+
+        @Override
+        public LoadManagerReport generateLoadReport() throws Exception {
+            return null;
+        }
+
+        @Override
+        public void setLoadReportForceUpdateFlag() {
+            // No-op
+        }
+
+        @Override
+        public void writeLoadReportOnZookeeper() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public void writeResourceQuotasToZooKeeper() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public List<Metrics> getLoadBalancingMetrics() {
+            return null;
+        }
+
+        @Override
+        public void doLoadShedding() {
+            // No-op
+        }
+
+        @Override
+        public void doNamespaceBundleSplit() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public void disableBroker() throws Exception {
+            // No-op
+        }
+
+        @Override
+        public Set<String> getAvailableBrokers() throws Exception {
+            return null;
+        }
+
+        @Override
+        public CompletableFuture<Set<String>> getAvailableBrokersAsync() {
+            return null;
+        }
+
+        @Override
+        public void stop() throws PulsarServerException {
+            // No-op
+        }
+
+        @Override
+        public void initialize(PulsarService pulsar) {
+            // No-op
+        }
+    }
+
+    @BeforeMethod
+    void setup() throws Exception {
+        executor = new ThreadPoolExecutor(5, 20, 30, TimeUnit.SECONDS,
+                new LinkedBlockingQueue<>());
+        // Start local bookkeeper ensemble
+        bkEnsemble = new LocalBookkeeperEnsemble(3, 0, () -> 0);
+        bkEnsemble.start();
+
+        // Start broker 1
+        ServiceConfiguration config1 = new ServiceConfiguration();
+        config1.setLoadBalancerEnabled(false);
+        config1.setLoadManagerClassName(MockLoadManager.class.getName());
+        config1.setClusterName("use");
+        config1.setWebServicePort(Optional.of(0));
+        config1.setMetadataStoreUrl("zk:127.0.0.1" + ":" + bkEnsemble.getZookeeperPort());
+        config1.setBrokerShutdownTimeoutMs(0L);
+        config1.setBrokerServicePort(Optional.of(0));
+        config1.setAdvertisedAddress("localhost");
+        createCluster(bkEnsemble.getZkClient(), config1);
+        pulsar1 = new PulsarService(config1);
+        pulsar1.start();
+
+        // Start broker 2
+        ServiceConfiguration config2 = new ServiceConfiguration();
+        config2.setLoadBalancerEnabled(false);
+        config2.setLoadManagerClassName(MockLoadManager.class.getName());
+        config2.setClusterName("use");
+        config2.setWebServicePort(Optional.of(0));
+        config2.setMetadataStoreUrl("zk:127.0.0.1" + ":" + bkEnsemble.getZookeeperPort());
+        config2.setBrokerShutdownTimeoutMs(0L);
+        config2.setBrokerServicePort(Optional.of(0));
+        config2.setAdvertisedAddress("localhost");
+        pulsar2 = new PulsarService(config2);
+        pulsar2.start();
+    }
+
+    @AfterMethod(alwaysRun = true)
+    void shutdown() throws Exception {
+        executor.shutdownNow();
+
+        pulsar2.close();
+        pulsar1.close();
+
+        bkEnsemble.stop();
+    }
+

Review Comment:
   Maybe we can add more UTs for unhappy cases(when metadata store read/write fails).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1045546483


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;

Review Comment:
   What's the consistency model of `LockManager` / `BrokerRegistry`? Does BrokerRegistry guarantee eventual consistency?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1045546483


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;

Review Comment:
   What's the consistency model of `LockManager` and `BrokerRegistry`? Does BrokerRegistry guarantee eventual consistency?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1048033740


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Can we do future.join().get()?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1066671971


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt ->

Review Comment:
   In most cases, the returned future of this.lookupAsync(brokerId) should be Completed, as we expect the lookup is mainly from the cache. However, if the returned future is Incomplete, we probably need to measure the latency of the lookup i/o to the metadata store.
   
   In general, I think every i/o to the dependent services needs to be tracked(preferably as metrics). I am not sure about this practice in the Pulsar repo, though.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1092737937


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataCache;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    private volatile CompletableFuture<Void> cacheReloadFuture;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataCache = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+        this.cacheReloadFuture = CompletableFuture.completedFuture(null);
+        this.lookupServiceAddress = pulsar.getLookupServiceAddress();
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            // Update all lookup data to cache
+            this.reloadAllBrokerLookupCacheAsync()
+                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+            this.started.set(true);
+        } catch (MetadataStoreException | ExecutionException | InterruptedException | TimeoutException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {

Review Comment:
   There is no description about the broker ID in the proposal. It might make readers confused. For example, an ID is usually an integer, e.g. the broker ID in Kafka.
   
   Could `getBrokerId()` return a unique string that does not represent the address? If not, I think keeping the `getLookupServiceAddress` not changed would be better,



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059490454


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt -> {
+                    if (lookupDataOpt.isPresent()) {
+                        map.put(brokerId, lookupDataOpt.get());
+                    } else {
+                        log.warn("Got an empty lookup data, brokerId: {}", brokerId);
+                    }
+                }));
+            }
+            return FutureUtil.waitForAll(futures).thenApply(__ -> map);
+        });
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.get()) {
+            return;
+        }
+        try {
+            this.unregister();
+            brokerLookupDataLockManager.close();
+            scheduler.shutdown();
+            this.listeners.clear();
+            started.set(false);
+        } catch (Exception ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }

Review Comment:
   Don't we need to reset `started.set(false)` when an exception is thrown?
   
   We don't expect this `register() / unregister()` (or `start() / close()` in similar classes) to be called competitively many times from multiple threads. What's the practice in Pulsar for concurrency control in such cases? (I think the simple `synchronized` is enough in these cases because threads waiting happens less likely )
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059202962


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt ->
+                        lookupDataOpt.ifPresent(lookupData -> map.put(brokerId, lookupData))));

Review Comment:
   If the lookup data is not present, we probably want to print logs.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt ->

Review Comment:
   Let's check if lookupAsync is complete or not.
   
   If not, we probably want to print a log with the i/o waiting time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1053921340


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   IMHO, a separate cache is redundant, as the lock values are already cached in the metadata store.
   
   I would like to hear others' thoughts, and I guess in the worst case, we could go forward with your original implementation. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1055019638


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   The selection method will be like this:
   ```java
   brokerRegistry.getAvailableBrokerLookupDataAsync()
           .thenCompose(availableBrokers -> {
               LoadManagerContext context = this.getContext();
   
               // Filter out brokers that do not meet the rules.
               List<BrokerFilter> filterPipeline = getBrokerFilterPipeline();
               Map<String, BrokerLookupData> availableBrokerCandidates = new HashMap<>(availableBrokers);
               try {
                   for (final BrokerFilter filter : filterPipeline) {
                       filter.filter(availableBrokerCandidates, context);
                   }
               } catch (BrokerFilterException e) {
                   availableBrokerCandidates = availableBrokers;
               }
               if (availableBrokerCandidates.isEmpty()) {
                   return CompletableFuture.completedFuture(Optional.empty());
               }
   
               return CompletableFuture.completedFuture(
                       brokerSelectionStrategy.select(availableBrokerCandidates.keySet(), bundle, context));
           });
   ```
   So the filter will not be blocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046666002


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)
+                .thenAccept(listLocks -> future.complete(Lists.newArrayList(listLocks)))
+                .exceptionally(ex -> {
+                    Throwable realCause = FutureUtil.unwrapCompletionException(ex);
+                    log.warn("Error when trying to get active brokers", realCause);
+                    future.complete(Lists.newArrayList(this.brokerLookupDataMap.keySet()));
+                    return null;
+                });
+        return future;
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    @Override
+    public void forEach(BiConsumer<String, BrokerLookupData> action) {
+        this.brokerLookupDataMap.forEach(action);
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        try {
+            this.unregister();
+        } catch (MetadataStoreException ex) {
+            if (ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                throw new PulsarServerException.NotFoundException(MetadataStoreException.unwrap(ex));
+            } else {
+                throw new PulsarServerException(MetadataStoreException.unwrap(ex));
+            }
+        }
+
+        this.listeners.clear();
+    }
+
+    private void handleMetadataStoreNotification(Notification t) {
+        if (t.getPath().startsWith(LOOKUP_DATA_PATH) && t.getPath().length() > LOOKUP_DATA_PATH.length()) {
+            try {
+                if (log.isDebugEnabled()) {
+                    log.debug("Handle notification: [{}]", t);
+                }
+                this.scheduler.submit(() -> {
+                    String lookupServiceAddress = t.getPath().substring(LOOKUP_DATA_PATH.length() + 1);
+                    this.updateBrokerLookupDataToLocalCache(lookupServiceAddress, t.getType());
+                    for (BiConsumer<String, NotificationType> listener : listeners) {
+                        listener.accept(lookupServiceAddress, t.getType());
+                    }
+                });
+            } catch (RejectedExecutionException e) {
+                // Executor is shutting down
+            }
+        }
+    }
+
+    private void updateBrokerLookupDataToLocalCache(String lookupServiceAddress, NotificationType type) {
+        switch (type) {
+            case Created, Modified, ChildrenChanged -> {
+                try {
+                    Optional<BrokerLookupData> lookupData =
+                            brokerLookupDataLockManager.readLock(keyPath(lookupServiceAddress))
+                                    .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                    if (lookupData.isEmpty()) {
+                        brokerLookupDataMap.remove(lookupServiceAddress);
+                        log.info("[{}] Broker lookup data is not present", lookupServiceAddress);
+                        break;
+                    }
+                    brokerLookupDataMap.put(lookupServiceAddress, lookupData.get());
+                } catch (Exception e) {
+                    log.warn("Error reading broker data from cache for broker - [{}], [{}]",

Review Comment:
   We can do the cache reload here: https://github.com/apache/pulsar/pull/18810#discussion_r1046665315



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047964792


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   The `LockManagerImpl ` cache only has key(`LookupServiceAddress`) list, but we don't have a cache to traverse the `key, value`, see:
   https://github.com/apache/pulsar/pull/18810/files#diff-830131453bc160555aeefc19581054f593d97be9848c4070f75a2031316b09fdR184-R186
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1047987341


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   But the `brokerLookupDataLockManager.readLock(keyPath(broker))` returned a `CompletableFuture<Optional<T>>`. How do we use this in the `forEach` method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1048037808


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Sorry. What's the usage of this `foreach`?
   can we run this `foreach` async (run on a different thread)?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1046665488


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"

Review Comment:
   Yes, updated!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
heesung-sn commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1050287035


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)
+                .thenCompose(listLocks -> CompletableFuture.completedFuture(Lists.newArrayList(listLocks)));

Review Comment:
   nit: can't this be ` .thenApply(listLocks -> Lists.newArrayList(listLocks));` ?
   



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)
+                .thenCompose(listLocks -> CompletableFuture.completedFuture(Lists.newArrayList(listLocks)));
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public void listen(BiConsumer<String, NotificationType> listener) {
+        this.listeners.add(listener);
+    }
+
+    @Override
+    public void close() throws PulsarServerException {
+        if (!started.compareAndSet(true, false)) {

Review Comment:
   nit: This close is not retryable because `started` is already false when unsuccessful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1053919039


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   > Unless necessary, we want to keep this path design for the following reasons.
   >
   >1. This path design change can further complicate version compatibility(uneasy to maintain the path design for different pulsar versions).
   >2. During version upgrade, if the shutdown is ungraceful, the new path design could incur two locks for the same >broker. However, currently, the restart will be blocked as the lingering lock will block it.
   
   Yes, I agree. I have the same concern after trying to implement the broker filter.
   
   > Here the readLock will return the up-to-date lock value (maximum 5 mins delay, except the first time) as the cache refreshes after 5mins, once the key-value is stored.
   > In this way, we can make this foreach non-blocking. When the readLock cache is initially empty, it will return the empty available broker set. In this case. the client should retry.
   
   Why don't we use a cache like the original implement? It will never block the client thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059318860


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }

Review Comment:
   I used `AtomicBoolean` in the first version. https://github.com/apache/pulsar/pull/18810#discussion_r1045516181



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1053937530


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    private static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerZNodePath;
+
+    private final String lookupServiceAddress;
+
+    @VisibleForTesting
+    protected final Map<String, BrokerLookupData> brokerLookupDataMap;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private final AtomicBoolean registered;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.brokerLookupDataMap = new ConcurrentHashMap<>();
+        this.listeners = new ArrayList<>();
+
+        this.registered = new AtomicBoolean(false);
+        this.lookupServiceAddress = pulsar.getAdvertisedAddress() + ":"
+                + conf.getWebServicePort().orElseGet(() -> conf.getWebServicePortTls().get());
+        this.brokerZNodePath = LOOKUP_DATA_PATH + "/" + lookupServiceAddress;
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() {
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+    }
+
+    @Override
+    public void register() {
+        if (registered.compareAndSet(false, true)) {
+            this.brokerLookupDataLock =
+                    brokerLookupDataLockManager.acquireLock(brokerZNodePath, brokerLookupData).join();
+        }
+    }
+
+    @Override
+    public void unregister() throws MetadataStoreException {
+        if (registered.compareAndSet(true, false)) {
+            try {
+                brokerLookupDataLock.release().join();
+            } catch (CompletionException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getLookupServiceAddress() {
+        return this.lookupServiceAddress;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        CompletableFuture<List<String>> future = new CompletableFuture<>();
+        brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH)

Review Comment:
   Or can we add a method to let the registry return the map of lookup data?
   
   ```
   public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync()
   ```
   
   And change the broker filter method as:
   
   ```
   List<String> filter(Map<String, BrokerLookupData> availableBrokerLookupData, LoadManagerContext context) throws BrokerFilterException;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
Demogorgon314 commented on code in PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#discussion_r1059212074


##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.BiConsumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.apache.pulsar.metadata.api.coordination.LockManager;
+import org.apache.pulsar.metadata.api.coordination.ResourceLock;
+
+/**
+ * The broker registry impl, base on the LockManager.
+ */
+@Slf4j
+public class BrokerRegistryImpl implements BrokerRegistry {
+
+    protected static final String LOOKUP_DATA_PATH = "/loadbalance/brokers";
+
+    private final PulsarService pulsar;
+
+    private final ServiceConfiguration conf;
+
+    private final BrokerLookupData brokerLookupData;
+
+    private final LockManager<BrokerLookupData> brokerLookupDataLockManager;
+
+    private final String brokerId;
+
+    private final ScheduledExecutorService scheduler;
+
+    private final List<BiConsumer<String, NotificationType>> listeners;
+
+    private volatile ResourceLock<BrokerLookupData> brokerLookupDataLock;
+
+    private volatile boolean registered = false;
+
+    private final AtomicBoolean started = new AtomicBoolean(false);
+
+    public BrokerRegistryImpl(PulsarService pulsar) {
+        this.pulsar = pulsar;
+        this.conf = pulsar.getConfiguration();
+        this.brokerLookupDataLockManager = pulsar.getCoordinationService().getLockManager(BrokerLookupData.class);
+        this.scheduler = pulsar.getLoadManagerExecutor();
+        this.listeners = new ArrayList<>();
+        this.brokerId = pulsar.getLookupServiceAddress();
+        this.brokerLookupData = new BrokerLookupData(
+                pulsar.getSafeWebServiceAddress(),
+                pulsar.getWebServiceAddressTls(),
+                pulsar.getBrokerServiceUrl(),
+                pulsar.getBrokerServiceUrlTls(),
+                pulsar.getAdvertisedListeners(),
+                pulsar.getProtocolDataToAdvertise(),
+                pulsar.getConfiguration().isEnablePersistentTopics(),
+                pulsar.getConfiguration().isEnableNonPersistentTopics(),
+                pulsar.getBrokerVersion());
+    }
+
+    @Override
+    public void start() throws PulsarServerException {
+        if (started.get()) {
+            return;
+        }
+        pulsar.getLocalMetadataStore().registerListener(this::handleMetadataStoreNotification);
+        try {
+            this.register();
+            this.started.set(true);
+        } catch (MetadataStoreException e) {
+            throw new PulsarServerException(e);
+        }
+    }
+
+    @Override
+    public boolean isStarted() {
+        return this.started.get();
+    }
+
+    @Override
+    public synchronized void register() throws MetadataStoreException {
+        if (!registered) {
+            try {
+                this.brokerLookupDataLock = brokerLookupDataLockManager.acquireLock(keyPath(brokerId), brokerLookupData)
+                        .get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = true;
+            } catch (InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public synchronized void unregister() throws MetadataStoreException {
+        if (registered) {
+            try {
+                brokerLookupDataLock.release().get(conf.getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+                registered = false;
+            } catch (CompletionException | InterruptedException | ExecutionException | TimeoutException e) {
+                throw MetadataStoreException.unwrap(e);
+            }
+        }
+    }
+
+    @Override
+    public String getBrokerId() {
+        return this.brokerId;
+    }
+
+    @Override
+    public CompletableFuture<List<String>> getAvailableBrokersAsync() {
+        return brokerLookupDataLockManager.listLocks(LOOKUP_DATA_PATH).thenApply(Lists::newArrayList);
+    }
+
+    @Override
+    public CompletableFuture<Optional<BrokerLookupData>> lookupAsync(String broker) {
+        return brokerLookupDataLockManager.readLock(keyPath(broker));
+    }
+
+    public CompletableFuture<Map<String, BrokerLookupData>> getAvailableBrokerLookupDataAsync() {
+        return this.getAvailableBrokersAsync().thenCompose(availableBrokers -> {
+            Map<String, BrokerLookupData> map = new ConcurrentHashMap<>();
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (String brokerId : availableBrokers) {
+                futures.add(this.lookupAsync(brokerId).thenAccept(lookupDataOpt ->

Review Comment:
   Can you explain it in detail?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on pull request #18810: [improve][broker] PIP-192: Implement broker registry for new load manager

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #18810:
URL: https://github.com/apache/pulsar/pull/18810#issuecomment-1384824151

   > The lookup data will not be too heavy for Zookeeper, right? Maybe we can do some tests to find out is necessary to use LoadDataStore to store the lookup data.
   
   IMO, it's ok. One broker will have one znode to store the lookup data. And we also use the znode as a mutex lock to avoid the different brokers registering with the same lookup data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org